Test Automation Glossary

flaky tests

noun.

Automated tests that fail randomly during a run for reasons not related to the code changes being tested. They are often caused by timing issues, concurrency problems, or the presence of other workloads in the system.

Flaky tests are a common problem for many development teams especially as test suites grow. Flakey tests are more common at higher levels of the Test Pyramid especially in UI tests and system tests.

Like the fictional boy who cried “wolf”, tests that send a false signal too often are sometimes ignored. Or worse, people spend really time and effort trying to diagnose a failure, only to discover that it has nothing to do with their code changes. When flakiness occurs with a lot of tests it can make people weary of all tests and all failures—not just flaky tests—causing a loss of trust in tests.

Tests that produce flaky results should be repaired or removed from the test suite.