Flaky tests originate from multiple sources such as poorly written tests, infrastructure issues, memory failures, and so on. However, each test failure requires additional time to identify the actual cause.
Classifying flaky tests requires remembering the history of a test suite history. Thus, in a large test suite, classifying flakes becomes a non trivial problem.
Once teams have identified flakes, they have policies to deal with them. These could include running tests multiple times until they pass, moving them out in their own runs, or testing them later in the pipeline.
Each option slows the developers and teams down. Something as simple as re-running the tests till you get a pass can drastically reduce velocity.
As entropy sets in, there are more and more flaky tests. Then a hero steps in - analyzes the failures, fixes some flakes. They get some wins and then entropy kicks in all over again.
- GG, CTO of a Web Retailer
As projects and tests grow, decay sets in, along with flaky tests. Keeping on top of flaky tests is a headache that every development team has to constantly work through.
Get your API key. Connect your builds. Connect your test results. Sit back, and let Launchable do the analysis.
