Unreliable tests are a pervasive problem eating into development velocity
Classifying tests as flaky takes time
Flaky tests originate from multiple sources such as poorly written tests, infrastructure issues, memory failures, and so on. However, each test failure requires additional time to identify the actual cause.
Classifying flaky tests requires remembering the history of a test suite history. Thus, in a large test suite, classifying flakes becomes a non trivial problem.
Flakes require test re-runs that slow down dev velocity
Once teams have identified flakes, they have policies to deal with them. These could include running tests multiple times until they pass, moving them out in their own runs, or testing them later in the pipeline.
Each option slows the developers and teams down. Something as simple as re-running the tests till you get a pass can drastically reduce velocity.
As entropy sets in, there are more and more flaky tests. Then a hero steps in - analyzes the failures, fixes some flakes. They get some wins and then entropy kicks in all over again.
- GG, CTO of a Web Retailer
Finding and fixing flakes is a never ending cycle
As projects and tests grow, decay sets in, along with flaky tests. Keeping on top of flaky tests is a headache that every development team has to constantly work through.
Flaky Tests Insights
Quickly identify and eliminate flakes
Identify the top flaky tests in a test suite to prioritize fixing the right tests
Identify if a test run failed because of flaky tests to hone in on the real problems
Get started in less than 30 minutes
Get your API key. Connect your builds. Connect your test results. Sit back, and let Launchable do the analysis.