Our core technology is a machine learning engine that predicts the likelihood of failure for each test case based on the source code changes being tested.
Integrating your test runners with Launchable’s SaaS service lets you run only the meaningful subset of tests based on the changes being tested in the order that minimizes feedback delay to developers.
Launchable can be applied in a variety of scenarios to reduce test feedback delay, such as:
We want to learn from you! What’s the biggest testing bottleneck in your environment?
After we build a prediction model using your historical test results, you’ll use our plugins to integrate your test runner with Launchable’s prediction APIs for real-time predictions based on the code changes being tested.
The plugin asks Launchable which tests to run based on the changes being tested. It then interacts with the test runner to run the right subset of tests in the right order.
How long it takes to train a model is really a function of the volume of test sessions you are executing in a given time frame, so we need to learn about your specific environment and use cases before we can make an estimation.
That said, with most enterprise software development teams we’ve been talking to, 4-12 weeks seem to be good baseline estimate to produce a sizable impact. As an example, we used three months of data to create the Spark model.
Time to First Failure TTFF refers to the wall clock time from the start of the tests to the end of the first test that fails when a set of tests are run. We believe this is a hugely important metric because the sooner a developer is aware of the first test failure, the sooner they can start working on a fix. Each potential application of Launchable’s predictive test selection capability is designed to reduce TTFF and overall test feedback delay.
Why is it time to first failure, not time to first feedback?
Most of the time, builds succeed. And you'd only know that after all the tests have run. So if we compute "time to first feedback" that includes success, then the difference we make will be a wash. What this means is that if people don't change their behaviors (of doing something after they hear the success), then we won't really make any difference. So we need to give people some cue when the test execution is out of the most risky part.
Ideally flaky test failures shouldn’t count toward TTFF. However, we don’t have a mechanism for classifying test failures, so all failures are treated the same. Classifying test failures is an important problem, but we are not focused on it today.
The flaky tests are the ones that fail, so you’ll continue to see them run. Launchable doesn’t make them better or worse.
Launchable won’t stop the test execution after the first failure has been found, but some CI tools and test runners can be configured to do this if desired.
Launchable will recognize that the tests are new and will make sure to run them.
We will work with you to analyze and the impact of dependencies on Launchable’s ability to make a prediction and resolve if needed.
Launchable’s predictive test selection service learns the relationship between code changes and the test cases impacted by those changes.
The two key inputs for this are:
1) Metadata ('features') about the code changes being tested, which includes:
2) Metadata ('features') about the test cases that were run, which includes:
This scope may change as we evolve the service.
The actual content of your source code is not needed, only metadata about changes.
There are two stages to a Product Advisor Program engagement: analysis and pilot.
In the analysis phase, you’ll install a program called the Launchable ingester (see below). The ingester works with Jenkins today, but we may add support for more CI tools as needed. The primary requirement is that your test results need to be made available in a machine-readable format such as JUnit/xUnit.
In the pilot phase, the Launchable plugin integrates your test runner or build tool with our cloud service to directly manipulate test execution. We will build a plugin for the test runner(s) or build tool(s) you use in preparation for a pilot in your environment.
The Launchable ingester runs in a Docker container in your environment. It interacts with your Jenkins server and test frameworks, scraping historical data to send to Launchable to train a prediction model.
Launchable is a cloud service that is not available on-premises.
Launchable is a multi-tenant SaaS product. Each customer’s data is kept separate from each other.