Cycle Time Reduction

Case study

A Silicon Valley icon reduces slow delivery cycles by testing faster and improves developer happiness

One member of Launchable’s beta program is a division within a major Silicon Valley software company. Here we refer to them as Rocket Software. Rocket Software has 1,500 engineers. 

Key challenges: The team has identified cycle time reduction as critical. Slow dev cycles make developers lives harder. Developer happiness has consistently shown up in their business objectives for the last three years.

Employees Mode of Development Scenario Languages & Tooling
20,000+ SaaS Pre-merge tests Java, Python, Jenkins, Gerrit, Bazel

Summary

Within Rocket Software, there is a 50 person team that delivers an analytics product based on the core product. We'll call them Rocket Insights. This team is a lighthouse within the greater organization following cutting edge development practices including tools like Bazel to perform build and test automation. Rocket Software looks to this team to lay the groundwork for wider improvements for developer productivity and happiness within the company.

With Launchable, Rocket Software can reduce test feedback time. We quantified the impact of slow feedback from tests (7.4k-10.2k developer hours per year). The result was that Launchable can reduce their build time by 40%.

The bottom line? Launchable can help this team within Rocket Software save up to $430k per year by reducing test feedback time to around 5.4k hours a year (a 40% reduction).

How does Launchable help? Launchable brings machine learning in test automation tooling to identify tests that matter to run for each change. Rocket Software uses Bazel for test automation and Launchable's improvement work on top of this tool. 

About Rocket Insights: A cutting edge team within an Future 50 software company

Rocket Software selected a division with the company called Rocket Insights to pilot Launchable.  You can think of Rocket Insights as a startup within a much larger company. Rocket Insights builds a product that provides analytics on top of the core product from Rocket Software.

The product is delivered on-premises. Thus, software is delivered on multiple operating systems with multiple product versions. There are a sizable number of tests. Developers usually work on multiple release lines at once, and cognitive cost of switching between lines is high. 

The team is distributed across three geographic locations. Software development and testing continues around the clock.

The key challenge: slow test feedback

At its heart, the Rocket Insights division is about delivering on-premises packaged software on multiple operating systems. The challenges they face with slow feedback are applicable to a wide range of teams. Rocket Insights is typical of a few Launchable beta partners who fit this mode of software development.

The quotes in this section are representative of issues within the larger company. However, we can still bring significant benefits even though Rocket Insights is a smaller team within Rocket Software.

Rocket Insights' primary challenge is slow feedback caused by long cycle times. Features often take multiple weeks to deliver. 

If things are slow and I’m not productive… these things [slow development times] make people dissatisfied. It’s not just about productivity it’s about satisfaction which drives retention.

VP of Engineering

Rocket Software

Rocket Insights also has a complicating factor. They use Bazel to cache tests and drive build times down. Hardware bandwidth is at a premium for Rocket Insights. 

We have honed in on the “pre-merge” test cycle for Rocket Insights. The build times are about 1-2 hours for this team, and that is considered fairly reasonable today. 

The cost of slow feedback for pre-merge tests

We were able to help Rocket Insights quantify the impact of slow feedback from tests. Their team of 50 developers spends approximately 7.4k-10.2k hours waiting on tests each year.

The team runs 36 builds per day or between 170-230 builds per week. With this build time data we were able to compute the daily and yearly hours spent waiting by all developers and the corresponding dollar impact.

On average, there is a wait time of 4 days for every day of work delivered by this team!

Cycle Time Reduction Data

40% improvement in developer feedback time by testing faster & smarter

We analyzed builds, code changes, and test results to create what we call a "Confidence Curve" for Rocket Insights (shown below). This curve represents how quickly a developer finds that her changes have a problem. It shows the percentage of tests that Launchable needs to run on the x-axis to achieve the confidence level on the y-axis. You can think of confidence as the likelihood that a regression will be detected.

40% improvement in developer feedback time by testing faster & smarter

The dotted line shows the pre-Launchable performance of the system. The key number on this curve is 90% confidence; 75% of the tests must run to get to 90% confidence (dotted line). With Launchable, only 20% of the tests must run to get to the same 90% confidence number (red line).

By using Launchable to run only the pertinent 10% subset of tests, the test execution time drops by 20 minutes. The build time reduces from an average 51 minutes to 31 minutes which includes the 29 minutes overhead that Launchable doesn't influence. This overhead comes from compilation time. 

This is a 40% reduction in build time for developers!

Furthermore, Rocket Insights has the option to choose any number on the red curve to optimize between feedback time and confidence. For example:

Test Execution TimeConfidence
55 seconds (2.5%)85.8%
2 minutes (10%)89.5%
4.4 minutes (20%)91.8%
11 minutes (50%)95.3%

Implications of the reduced cycle time for the development team

It requires multiple patch sets to get a change in (average number of patch set is 4.9). The throughput of organization is about 6 changes per day. 

A developer can get feedback approximately twice as fast with Launchable (51m → 31m) and consequently can hope to deliver a feature in a day or two versus the multiple it takes today.

Using Bazel and Launchable together

As mentioned earlier, the team uses Bazel to optimize builds. Bazel subsets test executions and thus test execution varies. The worst case scenario is around 36 minutes with an average of 22 minutes (left graph below). 

The Launchable model (right graph below) works in conjunction with Bazel’s incremental test selection. We bring the worst case down to 3.6 minutes and average down to 2.2 minutes.

Thus, the ML based test automation approach from Launchable works on top of smart test automation tools such as Bazel.

Using Bazel and Launchable together

The bottom line impact: cost savings

There are two ways to think about the benefits to Rocket Insights – bottom and top line impact. 

Here, we’ve focused on the bottom line impact, which is cost savings in terms of time spent waiting because it is easier to quantify. However, we posit that the bigger impact is on the top line, because faster cycle times with higher quality shippable code translate to more benefits delivered by Rocket Insights to its customers.

The bottom line impact is $427k saved each year.

This impact is achieved by:

Save developer hours waiting for feedback or save hours wasted by developers doing something other than their primary task. We think of this as “leaks” or “waste” in an organization that have always been hard to quantify, let alone get a handle on. 

Massive drop in costs waiting for results

A 40% drop in a test runs drives a corresponding drop in the yearly wait time. This was 3,400 hours saved (mean) for 50 engineers. 

Saving 40% time implies a 40% dollar savings or roughly ~$427k. This saving easily exceeds the stated goal from the team as they began their engagement with us.

If we are able to save 30-60 minutes per developer per day, the savings will be staggering!

VP of Engineering

Rocket Software

Massive drop in costs waiting for results

The top-line impact: test faster to deliver more customer facing value

For Rocket Software, the top-line impact of using Launchable is that they are able to ship features faster at higher quality. We reduce software delivery cycle times because we can identify the right tests to verify changes. Developers getting faster feedback means their eventual changes are higher quality. Consequently, developers can push through smaller changes faster and more often. The ultimate winner is the customer, who now gets more value earlier.

Happier developers

Wait times from tests is one of the biggest hurdles in a developer's day. Long wait times means longer days or even longer weekends at work.

This is where Launchable shines! Shorter test cycle times without as much context switching implies developers can pick up a task and continue working on it without interruption until it is done. This brings the employee satisfaction up. Ultimately, this helps engineering leaders drive down employee attrition. This is a potential benefit of bringing Launchable in for every company. 

You don't have time for slow tests!

Eliminate slow test cycles. Test what matters.