Meet Matt
Matt works for Easy Meals, a cloud-based restaurant management software company based in Boston, Massachusetts. Easy Meals operates a cloud-based and digital technology platform for the restaurant industry in the United States and Ireland. The company offers Easy Meals Point of Sale (POS), a hardware product based on Android; Order & Pay, which allows guests to order and pay from their mobile devices and other products.
If you live in the US and have eaten in a restaurant lately, it is likely you paid the bill on an Easy Meals device.
Matt is a senior engineering manager in the Engineering Excellence department at Easy Meals. He is involved with general DevOps practices at Easy Meals, which covers anything from build pipelines to testing and anything that teams use to help build software.
The problem: Long testing cycles are impacting release velocity and developer productivity at Easy Meals
The Easy Meals mobile team handles the monolith that houses the POS application. This is their core product, and release velocity is paramount to their success as a company.
Currently, the mean time to value from a code change to release for the mobile team is 96 days. That means for any code change, the average time it takes for that change to become available as a release and deployed is ~96 days. They want to reduce this to 14 days. A key part of reducing this time to value is by reducing test times throughout the build and release process.
We had an important customer file a bug. We fixed it immediately but it took 4 months to get the fix delivered. We lost this customer because of the delay—they thought we didn’t care enough!
Long testing cycles are causing delays with getting code to production
Easy Meals mobile is a “multi-project” Gradle application that consists of around 100 heterogenous projects. The build process is intertwined with many layers of testing. Along with the typical automated testing, there are necessary code reviews that have to occur and some manual acceptance tests they run as well. As stated above, the Easy Meals team is looking to reduce their overhead on this process wherever they can.
Currently, the Easy Meals team is parallelizing each test suite where applicable, but it is still not enough. Resource costs are high, and with ~60 devs submitting around 10-15 PRs per day, they are looking for other ways to improve testing times.
A layout of their CI and release process can be seen below:
Subsetting unit tests with Launchable Predictive Test Selection
Easy Meals team decided that their biggest opportunity lay in reducing their unit test suite times which takes around 40 minutes to complete. Before Launchable, developers were waiting ~40 minutes to see if the unit test suite passed. With multiple PRs per day run through this pipeline, it became a serious bottleneck for developer productivity. It also adds to the total overhead of getting code out to production. If developers got this feedback in a more timely fashion, they could resolve issues sooner and cut releases more frequently.
The solution: Run a subset of unit tests in place for faster feedback
With Launchable, the Easy Meals mobile team is able to subset and retrieve a ~10 minute collection of tests resulting with 90% confidence. This means they can run their unit test suite in 10 minutes instead of 40 minutes, and have 90% confidence that the subset will find the failing run.
Since we still want to achieve their normal code coverage, they run the rest of the tests outside of the subset immediately after the subset, later in the pipeline. This allows the developer to quickly check if his code change will cause any testing failures, and move on. In the background, the remainder of tests run with much less importance and a less likelihood of containing a relevant failure. The developer can continue writing code, and the development process now allows for more iterations with much less overhead.
How Launchable made an impact and where to next
The Easy Meals mobile team was able to reduce their unit test suite from ~40 minutes to ~10 minutes. At 10-15 PRs per day, and around 60 developers, the savings are immense. On average, this is anywhere from 5-7.5 hours of reduced runtimes (or savings) per day. With a monthly release cycle, that can sometimes take up to 3 months, the savings can be anywhere from 4-7 days (6k to 9k minutes) of testing time saved per month. Put differently, Launchable cuts a week off in test runtime from a 4 week release cycle. Annually, this would be reduction of 3 months of test runtime!
With the results seen for the unit test suite, Matt achieved his goal of reducing release cycle time and improving developer productivity. With all of that extra time going back to developers, they have plenty of opportunities to improve other aspects of their development and release cycle.