Mo is in the developer tooling automation group and supports about 200 developers. This set of developers develop applications for the in-car dashboard for the car launching in 2023. Mo is a pragmatic engineer who takes well informed, well researched bets that solve critical problems which get in the way of keeping the engine (software) running.
Mo’s manager has been in multiple monthly meetings where developer happiness has been an issue. Development is slow, frustrating developers. Work has been put in to make things faster but with limited success. It is like a two-wheel drive stuck in the mud and the wheels just keep spinning.
One level deeper, the build and test times are slow for the team.
Every build and test lands on the in-car dashboard emulator device. There is a hardware constraint because these devices are expensive so there is a limit to what can be available. As builds and tests queue up, developers become frustrated because their commits just sit idling.
Our productivity loss is about 30-40% every day. I log into see if my builds ran after I put my kids to bed. This is just frustrating.
Lead engineer on the dashboard team
In December, Mo also has to estimate the hardware resources requirement for the next year and put in budget requests. These devices are then managed in a multi-million dollar lab. It is always hard to predict resources required for the upcoming year. Too much and he wasted Euros that could be used elsewhere and too less leads to longer wait times.
Talk about being under stress!
The solution is very clear for Mo. His proposal is that if he can get more juice out of his hardware, he can drive faster build and test times which should speed up developers, reduce their frustrations and do so economically.
Mo was at BazelCon to see if he can reduce the build times and make build times faster. Bazel is a long journey ahead (one that should be undertaken). That said, test run times are way longer than build time. He currently has no solution to reduce test cycle times. He has parallelized his tests and hit the wall where he cannot parallelize them any more.
We asked Mo to fundamentally rethink how he can improve the efficiency of his system by bringing ML to testing. Launchable brings Predictive Test Selection to smartly identify the tests that are most likely to fail and prioritize them first.
Mo threw their code and tests to train Launchable’s ML. which then produced the following graph.
The graph shows that the ML can identify 90% of the failing builds by running 20% of tests. Thus, what used to take them 60+ minutes comes back in only 11. Note: Mo ultimately chose 50% subset to roll this out to production which gives him about 95% confidence.
Faster delivery The first is clearly apparent - developers are finding failures 50% faster or their speed is going up 2x. This implies lesser late night checkins to see if the build has run.
Optimized resource usage The second benefit is reduction of hardware resources.
Launchable saves us 25% of resources on the mainline build while going faster
Mo has achieved optimized usage of his hardware resources and dramatically eased the hardware constraints problem that he set out to solve. His developers are seeing their dev cycles sped up by 2x for this particular project. He is looking for more aggressive optimization targets to speed things up further.
In conclusion, Mo has brought in the power of AI and ML for his team, relatively effortlessly. Other teams are looking curiously at this lighthouse team to see how they can do something similar.
Mo and his boss are about to evangelize to the rest of the organization. Mo is a hero.
Pre-merge tests. Python, C, C++, Jenkins, Gerrit, Nosetest and Yocto Linux