How to Build a Data-Driven DevOps Decision Making Culture

There is no doubt that companies today are awash in information. By itself data is of little use, but businesses who correlate it, gain insights into their operations and can change themselves for the better. This is both the opportunity as well as the challenge in creating Data-driven DevOps. Development teams have an abundance of data to leverage, and therefore an abundance of potential to speed up application testing and delivery, using data to drive software quality and velocity. Regardless of your team or organization size, adopting a data-driven culture can start today. 

Take Advantage of Existing Data

The first step in building a data drive DevOps culture requires organizations and teams to understand what data points and metrics they already have and what they are missing. Tools including Jenkins, generate growing volumes of Continuous Integration (CI) data outputs. Development teams pull that data from hundreds of libraries, modules, and services, via scripts running during various parts of the workflow. The information sped up development’s front-end, but more work is needed on the back end. Taking advantage of existing data can come in different forms - one example is correlating pass/fail data of your tests to flaky tests. This existing data would be helpful to identify how flaky a test is likely to be 

Automate Data Delivery to the Right People

Holes are also evident as developers process feedback. Individuals adding new features and fixing bugs often do not get the data they need in a timely manner because it is generated at the end rather than during the testing process. Getting the developers the notifications when the first test failed would be an impactful adjustment. Another option is updating building notifications to be smarter - identifying and messaging the right developer. Running a git-blame on lines of code generates a build error and immediately notifies the Git author of the failure. They work on a fix immediately rather than wait hours for the full test results.

Simple Data-Driven Solutions Add Up

Incorporating data into your DevOps cycle doesn’t have to be a large undertaking. There are often low-hanging fruit opportunities that you can incorporate to improve productivity or efficiency. For instance, if notifications for test failures go to all engineers, take a look at your failure notifications and identify any that are caused by infrastructure failures (server disk space or database down). Those don’t necessarily need to go out to all of your engineers, but likely rather a specific group or person. Setting up a simple automatic scan of the last 50 lines of the build log for specific keywords around those error types, and sending an automatic notification to the right people would reduce notifications to your developers and lessen the notification fatigue or frustration caused by errors.

Factor in the Future of DevOps 

When adopting data-driven DevOps, we can all learn from leading organizations incorporating it at scale and the emerging trends. Companies including Google and Facebook have shared how they are harnessing machine learning to prioritize tests to run for each code change. A new approach to test impact analysis is Predictive Test Selection, which combats long cycle times by tracking your builds and selecting a subset of tests that are critical to run. We’ve also done a deep dive webinar into the state of the art of Flaky Tests, examining how teams are tackling the prevalent pest. 

Measuring Data-Driven DevOps Progress

There are at least two metrics to track when incorporating data drive DevOps into your organization. The first are the key performance metrics that are helpful to your team to track and celebrate internally, like test runtime, time from code push to build complete, and code coverage. The second are metrics to show progress to leadership, which can include lead time for code changes, time to restore service, and change failure rate.

Development teams are able to collect more information today than ever before. Artificial Intelligence and machine learning products enable them to correlate that information in new ways and find solutions to long-standing problems, such as slowdowns in Continuous Delivery. This expanding solutions enable teams to create a modern data-driven DevOps culture that streamlines development processes significantly, maximizes resources, and improves software quality.