More work is needed to achieve continuous testing, Forrester report shows
Forrester recently completed an interesting new research project exploring what separates DevOps leaders from DevOps laggards. After analyzing data collected from 603 global enterprise respondents responsible for their firms’ DevOps strategies, Forrester learned that firms with the most mature DevOps practices do several key things differently:
- Allocate proper testing budgets and focus on upgrading their testing skills.
- Implement continuous testing to meet the demands of release frequency and support continuous delivery.
- Include testers as part of their integrated delivery teams.
- Automate end-to-end functional testing.
- Shift left testing to earlier in the development life cycle.
Firms who follow these five best practices experience testing as a way to gain speed and efficiency, rather than a bottleneck to delivering faster.
This was not terribly surprising. On the one hand, study after study shows that testing is the No. 1 bottleneck to accelerated application delivery. On the other, we’ve seen firsthand how organizations that commit to transforming testing successfully shift from waterfall-style manual testing processes to automated continuous testing integrated into CI/CD pipelines.
What caught us off guard, however, was CXOs’ highly inflated estimation of their continuous testing, DevOps and Agile maturity.
According to the report, only 26 percent of non-CXOs reported that their organization was following all five of these best practices—but 57 percent of CXOs believed that they were. According to the report:
While most Agile and/or DevOps firms have implemented some of these five core testing practices, few have actually implemented all of them — only about a quarter of respondents say they somewhat or completely agree with all five best practice statements.
This is an important distinction. While many CXOs and high-level decision makers believe they are ahead of the pack, our research shows that only the small fraction of firms that follow these best practices have more advanced Agile and DevOps practices compared to their peers — and their behaviors and attitudes support that assertion.
Moreover, CXOs also reported much higher levels of Agile and DevOps maturity across the board than non-CXOs did.
The good news for DevOps professionals is that CXOs are your champions: They believe you’re doing a great job! The bad news is that, at some point, you need to tell them about the gaps that currently exist in the process. You also need to collect and share some of your own data to make the case that actually adopting these five proven best practices will help you increase delivery speed while reducing business risk.
Detailed Report Findings
This was just one of the many interesting findings from research project, which was led by Forrester Vice President and Principal Analyst Diego Lo Giudice.
As a high-level summary, the research uncovered the following:
- Understanding of business risk is the most important determining factor of DevOps and Agile maturity.
- Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).
- Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.
Detailed results and analysis are available in the complete report: “Forrester Research: What Separates DevOps + Agile Leaders from Laggards?”
Report Methodology
In this study, Forrester conducted an online survey of 603 enterprise organizations in North America, Europe and Asia Pacific to evaluate current software testing practices and metrics tracked during survey development. Survey participants included decision-makers and individual contributors responsible for their organizations’ Agile and/or DevOps development strategies. Questions provided to the participants asked about their firms’ attitudes toward software development automation, risk management and testing practices, as well as the metrics they track and value in the software development life cycle.
