Companies that want to better address market demand and more efficiently deliver value through speed, quality, and agility are embracing DevOps. This includes retailers, insurers, defense contractors, and most certainly software makers.
Consider the DevOps journey within the IBM Software Group, which is unarguably larger than most enterprise software groups: currently 519 major software product titles, 36,000 software engineers, 116,000 virtual machines supporting development and testing, and 44,000 physical servers. “This is DevOps at scale,” said Matt Ellis, vice president mobile and cloud DevOps at IBM, recently at InterConnect 2015.
Getting teams and processes of that size – while cutting waste associated with outdated ways and duplicate toolsets – is no easy feat. But if it can be done, the benefits are substantial. For IBM, the efforts do appear to be paying off – if the results recently shared over the past six months at the DevOps Enterprise Summit 2014 and the IBM InterConnect 2015 conferences are any indication.
According to Dibbie Edwards, VP DevOps for hybrid, continuous engineering and application lifecycle management development at IBM, IBM has gone from spending about 58% of its development resources on innovation to about 80%.
What did that shift in focus do to IBM’s time frame? In 2008, IBM’s time to project initiation took 30 days. That was reduced to between two and three days in 2014. Groomed backlog was reduced from 90 days to one day and sprint test times from five days in 2008 to 14 hours today. Overall time to development went from 120 days to three, with time between releases reduced to three months from 12.
“IBM is seeing the value of DevOps, and as you heard themes through each one of those sessions, it’s really about speed,” said Edwards at the DevOps Enterprise summit this past fall. Speed has always been important, to be sure, but with the rise of cloud, analytics, mobile, and social, it’s become more important now than ever.
How did IBM begin its DevOps journey?
According to Edwards, it was systematic. The teams examined the work they were doing: all of their processes – from how they worked with their business stakeholders to how they were working with clients – and eliminated all of the inhibitors and friction points. It hasn’t been a swift journey – a couple of years – but it has been a meticulous one.
According to Edwards, during her talk at the DevOps Enterprise Summit 2014, Real-life Experiences Leading Large Organizational Transformation at IBM, the software group focused on uncovering every manual process that it could – things such as how the tests were being created, building more efficient testing tools, and building a page object framework in selenium that its developers could leverage. “That framework itself has allowed us to more than double the number of automated test cases that we have,” she said. “That very significant ability gave us the option to do our function verification testing alongside our development.”
With manual inefficiencies identified and senior leadership agreement to prioritize how processes would be improved and how much will be invested on clearing technical debt, the IBM Software Group then had a foundation on which it could build development processes based on DevOps principles and current business objectives. “We proceeded to build a groomed backlog based on those business priorities and on the size of the development work,” Edwards said. “And if the teams came up against work that they didn’t understand, they would run an exploration phase within their sprint to better understand what needed to be implemented.”
With that groomed backlog in place, IBM could then make more informed decisions at the end of each development sprint – and even pivot, if needed. “If we had a customer need or a business change where we need to shift the focus of the individual teams, we had the flexibility to do that,” she said.
That ability to pivot had been a big benefit for the IBM Software Group. “I recall just how many times we would just churn over these decisions and remake them and make them again,” she said.
Continuous Testing Automation
The culture at IBM has moved to full automation ahead: everything should be automated, and it’s an exception if it isn’t. “I think this is so critical. I saw a real shift in our team as it started to see the value of this and recognized that doing things manually just was not acceptable and really not sustainable, given the level of speed that we were trying to achieve,” says Ellis.
Previously, IBM completed testing in following sprints, because testing teams just could not keep up. Getting to this point was an effort that took focus on continuous improvement. “One thing I always share with the team is how critical learning at the end of each one of our sprints is. And then planning for the next sprint to eliminate the friction points and the areas where we were less efficient than we should have been,” she said.
Finally, IBM automated the testing of its more complex customer usage scenarios. “It literally used to take my team 30 hours to set up an environment to do the testing because we did everything manually,” she said. “We would have to set it up, break it down, and reset it up, because we had all the different databases configurations. By using patterns and automation, we were now able to deploy automatically within three hours.”
