Jez Humble’s (@jezhumble) career has spanned roles through coding, infrastructure, and product development across three continents and organizations of varying sizes. To say he knows a lot about continuous delivery is a total understatement. In 2010, he and Dave Farley literally wrote the book on continuous delivery—and if you have not yet read it, I would suggest you pick up a copy. It’s a fascinating read.
I’ve heard Jez speak over the years at a number of DevOps conferences, and I was excited when he accepted our invitation to speak at All Day DevOps. At the conference, Jez presented “Continuous Delivery Sounds Great But It Won’t Work Here,” during which he addressed the four reasons he consistently hears from organizations on why continuous delivery won’t work in their organization:
- We’re regulated
- We’re not building websites
- Too much legacy
- Our people are too stupid
Jez also outlined the real reasons behind the stated reasons: Their culture sucks and/or their architecture sucks.
Jez spent his talk addressing each objection.
We’re Regulated
Jez noted it took Amazon four years to go from monolithic to service-oriented architecture (2001-2005) but the company did it. Do you think Amazon is the pillar of free enterprise, free of regulation? Not so. Being a publicly traded company, Amazon is heavily regulated; it is subject to Sarbanes-Oxley regulations; it handles personal and corporate financial and proprietary data; and it provides private cloud services to the U.S. government, including highly sensitive data for the Department of Defense.
Amazon Web Services’ GovCloud, in fact, provides the cloud platform for cloud.gov, the 18F project Jez worked on to address the authority to operate process within the government.
The old process had screened every software application before it was allowed on a federal IT system. It stifled innovation because it duplicated efforts multiple times over. There was no cross-agency assumption that if one agency approved a software package, it was okay at another agency. Cloud.gov moved controls from operations into infrastructure as a service. Now, agencies can receive software through cloud.gov to integrate into their applications.
The bottom line is continuous delivery is excellent for heavily regulated environments because it gives assurance to regulators by removing opportunities for human error.
We Aren’t Building Websites
Jez’s favorite example is HP LaserJet firmware. HP had a 400-person team spanning three continents. Obviously, the firmware was critical for new printers, but the team was moving slowly. Only 5 percent of their time was spent on innovation. HP tried hiring, firing, insourcing and outsourcing. With business school tried-and-true solutions exhausted, and knowing it had to eliminate non-value adding work to increase innovation, the company looked to continuous delivery with two business goals in mind: 1) Get firmware off the critical path; 2) Achieve 10X increase in development productivity, measured by percent spent on innovation.
HP had other goals:
- Reduce hardware variation
- Create a single package
- Implement continuous integration
- Implement comprehensive test automation
- Create a simulator
Note: the chart above doesn’t show the 23 percent of time spent managing the automation tests. Time well-spent.
In the end, between 2008 and 2011:
- Overall development costs were reduced by ~40 percent
- Programs under development increased by ~140 percent
- Development costs per program decreased 78 percent
- Resources now driving innovation increased by 8X
Too Much Legacy
Jez’s example here is Suncorp, one of Australia’s largest insurers (note: Also falls under the “We’re heavily regulated” excuse). Suncorp was burdened with loads of mainframes and legacy systems, but it built a set of test suites with open source tools to move into continuous delivery.
Our People Are Too Stupid
Jez looked outside of the software development industry into an old-line manufacturer: automotives. GM’s worst performing plant had been closed down. GM fired every one of the workers from the plant. The workers were notorious for problems ranging from sabotaging cars to drinking to gambling on the job.
Remarkably, GM’s union leaders convinced Toyota to hire the old GM employees to learn the Toyota processes, and Toyota hired the GM employees to run their new NUMMI plant. Toyota and the old GM employees showed it wasn’t the people that was the problem; it was the leadership, the management system and the use of obstacles. At the GM plant, an employee had a job and a certain amount of time to do it. If it didn’t get done, the assembly line kept running. At the Toyota NUMMI plant, employees were empowered to ask managers for help—without retribution. The car had to be in a good state before proceeded to the next stage of the production line.
In software development, this is continuous integration. Toyota’s NUMMI is the inspiration for Lean in the United States.
Jez’s take-home point for any organization is that results aren’t achieved when you adopt practices but don’t implement the culture. You need to:
- Agree and communicate measurable business goals
- Give teams support and resources to experiment
- Talk to other teams
- Achieve quick wins and share learning
- Keep going
Jez dives into more details in his talk, which you can watch here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.
