‘Everything as code’ can create opportunities for misadventure for DevOps teams
DevOps’ ultimate aim is to create efficiencies in the software delivery process. But it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. Following is a discussion submitted by Director of Engineering Laura Frank Tacho about deploying code out of step.
Scenario:
Running in parallel during CI/CD can speed up your automated testing and shorten your feedback cycles. Just make sure deployment is not executing as a parallel step. That misconfiguration will cause your code to deploy at the same time the tests are running, regardless of their exit codes, pretty much defeating the purpose of automated testing before a deploy.
This scenario touches on a theme we’ve seen throughout most of the “DevOoops” stories our colleagues have come up with. The theme is how the emerging concept of “everything as code” can create opportunities for DevOps teams to slip on proverbial banana peels.
In this case, we’re talking about sequencing problems in a process we can call pipeline as code.
To be successful, teams need to extend a quality-first approach in automated testing throughout their continuous delivery life cycle—from code to commit to test, as well as through the various tiers of the development life cycle. And with a quality-first approach, you need to apply testing and quality management to your environment configurations and infrastructure as code.
Laura’s example highlights how teams need to treat their CD pipelines as applications themselves. They need to check for and eliminate bugs and errors in CD pipeline code to ensure that code is operating as expected. Failure to properly manage the creation and evolution of complex CD delivery pipelines will result in significant cost or risk.
So, if you don’t have proper test coverage on application code, you could end up releasing an application that hasn’t been tested at all. If the app doesn’t pass through a security gate, as it’s supposed to, you could release unintended preproduction data.
While there are always risks with untested code, we have to give unique consideration to untested or unmanaged CD pipelines. We need to consider the risks of what missed gates could mean for our developers, our organization and our customers.
Teams should do the following as they release their CD pipelines:
- Build: Start from the left-hand side of process and work through the defining and coding of CD pipeline left to right in the direction your apps and development artifacts flow: from dev to test to deployment. While we may not be able to define an ideal automation solution completely on the first pass, going left to right encourages teams to focus on what the life cycle of an artifact looks like.
- Manage: Define your CD pipeline using an everything-as-code or pipeline-as-code approach. For this to add value, you must manage development of your CD pipeline as you would your application code and apply software best practices. Check your CD pipeline code into a source code repository and leverage language-specific technologies that can check your syntax. (If you’re using Groovy-based code or Jenkins-based code, there are versions of Lint that can perform these functions.)
- Create staging environments: You should establish staging environments or test pipelines to validate your CD process. These pipelines should go through all checks and gates against your real tools, using reference code that is representative of the applications your pipeline would be servicing but stops short of production into environments.
This will give you an opportunity to iterate changes against your application code, do some initial scans and checks to collaborate to define your CD pipeline in code-managing SCM. Execute initial validation on scans as pipeline as code and then validate expected operations in a staging environment before you deploy it as the basis of the pipeline for your production of business-critical applications.
Key Takeaways
- This story highlights how crucial it is to ensure that your application code and environments pass through certain gates and checks before being deployed out to the customer.
- You need to construct your CD pipelines so your parallel testing is managed as a gate to production, ensuring that properly tested and validated changes are not automatically deployed.
- Deployment happens at the end of all the processes within a pipeline, and all required tests, validations, checks and approvals have been done before any change is deployed to the customer.
To read our previous “DevOoops Moves” post, click here.
