Most development teams want to increase the pace of their software delivery. As such, continuous integration and delivery (CI/CD) has grown in importance, helping push code from build to production as seamlessly as possible. CI/CD pipelines often loop in many elements and may comprise a diverse stack of tools, automations and various languages. But because of this, the attack surface can be large. And as gaps in the CI/CD toolchain emerge, so do security concerns.
In recent years, we’ve seen several high-profile cases where CI/CD environments were compromised. For example, take the Circle CI breach, in which a bad actor accessed Circle CI production systems. Others include the compromise of the SolarWinds build system and the Codecov breach. CI/CD security issues are alarming because so much control and automation are given to a pipeline. Unauthorized CI/CD access could be leveraged to publish malicious code or steal sensitive information.
I recently met with Daniel Krivelevich, co-founder and CTO of Cider Security (acquired by Palo Alto Networks), to better understand the risks facing modern CI/CD. Krivelevich is a seasoned CI/CD security expert and co-author of the Top 10 CI/CD Security Risks framework, which OWASP adopted in late 2022. According to Krivelevich, there is a general lack of knowledge and visibility into these risks. “The CI/CD ecosystem as a whole is full of holes and opportunities for attackers,” he said.
Below, we’ll review some of the risks that CI/CD pipelines face and offer general ways to mitigate some of these concerns. Hopefully, with greater security awareness around CI/CD threats, both tooling providers and users can begin to diffuse the potential of attacks against their systems.
The Top Concerns Facing CI/CD Security
The OWASP Top 10 CI/CD Security Risks framework covers the most impactful risks within modern engineering systems and gaps in CI/CD. It was informed by analyzing many attacks in recent years and was compiled with insights from numerous experts. Krivelevich hopes the framework will spark discussions around the mindset defenders should have when they’re looking to protect the modern engineering ecosystem. The top 10 risks are as follows:
- CICD-SEC-1: Insufficient Flow Control Mechanisms
- CICD-SEC-2: Inadequate Identity and Access Management
- CICD-SEC-3: Dependency Chain Abuse
- CICD-SEC-4: Poisoned Pipeline Execution (PPE)
- CICD-SEC-5: Insufficient Pipeline-Based Access Controls (PBAC)
- CICD-SEC-6: Insufficient Credential Hygiene
- CICD-SEC-7: Insecure System Configuration
- CICD-SEC-8: Ungoverned Usage of Third-Party Services
- CICD-SEC-9: Improper Artifact Integrity Validation
- CICD-SEC-10: Insufficient Logging and Visibility
Before an organization can dissect these individual risks, however, they must overcome an even bigger problem—a critical lack of understanding regarding the CI/CD ecosystem and the many elements under the hood. Since CI/CD is not traditionally viewed as a security risk, this can result in insecure default configurations, insecure processes and systems not adopting a secure-by-design approach.
“There’s a big gap between the level of understanding and opportunities attackers have and the knowledge defenders have over what attackers can do,” said Krivelevich. “Organizations traditionally have not been proactive enough in understanding this, unlike other elements that are more traditionally recognized as part of the attack surface. The main gap is a mindset shift.”
Tips to Secure CI/CD
So, knowing the above risks, what are some best practices to secure CI/CD pipelines? Krivelevich shared some tips to keep in mind.
Bring CI/CD under the security umbrella. First, realize that CI/CD should be scrutinized. This will require an awakening in the community and more proactive effort than before. Krivelevich encouraged development teams to place it on their roadmaps and think about it.
Map the attack surface. Before implementing security measures, you need to know what you’re protecting. Therefore, it’s a good idea to map your CI/CD environment and produce an inventory of assets. Cataloging the total surface area of dependencies and integrations is necessary to understand risks and helps mitigate CICD-SEC-10: Insufficient Logging and Visibility.
Keep this updated regularly. Since the cadence of change is fast, teams will want constant visibility into their ecosystem and what’s under the hood. This may even lead to shadow IT, as per CICD-SEC-8: Ungoverned Usage of Third-Party Services. Therefore, Krivelevich recommended keeping a regularly updated inventory. This means continually mapping control systems, container registry artifacts, plugins and all other assets used as part of CI/CD.
Protect the software supply chain. Many building blocks make up the modern software delivery process. But, some dependencies may be abused or contain malicious code, as per risk CICD-SEC-3: Dependency Chain Abuse. Dependencies within the CI/CD process may be prone to a host of open source vulnerabilities.
Safeguard CI/CD credentials. CI/CD tools “possess secrets to most sensitive crown jewels in an organization,” Krivelevich explained. Therefore, access tokens and secrets related to CI/CD tools should be protected to avoid costly credential leaks. A zero-trust approach to CI/CD could help avoid CICD-SEC-2: Inadequate Identity and Access Management.
By creating a virtual heat map of areas to address, organizations will be more informed and better equipped to assess risk. And according to Krivelevich, by implementing these best practices, defenders have a lot of power to raise security expectations in the CI/CD space. “When defenders and users become more diligent about security, the requirements from vendors will increase.”
CI/CD is the Beating Heart, Don’t Let it Break
“The engineering ecosystem and pipelines themselves are the beating heart of engineering,” said Krivelevich. And as this system becomes a larger part of what makes a business tick, it will be more heavily relied upon for mission-critical tasks and retaining agility.
Yet, many building blocks comprise the contemporary software delivery process, and CI/CD pipelines can be a risky area due to the number of systems they touch. Further complicating things is the fact that the delivery ecosystem wasn’t traditionally viewed in security terms. This combination of many opportunities and limited security oversight makes CI/CD lucrative for attackers.
Rectifying the top 10 risks is a relatively easy gap to address, said Krivelevich—what’s not so easy is the mindset shift required to take CI/CD security seriously. Organizations that invested deeply into microservices and rapid engineering processes will most likely be more susceptible to attacks, he added, since they typically use a diverse array of tools and a complex, integrated delivery process.
