Today’s question in this recurring series comes from a developer in a typical large enterprise – risk averse, tightly controlled, highly secured – a major challenge to any DevOps transformation:
Q. I work in a large government department that is tight on security. We have strict policies on separation of duties, can’t share users IDs or passwords, struggle to share code, can’t use AWS or Azure, audit everything, and so on. How can DevOps work in my environment?
If DevOps is about collaboration and removing bottlenecks, then it must include security, from planning to coding to testing to operations; building, delivering, and running applications that are secure by design. Unfortunately, for many enterprises security is a major roadblock in the application delivery lifecycle.
Yet security does not need to be a DevOps bottleneck. Indeed, these specific issues are solved problems. While I do not boast of a deep security expertise, I can address the solutions based on my experience (and some further reading – see below):
Maintain separation of duties
Large enterprises must maintain a separation of duties, to prevent (for example) a rogue developer from adding a back door to new code and putting it into production themselves, with no checks and balances. But staff can still have the access they need when they need it – and no more. When roles change, such as developers accessing the operations ‘flight deck,’ or operators being seconded to development, identity management and access controls can provide the right access automatically, as needed, audited and policy-compliant.
Protect privileged user IDs and passwords
For large enterprises it is never acceptable to share passwords, especially to privileged accounts like cloud control panel admins. However, privileged identity management allows staff to ‘check-out’ privileged IDs from a secure pool, use them as needed (e.g. for break-fix), then check them back in when they are done. As policy dictates, access can be subject to additional approvals, time limits, access limits, keystroke logging, session capture, etc. This allows on-demand access to sensitive IDs while maintaining stringent audit and control.
Provide secure cross-team interfaces
Part of DevOps is sharing – processes, skills, knowledge … but also reusing code and services (or microservices) through shared APIs, whether public or proprietary. This sharing accelerates delivery and reduces costs, but can also introduce risk. To prevent unintended consequences when sharing code and services, API security can enable sharing both internally and externally, but with controls like user authentication (e.g. OAuth), granular access controls, volume/rate limiting, blocking and more.
Enable secure on-demand cloud access
Most enterprise staff cannot just fire up an AWS or Azure instance on the corporate card. Even applying for ‘approved’ access can be a multi-week process. Instead, security pros can set up automated self-service provisioning to ‘known good’ or ‘enterprise-grade’ services, pre-approved by security, HR, finance, etc. This helps to maintain necessary audit and control while enabling DevOps staff to quickly and easily adopt cloud and other services.
Use automation to maintain security and governance
Automation is a powerful way to reduce risk, ensure compliance, and maintain governance. For example, release automation allows developers to deploy new code without having personal access to production systems (the tool gets the access, not the individual). It can also restrict access to such systems through ‘known-good processes,’ to ensure deployment complies with policy. Tracking who initiates deployment maintains that essential audit trail for governance and reporting. Test automation, data management, process automation, configuration management and other automation tools all provide similar security benefits throughout the application delivery lifecycle.
Have a security culture
Of course, there is more to secure DevOps than technology solutions. Just as DevOps requires a new culture of collaboration, good governance ultimately requires a culture of security. Security is everyone’s responsibility at every step of the service delivery lifecycle. Security is still a specialist role, so security pros should have a place in agile planning, standup and triage meetings, but good governance is everyone’s role. Dev, ops, test, QA and others all need to step to the security challenge; and security pros should share their requirements, knowledge, best practices and tools with the rest of the DevOps team.
Further Reading
In compiling this post, I leaned on many others who have helped to establish the concepts of DevOpsSec / SecDevOps and Rugged DevOps. A great starting point for more specialist detail on this topic is the SecDevOps content here on DevOps.com, particularly the work of Rich Mogull (@rmogull) and Andrew Storms (@st0rmz). For an early intro, check the Gartner note, DevOpsSec: Creating the Agile Triangle and the seminal work of James Wickett (@wickett) on Rugged DevOps. For a more recent thought, try the Threatstack blog, and Edward Haletky (@texiwill) for a good end-to-end view. Finally, for a great resource specific to DevOps auditability, check the Gene Kim (@RealGeneKim) project, the DevOps Audit Defense Toolkit.
