When news broke earlier this month about the nation’s largest bond insurer exposing sensitive account information to Google’s indexing spiders due to a server misconfiguration, many an IT professional wondered how such a massive fail could have been neglected for so long. Security experts called it out-and-out negligence by the company, MBIA. But the truth is that plenty of other businesses with traditional IT departments could have just as easily suffered the same fate. MBIA was just unlucky enough to have caught the eye of security researchers and inquiring journalists.
The fact of the matter is that most IT shops today still stink at configuration management. According to a survey out by Avaya earlier this year, 82 percent of organizations today still experience network downtime due to human error around configuration changes. In MBIA’s case, it wasn’t downtime but a data exposure that MBIA suffered as the result of a misconfigured Oracle Reports server tied to a web application. The issue allowed back-end data such as account numbers, routing numbers and other sensitive data to be indexed by Google, along with administrative credentials that would let attackers access any other data not already out there for the world to see. Avoiding these kind of blunders stands as one of the big value propositions of DevOps processes and the orchestration toolsets that support them.
“If they’d used DevOps, they’d probably have caught this issue much sooner,” says Rajat Bhargava, CEO of JumpCloud. “They should have controlled configs and change management, and DevOps is generally useful for those things. You make a change, you make it in code and theoretically people ought to be able to document these changes and then apply them across the board. So it is a little more controlled than someone going in manually and changing a setting and forgetting about documenting it.”
However, when it comes to reducing the overall risk posture of infrastructure managed by the constant flux of DevOps’ fast iterations, Bhargava believes many organizations could stand to take their practices to the next level. He says they can do that by better embedding security testing into the DevOps cycles.
“Clearly, DevOps shops are already doing automated testing around their code,” he says. “So there’s no reason why we can’t embed automated security testing just as we’ve automated QA and so on.”
This means not only testing the security of the code itself, but also of the configurations and overall system security once that code goes live. Doing so acts as a backstop to the already automated configuration management process, which can be huge since a human error at the top of the orchestration stack could prove even more disastrous than the kind of one-off configuration error that hurt MBIA.
“With great orchestration and configuration management tools, comes great responsibility. A simple misconfiguration of a system is only going to be compounded to the nth degree when you introduce DevOps configuration tools,” says Andrew Storms, a security and DevOps expert.
Just as DevOps can speed up update cycles for software and configuration, adding security automation and testing into the equation can speed up response to problems—whether they come from internal human error or news of new threats that need immediate attention from both dev and ops teams.
“For example, I know of companies that patched their entire DevOps enterprise for Heartbleed in a few hours,” Storms says. “Security best practices dictate that the security of systems be continuously checked in all environments — from early in the development process all the way into the production environment. Security is struggling to retool their processes and mold their staff to fit into the new ever changing DevOps world. It’s not just DevOps shops that need to do a better job at incorporating security into their processes. However, combining DevOps automation tools with security products is a true force multiplier that is our next best hope to get security right.”
Rich Mogull, CEO for analyst firm Securosis, agrees. He sees the kind of embedded security testing that Bhargava is advocating for as a huge opportunity for DevOps organizations.
“If you’re using the DevOps pipeline, to be able to go ahead and add security testing to that is a massive opportunity, and there are a bunch of tools out there to do some awesome testing (whose effect) would be extremely difficult to replicate using non-DevOps, traditional deployment testing,” Mogull says. “Not only is there an opportunity to get security engaged with automated testing, but you combine that with the building of a secure DevOps pipeline and I think we can wipe out entire categories of security issues.”
For example, at Black Hat Europe this week, Mogull says he talked to a company that no longer has to deal with the traditional firefighting exercise of patching machines.
“When they have to patch for something they actually just push the updated instances through their automation,” Mogull says. “So they’ll rebuild a new image and when they’re done with that, Jenkins will push it out into production and terminate the old instances.”
Mogull also believes that security testing of code is a big part of the value proposition of embedding security into DevOps. Security vendors like Veracode are now starting to tie into Jenkins to let developers know code has security issues and where to find those issues before pushing them into production. And tools like Gauntlt are making it possible to spin up cloud environments and automate vulnerability assessment, network scans and configurations and flag policies through a Jenkins plug-in. Additionally, in the same vein as the patching example, incident response to viruses can be completely rethought, because “if an instance gets infected or something along those lines, they blow them away,” Mogull says.
On a different security front, Mogull says the early adopter he spoke with at the conference also disabled SSH for admin log-ins to their production systems.
“Because, with DevOps, nobody should ever log into a production system,” Mogull says. “Any changes you make, you make at the beginning of your pipeline and then that gets pushed out and flows into production.”
