Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
We have done amazing things with Agile and DevOps, increasing IT responsiveness to levels that most people would not have believed and our business counterparts only dreamed of even a decade ago. Think about it—we can check in a single source file and kick off a chain of events that involves half a dozen or a dozen software tools that collectively shepherd an entire application through the development process to deployment. Through all of that, only deployment might require human interaction, depending upon corporate policy. In many places, even deployment does not require a person. Just development.
We didn’t get to this point by moving slowly, and sadly, we did not get to this point by acting in a security-first manner. Thanks to the security-toxic mentality that DevOps was born in, we have had to go back and try to bolt on security where it was ignored initially. But we’ve largely managed that process. We have tools that handle privileges, monitor connections, scan source, look for vulnerabilities in support software, even scan entire container definitions before the build.
Know what most of you still don’t have? A truly secure toolchain. This is bad policy. A knowledgeable attacker is going to go after one of the aggregate locations. Places like container management servers that give them access to a lot of resources. Or the toolchain, which gives them access to every bit of software the organization develops. If an attacker wants to do code injection without detection, the toolchain in charge of builds and deploys is it.
And yet, for ease of use and keeping up with iterations, many of you have credentials for the toolchain stored in easily accessible places so that whichever tool is running the show has access to credentials for each piece of the toolchain. I wish I could say “No one keeps that data in flat files anymore,” But alas, I cannot.
One of the reasons it is not sufficiently secured is that build toolchains fall somewhere between Dev security and Ops security. That means our teams that came from a development background are not focused on security for the build chain, and our teams/team members that came from an ops background are not focused on the toolchain selected and often initially deployed by developers.
Which brings me to my recommendation. Put security professionals on the toolchain to verify it is locked down at every step. There are a lot of places for the toolchain to be insecure, not just poorly stored credentials. And the toolchain touches all of IT’s crown jewels. So make sure it isn’t wide open. If your organization doesn’t have dedicated security staff, bring in consultants to review the toolchain and make recommendations. There are some great security teams for hire out there that will be happy to lend a hand.
And keep rocking it. Your teams are building and deploying at a rate never seen before—even if DevOps is “slow” at your org, it is still likely faster than its predecessors—for doing anything. So keep working miracles, and lock down your castle’s the inner keep.
