Consider how much money your enterprise spends on audits. Smaller firms not subject to external regulation should consider how much money is spent on quality assurance—or the lack of it. A business that suffers from a customer-facing quality incident will, at minimum tarnish the brand of its services or at the high end, lose market share altogether. Larger firms that fail audits can suffer from brand tarnishing as well as high fees, not to mention further scrutiny by regulatory agencies.
Failure to perform or failure to follow proscribed processes inevitably leads to a black eye in front of customers. The entire discipline of governance is premised on avoiding both. Whether your governance practices are efficient or not—and whether your “process” as it sits today leads to quality or not—is not the subject of this review; it is how DevOps and automation can shine a spotlight on both and radically reduce the costs you spend today to achieve them.
The Spotlight of DevOps
The spotlight of DevOps begins with consistency that leads to predictability. Automating any given kind of process or technology the same way every time begins to give insight on any “holes” from things that do not work (or work well). If all that changes in your deployment process, for example, are pointers inside a configuration file and the kind of testing executed against this version of the code, you have achieved a build-once, deploy-many state of deployment automation. Operations staff who support the deployment of change or maintain the environments in the software development life cycle (SDLC) begin to isolate any issues faster because they become familiar with “how” a given kind of technology is automated to deploy. This reduces the total number of deployment errors and, in tandem, reduces the length of time it takes to fix them. Any “process” problems or weaknesses in your governance model become apparent quickly and can be fixed quickly. Limiting variation is the medicine to keep them away in future.
The spotlight of DevOps next turns to transparency. Offering your entire enterprise visibility or “read only” access to your entire DevOps tooling suite has a profound impact on the transparency of change. As an example, consider that I am a humble accountant in your firm who has submitted an agile story to improve our billing system. I may or may not, given the nature of my job, be co-located near the programming team who will fulfill this request. If our firm specializes in accounting software then perhaps I will co-locate, but if I am only requesting a change to an ancillary or support system, likely not. Without direct access to the programmers through traditional agile co-location, my visibility into the progress of change is limited to the tracking system I submitted the request within (assuming you have one, and don’t use email to do this).
If I know a programmer in MIS, perhaps I ask him at lunch how my change is coming along. But if I have read-only access to the testing systems, or to the release orchestration (RO) tooling, I may get a much better up-to-the-minute view of exactly where my change is, how it is doing from a quality assurance (QA) perspective, and when it is scheduled for release into production. Based on this news, I may make decisions about how we bill our customers in what time frame to increase overall revenue to the company. I would notify my boss of same (who also has access, but pays less attention), and of any changes that impact the new timeline I create. The marketing team (who also has access) begins to work with public relations (PR) and the advertising guys on customer announcements, etc. etc.
The net impact of having universal transparency is that audiences who are not traditionally notified or interested in the inner workings of change movement now make decisions based on it. Consider that prior to this systemic access, and the reliability of DevOps to ensure delivery, the only news you got before was from “the IT guy,” who was often wrong. It does not take long before IT loses credibility from being wrong about delivery timelines, and these other audiences begin to pay less attention (if at all). At that point, an “I’ll believe it when I see it,” attitude emerges. And this same attitude infects internal auditors because of what they hear and see from disaffected internal customers who have been misled repeatedly by the failed manual methods of the past (pre-DevOps).
Processes that are disinfected over time by the consistency of DevOps begin to change the internal company thinking about the delivery of innovation—namely that it can be counted on as accurate. Transparency offered to all audiences of a company increases the visibility into all aspects of innovation delivery and better ideas emerge, both about what to do with completed efforts and how to improve the innovation process itself. And before you know it, your governance and QA processes become efficient, reliable and predictable. Variation is eliminated or greatly reduced, and the costs of operation and innovation begin to decline.
The Forgotten Stepchild of Automation
What is rarely a matter of focus in most companies is the secondary ability to automate the compliance history of a given application. Most firms are content (translate, only able to) work with their testing tools to produce logs about a given version of an application now residing in production. A test log does show an auditor the majority of the quality related information about a given version of an application. But the log is a highly limited view of what has taken place. What is needed is the full context of innovation. Enter compliance automation.
Consider for a moment, that requirements drive the train. Why are we doing the innovation we are doing? Is this a bug-fix, a new feature, a client request, a regulatory requirement? Being able to trace the code itself back to the requirements system that drove the need adds significantly more context than is possible from mere test logs. Next, think about the number of attempts that finally resulted in this version of the application. An automated build tool (part of our DevOps suite) has this information. At this point you could evaluate how many builds and deployments took place in development before a worthwhile candidate emerged to traverse the remainder of the SDLC. An automated deployment tool (part of our DevOps suite) can trace the number of environments this version of the app was deployed into for testing and how long the overall process took to get something moved from development to production.
And finally, testing is often done from a variety of tools, with results compiled in a variety of result repositories. This information may be available from a solid RO tool that tracks the gated progress of this version of the application, but if it’s not, it still should be sitting in each of the testing tools used throughout the process. Depending on the level of inherent DevOps tool integration (i.e., using a vendor suite) much of this information may “flow” from one tool to the next. But if not, the information does still exist in the logs produced by each of the DevOps tools.
It may be well worth the application of internal programming resources to put together a tool that gathers all this information (automates the collection and presentation of) for audit purposes. Imagine the reduction in cost of supporting an audit that can gather a simple push-button report (or dashboard) of the entire life cycle of a given version of change. At that point, you have the entire context for a piece of innovation. You know what drove it (requirements). You know what it took to create it (build and deploy). You know the entire history of testing it (testing tools). And, finally, you have the entire governance process used in its deployment to production (RO tooling). But instead of examining multiple tools to collect and present it, you have created a push-button system to do so.
Impacts to the Bottom Line
The investment in adding automation to the compliance history of innovation pays off in more than one way. It is not just the ease with which history can be recited. It follows the same principles as DevOps itself. Automation introduces consistency in how audit information is gathered and presented (reducing variability from one audit to the next). Automation introduces transparency if the entire company is given “read only” access to this new system. Automation shows auditors we are not afraid to learn from our mistakes and proud of our accomplishments. And in effect, it reduces the need for audit altogether.
Typically, regulatory bodies investigate and audit firms where they are uncertain of how the results were obtained. An automated system like this one fully eliminates that need. Transparency as could be offered here, may convince auditors that attention is not warranted here. Keeping in mind the spotlight of DevOps has already improved quality and streamlined process, add to that full transparency of audit materials and the confidence of auditors regarding your company may increase significantly. Routine becomes the only reason to audit, rather than suspicion.
Invariably the “new” will sometimes result in an error that your current systems and processes (automated or not) will not catch. But this truism is recognized, and accounted for, in how DevOps enables innovation. And if you have this kind of automation in place, it can be demonstrated in how we fixed what we fixed (whether governance or QA). Your strategies in DevOps should help you achieve this level of confidence. (If you need help, feel free to contact me, I am currently looking.)
To continue the conversation, feel free to contact me.
