A survey of 200 executives working for organizations that have revenues in excess of $350 million finds industrial coding errors cause manufacturing shutdowns lasting 31 hours on average, costing $4.2 million per hour for a total of $126 million per shutdown.
Conducted by Copia Automation, a provider of an industrial DevOps platform, the survey finds half of all downtime is caused by industrial code changes, code confusion, lack of visibility into industrial code and issues with programmable logic controllers (PLCs), the survey finds.
The most common causes of unplanned downtime are cybersecurity breaches (47%) followed by hardware malfunction (45%), coding/software issues (41%), human error (32%) and environmental disasters (25%).
Gerry Abbey, director of product marketing for Copia Automation, said the survey makes it clear that as far as the application of best DevOps practices in the realm of industrial platforms much work still needs to be done. For example, more than three-quarters (79%) said ad hoc fixes in industrial programming are commonplace, the survey finds.
Additionally, survey respondents spend an average of 10X more time (45 hours per month) debugging code than reviewing it. Nearly all respondents (99%) said they have a code review process in place, but 80% conceded their organization spends less than four hours per month reviewing code.
There is also an unfortunate tendency to focus on temporary fixes that prioritize restoring operations over addressing root causes, he added. As a result, organizations are likely to encounter the same issue multiple times over, said Abbey.
In general, more organizations than ever are deploying software on a wide range of industrial platforms but much of that effort remains manual, which in turn leads to more errors, misconfigurations and outdated software that have known vulnerabilities for cybercriminals to exploit at a time when the overall size of the attack surface that needs to be defended continues to exponentially expand, noted Abbey.
On the plus side, the survey finds nearly all respondents (97%) said that they are aware of industrial DevOps. However, 44% cited competing priorities as the top challenge to adoption, followed closely by a lack of interest from management (39%).
That may become a larger issue as more organizations look to deploy artificial intelligence (AI) on industrial platforms. Those models should be deployed on a DevOps foundation that makes it possible to programmatically deploy them, said Abbey.
Edge computing platforms that include industrial systems are the next frontier for DevOps teams. As edge computing continues to evolve there may come a day when more workloads are running at the edge than there are in the cloud. Less clear is to what degree those workloads will be deployed by a DevOps or platform engineering team or an existing operational technology (OT) team that has embraced best DevOps practices.
One way or another, the amount of code being deployed at the edge and industrial platform will eventually force a DevOps discussion. Organizations are not going to be able to hire a small army of IT and OT professionals to manually manage these platforms, especially when platforms that automate DevOps processes across a wide range of IT environments are now readily available.
