Patch management has been a thorn in IT’s side for decades. While it is undeniably necessary for security and stability purposes, the staggering array of patches that are available make managing them effectively nearly a full-time job – or a job better left to users. Few IT professionals want to be the “patch person,” and yet, nearly all IT professionals want their systems protected by the latest patches. No one wants to find out an intrusion was easily preventable via patching. It goes without saying that the SolarWinds attack has left quite a few IT departments scrambling for an alternate solution.
The world of patch management has gone through several evolutions that we won’t cover in much detail here. Instead, we’ll focus on what’s available, what you should look for and where the market is headed. Suffice it to say that patch management as a practice has grown from an individual applying patches to servers and server apps while employees were encouraged to patch their equipment, to complex systems that manage the patching themselves, with IT oversight. Currently, patch management is often bundled with endpoint management, simply because many of the patches are going to end points (user machines and devices).
Out of this growth have come several market differentiators that you should be aware of when looking into a way to manage the avalanche of updates the average organization is seeing.
- Centralized or de-centralized? Increasingly, patch management is run on servers/cloud and pushes changes out to those machines/devices that need them, or agents on each machine manage updating themselves. Which is best for a given organization really varies. If a lot of machines are running at the edge of their capability, an agent running updates in the background might be too much to ask. In that scenario, a server-managed install environment is probably better. Conversely, if a centralized location to run updates from is considered an architectural weakness (because it requires access to the update pushing engine), agents might be the better plan.
- OS support. We’re way past the days when single-OS solutions were considered useful. List the operating systems used in your environment, and choose a product accordingly. Most products on the market support Windows/Mac/Linux, but there are variances. First off, make certain your chosen product supports all three. Then ask, “Which flavors of Linux?” Make certain the flavors running in-house are supported by the software.
- App support. There are a lot of applications out there. Which are supported by the tool? How much leeway does the customer have for adding applications by pointing the tool at update locations? Does it support FOSS updates? There are a lot of those too, and which ones matter. Make certain the tools the organization uses are supported or can be added.
- Many tools will scan machines for update issues. Understand what your chosen tools scan for, and how the scans impact performance on the target machines.
- Updates are not 100% perfect. There will inevitably be problems. Make certain the tool can roll back changes. Updating 50 core infrastructure machines, only to find, hours or days later, that the update breaks something demands roll-back capability.
- Inter-relationships. Updating a piece of software can require that another piece of software or a driver be updated, which can demand that UEFI/BIOS be updated. Or the CMOS on a RAID controller. Understand how cascading updates are handled by the tool, and if the cascading updates can be packaged, so that a failure automatically rolls back the updates applied thus far.
- Approvals. Speaking of driver and BIOS updates, what tools are available to allow a human to approve changes? No one wants willy-nilly updates to the drivers on core systems – or even to drivers on user systems.
- Security of the tool itself. We’re in perilous times where IT security is concerned, and frankly, an update command-and-control server is an attacker’s dream machine – take it over, put your back-door update into the mix, and tell it to update every machine it manages. It cannot be understated: an update management tool must be as protected as possible – both by the customer locking down the installation and by the developer making sure the code is solid.
Once the system is chosen and worked into your DevSecOps environment, updates will be more responsive, which leaves one more thing to consider – does it feed into your agile tools? If the update management tool notifies a project management tool like JIRA when it updates a server, then if code on the server breaks, the developers aren’t in the dark about changes, and will have breadcrumbs to help them track down the source of the problem.
Imagine if manual updates became a rarity instead of the norm. There is a cost associated with this process, but most of the products on the market can be had for a reasonable price – if you waste a couple hours a month (or for some products, a year) on updating, they’re paying for themselves. Of course, your time is a sunk cost, and the cost of these is an operational cost, so there is an evaluation your organization will have to undertake. Given how busy everyone in DevOps and DevSecOps is, it is likely worth the investment for all but over-staffed organizations. What? There are a few!
Find what suits your organization, and put it to use. Faster updates, less time investment. The idea is sound, you just need to figure out what suits your needs. And keep rocking it. You are managing updates already, this is just a way to make your life easier. And we could all use a bit of easier in a busy, complex IT environment.
