Patch Easier, Then Faster

(Day 8 of 22 in our Cybersecurity Awareness Month series. Today: patching)

There is a saying that “slow is smooth, and smooth is fast.” It’s usually credited to the military. It means that rushing leads to mistakes, and cleaning up after them takes longer than doing the work carefully in the first place. Thinking about it more, we realized this idea applies to patching, too.

The number of vulnerabilities has grown enormously, and much of that is due to AI and its ability to find vulnerabilities quickly. Put frankly, nobody can patch fast enough to stay ahead of all of it. Or, stated a different way, nobody wants to. It takes constant heroic effort, and teams that try burn out. The answer, then, is not to patch faster. It is to make patching so routine that it barely needs anyone’s attention. Who cares how fast patching is if you don’t have to do much to make it happen?

This post is a collection of ideas for getting there, collected from presentations and discussions over the last few years.

No More Picking and Choosing

Patching has to happen, and you need to take all the patches. That has actually always been true, but for years, organizations could get away with skipping updates, waiting a few quarters, or applying only the ones that looked scary. Attackers were not nearly as relentless as they are now.

That doesn’t work anymore. As we covered in our post about vulnerabilities, attackers chain small flaws together, and AI has made building those chains fast and cheap. A backlog of Low and Moderate issues is just as helpful to a modern attacker as a single critical issue.

Triaging vulnerabilities has also become much harder. A CVSS score can no longer tell you how useful a flaw will be to an attacker, and even experts struggle to judge that. You can’t always tell what an update fixes, either. Every detail a vendor publishes teaches attackers where to look, so release notes may not show how important an update really is. Picking and choosing patches by severity no longer works. Take them all.

At the same time, nobody can patch everything the moment it ships. That’s where defense-in-depth and the rest of your security design come in. Isolating management interfaces, controlling who has access, and hardening your systems all buy you time. That time lets patching happen on a steady schedule instead of in a panic.

Build Systems That Are Easy to Patch

A lot of what makes patching hard comes from how clusters were built, as well as organizational policies that let applications dictate how infrastructure has to behave. The current AI worries make this a good time to renegotiate some of that.

Trust vMotion

VMware vSphere vMotion is mature and reliable. For all but a small number of applications, it moves running workloads between hosts without anyone noticing. It is the foundation that lets you patch a host in a way that’s invisible to workloads and their downstream consumers.

Some workloads genuinely can’t be moved, such as some VMs that use passthrough hardware devices, or ones with real-time requirements (telco applications, etc.). Others have owners who insist they can’t be moved. Test that claim. Applications that cannot live-migrate present a serious issue to infrastructure security, and many applications that people think can’t move turn out to move just fine. Try vMotion on them in a controlled way, outside business hours, with the application owner involved. In many cases, organizations have found that the major improvements to vMotion in the vSphere 7 timeframe let the application move without disruption.

For the applications that truly can’t move, many organizations put them on their own cluster and patch it on its own schedule, with planned downtime. The rest of the environment can then stay current without waiting on them. That cluster will likely be patched less often, so treat it as higher risk. Give it tighter network isolation and stricter access controls than the rest of the environment.

Also keep in mind that VMware Tools has pre-vMotion hooks you can use to automate things, like a drain-stop of a cluster node in a load-balanced environment. Similarly, VMware vSphere Distributed Resource Scheduler (DRS) in VMware Cloud Foundation (VCF) 9.0 and later works to minimize the number of vMotions while patching, so any actual disruption to an application would likely be only once.

Keep Spare Capacity for Patching

Design every cluster with at least N+1 capacity, meaning enough spare resources to run every workload with one host out of service. That leaves room to take a host out for patching. Then protect that space for the infrastructure. A cluster that’s 100% full is at N capacity. The +1 is yours.

If it helps to have a technical limit to point to, let VMware vSphere High Availability (HA) admission control enforce it for you. When someone wants to power on more VMs than the cluster can protect, admission control says no, and you don’t have to.

Respect the vSAN Timers

When a host enters maintenance mode, VMware vSAN asks what to do with the data on it. For routine patching, use the default, “Ensure accessibility.” It moves only the data needed to keep every VM’s storage accessible, and then vSAN waits before it starts rebuilding anything. That wait is the object repair timer, which is 60 minutes by default. If each host finishes patching and rejoins within that window, vSAN does not need to rebuild data.

Don’t override this with “Full data migration” for routine patching. Evacuating every host’s data, one host at a time, turns a quick reboot into hours of data movement across the cluster. It adds load and risk to every patch cycle for very little benefit. It also wears out your storage faster. Flash devices, whether SSD or NVMe, are consumable components that can handle only a finite amount of writing over their lifetime. Copying a host’s worth of data around the cluster every patch cycle uses up a substantial amount of it. Save full migration for hosts that will be out of service for days, or that are being retired.

Keep the Image Lean

VMware vSphere Lifecycle Manager (vLCM) manages each cluster with a single image. The image holds a base VMware ESX version, your server vendor’s add-on, drivers and firmware, and any extra components. Keep that image lean. Hardware vendor ISOs often include custom drivers and applications for a wide range of hardware and uses, and you may not need all of them. Build the image to include only the drivers your hardware uses and the components your environment actually runs, and nothing else. Every extra component needs testing and has to stay compatible, and problems with extras like these are a common reason an upgrade stops partway through.

Update Firmware and Drivers Together

Server vendors generally test new firmware with new drivers. Running a current ESX build with firmware from years ago, or the reverse, puts you in a combination that nobody has tested. Updating both together leads to fewer surprises. Your server vendor and the Broadcom Compatibility Guide list which combinations of firmware, drivers, and ESX are supported.

vLCM can include firmware in the cluster image when your server vendor provides a hardware support manager, so firmware and drivers move together. A hardware support manager is an extra component you may not want to run, secure, back up, and audit, though. Another idea is that many servers’ out-of-band management controllers can stage a firmware update so it applies at the next reboot. The host then picks up the firmware during the same restart as the ESX patch.

If you stage firmware that way, turn off Quick Boot on those hosts. Quick Boot speeds up restarts by skipping the server’s Power On Self-Test (POST) initialization, so a staged firmware update may not apply.

It can also help to set one host to download and stage new firmware automatically. That gives you a place to see whether new firmware fixes a problem, or causes one, before the rest of the cluster gets it.

Roll Out Patches in Stages

Patch VMware vCenter first, then one host, and then wait a few hours before doing the rest. If a problem would only show up during a daily peak or a nightly batch job, wait until the host has run through that, too. Test environments are important, but a test environment is never quite like production, so treat that first production host as the real test. Watch that host closely, but don’t let nerves turn into not patching.

Some organizations also wait a few days after a release to see whether others report problems. For routine updates, that can be reasonable. For a vulnerability that is being actively exploited, waiting is usually riskier than patching. The VMware Security Advisory (VMSA) and its companion Q&A, when there is one, can help you decide.

DRS rules can help with these processes:

• Put low-priority VMs on the first host. Use VM-host rules to place some lower-priority VMs on the host you usually patch first. Test, development, and other Tier 3 workloads are good candidates. If something goes wrong, it affects workloads that matter less. Make them “should” rules, not “must” rules, so DRS can still move those VMs when the host enters maintenance mode.
• Keep redundant VMs on separate hosts. Use anti-affinity rules for VMs that provide the same service. Examples include a pair of domain controllers, your DNS servers, or the nodes of an application cluster. If the host you just patched has a problem, only one of them goes down with it.
• Pin important VMs to specific hosts or groups of hosts. Important VMs to a VCF environment are things like vCenter, VCF Operations, and perhaps your local identity provider and DNS resolver. Create a DRS rule that says they “should” run on a particular host, so that if something does go awry, you can find them without wasting time searching host to host.

Let the Automation Do the Patching

The main way that organizations are surviving the need to update more frequently is through automation. For VCF, vLCM remediates a cluster one host at a time, and you don’t need to sit there and watch it, though you should check in occasionally. If a host fails to remediate, vLCM is designed to stop and report the error. The hosts after it are left as they were. A bad patch does not roll across the whole cluster. If it stops, find out why, fix that, and run it again. A stop usually points at something that makes patching harder than it needs to be.

Shorter host downtime also means less to watch. Live Patch applies eligible ESX patches without restarting the host, and Quick Patch reduces downtime when you patch vCenter. Keep in mind that Live Patch works when you apply patches in sequence. If you skip one, catching up means a traditional restart, which is one more reason to stay current.

Make Patching a Standard Change

ITIL, which originally stood for IT Infrastructure Library, is a widely used framework for running IT services. Many organizations base their change management process on it, even if they don’t call it that. ITIL sorts changes into three types:

• Standard changes are low risk and follow a defined procedure. They are pre-approved, so they don’t need an individual review each time.
• Normal changes go through assessment and approval before they are scheduled, often by a change advisory board.
• Emergency changes need to happen as soon as possible, such as fixing an outage or closing a critical security hole. They get a faster approval process.

Infrastructure patching should be a standard change. Many organizations still treat every update as extremely risky and send every patch through a lengthy change advisory board review. The problem is that the only way to gather enough data to determine if there is truly a risk to your environment is to actually do the update. Without actual data you’re just sitting there letting attackers have a turn.

Patching a host is mostly vMotion and a restart, and even less now with Live Patch. vMotion is probably already a standard change in your organization, whether anyone has explicitly decided that or not. DRS moves workloads with vMotion all day long, and nobody files a change request for it. If moving the workloads off a host is routine, why should patching that host be much different? The risk is lower than it looks, and the way to show that is a track record. Patch in regular maintenance windows, as described below. Use Live Patch and Quick Patch where the patches support them. Keep notes on each patch cycle, such as what you patched, how long it took, and any problems. Bring those notes to the change advisory board. A string of boring patch cycles is the best argument you’ll have.

Lean into the change management definitions, and if your organization doesn’t use the ITIL terms yet, start. A critical vulnerability is exactly what an emergency change is for. Declaring an emergency change for a critical vulnerability is not an abuse of the process, because the risk of not patching is much higher than the risk of patching.

Plan for a Monthly Patching Cycle

When you plan your team’s activities, assume a full patching cycle every month, for everything. Some months will be quieter, and that’s fine, but it’s better to assume the worst. If it takes your team more than a month to patch everything, then work needs to go into finding the slow parts and speeding those up.

Define a weekly maintenance window for management components, such as vCenter and VCF Operations. ESX hosts don’t need one. With vMotion, workloads keep running while a host is patched, and in ITIL terms vMotion is a standard change, or should be.

Consider filing change records in advance, possibly even for the next year or two. You can always cancel them, and it’s worth it to get your maintenance on the schedule so that other activities don’t conflict. If your change managers disagree, get it in writing, and then lean into the emergency changes until they conclude that your planned, monthly approach is way easier to handle. This is doubly true for applications that refuse to vMotion. Those application teams need a specific, planned outage window so they can plan.

In the end, though, none of this makes any single patch go faster by itself. What it does is make patching easier and more routine, and those are the things that make this new reality tolerable in the long term. Slow becomes smooth, smooth becomes fast.