Dependencies Are a Boat Anchor

There was a time in this industry when having system uptime measured in years was viewed as a weird badge of honor or accomplishment. Unix admins would delight in the uptime command returning years, and brag about how stable something was because it could run that long uninterrupted. The modern security landscape has changed the reaction from positive amusement to dread and fear when systems go longer and longer without patching. When I talk to customers about why they have not been patched, I find it is often rather mundane dependencies that are getting in the way of keeping vSphere up to date.

I’m Waiting on…..

A common cause for patching gaps is that customers state they are waiting on a 3rd party server vendor to package the ESXi release into an ISO file. This is a curious one, because back in the day when you bought a server with vSphere pre-installed it would often come with a physical CD, but it has functionally been years since I’ve actually seen anyone with a CD burner, so this release mechanism feels slightly antiquated. The vendor ISO release is functionally a bundling of the “vanilla” ESXi release with its inbox drivers, but also supplemented with asynchronous drivers and VIBs for handling things like that OEM’s proprietary server management tooling. The challenge is that vendors do not keep up with hotpatches, and some release as rarely as quarterly. Waiting 90 days for bugfixes and critical security patches is simply not acceptable.

Other causes often include various management plugins for storage arrays.

There is a better way

For people who have installed using the vendor ISO, there is a common misbelief that you can only update it with subsequent vendor ISOs. In reality, all updates to ESXi are cumulative, and you can apply the latest patches to an install that began with a vendor ISO. It does not “break” the release, in the same way your Windows laptop from Dell may have started with a Dell image, but you used Windows Update for day 2 patching.

Inversely, you can also start with vanilla ESXi and layer on the vendor tooling using the OEM Add-On functionality baked into vSphere.

What’s really in that ISO?

Previously, the storage subsystems were heavily dependent on async drivers for RAID controllers and HBAs. Today, most storage locally attached to a server is handled by the VMware NVMe inbox driver, which is maintained by Broadcom and directly tested against the firmware. RAID controller drivers are commonly inbox now, but CLI management utilities for them may still ship in the vendor OEM VIBs. Network interface cards and fibre channel HBAs still see async drivers for bug fixes. If support directs you to a fix and you need an async version of a driver for a specific issue, it can also be layered into a vLCM image. In many (if not most) cases, a server will boot and operate just fine using the vanilla ISO. While you may need it for a legacy CNA app to work or for the vendor’s HSM to function, you will often find that many of the asynchronous drivers in the latest update are not actually for devices that you use.

Simplifying dependencies that block patching

A lot of the particularly slow-to-update third-party plugins are largely being replaced by REST API workflows. Monitoring using proprietary hooks is shifting to REST + Prometheus, and that agent that looked for events might simply be replaced by syslog feed-triggered actions. There’s also value in looking inward at the VMware Cloud Foundation (VCF) suite to see if solutions inside VCF can displace that dependency blocking a patch or upgrade. VMware NSX manages overlays and can replace complicated appliance-based solutions, VMware vSAN for storage can replace the need for storage provisioning plugins, and VCF Automation means you don’t need to wait on a CMP to approve a patch. Given that all VCF components are built together, tested together, and integrated together even as part of minor releases, it’s one fewer element of complexity since you know you can update them together. At no point in the last 10 years has vSAN been allowed to ship a new version that was incompatible with vSphere (or vice versa), and the delay for vCenter support of vSAN is precisely 0 days every time.

Are we making mountains out of mole hills?

I sometimes see requests like “has XXX vendor’s plugin or backup product  been validated against release 9.1.0.200?” At a certain point, there may be some mild uncertainty with a security patch and a third-party dependency, but it helps to understand:

1. The scope of what rolling back that patch entails. (so you can just go ahead and deploy it)
2. What a good mechanism for doing limited host patching and testing looks like.
3. Recognizing that interop matrices do not account for minor security hotpatches, as they are scoped small enough not to cause major API formatting changes (without it being called out in the release notes).

We also need to question if a plugin or third-party dependency that has a history of causing problems with security patches is worth maintaining, or if the time savings it provides could be delivered in another way. Balancing the risk to the business with the automation these plugins delivered should be done on a case by case basis if they turn out to be problematic with minor patches. The reality is bugfix level patches very rarely have impacts on 3rd party accessible API that plugins use, and the support stance is simply carried forward. 

Conclusion

One of the last projects I worked on was an environment with tens of thousands of users on an out-of-support version of Novell and Windows XP. In peeling back the onion of why this customer had not updated, they discovered the one “blocking issue” was a single application used by seven people. The security of a large organization was effectively being held hostage by an incredibly minor dependency that was shifting its budgeting problem into a risk problem for the rest of the organization. I encourage VMware admins and sysadmins to re-evaluate the dependencies they have and look for ways to unblock faster patching (or even live patching) as soon as possible.