The biggest issue with remediating CPU vulnerabilities in a virtual environment is that, in most cases, the virtual machines must be power cycled. Not just rebooted but powered off and then back on. To help our customers get this done easily VMware has added the vmx.reboot.PowerCycle advanced parameter to vSphere which automates this process.
Why does a VM need to be power cycled to be remediated?
Intel has chosen to remediate some of the vulnerabilities by shipping new CPU microcode (firmware) that adds new CPU instructions. For example, the mitigations for the MDS vulnerabilities add MD_CLEAR which a guest OS can detect and use to help protect itself. Just as with physical hardware, a VM only learns about new CPU and hardware features when it first powers on. Unlike physical hardware, though, a VM can vMotion to avoid outages. That means that a VM could be running for a very long time with an outdated view of the hardware underneath it.
Guest OS reboots, such as scheduled Windows Update patching, don’t help, either. At the most fundamental level a VM is just a process running on an ESXi host. It’s that process, called the virtual machine runtime or what we sometimes refer to as VMX, that emulates the hardware that a guest OS sees. That includes the CPU features, too. Until that process is completely stopped and started fresh it won’t know about new hardware and won’t be able to share the news with your guest OSes. A guest OS reboot doesn’t affect the VMX process, it just restarts the OS itself.
Beginning with vSphere 6.7 Update 3 and 6.5 Update 3 there is a new advanced parameter you can set for VMs, vmx.reboot.PowerCycle. If you set it to TRUE it will cause ESXi to cycle the power of a VM whose guest OS is rebooted. Once the VM has been power cycled, either via this mechanism or through a manual power cycle, ESXi will clear that parameter. This feature requires the VMware Tools to be installed and running so that it can detect the reboot correctly.
How Do You Use This?
While you can set advanced parameters for VMs in a few different ways the easiest way to use this feature, by far, is with PowerCLI:
You can also do this to whole groups of VMs, too. For instance, if I have a folder called Test Group 1 I can retrieve that object, then get the list of VMs, then pipe that into New-AdvancedSetting:
1
Get-Folder"Test Group 1"|Get-VM|New-AdvancedSetting-Name vmx.reboot.PowerCycle-Value TRUE-Confirm:$false
As with most commands you can also add the “-Force” flag to force the update, and control what happens when there’s an error with the “-ErrorAction” flag, too. If you’re new to PowerCLI try using the PowerShell ISE where you can easily see all the flags and their parameters as you type. You can also hit Tab in the PowerShell CLI to autocomplete commands, as well as cycle through parameters, which is very helpful.
Once set, you’ll be able to see the flag in the vSphere Client, right at the bottom of the Configuration Parameters view:
You’ll also need to ensure that the VMware Tools are installed and running in the guest OS. It’s the VMware Tools that detects the reboot and signals ESXi that it’s safe to cycle the power.
Using this with EVC
Remediating CPU vulnerabilities isn’t the only use for this feature. It’s also useful where you’ve enabled Enhanced vMotion Compatibility, or EVC. If you aren’t familiar with EVC it’s a feature that resolves differences between older CPUs and newer CPUs so that vMotion works. It’s very useful for expanding a cluster’s capacity, or doing rolling hardware replacements, too. Once you’ve replaced your hardware you’ll want to change the EVC compatibility level to take advantage of the new features. For a guest OS to see those changes the VM will need to be power cycled, too. With vmx.reboot.PowerCycle it’s no problem!
Even though CPU vulnerabilities aren’t something caused by VMware, vmx.reboot.PowerCycle is a great example of how VMware is continually working to help customers deal with challenges in their environments, baking security into the products and making it easy to be secure. If you have feedback or other ideas for how we can help you please reach out to us and let us know. As always, thank you for being our customers!
