Update (07/01/2019): This issue has been resolved in recent updates to Microsoft’s operating systems. Microsoft KB4497935 contains information on how to get this patch, but in short, it is available through normal cumulative Windows Updates.
——
Users of Virtualization-Based Security or the virtual I/O MMU features in vSphere should take note of a serious issue that has been discovered with the 1903, 19H1, and May 2019 updates to Windows 10, Windows Server, and Windows Server 2019 LTSC editions. In short, a new installation of the OS at these levels, or an update of an existing guest OS to these levels, will cause the guest VM to hang or emit a blue screen of death (BSOD) during the boot process if VBS or the I/O MMU is enabled.
What is Virtualization-Based Security?
Virtualization-Based Security (VBS) is a feature that enables Microsoft VBS, which is also known as Device Guard and Credential Guard. Since vSphere 6.7, ESXi supports running Windows VMs that require VBS. It enables a feature of Microsoft Windows 10/2016/2019 that uses part of the Hyper-V subsystem to create an isolated, secure space for credentials and other secrets. VBS helps prevent many credential theft attacks and thwarts many types of malware and persistent threats. Under normal circumstances VBS is a highly recommended tool that vSphere admin teams can use to protect their environments, especially crucial infrastructure components like Active Directory domain controllers.
Am I Affected?
This issue was first discovered in Windows Insider Build 18362 and is not a VMware issue, per se. It is an issue Microsoft must fix in Windows. VMware and Microsoft are collaborating on the fix, and we expect Microsoft to release a patch soon. In the meantime, there are workarounds if you are affected.
VMware KB 68043 has a list of questions to help you determine if you are affected. Put simply, if Virtualization-Based Security and/or I/O MMU options are enabled for a VM, and you are updating to these versions of Windows, there is a strong chance you will need to take action.
PowerCLI makes it very easy to find VBS- or IOMMU-enabled virtual machines:
This command finds VMs with either VBS (VbsEnabled) or I/O MMU (VvtdEnabled), and the OS type of Windows 10/2016/2019 (windows9*), then prints the results.
What Can I Do?
To start, please read VMware KB 68043 which has instructions for a reliable workaround. In short, you can disable VBS, update, and then re-enable VBS which will skirt the issue. Note, though, that a subsequent Windows Update may trigger the issue again if it applies another update, such as the 2019-04 or 2019-05 Cumulative Update, that doesn’t contain the future fix. Update your VMs completely before re-enabling VBS.
Also note that the workaround requires you to add the Hyper-V feature, which isn’t normally a requirement to enable VBS (after Windows 10 1507, and with 2016 & 2019, you haven’t needed to explicitly add Hyper-V when you enable VBS in Group Policy). If you do use this workaround remember to restore the guest OSes to their original & documented state once the fix is released. We encourage the use of snapshots and clones where possible to help reduce risk, and always encourage administrators to verify that VM backups are working properly.
Another course of action, while never optimal, is delaying the update. Review the issues that the latest updates resolve and discuss the implications of waiting a few weeks with your team, information security staff, and organization’s management. Determining the risk involved in delaying a software update is a business decision that can only be made by you and your organization’s management. This is something VMware cannot endorse, as timely and thorough patching is one of the biggest ways you can improve your organization’s security posture, and we cannot speak to your organization’s risk tolerance or security requirements. However, availability is a key part of information security and risk management, too, and an update that threatens system availability needs to be considered thoughtfully. These situations are where the idea of “defense in depth” is most useful.
The 1903 & 19H1 DVD/ISO media associated with these Windows releases will continue to contain this problem until Microsoft releases their next scheduled media update, typically every few months. Keep this in mind if your guest OS provisioning process includes fresh builds or you happen to be rebuilding VM templates. If you are building new VM templates we suggest enabling VBS & Device Guard, EFI Firmware, Secure Boot, and encrypted vMotion, all of which are very easy wins for guest OS security and compliance.
We are also very fortunate that the editions of Windows affected by this VBS issue do not have the critical Remote Desktop Services vulnerability, CVE-2019-0708, which would require them to be patched immediately to avoid a severe remote code execution issue. If you or your team are responsible for Windows XP or 7, or Windows Server 2003, 2008, and/or 2008 R2 systems you need to patch immediately (even XP and 2003, that is not a mistake!).
Thank You
As always, thank you for using VMware vSphere. We appreciate the amazing things our customers have built on the solid, efficient, and secure foundation that is vSphere. If you have concerns or questions about this issue or others please contact VMware Global Support Services, your account team, your technical account manager, or reach out through social media.
Update (07/01/2019): This issue has been resolved in recent updates to Microsoft’s operating systems. Microsoft KB4497935 contains information on how to get this patch, but in short, it is available through normal cumulative Windows Updates.
