I’ve talked about how vSphere has been moving towards a “secure by default” stance over the past few years. This can clearly be seen in the new vSphere 6.5 Security Configuration Guide where the number of “hardening” steps are growing smaller with every release. In this blog post we will go over another “secure by default” feature of vSphere 6.5 that provides hypervisor assurance, Secure Boot for ESXi.
One of the coolest things in 6.5, in my opinion, is the adoption of Secure Boot for ESXi. Now, you might say “But my laptop has had Secure Boot since Windows 8, what’s the big deal?”
Well, the “big deal” is that we’ve gone beyond the default behavior of Secure Boot and we now leverage the capabilities of the UEFI firmware to ensure that ESXi not only boots with a signed bootloader validated by the host firmware but that it also ensures that unsigned code won’t run on the hypervisor. Best of all, it’s simple to implement! Let’s dive in!
Secure Boot and UEFI
Let’s do a brief overview of UEFI firmware and Secure Boot.
UEFI, or Unified Extensible Firmware Interface, is a replacement for the traditional BIOS firmware that has its roots in the original IBM PC. I would highly recommend reading the Wikipedia overview on UEFI to get a better understanding of all the capabilities it can present. I can also recommend the Ubuntu blog article on how they use UEFI. I’ve consulted both for use in this blog. For the purposes of this article, I’ll focus on how UEFI and Secure Boot relates to ESXi.
In UEFI parlance, Secure Boot is a “protocol” of the UEFI firmware. This capability was designed to ensure that boot loaders are not compromised by validating their digital signature against a digital certificate in the firmware. A typical compromise on your desktop or laptop would be if malware installed a root kit. This would change the digital signature and the UEFI firmware would check and not allow further booting.
UEFI can store whitelisted/valid digital certificates in a signature database (DB) . There is also a blacklist of forbidden certificates (DBX), a Key Exchange Keys (KEK) database and a platform key. These form the basis of a root of trust that begins with the firmware installed on your host.
These digital certificates are used by the UEFI firmware to validate the boot loader. Boot loaders are typically cryptographically signed and their digital signature chains to the certificate in the firmware. The default digital certificate in just about every implementation of UEFI firmware is a x509 Microsoft UEFI Public CA cert. Most UEFI implementations also allow for the installation of additional digital certificates. A typical use for this would be if you were developing a custom boot loader that’s signed against your own certificate. You could install that certificate in the UEFI firmware and UEFI would validate your boot loader against it.
Default certificates are part of the firmware installation from your server vendor, not VMware. When you update your UEFI firmware on your server, the digital certificate(s) are included.
How ESXi builds upon UEFI and Secure Boot
With ESXi 6.5, we take this capability of the firmware storing digital certificates and validating the boot loader and we build upon that.
ESXi is comprised of a number of components. There is the boot loader, the VM Kernel, Secure Boot Verifier and VIBs, or “vSphere Installation Bundles”. Each of these components is cryptographically signed. Let’s step through each of these.
Boot Loader
As mentioned above, the UEFI firmware itself verifies the bootloader’s digital signature to validate bootloader integrity. Normally, with many operating systems, that’s the limit of what happens because the threat of root kits are now mitigated. But not so with ESXi. We go beyond and ensure that all content shipped is cryptographically signed.
The ESXi boot loader is signed with the Microsoft UEFI Public CA cert. This ensures that standard UEFI Secure Boot firmware can validate the VMware boot loader. The boot loader code also contains a VMware public key. This VMware key is used to validate the VM Kernel and a small subset of the system that includes the Secure Boot Verifier, used to validate the VIBs.
VM Kernel
The VM Kernel itself is also cryptographically signed using the VMware private key. The boot loader validates the kernel using the VMware public key it has. The first thing the VM Kernel runs is the Secure Boot Verifier.
Secure Boot Verifier
The Secure Boot Verifier validates every cryptographically signed VIB against the VMware public key. The VMware public key is part of the Secure Boot Verifier codebase. (You can see in the graphic that the VMware Public Key is in two places, the ESXi Boot Loader and the Secure Boot Verifier)
When ESXi boots, it creates a file system in memory that maps to the contents of the VIBs. If the file never leaves the cryptographically signed “package” then you don’t have to sign every file, just the package.
This means you’re signing an order of magnitude less files, thereby limiting the impact on boot times. And because we have already had that digital signature process in place for years, it was the logical way to extend the Secure Boot capabilities.
The VIBs are signed with the VMware public key and validated with the Secure Boot Verifier.
The boot process
Host Power On
UEFI Firmware validates the ESXi Boot Loader against the Microsoft digital certificate in the UEFI firmware
ESXi Boot Loader validates the kernel against the VMware digital certificate in the Boot Loader
Kernel runs the Secure Boot Verifier
Secure Boot Verifier validates each VIB against the VMware digital certificate in the Secure Boot Verifier
Management apps (DCUI, hostd, etc) now run
Upgrades .vs. Fresh Installs
Because of changes in signing older VIBs, you may encounter some issues if you are upgrading a host from previous ESXi versions to 6.5 and enabling Secure Boot. Also, you may also find out that you have unsigned code running on your older systems. For example, these could possibly be beta drivers for a specific hardware device and usually fall under the “if it ain’t broke, don’t fix it”. The reasons they were never updated are usually lost to folklore. Things like this could be an issue so we’ll go over some steps to help mitigate it.
Personally, I try to treat ESXi servers less like “pets” and more like “cattle” (to use the popular vernacular of the DevOps crowd). I like to build ESXi servers in my lab from scratch every time. Use of Host Profiles can help lessen the impact and there are various other methods for automating and configuring ESXi hosts. Building from scratch ensures everything is “clean” and helps tremendously with troubleshooting issues. Consider adopting this way of thinking if possible. Again, this is just my personal preference.
Possible upgrade issues
UEFI secure boot requires that the original VIB signatures are persisted. Older versions of ESXi do not persist the signatures, but the upgrade process updates the VIB signatures.
If your host was upgraded using the ESXCLI command then your bootloader wasn’t upgraded and doesn’t persist the signatures. When you enable Secure Boot after the upgrade, an error occurs. You can’t use Secure Boot on these installations and will have to re-install from scratch to gain that support.
If you upgraded using the ISO method then old VIBs may be retained and the Secure Boot process cannot verify the signatures for the old VIB(s) and the boot process will fail. The ISO you use must contain new versions of all installed VIBs that are on the host. This ensures that signatures are updated. You may encounter this if you upgrade a vendor installation with the VMware ISO. If you do this, you will have to reinstall ESXi using a fresh install to enable Secure Boot.
Community supported VIBs
Because these VIBs are not signed they are not able to be installed on an ESXi host that has Secure Boot enabled.
Post-Upgrade Secure Boot Check
A script to check your environment after you’ve upgraded is available on ESXi 6.5. Its purpose is to ensure you can enable Secure Boot after you have done the upgrade. One caveat: UEFI secure boot also requires an up-to-date bootloader. This script does not check for an up-to-date bootloader.
Prerequisites
Verify that the hardware supports UEFI secure boot. You may have to check for a firmware upgrade.
Verify that all VIBs are signed with an acceptance level of at least PartnerSupported. If you include VIBs at the CommunitySupported level, you cannot use secure boot.
If you have upgraded your host to 6.5 and haven’t tried enabling Secure Boot then you can run a validation script located on the ESXi host. The script is called:
/usr/lib/vmware/secureboot/bin/secureBoot.py -c
The output either includes Secure Boot can be enabled or Secure boot CANNOT be enabled.
If Secure Boot cannot be enabled then see “Possible upgrade issues” above. You may have a situation that requires an clean installation. ESXi will continue to run just fine, however you won’t be able to take advantage of Secure Boot for ESXi.
PSOD’s, unsigned VIBs and File Integrity Monitoring (FIM)
PSOD – Purple Screen of Death
If you already have unsigned VIBs on your ESXi host and you enable Secure Boot in the firmware then ESXi will boot into a purple screen and tell you which VIB is unsigned. The error should look similar to this:
To get out of this situation do the following:
Restart and turn off Secure Boot in the UEFI firmware and boot the host with Secure Boot turned off.
When booted, log into the host and remove the offending VIB and shutdown.
Re-enable Secure Boot and restart the host and the system should boot normally.
You can only get into this situation if you have pre-existing unsigned code installed.
File Integrity Management
A customer ask I’ve heard for many years has been the ability to install File Integrity Monitoring (FIM) on ESXi. You can’t do this because ESXi isn’t Linux. It’s structured differently and we don’t allow user-level processes to run. All users run as “root” on VMware and their permissions are controlled with the vSphere API. However, with 6.5 and Secure Boot, you can address File Integrity Monitoring by enabling Secure Boot and collecting SYSLOG output from the ESXi hosts. The combination of these two technologies ensures that only signed code can run and any changes are monitored. Remember, all shell actions are sent out via SYSLOG and can be reported on by your log collection system, like VMware Log Insight.
Another key feature of enabling Secure Boot for ESXi is that you cannot forcibly install unsigned VIBs if Secure Boot is enabled! Commands like the following just won’t work:
The question always comes up in customer conversations of “Does this require TPM or TXT??”. The answer is no. They are mutually exclusive. Secure Boot for ESXi is purely a function of the UEFI firmware and the validation of cryptographically signed code. Period.
Note that TPM 1.2 and TPM 2.0 are two vastly different implementations. They are not backwards compatible. There is support, via 3rd parties like HyTrust, for TPM 1.2 in ESXi 6.5.
TPM 2.0 is not supported in 6.5.
Standard BIOS firmware vs UEFI firmware
Typically, switching your hosts from their standard (legacy) BIOS firmware to UEFI firmware in some operating systems will cause issues. With ESXi, you can switch with no modification to ESXi. If you have installed 6.5 using standard BIOS and you want to try out Secure Boot then in the host firmware you can switch and ESXi will come up.
Secure Boot and enabling vSphere HA
When you configure vSphere HA vCenter attempts to install a (signed) VIB on each host in the HA cluster. Because vCenter is using the qualifier “nosigcheck” and Secure Boot disables that ability, you will run into an error. In this case, you can either disable Secure Boot (but that requires a reboot or two) or you can manually install the vmware-fdm VIB. More information and instructions are located at KB2148562. And yes, we are working on fixing this.
Auto Deploy Addendum
To use Secure Boot with vSphere Auto Deploy, you have to enroll/add the VMware certificate in the UEFI firmware whitelist (DB). This is because only the ESXi Boot Loader is signed with the Microsoft certificate and some of the PXE code, used before the boot loader starts, is signed with the VMware certificate. Follow the instructions in KB2148532. In there you’ll see that you’ll need to work with your hardware vendor to get the steps necessary for updating the whitelist DB. There’s also a pointer to the Auto Deploy Best Practices that need to be followed.
As part of our vSphere Lightboard series of videos, I recorded this video that goes over SecureBoot for ESXi. Note that what’s in this blog is more current than the video, but the process is very similar.
Video
Wrap Up
I know this has been another large blog post but I hope it covers all of the questions that might come up about ESXi and Secure Boot in 6.5. For those of you that are most concerned about security in your environment, the assurance that your ESXi host boots cryptographically “clean” with little change to your operations is a key feature of 6.5. As with any new feature, security or otherwise, there may be changes to your operations model. This is no different.
I look forward to your questions here or on Twitter or via email at mfoley–at–vmware–dot–com
To get out of this situation do the following:
