Using the ESXi 6.0 CPU Microcode Loading Feature

Authored By: Tim Mann

Abstract: Occasionally ESXi users want to run updated CPU microcode that isn’t yet available either from their hardware platform vendor in a BIOS update or bundled into ESXi. This article explains how to do that.

Note: This procedure is not supported by VMware, and in production environments VMware recommends using only microcode obtained from your platform vendor.

This article applies beginning with ESXi 6.0. For earlier versions of ESXi, the following third-party blog post seems generally accurate, though of course not officially endorsed by VMware: FAQ: CPU microcode updates and VMware ESXi.

What are microcode updates?

All modern CPUs include a writable microcode store that allows the CPU vendor to fix or work around certain CPU errata in the field by providing customers with updated microcode. Normally, CPU microcode updates are the responsibility of the CPU vendor (in the x86 world typically Intel or AMD) and the hardware platform vendor (such as Apple, Dell, HP, etc.). The processor vendor develops, tests, and digitally signs the microcode update, then makes it available to the platform vendors. The platform vendors test the microcode with their platforms. If all is well, the platform vendors then make the updated microcode available to end users by rolling it into a CPU firmware (BIOS) update.

Unfortunately, some platform vendors are slow about pulling the latest microcode into their BIOS updates, slow about releasing BIOS updates, or don’t release BIOS updates at all. Usually this is not a big problem, but occasionally you might need a microcode update that you can’t get from your platform vendor yet.

How does ESXi help?

ESXi includes a boot-time microcode loader that can help when the platform vendor hasn’t made an update available yet. ESXi ships with a very small number of microcode updates bundled in that are important for ESXi to run correctly on certain CPU models. During each boot, ESXi checks whether any of its bundled microcode updates is applicable to the CPUs in your machine and is newer than the microcode the CPUs are currently running. If such an update is found, ESXi loads it into the CPU, where it remains until the machine is powered off or rebooted.

By VMware policy, however, ESXi does not bundle in all the latest microcode for supported CPUs. Instead, only a very few critical updates are bundled in, and new updates are added in only after consultation with major hardware vendors. We do this because our hardware partners have asked us to avoid shipping microcode that they themselves have not yet tested on their platforms and are not yet ready to support.

What if that’s not enough?

But what if you really need an update that neither your platform vendor nor VMware has released yet? Perhaps the update corrects a CPU erratum that’s causing you a serious problem, but it wasn’t critical enough or wasn’t fixed soon enough for VMware to have bundled it into the ESXi release you’re running. This article explains how you can use ESXi’s CPU microcode loader with microcode VMware did not bundle into your ESXi release. Note: This procedure is not supported by VMware, and in production environments VMware recommends using only microcode obtained from your platform vendor.

Microcode vib and boot modules

Starting in ESXi 6.0, microcode updates are applied very early in boot, long before the ESXi filesystem is available. The 6.0 microcode updater can run only at this early stage. This design choice was made because, in general, some microcode updates are not safe to apply later in boot. For example, the VMkernel may have read and cached some information from the CPU that will be changed by the microcode update.

ESXi therefore carries its microcode updates in boot modules. If you look at the /bootbank directory of an installed ESXi system (or wherever the boot modules reside in other cases, such as a directory on the boot CDROM or a PXE deployment directory), you’ll see files named uc_amd.b00 and uc_intel.b00. These are, respectively, collections of AMD and Intel microcode in a compressed format. The microcode boot modules come from ESXi’s cpu-microcode vib.

The cleanest method to add your own microcode is to use the vibauthor tool to make your own version of the cpu-microcode vib, at the Community Supported acceptance level, and use it to replace the VMware-supplied cpu-microcode vib.

Alternatively, a quick shortcut method to add microcode is to build just the uc_amd and/or uc_intel boot module and replace it in your bootbank. This method has a number of drawbacks but can be useful, especially for initial testing.

We’ll describe both methods below. Both require a suitable Linux system or VM to work in. At this writing the current release of VMware’s VIB Author tool is packaged in RPM format for a SuSE Enterprise Linux 11 SP2 system, so that would be the most straightforward distro to use, especially if you’re going to use the vib method.

Procedure

Step 1: Get the microcode update you need

All the latest released Intel microcode can be obtained from their web site. The download contains all current Intel microcode in one file, in gzipped tar (.tgz) format. Try the following search link to find it: Intel microcode download. Sort the results with newest first and download. Then use “tar -zxf microcode-*.tgz” to unpack the .tgz file. This should give you a microcode.dat file.

Recently released AMD microcode can be found in the Linux kernel git repository. This repository currently contains three .bin files, all needed. Look for them at the following link: AMD microcode download.

If your update came through some other channel, check that it’s in the CPU vendor’s so-called “Linux” format. For Intel, the required format is ASCII, comma-separated lists of 32-bit hex numbers, with comment lines starting with “/”. The file extension usually is “.dat”. For AMD, the format is a binary container; if you look at it with a hex dump tool, you’ll see that it starts with 44 4d 41 00 (“DMA00”). The file extension usually is “.bin”.

Step 2: Make the boot modules

Here’s a small Python script to convert an Intel ASCII microcode.dat file to binary. Save this file as intelBlob.py.

Once you’ve saved the script, create either or both of the boot modules as follows:

Step 3: Quick shortcut method

If you’re using the vib method, skip to step 4. If you’re using the quick shortcut method, simply copy the file(s) you created in step 2 onto your ESXi system, then replace the existing microcode boot module(s) as follows:

Now skip to step 5.

Step 4: Make the vib

Get vibauthor and install it on a suitable Linux system or VM.

Grab the sample vib descriptor file cpu-microcode.xml. Edit it to have a more meaningful vendor, version number, description, etc.

Make the vib as follows. If you don’t want to include microcode for both CPU vendors, leave out the corresponding -p option.

One way to install the vib is with esxcli. Example commands:

(Other ways to install the vib require you to have put it into an online or offline depot. See the vibauthor documentation for instructions on making an offline depot.)

To check your work, look at the uc_*.b00 files in the /altbootbank directories immediately after the vib install to make sure they are the new gzipped binaries you made in step 2.

Step 5: Test

Reboot to get the new microcode to be used.

Look for MicrocodeUpdate messages in /var/log/vmkernel.log to see the results. Alternatively, use vsish to look at VSI node /hardware/cpu/cpuList/* for each CPU. Near the end are lines like the following:

On many processors, two or more logical CPUs share a single microcode store. ESXi updates the logical CPUs one by one, so on such systems, the first logical CPU that shares a microcode store will update it, and then the rest will show as being already up to date. Thus you’ll often see “Number of microcode updates: 1” on some CPUs and “Number of microcode updates: 0” on the rest, with different “Original Revision” numbers. But all CPUs will end up with the same “Current Revision”, which is what matters.

Additional notes

You can prevent ESXi from loading any bundled microcode by setting the VMkernel boot option microcodeUpdate=FALSE.

By default, ESXi will not attempt to downgrade microcode to an older revision.

Intel has the concept of “experimental” microcode. An experimental microcode update has the high-order bit of its 32-bit revision number set. Sometimes this is described as having a negative revision number. By default, ESXi will not attempt to install an experimental microcode update. In addition, if an experimental Intel microcode update has already been loaded before the kernel boots (presumably by firmware), by default ESXi will not update the microcode.

You can override the ESXi defaults with respect to downgraded and experimental microcode with the kernel boot option microcodeUpdateForce=TRUE. If this option is set, the kernel will attempt to apply any update it finds in the boot module that matches the current processor, regardless of whether that downgrades or applies an experimental update.

In general, if there is more than one update for the current processor model in the boot module, the microcode loader automatically picks the highest-numbered revision. For this comparison, the revision number is taken as unsigned, so experimental updates are preferred if they are enabled.

Your CPU will not necessarily accept a microcode update that ESXi attempts to load, even with microcodeUpdateForce=TRUE. All modern CPUs refuse to accept microcode that does not have a valid embedded digital signature from the CPU vendor. Most CPUs will probably refuse to accept a microcode downgrade as well. To accept an experimental update, Intel CPUs generally require “debug mode” or “debug interface” to be enabled in the BIOS settings. This setting may be available only on Intel prototype platforms. After enabling debug, a full power cycle of the system is typically required for it to take effect.