In a previous blog post I went over the details on how ESXi uses a TPM 2.0 chip to provide assurance that Secure Boot did its job and how that “attestation” rolls up to vCenter to be reported on.
In this blog article I’m going to go over some of steps necessary to configure the ESXi host to use TPM 2.0 chip. Now, I have only a limited number of hardware systems in my lab from which to do this, but the steps should be familiar, regardless of the server model.
As called out in the documentation, there are a few prerequisites you need to meet before starting this process.
To use a TPM 2.0 chip, your vCenter Server environment must meet these requirements:
vCenter Server 6.7
ESXi 6.7 host with TPM 2.0 chip installed and correctly configured in the UEFI bios
UEFI Secure Boot enabled
Server BIOS settings
Correctly configuring the TPM 2.0 devices in the BIOS involves ensuring a number of settings are correct.
The TPM is set to use SHA-256 hashing
If available, it must also be set to use the IS/FIFO (First-In, First-Out) interface and not CRB (Command Response Buffer)
TXT must be disabled
Yes, we use TXT when using TPM 1.2 but it is not yet implemented in TPM 2.0 on ESXi (and yes, I ran in to this specifically!)
My Servers
The servers I have in my lab are Dell PowerEdge R630’s. They originally came with TPM 1.2 devices but I had them upgraded to TPM 2.0 and they are running BIOS version 2.6.0.
TPM Settings
Here are the settings in the System Security part of my servers BIOS. Your systems may look different but the options should be similar.
RTFM?
When I first started this process I did what most do. I didn’t read the docs. I like to break things and see if I can fix them. And then ask questions of the engineers. Why do I do this? Well, for one, I believe I learn faster by breaking and fixing and besides, it’s a lot more fun for me. Also, I’m trying to replicate what customers may encounter. Oh, sure, 99% of you actually read the docs before jumping on to Twitter to ask a question, right? RIGHT? Well, I’m there for that 1% who don’t!
When I started, I got the TPM 2.0 devices installed and I then installed 6.7 (after updating my VCSA first of course!). What resulted next was an error on the summary page of the ESXi host.
Note: I do not have 117 ESXi hosts at my disposal. Yes, I have been asked that.
I went in to the BIOS and started playing around with settings. I cleared the “TPM Hierarchy” (the contents of the TPM) but that didn’t do it. I was getting an alarm that things weren’t configured correctly.
Hashing
One of our engineers, Sam, was awesome. I have to give her credit for maintaining her patience with me. She had me look at the logs and sure enough, we found something interesting:
1
2
3
2018-05-09T21:30:21.060Zcpu23:2099722)WARNING:tpmDriver:TpmDriverInitImpl:532:TPM2SHA-256PCR bank notfound tobe active.
While going through this process I was sharing my experiences on the vExpert Slack channel and others had come across the “Tpm2ResMgrProcessResponse:846: Error: TPM command error code 0x18b” error as well.
It was at this time that I was told by Engineering to disable TXT. TXT has not been implemented it in our current TPM 2.0 code.
Time to file a bug report
Reboot number, oh, I don’t know, 3? 4? I still encountered a failure. So I filed a bug. This time the host was reporting a “Failed” attestation and there was nothing in the kernel log stating why. Another one of our engineers looked at the bug and the vCenter and ESXi support bundles and found the latest culprit.
vpxd-11.log:2018-05-10T17:41:05.588Zinfo vpxd[05828][Originator@6876sub=Attestation opID=HB-host-29@74-be7ec9-SWI-4843ebf6]No cached identity key,loading from DB
A-Ha! “No identity key in DB, try to reconnect host” explains it! What this means is that the host was added to vCenter without a TPM 2.0 chip enabled in the bios. After it was added was when the TPM 2.0 chip was enabled in the BIOS. In my case, my hosts were added a couple of years ago, I installed a TPM 2.0 device after the fact. What this error means is that there is no TPM Endorsement Key stored in the VCDB. This trust is set up when vCenter first adds the host to a cluster.
Disconnect…Reconnect
The solution was simple. Disconnect and reconnect the host. Put the host into Maintenance Mode, right click and select Connection…Disconnect and then right click again and select Connection…Connect. No need to remove the host from inventory.
Documentation fixes
Unfortunately, when I looked in the documentation (after the fact, naturally) to see if the error and solution was documented the response was “Call support”. We quickly got that fixed and now the documentation says the following:
Note: If you add a TPM 2.0 chip to an ESXi host that is already managed by a vCenter Server, you must first disconnect the host, then reconnect it. See vCenter Server and Host Management documentation for information about disconnecting and reconnecting hosts.
In fact, we even added a section on troubleshooting based directly on my experiences that led to this blog!
Passed Attestation
At this point the host showed up as having passed attestation! Woo-Hoo! Secure Boot has done its job and I can provide a report that says so, based on TPM 2.0 trust.
Wrap Up
I hope this has been helpful for you in setting up your ESXi host to use TPM 2.0. I think out of this whole process of NOT looking at the documentation and fumbling my way through the setup and configuration helped us end up with much better documentation and a better understanding of where things could go wrong. That’s #winning in my book.
I want to thank all the engineers that helped out on this. It really helped me understand what’s going on under the covers and enabled me to write these blogs.
If you have questions that haven’t been answered you can reply here, send them to mfoley at vmware.com or via Twitter to @vspheresecurity or my personal Twitter account: @mikefoley
@vspheresecurity is a curated list of vSphere Security specific tweets.
