Now that vSphere 7 has shipped and support for vSphere 6.0 has ended it’s time to revisit a lot of the certificate management methods and techniques we use when managing vSphere environments. Certificates are what drive the TLS encryption that protects all network communication to & from vSphere.
First, vCenter Server 7.0 has done some interesting things to help make certificate management easier. To start, the solution certificates are deprecated, being replaced under the hood with a less complex but equally secure method of connecting other products like vRealize Operations, vRealize Log Insight, etc.:
Second, there are now REST APIs for handling vCenter Server certificates, as part of the larger effort to ensure APIs are present for nearly everything in vSphere:
There are also additional simplifications around certificates for services in both vCenter Server and ESXi, so that the number of certificates to manage is much lower, whether you are managing them manually or allowing the VMware Certificate Authority (VMCA) that is part of vCenter Server to manage the cluster certificates for you. The VMCA is “just enough certificate authority” to manage the vSphere cluster’s cryptographic needs. It should not be confused with a general-purpose certificate authority (CA) like those that are often found as part of enterprise PKI infrastructure. You cannot ask the VMCA for a certificate for your company’s blog, for example.
vSphere Certificate Management Modes
The certificate management changes in vSphere 7 are evolutionary, smoothing our management activities for us. In vSphere 7 there are four main ways to manage certificates:
Fully Managed Mode: when vCenter Server is installed the VMCA is initialized with a new root CA certificate. This is used to manage the intra-cluster certificates (protecting communications between ESXi hosts, and between ESXi hosts and vCenter Server), as well as what is called the “Machine Certificate.” The Machine Certificate, despite its name, is what us humans see in our browsers when we log into the vSphere Client. We can download the VMCA root CA certificate from the main vCenter Server web page and import it into our PCs in order to establish trust. We can also regenerate the VMCA root certificate if we want, using our own information instead of the default text values like “VMware Engineering” and such.
Hybrid Mode: the VMCA does a tremendous job automating the certificate management inside the vSphere clusters, and it saves us enormous time and frees us from the possibility of errors, like when we forget to renew a certificate. However, if we have a lot of people that access the vSphere Client it is often impractical to ask them all to import the VMCA root CA certificate. Instead, we can replace the certificate that the vSphere Client uses so that it is accepted by default by client browsers. This is the best of both worlds – deep automation for the security inside the infrastructure and minimal management effort for vSphere Client users. However, vSphere Admins will still want to import the VMCA root CA certificate in order to establish trust with the ESXi hosts, whose management interfaces will have certificates signed by the VMCA. In most cases the vSphere Admin team is small(ish), making this task is very manageable:
Note that in both hybrid mode and the default, fully managed mode neither the ESXi hosts nor the vSphere Client have self-signed certificates, which is a common misconception. They are signed by the VMCA. The VMCA is an integral part of vCenter Server. We trust vCenter Server to manage the core of our infrastructure, and therefore we implicitly trust the VMCA, too. To say that the VMCA is untrustworthy is to call into question the trustworthiness of vCenter Server as well. Is the VMCA root CA certificate more or less trustworthy than all the other root CA certificates that appear without our consent in our browsers and operating systems? Many thousands of VMware customers answer that as “more trustworthy,” especially if they regenerate it with their own information. Similarly, many customers enjoy the separation of infrastructure trust from the rest of the enterprise PKI infrastructure, from a “separation of duties” perspective as well as avoiding potential dependency loops if parts of the enterprise PKI infrastructure run inside vSphere.
Subordinate CA Mode: the VMCA can operate as a subordinate CA, delegated authority from a corporate CA. This allows vCenter Server to continue automating the certificate management, just like in the fully managed mode, except the certificates it generates are trusted as part of the organization. This is appealing to some organizations, but it requires importing key material into the VMCA that, if misplaced (or secretly stored, just in case) in transit, could be used by an attacker to impersonate the organization and conduct attacks like man-in-the-middle. In most cases, organizations — both enormous and small — that seek this level of automation find themselves using the Hybrid Mode instead because it helps isolate potential fault domains.
Full Custom Mode: in this mode the VMCA is not used, and a human must install and manage all the certificates present in a vSphere cluster. Even with the simplifications in vSphere 7 this can still amount to dozens of certificates, and the potential for operational issues and outages should a certificate be allowed to expire. Furthermore, because vCenter Server uses certificates to establish trust with the hosts, the replacement of certificates on ESXi hosts involves disconnecting and reconnecting them to vCenter Server. This can be rather onerous in the face of distributed switches and vSAN storage, which don’t like to be disconnected like that. Generating hundreds of keys, CSRs, and signing certificates is also error prone and time-consuming, not just for vSphere Admins but also the enterprise PKI teams.
A Clear Winner
It’s probably clear which mode we recommend in vSphere 7: Hybrid Mode. It let’s us take advantage of the automation and the trust we have in our vCenter Server installations but replace the machine certificate so that humans have a better experience in their browsers.
To be clear, even though we feel strongly about hybrid mode, all four modes are documented and fully supported. One size does NOT fit all in this world.
The automation with the VMCA is very compelling, especially for large institutions, and especially ones with heavy compliance & security burdens. This might seem counterintuitive, but the truth is that, for most people, discussions around certificates conflate encryption and trust in very dangerous ways. This is especially true now with certificate authorities like Let’s Encrypt, where the emphasis is less on trust and more on enabling encryption. Take all that, mix in a cup of “best” practices from a decade ago, a gallon of compliance framework & auditor, two cups of confusing jargon, and a few condescending tablespoons of “that’s not how we do things around here” and you have a recipe for trouble, endangering staff time, morale, uptime, and actual security. Keep it simple and you keep it safe.
Certificate management is possibly the single most confusing topic we encounter, and so we’ve got much more to come on these topics. Stay tuned!
We are excited about vSphere 7 and what it means for our customers and the future. Watch the vSphere 7 Launch Event replay, an event designed for vSphere Admins, hosted by theCUBE. We will continue posting new technical and product information about vSphere 7 and vSphere with Kubernetes Monday through Thursdays into May 2020. Join us by following the blog directly using the RSS feed, on Facebook, and on Twitter. Thank you, and please stay safe.
