Replacing vCenter 6.0’s SSL Certificate

[Update 2] You might also be interested in the post “10 Things to Know About vSphere Certificate Management” which covers a lot of the common questions about certificates, including the pros & cons of the different VMware Certificate Authority modes, why hybrid mode is so popular, why self-signed certificates aren’t evil, and how to explain it all to a CISO or an auditor who might be mandating certain things. Check it out.

[Update 1] So far, this blog has been one of my most popular here on the vSphere blog. Since publishing this over 18 months ago we have learned quite a bit about the different requirements and considerations customers are under. This “Hybrid Mode” of certificate configuration, where you replace the externally facing reverse proxy certificate and let VMCA manage the hosts and solution users, addresses a number of issues that customers have encountered. It’s not only easier to manage from an IT standpoint but it has also stood up as an acceptable method of securing your vSphere environment by many security teams.

Because so many read this blog, it only made sense to update to link so something I’m sure you’ll all find very useful. So, I’m happy to announce that Adam Eckerle has posted a blog that links to one of our Product Walkthrough demos. This PWT will guide you, click by click, through the process of replacing the Machine SSL certificate. Check out this great work by Adam here.

A customer recently asked me “How do I replace the “external” SSL certificate of vCenter but still use VMCA in default mode?” Ever curious, I asked “Why?”. His security team required that any “externally” facing management web pages needed to have a custom certificate that chained up to the corporate PKI. But behind that, they were totally cool with using VMCA in default mode (with the self-generated root certificate) for things like ESXi servers and solution users.

Here’s a description of how this should look.

Lab setup

Windows Domain Controller (DC) – DC1.lab.local. This was built using my DC-Builder Powershell script I wrote for creating fully functional lab-based domain controllers. I referenced this in my previous blog, “Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 6.0” When this CA was set up, the issuer became “lab-DC1-CA” under “lab.local”. In the certificate this will be referenced as

CN = lab-DC1-CA, DC = lab, DC = local

VMware vCenter Service Appliance (VCSA) 6.0 – vcsa.lab.local. This is a fully embedded installation. VMCA will issue the certificates and they will be referenced as

O = vcsa.lab.local, C = US, DC = local, DC = vsphere, CN = CA

Note: Although the two CA’s come from systems in the same “lab.local” domain name in this example, they are two entirely different CA’s with no shared chain between them. As such, the Windows CA could just as easily be in an entirely different domain and this would still work.

VMware ESXi server – esxi-a.lab.local. This is part of the cluster that is managed by vcsa.lab.local. The VMCA on vcsa.lab.local manages the certificates on esxi-a.lab.local.

Create a folder that you’ll be able to download and upload files to during the exercise. On the VCSA I created a folder in the root account. “/root/Machine_SSL”. I’ll be connecting using WinSCP later. If you’re using WinSCP to connect to VCSA, read KB 2107727 and save yourself some frustration!

Generate the Certificate Signing Request

The first thing we are going to do is generate the Certificate Signing Request (CSR) for the “Machine SSL” certificate. As mentioned in the previous blog, “The Machine SSL certificate is the certificate you get when you open the vSphere Web Client in a web browser. It is used by the reverse proxy service on every management node, Platform Services Controller, and embedded deployment. You can replace the certificate on each node with a custom certificate.”

To generate the CSR we’ll use the Certificate Manager utility. This is located in:

•Windows
– C:Program FilesVMwarevCenter Servervmcad certificate-manager

•VCSA
– /usr/lib/vmware-vmca/bin/certificate-manager

Run the utility and select Option 1

Now select Option 1 again to generate the CSR and provide the output directory path to write out the files created. In my example I used the “/root/machine_ssl” folder I previously mentioned.

Upload and generate the new certificate

Next, I logged into my Windows system (DC1.lab.local), fired up WinSCP and downloaded the Machine_SSL csr and key files.

Open the CSR file in your favorite text editor and copy the contents to the clipboard

Open the web page of the Microsoft Certificate Authority and select “advanced certificate request”. Paste the contents of CSR and select the previously created “vSphere 6.0” template. Submit the request. Note that in some cases, your CA Administrator may have an approval process in place that will require additional steps. These haven’t been implemented in this lab-focused environment.

Download the new certificate and root certificate

After submitting the CSR, you’ll be presented with a download page.

On the download page, Select “Base 64 encoded” and click on “Download Certificate”. The downloaded file will be called “certnew.cer”. Rename this to “machine_ssl.cer”

Go back to the download web page and click on “Download certificate chain” (ensuring that “Base 64 encoded” is still selected). The downloaded file will be called “certnew.pb7”. Rename this to “cachain.pb7”

We are now going to export the CA Root certificate from the cachain.pb7 files. Right-click on the cachain.pb7 file and select “Open”

Expand the list and click on the Certificates folder. Right-click on the CA root cert (lab-DC1-CA in this example), select All Tasks…Export

You’re now presented with the Certificate Export Wizard. Click Next and then select the format you want to use. Select “Base 64 encoded X.509 (CER)”

In the next window, click on Browse… and provide a file location and filename. I used “root-64.cer” in the c:tempvcsa.lab.localMachine_SSL” folder I had previously created.

After successfully saving and exporting the root-64.cer file, it’s time to upload it to vCenter. Here I’ll use WinSCP again to copy the machine_ssl.cer and root-64.cer file to VCSA.

Replace the existing certificate with the newly generated certificate

Now that the files have been copied, open up the Certificate Manager Utility and select Option 1, Replace Machine SSL certificate with Custom Certificate. Provide the password to your [email protected] account and select Option 2, “Import Custom Certificate(s) and key(s) to replace existing Machine SSL certificate” You will be prompted for following files:

machine_ssl.cer
machine_ssl.key
root-64.cer

Select “Y” to continue the operation. This may take a few minutes, depending on how your systems are configured.

Verification Time

Now it’s time to check to see if all this work has paid off. One thing to remember before we start. Because the new Machine SSL cert has been issued by the CA on the domain controller, browsers that use the Windows certificate store will automatically recognize the vCenter web page. In my experience, Internet Explorer and Google Chrome will use the Windows certificate store. Mozilla Firefox does NOT use the Windows certificate store and as such you need to import the root certificate. Import the previously created root-64.cer into Firefox’s Trusted Root Authority and then open the vCenter web page and you should be all set.

Now open your vCenter login page and check the certificate being used to protect it.

You’ll see that the certificate has been verified by “lab-DC1-CA”. This is the CA running on the Windows domain controller. If I click on More Information and then View Certificate and scroll down to Issuer, you’ll see the hierarchy details of the certificate I referenced above in Lab Details.

So now that we’ve confirmed we’re using the Microsoft CA issued cert for vSphere Web Client logon, let’s see what’s going on after we log in.

VMCA in Default Mode

In a previous blog post I went over the different modes you can run VMCA in. The “Default Mode” is when VMCA uses its own root certificate and issues certificates that change to the root. You can install that root certificate in your browsers if you don’t want to see things as “untrusted”. Read more about the default modes and installing the root cert.

When you add an ESXi server to vCenter, VMCA will automatically issue it a certificate.

You can see that the issuer matches the values from the Lab Setup portion of this blog.

And the SSL thumbprint/fingerprint (SHA256) has a specific value ending in 1C:0D:E8

Now we’ll go back to vCenter and renew the ESXi certificate. Open vCenter, right click on the host and select Certificates…Renew…

Confirm by clicking on Yes.

Refresh your browser and repeat the steps to display the certificate properties. You’ll see in this example that the SHA256 thumbprint/fingerprint value has changed. The new ending values are now E8:E5:E1. VMCA has re-issued a completely new certificate.

Wrap up

To wrap up, let’s review what we’ve done.

Using the Certificate Manager utility and the SSL Certificate Template we created in a previous blog article, we generated a Certificate Signing Request for the externally-facing vSphere Web Client login page (The Machine SSL certificate).
We submitted the CSR to the Microsoft Certificate Authority that was installed on our Domain Controller and downloaded the newly generated certificate and root CA certificate
Using the Certificate Manager utility we replaced the Machine SSL certificate with the certificate generated by the Microsoft CA
We verified that the vSphere Web Client login page is now using the Microsoft CA-issued certificate
We then verified that the ESXi host was using a VMware CA issued certificate
Finally, we renewed the VMCA issued certificate and verified the change.

Here’s a video of the whole process:

Best of Both Worlds

There you have it. What one customer called “The best of both worlds”. Externally-facing certificates that chain to the corporate PKI and VMware CA certificates securing the vSphere components so we can retain all the VMCA functionality.

A future blog will change this up. We’ll issue a subordinate signing certificate to VMCA that chains directly up to the corporate PKI (In our case, the Microsoft CA) allowing VMCA to issue certificates chained up to the corporate PKI.

One thing about security you can always count on is that everyone does it different. One customer is totally cool with one way and the other is vehmently against it. I guess this is my burden to bear. 🙂

Please provide feedback either below or in email. I want to ensure that what I blog about helps you do your job better and/or helps you meet the goals set forth by the infosec team.

Thanks for reading!
mike