posted

5 Comments

[UPDATE] So far, this blog has been one of my most popular here on the vSphere blog. Since publishing this over 18 months ago we have learned quite a bit about the different requirements and considerations customers are under. This “Hybrid Mode” of certificate configuration, where you replace the externally facing reverse proxy certificate and let VMCA manage the hosts and solution users, addresses a number of issues that customers have encountered. It’s not only easier to manage from an IT standpoint but it has also stood up as an acceptable method of securing your vSphere environment by many security teams.

Because so many read this blog, it only made sense to update to link so something I’m sure you’ll all find very useful. So, I’m happy to announce that Adam Eckerle has posted a blog that links to one of our Product Walkthrough demos.  This PWT will guide you, click by click, through the process of replacing the Machine SSL certificate. Check out this great work by Adam here.

Adam is one half of our dynamic duo of vCenter Tech Marketing guys. Emad Younis is the other and his focus is on migration from Windows vCenter to the VCSA.  Their blogs are a must read!


A customer recently asked me “How do I replace the “external” SSL certificate of vCenter but still use VMCA in default mode?” Ever curious, I asked “Why?”. His security team required that any “externally” facing management web pages needed to have a custom certificate that chained up to the corporate PKI. But behind that, they were totally cool with using VMCA in default mode (with the self-generated root certificate) for things like ESXi servers and solution users.

Here’s a description of how this should look.

3rd Party Certificate for External Access
3rd Party Certificate for External Access

Lab setup

CN = lab-DC1-CA, DC = lab, DC = local

  • VMware vCenter Service Appliance (VCSA) 6.0 – vcsa.lab.local. This is a fully embedded installation. VMCA will issue the certificates and they will be referenced as

O = vcsa.lab.local, C = US, DC = local, DC = vsphere, CN = CA

Note: Although the two CA’s come from systems in the same “lab.local” domain name in this example, they are two entirely different CA’s with no shared chain between them. As such, the Windows CA could just as easily be in an entirely different domain and this would still work.

  • VMware ESXi server – esxi-a.lab.local. This is part of the cluster that is managed by vcsa.lab.local. The VMCA on vcsa.lab.local manages the certificates on esxi-a.lab.local.

Create a folder that you’ll be able to download and upload files to during the exercise. On the VCSA I created a folder in the root account. “/root/Machine_SSL”. I’ll be connecting using WinSCP later. If you’re using WinSCP to connect to VCSA, read KB 2107727 and save yourself some frustration!

Generate the Certificate Signing Request

The first thing we are going to do is generate the Certificate Signing Request (CSR) for the “Machine SSL” certificate. As mentioned in the previous blog, “The Machine SSL certificate is the certificate you get when you open the vSphere Web Client in a web browser. It is used by the reverse proxy service on every management node, Platform Services Controller, and embedded deployment. You can replace the certificate on each node with a custom certificate.”

To generate the CSR we’ll use the Certificate Manager utility. This is located in:

•Windows
– C:Program FilesVMwarevCenter Servervmcad certificate-manager

•VCSA
– /usr/lib/vmware-vmca/bin/certificate-manager

Run the utility and select Option 1

Certificate Manager Utility
Certificate Manager Utility

Now select Option 1 again to generate the CSR and provide the output directory path to write out the files created. In my example I used the “/root/machine_ssl” folder I previously mentioned.

Generate CSR
Generate CSR

Upload and generate the new certificate

Next, I logged into my Windows system (DC1.lab.local), fired up WinSCP and downloaded the Machine_SSL csr and key files.

WinSCP copy CSR from VCSA
WinSCP copy CSR from VCSA

Open the CSR file in your favorite text editor and copy the contents to the clipboard

Copy CSR contents to Clipboard
Copy CSR contents to Clipboard

Open the web page of the Microsoft Certificate Authority and select “advanced certificate request”. Paste the contents of CSR and select the previously created “vSphere 6.0” template. Submit the request. Note that in some cases, your CA Administrator may have an approval process in place that will require additional steps. These haven’t been implemented in this lab-focused environment.

Submit Certificate Request to MS CA
Submit Certificate Request to MS CA

Download the new certificate and root certificate

After submitting the CSR, you’ll be presented with a download page.

Certificate Download Page
Certificate Download Page

On the download page, Select “Base 64 encoded” and click on “Download Certificate”. The downloaded file will be called “certnew.cer”. Rename this to “machine_ssl.cer”

Go back to the download web page and click on “Download certificate chain” (ensuring that “Base 64 encoded” is still selected). The downloaded file will be called “certnew.pb7”. Rename this to “cachain.pb7”

We are now going to export the CA Root certificate from the cachain.pb7 files. Right-click on the cachain.pb7 file and select “Open”

Open cachain.pb7 file for export
Open cachain.pb7 file for export

Expand the list and click on the Certificates folder. Right-click on the CA root cert (lab-DC1-CA in this example), select All Tasks…Export

Export CA cert
Export CA cert

You’re now presented with the Certificate Export Wizard. Click Next and then select the format you want to use. Select “Base 64 encoded X.509 (CER)”

Export cert wizard - select Base 64 encoded
Export cert wizard – select Base 64 encoded

In the next window, click on Browse… and provide a file location and filename. I used “root-64.cer” in the c:tempvcsa.lab.localMachine_SSL” folder I had previously created.

Save root-64.cer file
Save root-64.cer file

After successfully saving and exporting the root-64.cer file, it’s time to upload it to vCenter. Here I’ll use WinSCP again to copy the machine_ssl.cer and root-64.cer file to VCSA.

Copy cer files to VCSA
Copy cer files to VCSA

Replace the existing certificate with the newly generated certificate

Now that the files have been copied, open up the Certificate Manager Utility and select Option 1, Replace Machine SSL certificate with Custom Certificate. Provide the password to your administrator@vsphere.local account and select Option 2, “Import Custom Certificate(s) and key(s) to replace existing Machine SSL certificate” You will be prompted for following files:

  • machine_ssl.cer
  • machine_ssl.key
  • root-64.cer
Import Custom Certificates via Certificate Manager Utility
Import Custom Certificates via Certificate Manager Utility

Select “Y” to continue the operation. This may take a few minutes, depending on how your systems are configured.

70 percent completed
70 percent completed
100 percent completed
100 percent completed

Verification Time

Now it’s time to check to see if all this work has paid off. One thing to remember before we start. Because the new Machine SSL cert has been issued by the CA on the domain controller, browsers that use the Windows certificate store will automatically recognize the vCenter web page. In my experience, Internet Explorer and Google Chrome will use the Windows certificate store. Mozilla Firefox does NOT use the Windows certificate store and as such you need to import the root certificate. Import the previously created root-64.cer into Firefox’s Trusted Root Authority and then open the vCenter web page and you should be all set.

Import root-64.cer into Firefox
Import root-64.cer into Firefox
Root cert imported into Firefox
Root cert imported into Firefox

Now open your vCenter login page and check the certificate being used to protect it.

vCenter web certificate in Firefox
vCenter web certificate in Firefox

You’ll see that the certificate has been verified by “lab-DC1-CA”. This is the CA running on the Windows domain controller. If I click on More Information and then View Certificate and scroll down to Issuer, you’ll see the hierarchy details of the certificate I referenced above in Lab Details.

vCenter Certificate Hierarchy
vCenter Certificate Hierarchy

So now that we’ve confirmed we’re using the Microsoft CA issued cert for vSphere Web Client logon, let’s see what’s going on after we log in.

VMCA in Default Mode

In a previous blog post I went over the different modes you can run VMCA in. The “Default Mode” is when VMCA uses its own root certificate and issues certificates that change to the root. You can install that root certificate in your browsers if you don’t want to see things as “untrusted”. Read more about the default modes and installing the root cert.

When you add an ESXi server to vCenter, VMCA will automatically issue it a certificate.

ESXi server certificate
ESXi server certificate

You can see that the issuer matches the values from the Lab Setup portion of this blog.

 

And the SSL thumbprint/fingerprint (SHA256) has a specific value ending in 1C:0D:E8

ESXi Original SSL thumbprint
ESXi Original SSL thumbprint

Now we’ll go back to vCenter and renew the ESXi certificate. Open vCenter, right click on the host and select Certificates…Renew…

Renew ESXi Certificate
Renew ESXi Certificate

Confirm by clicking on Yes.

Confirm ESXi renew certificate
Confirm ESXi renew certificate

Refresh your browser and repeat the steps to display the certificate properties. You’ll see in this example that the SHA256 thumbprint/fingerprint value has changed. The new ending values are now E8:E5:E1. VMCA has re-issued a completely new certificate.

ESXi Renewed SSL Thumbprint
ESXi Renewed SSL Thumbprint

Wrap up

To wrap up, let’s review what we’ve done.

  • Using the Certificate Manager utility and the SSL Certificate Template we created in a previous blog article, we generated a Certificate Signing Request for the externally-facing vSphere Web Client login page (The Machine SSL certificate).
  • We submitted the CSR to the Microsoft Certificate Authority that was installed on our Domain Controller and downloaded the newly generated certificate and root CA certificate
  • Using the Certificate Manager utility we replaced the Machine SSL certificate with the certificate generated by the Microsoft CA
  • We verified that the vSphere Web Client login page is now using the Microsoft CA-issued certificate
  • We then verified that the ESXi host was using a VMware CA issued certificate
  • Finally, we renewed the VMCA issued certificate and verified the change.

Here’s a video of the whole process:

 

Best of Both Worlds

There you have it. What one customer called “The best of both worlds”. Externally-facing certificates that chain to the corporate PKI and VMware CA certificates securing the vSphere components so we can retain all the VMCA functionality.

A future blog will change this up. We’ll issue a subordinate signing certificate to VMCA that chains directly up to the corporate PKI (In our case, the Microsoft CA) allowing VMCA to issue certificates chained up to the corporate PKI.

One thing about security you can always count on is that everyone does it different. One customer is totally cool with one way and the other is vehmently against it. I guess this is my burden to bear. 🙂

Please provide feedback either below or in email. I want to ensure that what I blog about helps you do your job better and/or helps you meet the goals set forth by the infosec team.

Thanks for reading!
mike

About the Author

Mike Foley

Mike Foley is a Senior Technical Marketing Architect at VMware. His primary focus is on security of the core platform, vSphere. His primary goal is to help IT/VI Admins build more secure platforms that stand up to scrutiny from security teams with the least impact to IT Operations. Mike is also the current author of the vSphere Security Configuration (formerly Hardening) Guide. Previously, Mike was on the evangelist team at RSA where he concentrated on virtualization and cloud security and contributed as a member of the product architect team. Mike has a personal blog at https://yelof.com and contributes to the VMware vSphere and Security blogs as well. Follow him at @vSphereSecurity on Twitter