VMware Cloud Foundation

Replacing VMware ESXi SSL Certificate in VMware Cloud Foundation

This blog post describes the steps to replace an SSL certificate for ESXi hosts in VMware Cloud Foundation (VCF). The steps mentioned in the blog are verified against ESXi 6.7, 7.x, and VCF 4.x versions and applicable to Management domain and Workload domains.

By default, vSphere components use the VMCA-signed certificate and key that are created during installation. Replace the default ESXi SSL certificate with certificates from a trusted CA, either a commercial CA or an organizational CA, if your company policy requires it.

Self-signed certificate using OpenSSL is used in this post to demonstrate SSL certificate replacement for ESXi hosts in VMware Cloud Foundation.

Create a self-signed certificate using OpenSSL

Pre-requisites: Before we begin, ensure that the OpenSSL binaries are installed before executing OpenSSL commands mentioned below.

The following example shows the command to verify whether OpenSSL libraries are installed in a Linux system.

If your environment doesn’t have the required OpenSSL libraries, you can download and install from here

Generate a self-signed certificate using OpenSSL command. In the subsequent steps, we will be using this self-signed certificate for generating a certificate for the ESXi host.
Refer OpenSSL official documentation in case if you are looking for any additional information related to OpenSSL commands.

Step 1: Execute the below command to generate a self-signed certificate using OpenSSL

After executing the above command, it will prompt the user to input the data for Country Name, State or Province Name, Locality Name, Organization Name, Organizational Unit Name, Common Name, and Email Address. Enter the details as appropriate.

For example, here is the command to generate a self-signed certificate.


Step 2: After executing the above command, it generates a self-signed certificate i.e. cert.pem and private key i.e. key.pem.

Copy cert.pem and key.pem files to all the ESXi hosts of a domain for which the certificate needs to be replaced. In the below snippet, for demonstration purpose, we will be copying cert.pem and key.pem files to only one ESXi host.

For example, here are the commands to verify the existence of cert.pem and key.pem files and to copy these files to the ESXi host.

Publish certificate to certificate store in vCenter Server

Add the certificate to the truststore in the vCenter Server system that manages the host to ensure that the vCenter Server and the ESXi host have a trust relationship. VMware Endpoint Certificate Store (VECS) is a repository for certificates and private keys. It contains all the Machine SSL Certificates, Trusted Roots and Solution Users Certificates.

Note: Add certificate i.e. cert.pem to the respective domain’s vCenter Server. i.e. if you are replacing the certificate of ESXi hosts for the Management domain, add the certificate to the respective vCenter Server which is part of the Management domain.
Similarly, if you are replacing the certificate of ESXi hosts for any workload domain, ensure to add the certificate to vCenter Server which is part of the workload domain.

Follow these step-by-step instructions to publish a certificate to VECS in vCenter Server.

Step 1: Copy the certificate i.e. cert.pem to vCenter Server. Note that cert.pem was generated in Step 1 in ‘Create a self-signed certificate using OpenSSL’ section.
(Note: If you’re unable to copy cert.pem to vCenter Server, follow the steps mentioned in KB article and then retry to copy the file again)

Log in to vCenter Server via SSH. Type shell and press the ‘Enter’ key, and verify whether cert.pem is copied to vCenter Server.

For example, here are the commands to enable shell access and to verify the existence of cert.pem file in vCenter Server.


Step 2:  Execute the below command to add the certificate to VMware Endpoint Certificate Store (VECS) in vCenter Server (Enter the password for [email protected] when prompted).

<path_to_certficate_chain>:  Mention the path of the certificate file that needs to be added to the truststore

For example, here is the command to publish cert.pem to VECS.


Step 3: Verify whether the certificate is added to the truststore.

Ensure that the certificate that was added in the previous step is listed when the below command is executed.
(Note: This can be identified by looking at the CN (Common Name) of the certificate. CN was provided while creating a self-signed certificate in the very first step i.e. ‘Create a self-signed certificate using OpenSSL’)

Execute the below command to list all the trusted certificates (Enter the password for [email protected] when prompted).

For example, here is the command to list all trusted certificates.

Publish certificate to SDDC Manager and Common Services truststore

Follow these step-by-step instructions to import the certificate to SDDC Manager and Common Services truststore.

Step 1: Copy certificate i.e. cert.pem to /tmp directory of SDDC Manager. Note that cert.pem was generated in Step 1 in ‘Create a self-signed certificate using OpenSSL’ section.


Step 2: SSH to SDDC Manager as a vcf user. Switch to the root user, and then execute the below command to get commonsvcs truststore password.

For example, here are the commands to switch to the root user and to obtain the password for the common services truststore.


Step 3: Execute the below command to import the certificate to the commonsvcs truststore.

<aliasname>: This can be any user-defined name
<Certificate_file>: Provide the correct path of the certificate file that needs to be imported to truststore

(Note: User will be prompted to provide a password while adding the certificate to the truststore. Enter the password obtained in the previous step (i.e. Step 2).  Also, type ‘yes’ when prompted to confirm to trust the certificate)

For example, here is the command to import cert.pem to common services truststore.


Step 4: Execute the below command to verify if the new certificate is added to the common services truststore (Enter the password for the truststore key obtained in Step 2).


Step 5: Execute the below command to import the certificate to the SDDC Manager truststore.

<aliasname>: This can be any user-defined name
<Certificate_file>: Provide the correct path of the certificate file that needs to be imported to truststore

(Note: User will be prompted to provide a password while adding the certificate to the truststore. Enter the default password  “changeit”. Also, type ‘yes’ when prompted to confirm to trust the certificate)

For example, here is the command to import cert.pem to SDDC Manager truststore.


Step 6: Execute the below command to verify if the new certificate is added to the SDDC Manager truststore (Enter “changeit” when prompted for a password, note that this is a default password).


Step 7: Restart all the SDDC Manager services.
Execute the below command to restart the SDDC Manager Services. Type letter ‘y’ when it prompts the user to confirm the restart of SDDC Manager services. Wait for about 5 to 10 minutes for all the SDDC Manager services to come up and running.

Change certificate mode to custom in vCenter Server

Note: Change certificate mode to custom in respective domain’s vCenter Server. i.e. if you are replacing the certificate of ESXi hosts for the Management domain, you need to change the certificate mode to custom in the respective vCenter Server which is part of the Management domain.
Similarly, if you are replacing the certificate of ESXi hosts for any workload domain, ensure to change the certificate mode to custom in vCenter Server which is part of the workload domain.

Follow these step-by-step instructions to change the certificate mode to custom in vCenter Server.

  1. Log in to vCenter Server using the vSphere Web Client.
  2. In the vSphere Web Client, select the vCenter Server that manages the hosts.
  3. Click Configure, and under Settings, click Advanced Settings
  4. Click EDIT SETTINGS
  5. In the Filter box, enter certmgmt to display only certificate management keys.
  6. Change the value of vpxd.certmgmt.mode to custom
  7. Click SAVE

    Here are the screenshots for reference which shows the navigation steps in vSphere WebClient UI to change the certificate mode in vCenter Server.Step 1: Change certificate mode to custom in vCenter Server Advanced Settings
    Step 2: Change certificate mode to custom in vCenter Server Advanced Settings
  8. Restart the vCenter Server services.
    Log in to vCenter via SSH. Type shell and press the ‘Enter’ key. Execute the below commands to restart the vCenter Server services.

    Note: Wait for all the vCenter Server services to come up. Roughly, it may take around 10 to 15 minutes for all the vCenter Server services to come up and running.

    For example, here are the commands to enable shell access and restarting vCenter Server services.

Steps to replace certificate for ESXi host

Generate Certificate Signing Request (CSR) for ESXi host

Step 1: Create an openssl.cfg file in ESXi host, and update CommonName and subjectAltName in openssl.cfg with appropriate FQDN of ESXi host. Here are the steps to follow.

  • Log in to ESXi host via SSH as the root user for which the certificate needs to be replaced.
  • Create openssl.cfg file with the content mentioned below.
  • Replace <FQDN> in the above openssl.cfg file with a fully qualified domain name (FQDN) of the ESXi host.
    The commonName and subjectAltName attribute must be set to FQDN of your ESXi host.
    Also, update the other attributes such as countryName, stateOrProvinceName, stateOrProvinceName, 0.organizationName and organizationalUnitName as appropriate.For example, here is the sample openssl.cfg file for reference after updating all the required attributes.


Step 2: Execute the below command to Generate a Certificate Signing Request (CSR) for the ESXi host.

The above command will generate the following two files:

rui-orig.key: Private key for the host
rui.csr: Certificate signing request file


Step 3: Execute the below command to convert the key to be in RSA format.

For example, here is the command to verify the existence of rui.csr and rui.key files in the ESXi host.

Generate Certificate for ESXi host

Step 1: Create an openssl.ini file in ESXi host, and update DNS in openssl.ini with appropriate FQDN of ESXi host. Here are the steps to follow.

  • Log in to ESXi host via SSH as the root user for which the certificate needs to be replaced.
  • Create openssl.ini file with content mentioned below.
  • Replace <FQDN> in the above openssl.ini file with a fully qualified domain name (FQDN) of the ESXi host.
    The DNS attribute must be set to the fully qualified domain name (FQDN) of your ESXi host.For example, here is the sample openssl.ini file for reference after updating the DNS attribute.

Step 2: Ensure that the root certificate i.e. cert.pem and private key i.e. key.pem files are already copied to ESXi host which was generated in very first step i.e. ‘Create a self-signed certificate using OpenSSL’ section.

For example, here is the command to verify the existence of cert.pem and key.pem files in ESXi host. If these files are not present then ensure to copy them before proceeding to the next step.


Step 3: Execute the below command to generate the certificate for ESXi host

The above command will generate the following file.
rui.crt : Certificate file

For example, here is the command to generate a new certificate for the ESXi host.

Replace Certificate for ESXi host

Step 1: Put the ESXi host in Maintenance Mode.

  • Log in to vCenter via vSphere WebClient.
  • Select ESXi host for which the certificate needs to replaced, and put the host in maintenance mode.
  • Disconnect the ESXi host from vCenter Server


Step 2: Log in to ESXi host via SSH as root user and navigate to directory /etc/vmware/ssl/


Step 3: Take a backup/rename the original certificate files such as rui.crt and rui.key which are present in ESXi host at location /etc/vmware/ssl

For example, here are the commands to take backup of rui.crt and rui.key files.


Step 4: Copy rui.crt (new certificate) and rui.key to /etc/vmware/ssl
(Note: The new certificate rui.crt was generated in Step 3 in section “Generate Certificate for ESXi host”, and rui.key was generated in Step 3 in section “Generate Certificate Signing Request (CSR) for ESXi host”)

For example, here is the command to copy the new rui.crt and rui.key files to /etc/vmware/ssl directory.


Step 5: Execute the below command to append the content of cert.pem to /etc/vmware/ssl/castore.pem
(Note: cert.pem was generated in very first step i.e. Create a self-signed certificate using OpenSSL, and it was already copied to ESXi host )

For example, here is the command to append the content of cert.pem to /etc/vmware/ssl/castore.pem


Step 6: Restart the ESXi host or restart the host’s management agents. In the below step, we will be restarting the ESXi host.
(Note: Restart can take some time, roughly up to 10 minutes. Wait for the host to come up before proceeding with next step)

For example, here is the command to restart the ESXi host.


Step 7: Connect ESXi host in vCenter Server System, and verify if the new certificate is shown for ESXi host

  1. Log in to vCenter via vSphere WebClient.
  2. Reconnect the ESXi host for which the certificate is replaced (accept the new thumbprint while reconnecting the host if there is a prompt).
  3. Exit the host from Maintenance Mode.
  4. Select Host > Configure > Certificate, and verify whether the new certificate is applied or not.
    (Note: In case if the new certificate is not reflected, disconnect and connect the host once again)

Here is the screenshot for reference to see the navigation steps in vSphere WebClient UI to check the new certificate details of the ESXi host.


***IMPORTANT NOTE: Repeat the above steps i.e. Generate Certificate Signing Request (CSR) for ESXi host, Generate Certificate for ESXi host and Replace Certificate for ESXi host listed under section ‘Steps to replace certificate for ESXi host‘ for all the remaining ESXi hosts of a domain.

References

For any additional information, please refer to the official VMware vSphere documentation. Here are few links for quick reference –

Configuring OpenSSL for installation and configuration of CA signed certificates in the vSphere environment
Update the vCenter Server TRUSTED_ROOTS Store
Change the Certificate Mode
Configuring CA signed certificates for ESXi 6.x/7.0 hosts
Replacing ESXi SSL Certificates and Keys

Related Articles