Automating CA Self-Signed Certificates for ESXi 5.1 for use with resxtop

Last week I wrote an article about resxtop failing to connect to an ESXi 5.1 host due to SSL Certificate validation which has been implemented in resxtop and I provided a few workarounds that you can use until a fix is released for resxtop. As promised at the end of that article, I will show you how you can automate the creation proper certificates for environments using CA self-signed SSL Certificates so you can continue using resxtop with ESXi 5.1 until a fix is released.

Note: VMware definitely recommends as a best practice to replace the self-signed SSL Certificates that are automatically generated during an ESXi installation with proper CA signed SSL Certificates for additional security. For more details on SSL Certificate replacement, please refer to the following VMware KB.

Disclaimer: These script are provided for informational/educational purposes only. It should be thoroughly tested before attempting to use in a production environment.

Here is a high level overview before using the script:

1. The script will generates a one time CA cert & key which will then be used to to sign the SSL Certificates that are generated for each ESXi 5.1 hosts all using the openssl utility.
2. You will then need to upload the SSL Certificates (rui.crt & rui.key) generated for each of the respective ESXi 5.1 hosts under /etc/vmware/ssl directory
3. You will then need restart the rhttpproxy service on each of the ESXi 5.1 hosts for the SSL Certificates to take effect
4. OPTIONAL: If you are using vCenter Server to manage your ESXi hosts, there is an additional required step of re-connecting the ESXi hosts as they will automatically be disconnected from your vCenter Server. This is due to new SSL Certificates on the hosts and the thumbprint that vCenter Server knows about is no longer valid
5. Set the two required environmental variables HTTPS_CA_DIR and HTTPS_CA_FILE to the CA Cert that was generated in step 1 which will allow us to connect to our ESXi hosts that have had their SSL Certificates replaced

Note: Steps 2-4 are currently manual as it depends on when you want the SSL Certificates to go live on the ESXi hosts which will also require a service restart for the changes to go into effect.

You can download the script called generateSSLSelfSignCert.sh which is a shell script that can be executed on any UNIX/Linux system. In the example below, I running the script using the vMA appliance.

The script accepts a single command-line argument which is a file that contain the hostnames (FQDN) of all the ESXi 5.1 hosts you wish to replace SSL Certificates for. In the example below, I created a file called hostlist which contains the following:

pod23-esx-01a.pml.local
pod23-esx-02a.pml.local

Make sure you set the execute permission on the script before running. Here is a screenshot of the script executing based on the input file that I have provided:

The first section which is highlighted in RED, is creating a temporarily directory called ssl-certs in the current working directory and generates a one time CA Cert & Key which will then be used to self-sign the SSL Certificates for each ESXi 5.1 hosts. Please note that you will need the cacert.pem file which is generated from this step to use resxtop to connect to an ESXi 5.1 host. If you delete or lose this file, you will need to re-generate all new SSL Certificates.

Note: You also have the ability to run the script for additional hosts afterwards and if the cacerts.pem already exists, then it will not re-regenerate a new one and just use the existing one to sign new SSL Certificates.

The next section that is highlighted in BLUE is creation of the SSL Certificates (rui.crt & rui.key) for each of the ESXi hosts which is stored in their own directory based on the hostname.

Once all the SSL Certificates have been generated, you will now need to copy the rui.crt and rui.key files to each of the respective ESXi hosts under the /etc/vmware/ssl directory as noted in the GREEN highlighted section and then restart the rhttpproxy service for the SSL Certificate to go live on the ESXi host.

You can either use SCP which relies on SSH being enabled on the ESXi host or you can leverage the vifs utility which is part of the vCLI to upload your SSL Certificates. Take a look at this article for more details. In this example, I am using SCP and here is the command that I am using:

scp ssl-certs/pod23-esx-01a/rui.* root@pod23-esx-01a:/etc/vmware/ssl
scp ssl-certs/pod23-esx-02a/rui.* root@pod23-esx-02a:/etc/vmware/ssl

Here is a screenshot of SCPing the SSL Certificates to the ESXi host:

Once you have uploaded the SSL Certificates to each of the ESXi hosts, we then need to restart the rhttpproxy service. To do so, you will need to login into the ESXi Shell and running /etc/init.d/rhttpproxy restart or restart the management service via the DCUI. In this example, I will be SSHing into the host and restarting the service using the following command:

ssh [email protected] “/etc/init.d/rhttpproxy restart”
ssh [email protected] “/etc/init.d/rhttpproxy restart”

Here is a screenshot of restarting the rhttpproxy for both ESXi hosts:

Note: If the ESXi hosts are being managed by vCenter Server, after the restart of the rhttpproxy or management service, the hosts will disconnect from vCenter Server. This is due to a change in the SSL Thumbprint that vCenter Server had recorded about the ESXi host when added to vCenter. You will need to re-connect each host back into vCenter Server.

The final step which is highlighted in ORANGE is to set the two required environmental variables HTTPS_CA_DIR and HTTPS_CA_FILE to specify the location of the CA Cert that was generated earlier before using resxtop. To set the two variables, run the following command:

export HTTPS_CA_DIR=/home/vi-admin/ssl-certs
export HTTPS_CA_FILE=/home/vi-admin/ssl-certs/cacert.pem

Here is a screenshot of exporting the two environmental variables and running resxtop to connect to one of the ESXi 5.1 hosts:

Note: The exact path will be automatically displayed for you in the script output, you just need to copy and run the command. Once you have finished, you can then connect to any of your ESXi 5.1 hosts using resxtop and you no longer will get an error connecting to your ESXi 5.1 host. Please note that it is important you keep the cacert.pem safe, do not lost it or accidentally delete it, else you will need to re-generate new SSL Certificates.

What I have shown here is an example on how you can easily automate the creation of your own CA self-signed SSL Certificates for your ESXi 5.1 hosts (especially after hostname or IP Address has changed on the host and the default VMware self-sign SSL Certificates no longer match). Though there are a few places where it requires some manual intervention, these can all be automated but will depend on how you want to integrate this within your build process. The example above shows a post-install/replacement of SSL Certificates, but you can also generate these certificates in advanced and copy them into your ESXi hosts as part of the actual deployment.

Get notification of new blog postings and more by following lamw on Twitter:  @lamw