The security of the communication between VMware Cloud Director cells and ESXi hosts has been enhanced in the latest 10.4 version. This impacts the vCenter Server registration process as the ESXi certificate chain (usually signed by VMware Certificate Authority (VMCA)) must be trusted otherwise certain features that require direct ESXi communication will stop working (console proxy, OVF import/export, guest customization).
This further enhances the previous security changes such as the ability to disable hostname verifications for vCenter Server or NSX Managers and aligns with the industry security guidelines.
If you wish to know more about the previous feature enhancements and explanations, please refer to the blog post created by Daniel Paluszek.
In this blog, I will discuss the enhancements made to the VMCA certificate handling for VMware Cloud Director 10.4 which is generally available since 14th July 2022.
Before going further, let’s recap what VMCA certificate is:
vSphere provides security by using certificates to encrypt communications, to authenticate services, and to sign tokens.
vSphere uses certificates to:
- Encrypt communications between two nodes, such as a vCenter Server and an ESXi host.
- Authenticate vSphere services
- Perform internal actions such as signing tokens
vSphere’s internal certificate authority, VMware Certificate Authority (VMCA), provides all the certificates necessary for vCenter Server and ESXi. VMCA is installed on every vCenter Server host or Platform Services Controller, immediately securing the solution without any other modifications. Keeping this default configuration provides the lowest operational overhead for certificate management. vSphere provides a mechanism to renew these certificates in the event they expire.
vSphere also provides a mechanism to replace certain certificates with your own certificates. However, it is recommended to replace only the SSL certificate that provides encryption between nodes, to keep your certificate management overhead low.
For more details, please refer to VMware Documentation.
vCenter Server Registration Changes
The vCenter Server registration process consists of three steps:
- Retrieve the vCenter Server endpoint certificate and either explicitly or implicitly trust it
- Register vCenter Server as IaaS/SDDC endpoint (optionally with NSX-V Manager)
- After vCenter Server is attached, VMware Cloud Director retrieves VMCA certificate from the Certificate Management section of the vCenter Server. In case this certificate is not already trusted by VCD, you will be prompted to trust that certificate as demonstrated above.
Note that the assumption is that ESXi host certificates are signed by VMCA. In rare cases where a different CA is used to sign ESXi host certificates such CA certificate must be imported into VCD certificate trust store manually.
When using UI, you will be guided through the three-step registration workflow. However, when using API, the third step must be done after the vCenter Server registration. The VMCA certificate can be retrieved with this new API (v37.0):
GET /cloudapi/1.0.0/virtualCenters/{vcUrn}/certificateAuthority/vmca
The vCenter Server must be already registered as you must supply its URN in the API call. Then the VMCA certificate can be added to the VCD certificate trust store:
POST /cloudapi/1.0.0/ssl/trustedCertificates
Please note that the latest API for the certificate handling only works with vCenter Server 7.0 or later.
If you are running an older version of vCenter Server 6.7, you will not get the prompt to trust the VMCA certificate and will be able to attach the vCenter Server.
However, you will observe an error message in VMware Cloud Director as mentioned below:
This issue is addressed later in this blog.
Walk-through attaching a vCenter with distinct endpoint and VMCA certificates:
When attaching vCenter with VMware Cloud Director, the administrator will be presented with the prompt to trust the vCenter certificate (CA Signed Issued).
Complete the wizard to connect to the vCenter (after providing other necessary details), then you will be prompted to trust another certificate. This is the VMCA certificate (Self Signed as per my lab).
What if the VMCA certificate is not trusted?
If the VMCA certificate isn’t trusted, then following features won’t work:
- Console proxy.
- Powering on a VM with guest customization.
- OVF/Media Uploads.
What if you are running older versions of VMware Cloud Director. i.e., 10.3 with vCenter Servers attached and you are planning to upgrade VMware Cloud Director to 10.4?
Once you upgrade to VMware Cloud Director to 10.4, an advisory will be presented, referring you to KB 78885 for the changes in the vCenter Integration. for the changes in the vCenter Integration.
The following simple procedure will retrieve VMCA certificates and import them to the VCD trust store:
- In the upgraded VCD 10.4 go to Resources > Infrastructure Resources > vCenter Server Instances
- Select the vCenter Server which is already registered
- Click Edit.
- Click Save without making any changes. You will be asked to Trust the VMCA certificate
- Review the certificate and click Trust.
Note that the above procedure will work only for vCenter Server instances that are on version 7.0. If you have vCenter Server 6.7 in your environment, you will need to retrieve their VMCA certificate manually and import it to the VCD trust store.
Locate the VMCA in the zip file contents and add it to VCD’s trusted certificates as follows:
Alternatively, you can run the below cell-management-tool command to retrieve and trust certificates from all configured vCenter Server and NSX servers as well as the VMCA certificate.
/opt/vmware/vcloud-director/bin/cell-management-tool trust-infra-certs –vsphere –unattended
The above command works both for vSphere 7 and 6.7 environments.
However, if the above cell-management-tool option is used then you should audit the trusted certificates and remove the ones unnecessary for VMware Cloud Director.
Thanks to Ankit Shah & Tomas Fojta for their support and collaboration in this effort.
