Posted by Ken Werneburg Tech Marketing Twitter @vmKen
Why does my "VR Management" service show with a red alert error in the "vCenter Service Status" tab?
When an extension (SRM, VR, etc.) is registered in vCenter it passes an SSL thumbprint that is used to establish the validity of the connection, and that is stored in the VC ADAM database.
The VRMS can generate its own certificate and will use that one with VC if chosen. If using an external cert, it will pull the external cert and replace the self-generated one. A thumbprint is generated for this cert, and this is used to registers the service correctly with the vCenter Server, and allows it to communicate for normal operations.
Retrieval of the service status, however, is done from vCenter by establishing an SSL connection to a different service, the VRMS "lightweight web server", which uses a different certificate and thumbprint.
If this thumbprint is not registered correctly for the health check when the VRMS service is registered, vCenter will be using the wrong thumbprint for the SSL cert when checking health.
The net result is certificate errors during usage, and your VC can not query the vSphere Replication management appliance to find out its health. Both the VC and VRMS believe they are using correct certs, so they carry on running normally, but the communication channel for querying health is not handshaking correctly.
Therefore you see things like red alerts on the VRMS service status in your vCenter, and errors like "unable to retrieve health data from https://<VRMS ip address>:8080/service/hms/cgi/health.py"
How do we fix this?
1) Obtain the lighttpd server.pem thumbprint on your VRMS appliance. This is what the VRMS believes should be used as the certificate thumbprint for SSL queries of the health service.
Log into the console of the appliance.
Examine the "serverThumbprint" line of /opt/vmware/hms/conf/extension.xml. E.g. "cat /opt/vmware/hms/conf/extension.xml | grep -i thumbprint"
Take note of this, but we will not use this again. This is the potentially incorrect thumbprint that was originally used but not updated. This is the VRMS thumbprint itself.
Change directory to /opt/vmware/etc/lighttpd
Execute "openssl x509 -in server.pem -noout -fingerprint". Take note of this, copy and paste, write it down somewhere, do all of the above. This is the VC SSL thumbprint we are expecting to use for service health queries.
2) Examine what is actually registered in VC
Download "ldp.exe" from Microsoft into your vCenter Server
Execute ldp.exe on that system.
Connect to your vCenter Server "Connection -> Connect". Make sure you connect to the vCenter Server to which your VRMS is registered!
Bind the connection "Connection -> Bind" Make sure you use an account with admin credentials.
View the tree "View -> Tree" Don't give it a BaseDN, just click "OK"
Double click the entry that begins with "CN=com.vmware.vcHms,…"
Examine the output on the right, and you will see one field tagged "vmw-vc-SSLThumbprint"
Check which thumbprint it is, and if you were having errors most likely it is filled with the entry from the extension.xml file we looked at first, rather than the actual signature from the openssl retrieval we did from server.pem.
3) Fix the SSL Thumbprint to reflect the correct entry
Right-click on the entry that starts with com.vmware.vcHms in the left navigation screen, and select "Modify" View this photo
Change the operation from "Add" to which it defaults, to "Replace". This is important and easy to overlook, so we'll do this first.
In the field titled "Edit Entry Attribute" type "vmw-vc-SSLThumbprint"
In the "Values" field, enter the correct thumbprint retrieved earlier from the server.pem file on the VRMS appliance that we retrieved with the openssl command.
In the results window on the right you will see a modify call is executed. If this successfully completes, you may click "close" on the Modify window. View this photo
Double click on the "CN=com.vmware.vcHms…" field in the navigator pane on the left and verify that the "vmw-vc-SSLThumbprint" value has updated correctly.
4) Verify it worked
Restart the VirtualCenter Management Webservices
Launch your vSphere Client, check the services, and see green.
Enjoy your ability to track and monitor the vSphere Replication service!
