On April 9, 2020 VMware published VMSA-2020-0006, outlining a serious vulnerability which may affect vCenter Server 6.7 and external Platform Services Controllers (PSCs) if certain criteria are met. This post is intended to help VMware customers and partners understand the issue better by collecting common questions. It is not intended to replace official VMSA communication. All customer environments are different and critical security issues such as this should start an immediate conversation between your organization’s management, information security personnel (CISO, etc.), and IT operations.
Please Read
The following links will help you determine if you are affected by this issue. Please read them before continuing.
What it is: Please read the VMware Security Advisory for the full description, and while you’re there sign up for the security advisory mailing list using the form on the right. In short, if your vCenter Server 6.7 instances and/or external Platform Services Controllers (PSCs) were created as part of an upgrade from a previous release line a malicious actor with network access to the appliance may be able to compromise vCenter Server or other services.
What you need to do: Find all instances of vCenter Server 6.7 and external 6.7 Platform Services Controllers that were upgraded from a previous vSphere version, as described in the VMSA, and update them using the standard patching process. As the vCenter Server Appliance VAMI may offer several patches please ensure you are selecting the latest one. Ensure that you are updating ALL vCenter Server Appliances as well as ALL Platform Services Controller instances. As always, ensure you have a backup of the affected systems prior to patching.
If in doubt, assume you require the update and act accordingly.
What effect will it have: Patching will close the vulnerability directly, and you can move on knowing you are secure.
So… should I be worried?
Organizations that don’t have a regular cadence for patching & updates or whose organizational structures make it nearly impossible to make changes in a timely fashion should be very worried. Even rigidly structured frameworks like ITIL support the concept of “emergency changes” and this would certainly qualify.
The best way to handle this issue is to patch it directly using the normal update procedures, such as through the VAMI on the vCenter Server Appliance.
Compensating controls, such as firewall rules on the vCenter Server Appliance, makes an environment more complicated and do not actually fix the underlying problem. Compensating controls and workarounds are like a cast on a broken leg. They don’t heal the wound; they just cover it up to help protect it. Over time the cast becomes a problem of its own. These workarounds make you less flexible, just as a cast does. They slow you down, just as a cast & crutches do. They also complicate operations by having their own rules and requiring their own maintenance. For example, “don’t get your cast wet” means a lot of normal activities are now really, really hard. Your cast will require its own maintenance, too, as it gets smelly and itchy and breaks down. Same for the workarounds.
You WILL spend much less time and effort overall if you just fix the problem outright by patching. This is borne out in experience and studies of IT over the last 40 years.
While we always recommend running current versions of all infrastructure software, you can patch vCenter Server and the Platform Services Controllers independently of ESXi. A common misconception by non-technical managers, and change management staff unfamiliar with vSphere capabilities, is that workloads are affected by vCenter Server patching. Workloads do not stop running when vCenter Server and the Platform Services Controllers are rebooting. Only VM management operations like console access, power on/power off, resizing, etc. would be affected for the duration of the update process. Direct access to VMs, such as via RDP or through an application’s normal interface, continue unaffected. It is often worth outlining this directly in proposed change documentation.
Frequently Asked Questions
How do I know if I am affected?
Read the VMSA and the associated KB article for details. If you are unsure, always proceed as if the patch is required.
I cannot patch my environment. Is there a workaround for this?
As the VMSA states there are no workarounds. There may be compensating controls, drawing on any defense-in-depth your environment has (isolation, etc.).
When you say “compensating controls” what do you mean?
As the PCI Security Standards Council defines it, “compensating controls may be considered when an entity cannot meet a requirement explicitly as stated, due to legitimate technical or documented business constraints, but has sufficiently mitigated the risk associated with the requirement through implementation of other controls.”
In short, they are other, more roundabout ways of achieving the same security goal without using the specific control that’s listed. In this case, using isolation techniques (network segmentation, firewalling, etc.) might be a compensating control, but that is a discussion between you and your information security staff.
When you say “isolation” what do you mean?
Isolation is where you use technologies like network segmentation, VLANs, firewalls, and such to separate assets based on the function of the asset. For instance, the VMware Validated Design is the documentation for a reference implementation of VMware vSphere and other products like the vRealize Suite. It outlines the use of VLANs to separate hypervisor management traffic from workload traffic. This separation would then be further enforced with firewalls. Implementing controls like this is a discussion between you, your information security staff, and often your network administrators.
How do I create firewall rules on the vCenter Server Appliance?
Please refer to the documentation, Edit the Firewall Settings of the vCenter Server Appliance, or contact VMware Global Support Services. Please test this in a dedicated test environment, and ensure you create a rule ALLOWING access for yourself prior to creating rules that deny access.
Can I just disable vmdird?
No. It is a required service.
What network port does vmdird use?
389/tcp.
What firewall rules should I put in place?
As explained above, adding compensating controls to your environment increases its complexity. We urge vSphere Admins to avoid unnecessary complexity. A firewall rule merely hides the issue, and does not fix it. As such, our recommendation is to patch in order to solve the problem outright.
That said, make sure that all linked vCenter Servers, and all PSCs in the domain, and any other products installed that use vCenter SSO authentication, have access to each other.
I inherited a vSphere environment. How can I tell if it was upgraded or a clean install?
Given the severity of this it is best to assume that it requires the patch if there is not evidence or documentation to prove otherwise.
My vCenter Server Appliance has been online for a long time and the logs have rotated and are compressed. How do I search files ending in .gz?
1
zcat/var/log/vmware/vmdird/*.gz|grep ACL
My vCenter Server Appliance has been online for a long time and old logs have been deleted. How can I tell if I need the patch?
If you restart the PSC or vCenter Server it will generate the message again. However, if you’re going to restart them you could also patch them…
Can I just restart vmdird to see the if I get the ACL MODE message?
We do not support or condone this without the guidance of a VMware Global Support Services engineer. Please open a support case.
My external PSC needs to be patched. Does that mean I don’t need to patch vCenter Server?
For support and compatibility reasons you need to keep vCenter Server and the PSCs at the same release level. Plan to patch them all.
Does this affect vCenter Server instances with SSH? If I disable SSH will that fix it?
We always recommend disabling SSH because, in most cases, it is not needed for management and configuration and disabling it eases compliance audits and improves security. However, in this case changing the state of SSH will not affect this particular vulnerability.
Does this affect Windows installations of vCenter Server?
Yes, the VMSA describes the versions affected.
Does this affect vCenter Server 6.5?
No. The VMSA describes the versions affected.
What if I migrated from Windows vCenter Server 6.5 to the appliance version of vCenter Server 6.7? Do I still need to patch?
Yes. The VMSA describes the versions affected.
How can I get the fix for this issue without updating all of vCenter Server?
As the VMSA states there are no workarounds. As such, the patch for this vulnerability is only found in the complete update to the version of vCenter Server listed in the VMSA.
Do you have a list of the vCenter Server releases between my build and the current one?
Can I update my Platform Services Controller and not patch vCenter Server?
During normal operations vCenter Server and the Platform Services Controllers need to be kept at the same versions to ensure compatibility and supportability.
Will Skyline Health proactively warn me if I’m affected?
VMware Skyline is a wonderful service available to all Production Support and Premier Services customers. The Skyline team at VMware has added this issue as a warning, but Skyline cannot tell directly if your PSC or vCenter Server is affected. Please use the method described in the KB and VMSA linked above. When in doubt assume that you require the patch.
Are other products that use vSphere affected?
Potentially. VMware Cloud Foundation is updated rapidly once the patch is available and tested. For other integrated solutions please ask the vendor directly.
On April 9, 2020 VMware published VMSA-2020-0006, outlining a serious vulnerability which may affect vCenter Server 6.7 and external Platform Services Controllers (PSCs) if certain criteria are met. This post is intended to help VMware customers and partners understand the issue better by collecting common questions. It is not intended to replace official VMSA communication. All customer environments are different and critical security issues such as this should start an immediate conversation between your organization’s management, information security personnel (CISO, etc.), and IT operations.
