VMware has released patches that address a new critical security advisory, VMSA-2021-0010 (CVE-2021-21985 & CVE-2021-21986). This needs your immediate attention if you are using vCenter Server (if you didn’t get an email about it, please subscribe to our Security Advisories mailing list). In most cases a security advisory is straightforward, but sometimes there are nuances that are worth extra discussion. That is the case here, and the goal of this post is to help you decide your course forward.
There are also tips for patching vCenter Server at the end which you may find helpful.
First, if you haven’t read the original advisory or are a returning visitor here are some links to the different resources available:
This post & the questions & answers document will be updated as new information develops.
Who is affected?
VMware Security Advisories always list the specific product versions that are affected. In this case it is vCenter Server 6.5, 6.7, and 7.0.
When do I need to do something about this?
Right now. These updates fix a critical security vulnerability, and it needs to be considered at once. Organizations that practice change management using the ITIL definitions of change types would consider this an “emergency change.” All environments are different, have different tolerance for risk, and have different security controls & defense-in-depth to mitigate risk, so the decision on how to proceed is up to you. However, given the severity, we strongly recommend that you act.
Why am I affected?
The VMSA outlines two issues that are resolved in this patch release. First, there is a remote code execution vulnerability in the vSAN plugin, which ships as part of vCenter Server. This vulnerability can be used by anyone who can reach vCenter Server over the network to gain access, regardless of whether you use vSAN or not.
Second, improvements were made to the vCenter Server plugin framework to better enforce plugin authentication. This affects some VMware plugins, and may also cause some third-party plugins to stop working. VMware partners have been notified and are working to test their plugins (most continue to work), but there may be a period after updating when a virtualization admin team may need to access backup, storage, or other systems through their respective management interfaces and not through the vSphere Client UI. If a third-party plugin in your environment is affected, please contact the vendor that supplied it for an update.
What should I do to protect myself?
First, if you can patch vCenter Server, do it. In general, this is the fastest way to resolve this problem, doesn’t involve editing files on the vCenter Server Appliance (VCSA), and removes the vulnerability completely. From there you can update any plugins as vendors release new versions.
If you can’t patch right away, and you don’t use vSAN, there are workarounds linked from the VMSA for disabling the plugins affected by this VMSA. This involves editing a text file on the VCSA and restarting services. Remember to re-enable those plugins after you update. If you disable the vSphere Lifecycle Manager plugin on vSphere 7 you will lose that functionality until you patch vCenter Server from the Virtual Appliance Management Interface (VAMI).
If you use Site Recovery you can disable the associated plugin and continue to manage your environment through the Site Recovery interface itself.
If you are amidst an upgrade to vSphere 7 this patch causes a “Back in Time” upgrade restriction, which will be a consideration for whether you choose the patch or the workaround. The course forward will depend on many things, including where you are in the upgrade process, if you use vSAN, and if you have other security controls in place to help mitigate this issue. There is more information in the Questions & Answers linked above and below.
If you ARE a vSAN customer, disabling the vSAN plugin will remove all ability to manage vSAN. No monitoring, no management, no alarms, nothing. This might be fine for your organization for very short periods of time but we at VMware cannot recommend it. Please use caution.
You may have other security controls in your environment that can help protect you until you are able to patch. Using network perimeter access controls to curtail access to the vCenter Server management interfaces, for example.
In this era of ransomware it is safest to assume that an attacker is already inside the network somewhere, on a desktop and perhaps even in control of a user account, which is why we strongly recommend declaring an emergency change and patching as soon as possible.
Tips for Patching
This advisory is only for vCenter Server, so that lessens the impact of patching (versus updating all hosts). vCenter Server is the management interface for vSphere, and restarting it does not impact workload availability, just the ability to manage the workloads. This is an important point to convey to change managers, too, as they may not understand that the workloads will continue running.
Some other thoughts on ensuring success while patching:
Ensure that your organization has the VCSA root & [email protected] account passwords stored correctly and are not locked out. By default, the VCSA root account locks itself after 90 days, which may be an unwanted surprise if you need it in an emergency. Prior to patching we suggest verifying these accounts work correctly, recovering the passwords if needed (which sometimes takes a restart of vCenter Server), and then changing them after the work is done as a good security practice.
Ensure that time settings are correct on the appliance. Many issues on systems can be traced to incorrect time synchronization. The maintainers of the open-source NTP software suggest configuring four NTP servers (but never two – if one is wrong you’ll never know which one!).
Ensure that there is a properly configured A (forward) and PTR (reverse) record for vCenter Server in DNS. You might think “these are basic and were set up long ago” but it only takes a second to check, and sometimes you learn interesting things. PTR records are required for vCenter Server and omitting them is not an accepted security practice.
Ensure that vCenter Server’s file-based backup & restore is configured and generating scheduled output. You can configure this through the Virtual Appliance Management Interface (VAMI) on port 5480/tcp on the VCSA.
Take a snapshot of the VCSA prior to the update, and preferably from the ESXi host client after the VCSA has been shut down gracefully & cleanly. Snapshots have performance impacts, so ensure that you delete it soon after the upgrade is verified.
An experienced sysadmin once suggested that if it has been a while since a system has been restarted it is often a good idea to simply restart it in place, let it come back up again and prove that it’s working well. Otherwise, you won’t be able to tell whether a problem was pre-existing or because of the work that just happened. An extra reboot does add management interface downtime, but if corrective action is needed it shortens the time to restore the service.
If vSphere HA has been configured with a custom isolation address (das.isolationaddress) ensure that it is NOT set to the vCenter Server itself, or that multiple addresses are specified (das.isolationaddress0 through das.isolationaddress9) so that one address becoming unavailable does not trigger HA failover.
Where possible, minimize the number of plugins installed in vCenter Server. Modern zero-trust security architecture practices discourage connecting systems in these ways, to make life harder for attackers (a single pane of glass is nice for cybercriminals, too, especially if it allows access to your backups). Fewer things installed also means fewer things to worry about from a compatibility perspective and makes upgrades and patching much less work.
Use DRS groups & rules to keep vCenter Server on a particular ESXi host so that if there is an issue you can find the VCSA easily using the ESXi host client. Create a host group that has only that one host in it and create a “should” affinity rule to keep that VM on that host. By using “should” you enable the VCSA to be automatically moved by DRS for normal host patching. Ensure that a management workstation can get to the host client interface on that ESXi host, and that you can log in.
Where should I go for more information?
There are a number of other places to look for more information on this issue, as well as guidance about security in vSphere:
Critical security advisories are always difficult conversations, and unfortunately part of the landscape in IT. We at VMware are always looking at what we need to do to our products to keep these advisories as uncommon occurrences, so we can go back to talking about all the positive security that vSphere offers. Please let us know what other questions we can answer. We appreciate you very much, thank you for being our customers, and we look forward to seeing you more as VMUGs and other events resume this fall.
