Security with VMware Tools

VMware vSphere offers a number of tools to improve the security of guest operating systems, like UEFI Secure Boot, Virtualization-based Security (which enables Microsoft Device Guard & Credential Guard), vTPM, encrypted vMotion, VM encryption, vSAN encryption, and more. Let’s look INSIDE the guest OS at some options available through the VMware Tools.

Security is a process and a mindset, and to secure anything we need to consider the attacker’s perspective. Who is the attacker? Could be someone who has gained access to your organization’s network through malware or other means. Could also be what’s known as an ‘insider threat’ which could be a malicious employee, an employee being blackmailed, or even an ex-employee who continues to have access to the organization’s resources. The second question to think about is: what does the attacker want? It is likely that they want to gain access to data that is secured on a virtual machine. They may also be attempting to use one virtual machine to attempt to access another.

The longer an attacker works inside your organization’s environment the more likely it is they’ll be discovered, either by tools like those from VMware Carbon Black, log reviews and alarms from products like vRealize Log Insight, or just by an inquisitive admin hunting down what looks like anomalous behavior (like “The Cuckoo’s Egg” – great book, by the way). To avoid that, an attacker might choose to exfiltrate – “withdraw surreptitiously, especially from a dangerous position” – the whole virtual machine. This is an especially common goal when it comes to virtualized Microsoft Active Directory domain controllers. The attacker can simply take a copy of the virtual machine to their own infrastructure where they can attempt to break in and crack passwords without fear of discovery or lockout.

What can we do about it? We can use the vCenter Server role-based access control to help limit who has rights to clone & decrypt VMs, for instance. We can also use techniques I wrote about in my post “The Importance of Isolation for Security” to add layers of defenses, commonly called “defense in depth,” to help create additional obstacles for attackers. VMware Tools has some options that are intriguing as well.

Before We Begin

This post was written with VMware Tools 11.1.0 in mind. Many of these ideas show CLI commands and you can use the following executables:

C:\Program Files\VMware\VMware Tools\VMwareToolboxCmd.exe (Windows)
/usr/bin/vmware-toolbox-cmd (Linux, using the open-vm-tools packages)

The examples will reference the Windows executable, but you can simply replace that with the Linux command and use the same parameters.

Many of these options can be configured from the vSphere Client, too. However, if you are working to add isolation, setting them at the guest OS level reinforces these controls. An attacker that gains access to the vSphere infrastructure could simply reenable them, but it may be harder for them if the guest OS has them disabled, too.

Several Words of Warning

As with all things, please test changes, and do staged rollouts of changes like these (one machine, then 5, then 15, then 150, etc., waiting for a day or two between). Remember that just because you CAN do something, it does not mean you SHOULD. Security and usability are often intentionally at odds with each other and adding controls where they are not necessary might limit your ability to easily manage your VMs or implement tools. For instance, vRealize Operations Manager uses a number of these features to monitor and manage VMs, which is a net gain, as monitoring is very important to security, too (the ‘A’ in the CIA triad!). There are always tradeoffs & exceptions; please choose wisely.

Similarly, if you have administrator/root rights on a VM and reading this you should talk to your vSphere Administrators first before locking them out of things. While it can be used that way, this post is not meant to be a guide to thwarting your vSphere Administrators. Talk to them first, if only to give them the courtesy of a heads-up. You might find they are receptive to the ideas, so long as there is a way to change them in the future should the organizational needs change. You also might find that your vSphere Administrators are using some of these features to help with auditing, patching, reporting, and such, things that are important to your organization but you might not want to get involved with personally.

Last, while we always strive for accuracy, this is meant as a guide. Please engage your own professional security & compliance auditors when it comes to securing your environments. All environments are different, and the architecture and details of implementations matter immensely.

Ensure Time Synchronization

Time synchronization is very important for security. Cryptographic operations need accurate time to connect to other hosts, and accurate time means accurate timestamps in logs, which is important for diagnostics and developing timelines if a breach should happen. It is considered a best practice to not synchronize guest OS time to the ESXi hosts, but instead let the VMs sync with NTP or other time sources. It is also considered a best practice to use at least four NTP sources to achieve N+1 redundancy (three sources being the minimum, plus one extra for redundancy). Never configure two sources, because you cannot tell which one is correct!

To ensure that time synchronization is disabled in-guest use the command:

Control Updates to VMware Tools

It is important to keep VMware Tools updated, like you update other software. Issues are fixed, improvements are made, and features are added. vSphere Administrators can influence the VMware Tools upgrade process from the infrastructure itself which can be a huge timesaver. However, them doing so means that they are loading executables on virtual machines from the infrastructure itself, and this may be something that a VM or application admin may wish to control. Similarly, vSphere Administrators can control some VMware Tools features, which may or may not be of interest.

If your organization has an official way to update & configure VMware Tools that is not through the vSphere infrastructure (using a separate system management tool, for instance) you should consider disabling those infrastructure capabilities.

Prevent Customization of a VM

There are some interesting opportunities for both human error and attackers when you consider cloning a VM and applying a customization specification to it. Customization specs allow you to change administrator passwords, reset network settings, change domain membership, and even set commands to run once when someone logs in for the first time. The VMware Tools allow you to disable customization, and it is probably a good idea, in general, to disable it once a VM is deployed from a template, to prevent someone from being able to apply customizations to it either in your environment or in theirs.

Control Information Offered via AppInfo

Appinfo is a method to do application discovery through the VMware Tools. It is a great way to get information about running processes, and tools like vRealize Operations Manager use it to help monitor an environment (the vSphere Cloud Community blog has a post on using it). You can also use it to assess patching state, as it can return versions of the executables, too, which means an attacker who doesn’t necessarily have access to a VM itself may have an easier time determining which exploits to use to gain access. If you are not using it consider disabling it.

Disable Guest Operations (Invoke-VMScript)

Guest Operations are a powerful management capability within vSphere, where a vSphere Admin can use the PowerCLI cmdlet Invoke-VMScript to interact with the guest operating system from the infrastructure itself. Invoke-VMScript uses guest authentication to enforce this, so someone trying to use it would need to know an account that has access to the host (consider domain-joined guests, though). Organizations often already have a way to issue ad-hoc commands to their managed guest OSes, and if so you might consider disabling this capability with the following command:

Consider Installed Modules

VMware Tools are meant to serve a wide community of VMware customers, running VMs on Workstation, Fusion, ESXi, and in the VMware Cloud, and the capabilities are different everywhere. For example, Workstation does not use the Service Discovery features, and ESXi does not implement Shared Folders. To help cope with this the VMware Tools developers have modularized a lot of the Tools so that customers can be granular about what they install.

On vSphere we strongly recommend the paravirtualized device drivers (vmxnet3 and pvscsi) for performance reasons, and the other drivers give you other functionality that makes like better without much tradeoff for security. However, other modules like Service Discovery, AppDefense, Shared Folders, and the like might not be necessary if you are not using those features (the VMCI component itself is what handles hypervisor-to-guest communications, including version reporting, Tools status, etc. so leaving that installed is a good idea). You can certainly customize the features from the VMware Tools Installer GUI, or through command line flags for a silent installation.

Is it bad to accept the defaults for the installation? No. Most of our customers do not need to deal with that level of granularity, and if you keep VMware Tools updated in a timely manner you are covered in most situations. For customers interested in higher levels of security, at the expense of added complexity, these are considerations that can be made. As I mentioned at the beginning, it’s possible to go too far with security and make your own lives, and the lives of all types of IT staff, difficult. Choose wisely, communicate well, test, and do staged rollouts.

Good luck and stay safe.