A great question crossed my desk today from a customer. “Can a VI Admin who has root access to ESXi “abuse” their privileges and “peek” inside the guests of VM’s hosted on the server?”
The short answer? If your ESXi admin has root or full administrator privileges, they can do anything. Nobody should be surprised by this! HOWEVER, you can mitigate, limit and monitor what is being done.
But first, let’s quickly review what is meant by “peek inside the guest”. In the human world, introspection is the examination of one’s own thoughts and feelings. In the virtual datacenter world, it is the examination of the CPU and memory contents of a virtual machine running on a hypervisor. Why is this an issue? Because the contents of a virtual machines memory will contain all sorts of sensitive information! Things like passwords and encryption keys for example. (this should also not be a surprise to any of you!)
In the examples below, we’ll touch on two types of access. The first is primarily one where you examine static contents e.g. a snapshot of the VM’s memory and the second would allow monitoring in real time. The first would require just ESXi root privileges and the second is FAR more complex and less probable or practical.
Console level
At the console I can do just about anything. I could snapshot a VM and rummage through the .vmss memory file just like a memory dump. Because of the danger of “root” level permissions, what you really have to rely on is what I call “Reagan’s Rules of Virtualization Security”.
Trust, but verify
At some point, a level of trust needs to be made between you the business owner or security guy and the administrators with full admin privileges. And in order to maintain that trust, verification of actions being done is necessary. And there’s no better tool for this at the console level than syslog output. As I showed in a previous blog, all commands executed at the console are logged to syslog. A good syslog tool, like VMware Log Insight, can scan the syslog output for custom strings like “rm –f *” or “vi /etc/passwd” or “createsnapshot” and alert you to these actions. Here’s an example of commands being captured.
Ideally, the best case is to limit access to ESXi to only your most trusted administrators. Accessing the ESXi console should be, IMHO, a “break glass scenario”. Leveraging higher-level API functions via clients or command lines (PowerCLI, Web Client, etc) allows you to leverage fine-grained role-based access controls already built into the vSphere API used by ESXi and vCenter.
VMsafe
Back a few releases ago, VMware came out with a restricted API under the moniker of “VMsafe”. In this program there was a CPU and Memory introspection API. As mentioned, this was a restricted API that was only available to VMware partners. This API allowed a partner solution to read the contents of memory or CPU, essentially giving the partner solution “x-ray vision”.
For vSphere 5.5, this API is no longer supported. See KB 2058911 for more details.
There are settings in the vSphere Hardening Guide that can be set to enable or disable and configure the use of the API. These use the VM Advanced Settings of “vmsafe.enable”, “vmsafe.agentAddress” and “vmsafe.agentPort”.
In vSphere 5.5, vmsafe.enable is disabled (value=”False”) by default.
The Vulnerability Discussion for these settings is as follows:
“The VMsafe CPU/memory API allows a security virtual machine to inspect and modify the contents of the memory and CPU registers on other VMs, for the purpose of detecting and preventing malware attacks. However, an attacker might compromise the VM by making use of this introspection channel; therefore you should monitor for unauthorized usage of this API. A VM must be configured explicitly to accept access by the VMsafe CPU/memory API. This involves three parameters: one to enable the API, one to set the IP address used by the security virtual appliance on the introspection vSwitch, and one to set the port number for that IP address. If the VM is being protected by such a product, then make sure the latter two parameters are set correctly. This should be done only for specific VMs for which you want this protection. “
These are the steps that need to be done:
Explicitly enable the vmsafe.enable setting
Set an IP address in the vmsafe.agentAddress
Set the port number in vmsafe.agentPort.
Stand up a VM running the restricted API
So, if I’m the malicious admin and I want to do all this work, I’ll need access to the restricted API and I need the ability to write code at a very advanced level. See where I’m going? It also means 3 configuration changes to a VM to enable this functionality and having a rogue VM on the network being ready to accept that introspection IP traffic. Now, assuming, however unlikely, that said administrator is that talented and can get access to the API and write all that code, it still means that a LOT of configuration changes will need to be made that could be captured by a log monitoring solution (see above) or viewable in the event log of vCenter.
RBAC – Role Based Access Control
Many of these kinds of configuration issues can be addressed by not allowing people to do things they have no business doing. In the (improbable) scenario above, making sure all administrators can’t change VM advanced settings would limit your exposure.. You can do that by creating an Role that disables Virtual Machine..Configuration..Settings. Assign this role to the admins you are concerned about and then apply this permission to the VM’s that you want protected.
Probable .vs. Impractical
We’ve talked about two scenarios. Now let’s talk about what’s more likely. Do you have a super-genius administrator with access to a restricted API and the means to set up a no longer supported solution to grab memory and CPU contents in real time?
Probably not.
But do you have a bunch of administrators with root level privileges at the ESXi console and are you not capturing syslog traffic? And are they all using a shared root password?
Yea, that’s the more likely scenario.
And how easy is it to compromise them using malware? It’s cheap AND easy. Yikes!
So, this is probably a good time to think about just who REALLY needs access to the ESXi console. And to monitor what gets done when someone is logged into it. If logging into ESXi directly is a “break glass scenario” then the act of someone logging in should be a red flag, right?
Good operational hygiene is paramount to security in the scaled out world of the SDDC. The hypervisor and vCenter are the nexus of your datacenter. They are your keys to the kingdom and as such, access should be limited and monitored. You don’t let just anyone walk into the computer room, right? That means limited access to configuration changing AND limited access to the management consoles. It means using named accounts for all admins.
In my recently released ESXi whitepaper I address a lot more about security at the hypervisor level. I’d encourage you, the VI Admin, to read through it. In addition, PLEASE forward it on to your security team. It’s my hope that it can help up-level the conversation about hypervisor security and limit the defensive discussions that sometimes come up. When both sides have a common level of understanding you can better focus on the task at hand.
