VMTN Blog

From the new VMware Security Blog, Nand Mulchandani responds to the article by Ron Oglesby and Dan Pianfetti at virtualization.info about the number of patches that VMware has released for VI3.

Link: VMware Security Blog > ESX patching questions.

Recently there was an article on “Patch Tuesday for VMware” over at Virtualization.info. It is an interesting article that raised some questions that we thought we might be able to shed some light on. The article was more focused on patching and not security alone, but since patching has now been so closely associated with security, so I'll jump in and provide a response on our security blog.

As the article points out, "patching is a necessary evil" - and that the existence of ESX patches should not come as a shock to anyone. So let’s talk about the sinister plan behind the increase in ESX patches. ...

You should read the whole thing. (Seriously. Nand explains it well.) One gee-whiz part for me is with the new Update Manager -- and even pre-3.5 with just DRS and VMotion -- how the end-user and admin experience for VI patches is very much not like MS Patch Tuesday. The other gee-whiz is the percent of patches that have been going to the Red Hat-derived Service Console, which of course with 3i is now  gone.

Link: Virtualization Security Guidelines - blog.scottlowe.org

The Center for Internet Security (CIS) has released some security benchmarks for VMware ESX Server 3.0.x.  The ESX security benchmark joins recommendations and guidelines for Windows 2000, Windows XP, Windows Server 2003, Red Hat Linux, and Mac OS X that are also available from the CIS.  The CIS has also published a generic virtual machine (VM) security benchmark as well. Taken together, the ESX benchmark and the VM benchmark provide good, solid recommendations around virtualization security.

The ESX/VM benchmarks are available for download here.

With all the hype around virtualization’s impact on security—positive or negative—it’s good to see some concrete and actionable recommendations.  If nothing else, these documents will at least provide a starting point for security professionals and virtualization experts to discuss the best way to architect solutions without compromising the security of the network/servers.  In fact, we might even find ways to enhance the security of the network/servers.

Link: Security Virtualization myths dispelled? | Virtual Scoop.

In an article titled VMware dispels virtualization myths, Bridget Botelho  wrote:

"One significant issue with virtual machine security is with virtual switch isolation," said Burton Group's Wolf."The current all-or-nothing approach to making a virtual switch 'promiscuous' in order to connect it to an IDS/IPS is not favorable to security." ...

This is an overall decent article but parts are very misleading.

I got in touch with Andrew Lambeth of VMware's Networking team for clarification. This is what he had to say:

... The vswitch-wide setting that probably confused him is not the only way to enable promiscuous mode. The right way to configure a vswitch for IDS/IPS is to create a separate portgroup from those used for normal VMs and configure only that portgroup for "Promiscuous Allowed". This prevents any normal VMs connected to the other portgroups on the vswitch from being allowed to sniff traffic not intended for them while allowing only the IDS/IPS VM to sniff.

Link: VMware shares secrets in security drive | CNET News.com.

VMware has traditionally restricted access to its hypervisor code and, while the vendor has made no official announcement about the API sharing program tentatively called "Vsafe," VMware founder and chief scientist Mendel Rosenblum said that the company has started sharing some APIs (application program interfaces) with security vendors.

"We would like at a high level for (VMware's platform) to be a better place to run," he said. "To try and realize that vision, we have been partnering with experts in security, like the McAfees and Symantecs, and asking them about the security issues in a virtual world."

Rosenblum says that some of the traditional tools used to protect a hardware server work just as well in a virtualized environment, while others "break altogether."

"We're trying to fix the things that break, to bring ourselves up to the level of security where physical machines are," he said. "But we are also looking to create new types of protection."

Rosenblum said the APIs released as part of the initiative offer security vendors a way to check the memory of a processor, "so they can look for viruses or signatures or other bad things."

Others allow a security vendor to check the calls an application within a virtual machine is making, or at the packets the machine is sending and receiving, he said.

"I don't want to be reverse engineering our products to find exploits or figure out signatures," Rosenblum said. "Fundamentally, that means we have to partner. Fortunately, there is a bunch that are happy to partner and I encourage that."

Posted by Charu Chaubal
Technical Marketing Manager for Datacenter Management

Recently, security consulting company Intelguardians presented at NDSS claiming they could execute malicious code on the host OS of a computer running VMware hosted virtualization software, such as the free VMware Player or the licensed VMware Workstation. Their subsequent presentation at SANSFIRE 2007, which was reported on in a number of blogs (such as PaulDotCom), apparently extends the NDSS presentation and talked about some other similar exercises (the slides from SANSFIRE 2007 are not yet posted anywhere so the exact details can’t be referenced).

It’s important to understand that this whole issue of guest-to-host compromise only exists for hosted virtualization platforms, such as VMware Player, Server, and Workstation, which run on top of general-purpose commodity operating systems. It specifically does NOT pertain to VMware ESX Server, for reasons that will be apparent below.

Although this series of demonstrations of guest-to-host attack does legitimately show one of the risks of virtualization, it is not due to flaws in virtualization technology as some might like to claim. Rather, it shows the dangers of using a product feature without fully understanding it.

So what should one make of these supposed new security exploits? All of them are based on the ability to pass information from the guest OS to the host OS in a trusted manner. Specifically, there are several ways of doing this in VMware’s hosted virtualization products:

• Host-Guest Shared Folders
• Host-Guest Drag-n-Drop
• Host-Guest Cut-n-Paste

The security of each of these ultimately depend on the host OS trusting the guest OS not to behave badly. If you trust the guest OS, then you believe that information that it passes to the host OS is not malicious or compromised. If the security of the guest OS cannot be guaranteed, then one should inspect any transferred file before doing anything with it. Alternatively, this type of functionality should be disallowed. For this reason, VMware has provided configuration settings which allow you to disable each of these. These settings are described in the VMware Workstation User’s Manual, in the section appropriately titled “Transferring Files and Text Between the Host and Guest”.

Bear in mind: these channels for passing information from the guest to the host are not related to the core virtualization layer. They are added on top of it, to provide ease of management and operation. The ability to transfer files from the guest to the host, e.g. without using scp, can be very convenient. By disabling these mechanisms, you lock out VMware-specific means of transferring malicious files to the host OS. (There are, of course, plenty of other ways of achieving the same thing – one cute way would be to have a hacked web server in the guest serve up an AJAX exploit to a browser on the host.)

As alluded to earlier, this issue doesn’t exist with ESX Server, since the “host” in this case is a purpose-built, thin, light-weight kernel which only runs code that ships with the product (for more information, see this white paper on the security architecture of VMware Infrastructure 3). There is neither any mechanism nor any reason for transferring files from the guest to this “host” OS.

The real lesson, however, is that VMware needs to be more prolific in educating the market about what features require implicit trust. (It doesn’t help that Drag-n-Drop and Cut-n-Paste are enabled by default when a VM is created.) Instead of “Escaping the Cave,” as PaulDotCom says, it’s more like “Being Escorted Out of the Cave.” In the end, security exists only to the extent of your least trusted component, and VMware will strive to provide more proactive guidance in the future that clearly details this.

We seem to be writing a lot about Blue Pill for something that's pretty hypothetical at this point.

A bit of background if you haven't been following this: Blue Pill is theoretical/proof of concept rootkit that uses virtualization -- a hypervisor architecture -- to insert itself and hide under your operating system.  Previous coverage on the VMTN Blog here, here, and here.

Here are excerpts from a longer background explanation by VMware's Beng-Hong Lim.

First off, it is important to understand that this threat targets the operating system.  It is not about vulnerabilities in virtualization. ... An interesting implication of the study is that operating systems that are already running in virtual machines are actually less vulnerable. ... A bare-metal style virtualization system, such as VMware ESX Server, does not have a general-purpose host OS, and is not vulnerable to the same attack points as on Windows and Linux operating systems.

A lot of claims have been made at this point, with hype as well as scoffing about undetectable rootkits. I know that you can't prove that something will never happen, but the computer scientists I talk to say this is very unlikely. The basic argument, most recently laid out by VMware's own Keith Adams and collaborators, is that it's easy to detect that you're inside a virtual machine, and in fact it's much easier to detect a hypervisor than to hide one. The disparity is so great that this isn't the same cat-and-mouse game that is being played with current malware. Here the good guys always stay ahead.

Recent work on applications ranging from realistic honeypots to stealthier rootkits has speculated about building transparent VMMs -- VMMs that are indistinguishable from native hardware, even to a dedicated adversary. We survey anomalies between real and virtual hardware and consider methods for detecting such anomalies, as well as possible countermeasures. We conclude that building a transparent VMM is fundamentally infeasible, as well as impractical from a performance and engineering standpoint.

Joanna Rutkowska, Blue Pill author, now has released a new version of Blue Pill as well as this blog post, wherein she claims that (and here I am paraphrasing in a slightly snarky way):

• her real point is that a monolithic kernel like Windows Vista is always going to be vulnerable to some sort of attack (OK)
• just detecting you're on a hypervisor is different than detecting that you're on an evil hypervisor (OK, but if you're on a hypervisor, first of all, you're now talking about vulnerabilities in ESX Server vs Windows, and eventually the hypervisor has to talk to the physical hardware and can detect that, as Thomas Ptacek explains here. Thomas seems to have taken the title of chief blue pill debunker along with colleagues Nate Lawson and Peter Ferrie. )
• some theoretical methods of detecting a hypervisor don't work so well in the real world, or at least in her hands (OK, I buy that as well -- theory doesn't always do well meeting reality; however, as Keith explains in here, they are really defending themselves against a straw man, not a real detection method)
• and if we have to resort to building in Symantec anti-rootkit technology into a hypervisor we've failed as well. (And again, with no disrespect to the fine ladies and gentlemen of Symantec, I'll agree with that too.)

OK, I've agreed with all of Joanna's points, but I don't think they've done much to convince me, the technical layman, that a completely undetectable rootkit is possible. 

If you want to dive deep on this topic, don't stop with misleading articles in the tech press. Go straight to the sources like this and this and this.

Just search for blue pill hypervisor to get the background. We've talked about it here before. VMware's Beng-Hong Lim and Keith Adams weighed in. Now Thomas Ptacek and Nate Lawson have challenged Joanna Rutkowska to a cage match at this year's Black Hat Briefings conference. I think at this point the argument is more about theoretical detectability in the field, Joanna having conceded that one can always check timing, but arguing that you can't prove the negative that a virtualization-based rootkit will never be invented. I am not losing any sleep.

Greg Ness, VP of Marketing for Blue Lane Technologies, wrote an article that talks about the increased security complexity that comes with virtualization. Not so coincidentally, Blue Lane has a product that can address these complexities! (Disclaimer: Blue Lane is a VMware partner, has a very cool product, and is going to release a virtual appliance.) Link: Virtualization: The Beginning of the End of Static Security

One of the more subtle outcomes of the hypervisor layer is that the network is now exposed on the server. This is good news and bad news – good in that it allows a new guard post on the servers, which can provide “zone defense” for the VMs without any footprint on the VMs; bad in that it presents a new target that can be exploited by hackers. It has been said that virtualization is changing everything. Security is obviously no exception.

In the virtual world, vulnerability scans can be rendered obsolete in an instant as new server images move from offline to online. Server sprawl means security solutions built on the assumption of the slower and more orderly changes inherent in the hardware-driven world will have a lot of catching up to do. You don’t want to be the last on your team to know that you’re not in Kansas anymore.

By de-coupling hardware from the operating system, virtualization challenges traditional network security solutions with location-specific rules of protection. For example, when new virtual servers are created and dynamically moved behind this important layer, they can inadvertently break static firewall rules. Security solutions for the virtual environment must automatically address dynamic moves and changes.

These are actually insightful observations around a new technology (virtualization) enabling new behaviors (resources coming on- and offline dynamically) which can have unintended consequences (security and monitoring applications may not know about these new machines on the network).  However, many times when an article talks about virtualization and security they start going on about patching all your Windows boxes, which seems to be exposing holes in your business processes and your virtual server sprawl more than anything inherent in virtualization (other than the aforementioned increased dynamicism). Scott Lowe, who evidently has his servers under control, weighs in. Link: Virtual Security Concerns

Generally speaking, anything that adds security to the infrastructure—virtual or physical—is usually a good thing, so I’m excited to see more vendors creating security solutions that are aware of virtualization solutions.  What I’m not so keen to see, though, is the trend among security vendors (and some analysts) that the addition of server virtualization completely changes the security picture. ...

“Special consideration for patching and updates”?  Huh?  How is patching a virtual instance of Windows Server 2003 any different from patching a physical instance? Administrators will still need to maintain virtual instances just like they maintain physical instances—both will need to be patched, reviewed for insecure configuration, scanned for malicious software, etc., generally using the exact same processes in both cases.

So go over to his site and let Scott know what you think. IANAITSE (I Am Not An IT Security Expert), but it seems to me that Scott is precisely correct, until you reach the dynamic resource pool stage of your virtual infrastructure, where you may not be able to ensure that all those dormant images sitting on your SAN somewhere are fully patched.

[Updated twice below.]

Check out Alex Bakman's VMworld 2007 presentation here as well on Top 10 Recommendations for Improving VMware ESX Security.

1. Use Firewall and Antivirus software for COS. Just as in any other operating system, this provides basic protection
2. Use VLANs to segment the physical network so only machines that are required to see each other are able to do so
3. When installing ESX, use security=high
4. Do not allow root level access over SSH and use secure commands
5. Disable all unnecessary services in console OS
6. Use VirtualCenter to help you manage granular security access
7. Stay current with ESX patches
8. Harden Guest Operating Systems
9. Control User Level Access using VirtualCenter
10. Document and monitor configuration changes in your environment, especially changes in security settings

[Update: from Schley Andrew Kutz in the comments:

I disagree about installing an AV scanner in the COS. There should be limited access to the COS to begin with, and an AV scanner provides unnecessary overhead. Remember, most ESX installations do not allocate the COS that much memory, and an AV scanner will just bog things down, and honestly is not all that helpful.

I appreciate Alex's work on this, but I work very hard to encourage people to think of VMs as physical machines. Hence, I think when designing documents like this we, as public voices for ESX, should separate guidelines and suggestions into 2 categories -- 1) topics related directly to VMs and 2) everything else. Steps 4, 5, 7, 8, 10 are all steps that should be applied to any server or operating system. They are not specific to ESX, its VMs, or guest OSs.

See also the two new papers just published on VMTN: VMware Infrastructure 3 Security Hardening and Security Design of the VMware Infrastructure 3 Architecture.]

[Update 2: From one of our internal security gurus:

Also as a general recommendation we suggest putting the Service console or Console OS (COS) on a separate network or VLan.  The  COS should only be accessible people who administer ESX.  This will significantly reduce the attack surface.  In fact if you can you should have the COS on its own private physical nic card and virtual switch.  Since you restrict permissions at the network layer the more common 'privilege escalations' are not possible.

As far as an anti-virus...  If you have a physically separate COS and you don't use the COS as a "general purpose" OS, you shouldn't need an anti-virus. If your environment requires anti-virus, you need one that will work with our glibc version.]

Pascal Meunier is covering his VMworld experience, mostly about security topics.

ReAssure (CERIAS), VIX and Lab Manager (VMware)

The VIX API on Tuesday morning was a very interesting session. It will enable the remaining automation functionality of ReAssure. It allows to automate the powering on and off of virtual machines, the taking of snapshots, transfering files (e.g., results) between the host and guest OS, and even starting programs in the guest OS! It was introduced with VMWare server 1.0 last summer, but I hadn’t noticed. It is still work in progress though; there’s support only for C, Perl and COM (no Python, although I was told that there was a source forge project for that).

Teaching (security) using virtual labs

There are of course other teaching labs using virtualization that have been developed at other universities and colleges; the challenge is of course to be able to design courses and exercises that are portable and reusable. We can all gain by sharing these, but for that we need a common infrastructure where all these exercises would be valid.

How virtualization changes the security equation

As a member of the panel argued, virtualization doesn’t make things better or worse, it still all depends on the practices, processes, procedures, and policies used in managing the data center and the various data security and recovery plans. Another pointed out that people shouldn’t assume that virtual appliances or virtualization provide security out-of-the-box. Out of all malicious software, currently 4-5% check if they are running inside a virtual machine; this may become more common.