VMware Security Blog

Recently ESX and VirtualCenter (VC) patches were released which - among others - fix several security issues. These issues are detailed in a new advisory, VMSA-2008-0007, and in the updated advisories VMSA-2008-0002.1, 0003.1, 0004.1, 0006.1. Please take some time out of your busy schedule to review your deployments and update where appropriate.
In an effort to improve our advisories, we have added a change log which should help identifying what has changed. Your feedback on this and on other possible additions to our advisories is highly appreciated. You can reach us by e-mailing security@vmware.com; our PGP key can be found here.

I posted a few weeks ago about VMware's booth at RSA next week and Steve Herrod's panel session next Friday.  However, virtualization and security is not just a VMware thing, it's an entire security community thing, so expect to hear a lot of security vendors at the RSA Expo this year discussing their plans around virtualization.

Many vendors have recently announced or are announcing virtual appliance editions of their products and will be showing them off at RSA.  For example, Symantec will be demonstrating their Mail Security 8300 gateway security product for messaging threats as a virtual appliance.

Some notable booths and events to check out with VMware partners at the RSA Expo:

• Tuesday, 1:15 pm, at the Tripwire booth - Dwayne Melancon and Gene Kim of Tripwire will be discussing "Controlling Risk in Virtual Environments"
• Wednesday, 1 pm, and Thursday, 2 pm, at the Shavlik booth - Shavlik CTO Eric Shultze and VMware product management will be speaking about virtualization security and product roadmaps
• Ongoing, at the McAfee booth - McAfee will be doing demos in their booth of potential new anti-malware capabilities using the VMware VMsafe API
• Ongoing, at the RSA Partner Pavilion - RSA and VMware will have a full Virtual Desktop Infrastructure (VDI) deployment showing the out-of-box integration of the Virtual Desktop Manager with RSA Authentication for secure two-factor end-user authentication

In addition, we plan on having several partners at VMware booth to discuss their roadmap for VMware integration or the VMsafe API.  So far we have McAfee coming by at noon on Tuesday and Wednesday and Tripwire at 3:30 pm on Tuesday.  Look for additional announcements onsite at our booth #339.

Finally, to unwind after a full day of conference activities, we've heard from some folks active on our Security and Compliance Communities forum that they're planning to meet up informally for drinks Wednesday from 5-8 pm, at the Thirsty Bear Brewing Co. at 661 Howard (just past the W Hotel).  The VMware security team will be dropping in for a drink as well, several of us right after the Expo closes at 6 pm.  We look forward to chatting with customers and other security community folks there - look for us in light blue VMware polo shirts.

Check out the VMware Communities post here RSA 2008 Conference in San Francisco. 

See you at RSA!

Last Monday, we released an advisory on the security fixes that went into the latest releases of Workstation, ACE, Server, Player and Fusion. We want to make sure that you have seen the advisory and we advise to review your VMware deployments and update where appropriate.
The fixed security issues range from denial of service to overwriting of sensitive  files on the host. One of the fixed issues is the path traversal issue that was made public late February by Core Security Technologies. The OpenSSL and libpng libraries have been updated and a default setting affected the way  debuggers authenticate to the guest has been changed to a more secure  value.
As always, comments and questions on security issues in VMware’s  products are very welcome at security@vmware.com.

Just a headsup to the infosecurity community - drop by our booth at this year's RSA Conference, April 7-11, 2008, to learn more about the recent VMsafe technology announcement, the new Update Manager product for online and offline VM patching, or just to talk about VMware's security initiatives in general.  We'll be in booth #339 on the Expo floor.

Steve Herrod, VMware's CTO and VP of R&D, will also be speaking on a panel on future trends in virtualization and security on Friday, April 11, at 10 am (session EXP-402).  Don't miss it!

Recently a researcher published a proof-of-concept called Xensploit which allows an attacker to view or manipulate a VM undergoing live migration (i.e. VMware’s VMotion) from one server to another. This was shown to work with both VMware’s and Xen’s version of live migration. Although impressive, this work by no means represents any new security risk in the datacenter. It should be emphasized this proof-of-concept does NOT “take over the hypervisor” nor present unencrypted traffic as a vulnerability needing patching, as some news reports incorrectly assert. Rather, it a reminder of how an already-compromised network, if left unchecked, could be used to stage additional severe attacks in any environment, virtual or physical.

On an insecure network, man-in-the-middle attacks can target both virtual and physical machines. The techniques published are novel in that they go after the contents of migrating VM memory to target credentials and data, rather than going after similar information flowing across internal network transactions. Putting aside the question of whether it’s even worthwhile to target memory instead of network traffic directly, the sensitivity of VM memory was never the question.

Encryption of all data-in-transit is certainly one well-understood mitigation for man-in-the-middle attacks.  But the fact that plenty of data flows unencrypted within the enterprise – indeed perhaps the majority of data – suggests that there are other adequate mitigations. Unencrypted VMotion traffic is not a flaw, but allowing VMotion to occur on a compromised network can be. So this is a good time to re-emphasize hardening best practices for VMware Infrastructure and what benefit they serve in this scenario.

1. The most important VMotion best practice is to isolate your VMotion activity from all production network traffic. The current design of VMotion assumes that the VMotion network is secure within a data center, certainly within a rack or set of adjacent racks.  In a typical situation, servers in one or more co-located racks would each have one or two network cards dedicated for VMotion; these would be connected to a switch or VLAN that has no other endpoints connected.
   
   Isolating VMotion takes away that most common of staging points for man-in-the-middle: some unpatched box anywhere on the production network that has already been taken over by malware.  Indeed why any non-ESX box, compromised or not, would be on this network at all would be immediately in question. The researcher’s assumption is that long-haul VMotion over wide area networks might become popular in the future. However, most companies today already use encrypted links for inter-datacenter traffic.

2. Tightly restrict access to VI administrative accounts and roles.  With VMotion isolated, a virtual rogue presence is more plausible than a physical one, but even a compromised guest VM does not have a virtual NIC on the VMotion network, only on the production network. Therefore the rogue VM must be configured in VI to have a vNIC on the VMotion network.

3. Don’t enable promiscuous mode on vswitches. Unlike a physical network card, someone who has taken over a guest VM cannot cannot configure a vNIC to be promiscuous. Another VI admin setting, promiscuous mode (off by default) is configured on the virtual switch port separately from a VM.  Also, to manipulate rather than snoop, the proof-of-concept technique requires traffic actually route through the rogue VM, which would not occur naturally on the vswitch.

Warren Wu
Security Product Management

Recently there was an article on “Patch Tuesday for VMware” over at Virtualization.info. It is an interesting article that raised some questions that we thought we might be able to shed some light on. The article was more focused on patching and not security alone, but since patching has now been so closely associated with security, so I'll jump in and provide a response on our security blog.

As the article points out, "patching is a necessary evil" - and that the existence of ESX patches should not come as a shock to anyone. So let’s talk about the sinister plan behind the increase in ESX patches. Fortunately, the answer is in the article itself. Our patches contain a lot of different things, from hardware compatibility updates, feature enhancements, security fixes, etc. Based on customer feedback, we started creating more discrete patches instead of creating specific update releases that included all the changes and fixes in one package. By creating discrete packages for each of these features, customers have more choice on what to deploy quickly and what to deploy later. This fits nicely with the best practices that customers already have in place around patching. Many customers already have a triage process that helps them evaluate what patches need to go in now and which ones can be bundled together for deployment later, like once a quarter. When these patches get released is important, but not as important as when a customer needs to actually deploy them.

We also want customers to view ESX as an appliance - or more accurately, as a product that has appliance-like characteristics. So what makes VMware different from other vendors on the patching issue? The good news is that there are specific features in our products that enable us to help customers do their patching without the pain they might be used to. VMotion (also called Live Migration) is a key feature that customers use to move their existing VM's to other systems, patch ESX, and transition those VM's back to the patched ESX system. Not easy enough? We've now added Update Manager that automates this entire process for the administrator, including downloading the patches and applying them directly to ESX. This feature is not available in any other products in this class, so we we're offering customers something unique and valuable to address this pain.

Speaking of appliances, another thing to consider is that we are now offering ESX in a number of different form-factors, including the brand new ESX Server 3i. 3i will have a significantly different patch characteristics - it does not have a Console OS and has a different patching mechanism than ESX that will be very attractive to customers.

So, in summary – we’re not trying to surprise anyone by issuing patches for ESX, and are doing a lot of work to make sure that customers have the best patching experience when it comes to ESX. Just like any other product, we hope that customers will regularly patch their ESX systems based on the priority and criticality of the patches and find ways to schedule these to fit their business needs. With VMotion and Update Manager, we really do think that customers have a lot of ways of addressing most of the pain associated with patching software products.

Nand Mulchandani

Senior Director, Security Product Management & Marketing

VMware