VMTN Blog

ESXi is not your father's ESX. There is no Service Console, so trying to fit it into the exact same processes that you're used to with ESX isn't recommended. I know, I know, you have all those scripts you're used to running in the console. VMware is building tools to manage and administer your ESXi from outside the box, and while they're not quite feature complete, they're well on their way. So don't pop the hood; it's welded shut for a reason.

Link: Keep your VMware ESXi warranty: Don't break the security shell.

Working with VMware ESXi can be frustrating; you're not supposed to enable the Dropbear SSH client or use its technical support mode without the assistance of a VMware support representative. System administrators, however, may be tempted to use tech support mode (or enable Dropbear) to fix problems or manage connections on the fly. Cracking this security shell, however, can void the VMware ESXi warranty and break support contracts. In this tip, I'll explain alternatives that allow you to manage your ESXi virtual machines without compromising its security -- and possibly breaking a support contract.

Rob Randell, one of our security specialists here at VMware, is guest-posting over at Mike D's blog. (Guys, you're welcome over here as well.)

Link: Mike D's Virtualization Blog: Virtualization Team vs. Security Team: It is important to remove the “vs.”!.

Unfortunately, very often this situation is the exception and not the rule. Many of the customers that I talk to are only talking to me because they have started a widescale deployment of VMware VI and the security team gets wind of it once it is well underway or worse some sort of audit is initiated (PCI, Sarbox, HIPAA, etc…). At this point the entire architecture needs to be reviewed and very often rearchitected to meet the necessary security and audit requirements. See the following article for a great example of this.

(Emphasis mine.) Sounds like a nightmare, so my guess is that you don't want that to happen to you. Always consult your friendly neighborhood security team first.

From the VMware Security Blog, which should be on your short list. (Note that the blog is more for news and updates, but you can get security notifications emailed to you -- check the right sidebar of the blog or the Security Center. Note also that this page is separate from the Security Technology page Charu mentions below.)

Link: VMware: VMware Security Blog: What's New in Security at VMware.com.

• The new VMware Compliance Center includes an overview of the issues involved with virtualization and compliance, a comprehensive listing of partner virtualization compliance solutions, and references such as white papers and recorded webcasts.
• There is a new listing of Free Security and Compliance Utilities.
  These tools are provided by VMware partners, and can be downloaded and
  used right away to help assess and monitor your VI deployment
• The Overview section of the Security Technology site has been updated to present the core issues of virtualization and security in a more streamlined way.  The Resources listing has also been enhanced to include more external resources.
• Although not new, the VMsafe section had received some updates over the summer which you might not have seen.
• Finally, something else that's not new but worth pointing out is the Security Certifications
  page.  We will be listing all security-related certifications that
  VMware products receive, so you can check here to see ones we have
  received.

Link: VMware: VMware Security Blog: Update to VI3 Security Hardening Guide.

Update to VI3 Security Hardening Guide

We have recently released an update to the VI3 Security Hardening guide.  The main changes are:

• new content for ESX 3.5 and VirtualCenter 2.5, including VirtualCenter plug-ins
• a section specific to hardening for ESXi.
• new sections for VM configuration as well as client software
• a greater level of depth for the existing recommendations

And if you missed it, see also: DMZ Virtualization with VMware Infrastructure.

From Eric Betts at the VMware Security Blog -- something we're very proud of.

Link: VMware: VMware Security Blog: VMware Infrastructure Earns Common Criteria EAL4+ Certification.

On May 20, 2008, VMware VI3 (ESX Server 3.0.2 & VirtualCenter 2.0.2) achieved Common Criteria certification at EAL4+ under the Canadian Common Criteria Evaluation and Certification Scheme (CCS).  EAL4+ is the highest assurance level that is recognized globally by all signatories under the Common Criteria Recognition Agreement (CCRA). ...

VMware is the first and only virtualization vendor for industry standard x86 hardware to successfully complete the rigorous Common Criteria certification process. Although several operating system vendors bundle virtualization technologies as part of their products, to date, none have included virtualization technology as part of their Common Criteria security certifications.

From the TripWire ConfigCheck site:

Tripwire® ConfigCheck^TM is a free utility that rapidly assesses the security of VMware ESX 3.5 hypervisor configurations compared to the VMware Infrastructure 3 Security Hardening guidelines. Developed by Tripwire in cooperation with VMware, Tripwire ConfigCheck ensures ESX environments are properly configured—offering immediate insight into unintentional vulnerabilities in virtual environments—and provides the necessary steps towards full remediation when they are not.

internetnews.com - TripWire Cures Virtual Misconfiguration:

"There haven't been any attacks against the hypervisor that could be demonstrated to break through, but misconfiguration could put you in a situation where you can get attacked even if you have no vulnerabilities or are fully patched," [VMware's Nand Mulchandani] added.

There are about 100 configuration settings in VMware that need to be set to ensure the most hardened environment possible, and these have, up to now, had to be manually checked.

NetworkWorld - Did you say: FREE, SECURITY and VIRTUAL SERVERS?

The ConfigCheck tool is based on VMware's own security hardening guidelines for ESX Server and future releases will also support VMware's Infrastructure 3 products. The free tool notifies IT managers of potential conflicts in configurations and also offers fixes to the incompatibilities between actual and desired configurations. The tool links back to the vendors' virtual security resource center

SearchSecurity.com - Virtualization tool assesses VMware security configurations

"It will be eye-opening when they run ConfigCheck against their systems and gauge that relative to best practices," said Mulchandani. "It will get them thinking about configuration and patching in key areas for security."

Hezi Moore, co-founder and CTO of Reflex Security, has a nice 3-part primer on how to start thinking about your virtual networks as a guest post on VMblog. While Hezi does mention virtual appliances, he avoids turning this into an ad for Reflex.

Best Practices for Securing Virtual Networks - Part One of Three 

However, virtualized environments face unique network security challenges that can affect the entire organization. Adding security to your virtual network, such as a virtual security appliance, can protect critical resources from intrusion, theft, service denial, regulatory compliance conflicts or other consequences. 

Fortunately, by combining prudent security measures with advancing virtualization technologies, organizations can adopt and deploy “defense in depth” best practices without the traditional high costs and complexities associated with physical infrastructure and enjoy the benefits of a virtualized architecture while avoiding excessive risks. ...

Virtualized environments are difficult to visually inspect and due to virtual server mobility and related issues, they often have dynamic configurations and server populations. In this context, threats can easily spread, devices can be overlooked, and inappropriate activity can be concealed. To prevent configuration oversights, rogue devices, auditing omissions and other issues, the security system should maintain persistent awareness of all virtualized devices, services and communications. 

Best Practices for Securing Virtual Networks - Part Two of Three

Primarily, organizations have four alternative or complementary approaches to secure virtualized environments: physical network security devices, physical device / VLAN configurations, host intrusion prevention systems and virtualized network security systems. 

Best Practices for Securing Virtual Networks - Part Three of Three

Leverage virtualization platform to enable security

Though virtualization can present new security challenges, it is a powerful technology that can have a significant impact on an organization’s ability to become more efficient, effective and productive. Organizations should determine not only what business applications can benefit from virtualization but also what IT applications can benefit from virtualization and use this trusted platform as an enabler. Determine which physical devices make most sense to deploy in virtualization and utilize complementary software like virtual security appliances to provide the following capabilities in the virtual environment:

• Security
• Visibility
• Control
• Manageability
• Policy enforcement
• Deployment

(And thanks, Dave, for getting this kind of original article out alongside the comprehensive industry and blog news you can find at VMblog.com)

Virtualization is mind-blowing stuff, but I have never seen the metaphors get so intricate or the prose get so purple as the blog posts on VMsafe over the past week. Either VMsafe (see our previous post) has touched a nerve, or rhetoric in the security industry  is even more heated than the virtualization industry. I suspect both.

Link: Chris Wolf: VMsafe is cool because … — Server Virtualization Blog.

“VMsafe is a very important technology in my opinion, as it changes how virtual environments are secured. Today, security appliance virtual machines (VMs) typically monitor other VMs by connecting to them over a virtual switch. The result is virtual network monitoring that resembles physical network monitoring,” Wolf said. “The current model is fine until VMs begin to dynamically move across a virtual infrastructure.  ...

Wolf continued, “VMsafe also provides the framework for offloading many security activities to special-purpose security VMs, including roles such as antivirus monitoring. As we move to an automated or dynamic data center, having special-purpose security appliances that are capable of enforcing security policies at the hypervisor level can ease security management in an environment that will be constantly changing.

Link to another coffee spit-take rant at: Rational Survivability: VMWare's VMSafe: Security Industry Defibrilator....Making Dying Muscle Twitch Again.

As I mentioned in a prior posting, VMware's VMsafe has the potential to inject life back into the atrophied and withering heart muslce of the security industry and raise the prognosis from DOA to the potential for a vital economic revenue stream once more. ... For the purpose of this post, I'm going to focus on the security implications of virtualization and simply summarize by suggesting that virtualization up until now has quietly marked a tipping point where we see the disruption stretch security architectures and technologies to their breaking point and in many cases make much of our invested security portfolio redundant and irrelevant. ...

So, we've got this fantastic technological, economic, and cultural transformation occurring over the last FIVE YEARS (at least,) and the best we've seen as a response from most traditional security vendors is that they have simply marketed their solutions slimly as "virtualization ready" or "virtualization aware" when in fact, these are simply hollow words for how to make their existing "square" products fit into the "round" holes of a problem space that virtualization exposes and creates. ...

VMSafe represents a huge opportunity for these vendors to claw their way back to life, making their solutions relevant once more, and perhaps even more so.

And then in the comments to Hoff's post, Greg Ness (VP Marketing for Blue Lane) says about the VMsafe introduction that "It felt like the IT industry's equivalent of the Beatles first performance on Ed Sullivan." and then posts at his own blog. Link: Dispelling Virtsec Myths « ARCHIMEDIUS.

The hardware infrastructure that emerged with the rise of desktop computing and the internet is about to collapse back into the server. That model is infinitely more scalable, more dynamic and more flexible than the world of pipes, racks and screwdrivers. That is why virtualization will win out over daisy chains of specialized hardware. ...

Some deep security experts suggest that there are new hypervisor-specific attacks that pose real, catastrophic threats. As I commented while on an InformationWeek panel last month, the hypervisor is modern code with a very lean attack surface. Compare that lean hypervisor code to the layers of code and sizable population of known vulnerabilities in any leading operating system or application/database. Then look at the rate of change now possible in a virtual infrastructure. ...

Let’s use the hypervisor layer to deliver improved security. After all, it is a standardized inflection point that can scale with the servers and the traffic ...

From the VMware Security Blog: Keeping Your VMotion Traffic Secure.

Recently a researcher published a proof-of-concept called Xensploit which allows an attacker to view or manipulate a VM undergoing live migration (i.e. VMware’s VMotion) from one server to another. This was shown to work with both VMware’s and Xen’s version of live migration. Although impressive, this work by no means represents any new security risk in the datacenter. It should be emphasized this proof-of-concept does NOT “take over the hypervisor” nor present unencrypted traffic as a vulnerability needing patching, as some news reports incorrectly assert. Rather, it a reminder of how an already-compromised network, if left unchecked, could be used to stage additional severe attacks in any environment, virtual or physical. ...

Encryption of all data-in-transit is certainly one well-understood mitigation for man-in-the-middle attacks.  But the fact that plenty of data flows unencrypted within the enterprise – indeed perhaps the majority of data – suggests that there are other adequate mitigations. Unencrypted VMotion traffic is not a flaw, but allowing VMotion to occur on a compromised network can be. So this is a good time to re-emphasize hardening best practices for VMware Infrastructure and what benefit they serve in this scenario.

Christofer Hoff talks about how NAC (Network Access Control) appliance vendors are coping in a world where all compute nodes are virtualized, all nodes are flying around with VMotion, all traffic is going through virtual switches, and you're trying to protect access to the cloud -- is that like nailing Jello to the wall?

Link: Rational Survivability: UPDATED: How the Hypervisor is Death By a Thousand Cuts to the Network IPS/NAC Appliance Vendors.

Virtualization is causing IPS and NAC appliance vendors some real pain in the strategic planning department.  I've spoken to several product managers of IPS and NAC companies that are having to make some really tough bets regarding just what to do about the impact virtualization is having on their business. ...

It's especially hard for vendors whose IPS/NAC software is tied to specialty hardware, unless of course all you care about is enforcing at the "edge" -- wherever that is, and that's the point.  The demarcation of those security domain diameters has now shrunk.  Significantly, and not just for servers, either.  With the resurgence of thin clients and new VDI initiatives, where exactly is the client/server boundary? ...

...and it's going to get even more hairy as the battle for the architecture of the DatacenterOS also rages.  The uptake of 10Gb/s Ethernet is also contributing to the mix as we see customers:

• Upgrading from servers to blades
• Moving from hosts and switches to clusters and fabrics
• Evolving from hardware/software affinity to gird/utility computing
• Transitioning from infrastructure to service layers in “the cloud”

He also points to Chris Silva @ Forrester with much the same concerns:

Server virtualization blurs segmentation models. ... Client virtualization proliferates MAC addresses and blurs endpoints.  ... Application virtualization hides setting and blurs endpoint status.