Home > Blogs > Virtual Reality > Monthly Archives: February 2011

Monthly Archives: February 2011

Hypervisor Memory Management Done Right

Memory management in VMware vSphere

Sophisticated and effective memory management has always been a key strength of the ESX (and now, ESXi) hypervisor that powers VMware vSphere.  Back in 2001, when ESX first came out, 2GB was a lot of RAM in an x86 server, so it was essential for a hypervisor to economize on memory to deliver the server consolidation benefits customers were so eager to realize.  Back then, the big attraction of server virtualization was running several lightweight utility and test/dev servers in VMs on a single host and memory was usually the resource that limited consolidation ratios.  Today, x86 machines are pushing 1TB of RAM, but because our customers have now virtualized their most memory-hungry production database, messaging and application servers, memory resources are just as critical as ever.

ESX and ESXi have been able to keep up with these huge increases in memory demands because our hypervisor has always employed a multi-layered approach to memory management that delivers the most efficient memory management, best VM densities and lowest costs to our users.  The methods employed are detailed in an excellent VMware technical paper and I’ve summarize them below.

Let users provide critical VMs with guaranteed memory.  The memory “Shares” and “Reservation” settings let you prioritize memory allocated to each VM and reserve enough host RAM for the active working memory of each guest OS.

Efficiently reclaim memory from VMs when needed.  Simply carving off a VM’s full memory allocation from the host’s physical RAM is tremendously wasteful since VMs rarely use all their virtual RAM.  Instead, ESXi borrows or “reclaims” memory from less active VMs when it is needed by other VMs.  We use four techniques for memory reclamation:

  1. Transparent Page Sharing.  Think of it as de-duplication for your memory.  During periods of idle CPU activity, ESXi scans memory pages loaded by each VM to find matching pages that can be shared. The memory savings can be substantial, especially when the same OS or apps are loaded in multiple guests, as is the case with VDI.  Transparent Page Sharing has a negligible effect on performance (sometimes it evens improves guest performance) and users can tune ESXi parameters speed up scanning if desired.  Also, despite claims by our competitors, Transparent Page Sharing will in fact work with large memory pages in guests by breaking those pages into smaller sizes to enable page sharing when the host is under memory pressure.
  2. Guest Ballooning.  This is where ESXi achieves most of its memory reclamation.  When the ESXi hypervisor needs to provide more memory for VMs that are just powering on or getting busy, it asks the guest operating systems in other VMs to provide memory to a balloon process that runs in the guest as part of the VMware Tools.  ESXi can then loan that “ballooned” memory to the busy VMs.  The beauty of ballooning is that it’s the guest OS, not ESXi, that decides which processes or cache pages to swap out to free up memory for the balloon.  The guest, whether it’s Windows or Linux, is in a much better position than the ESXi hypervisor to decide which memory regions it can give up without impacting performance of key processes running in the VM.
  3. Hypervisor Swapping.  Any hypervisor that permits memory oversubscription must have a method to cope with periods of extreme pressure on memory resources.  Ballooning is the preferred way to reclaim memory from guests, but in the time it takes for guests to perform the in-guest swapping involved, other guests short on memory would experience freezes, so ESXi employs hypervisor swapping as a fast-acting method of last resort.  With this technique, ESXi swaps its memory pages containing mapped regions of VM memory to disk to free host memory.  Reaching the point where Hypervisor swapping is necessary will impact performance, but vSphere supports swapping to increasingly common solid state disks, which testing shows can cut the performance impact of swapping by a factor of five.
  4. Memory Compression.  To reduce the impact of hypervisor swapping, vSphere 4.1 introduced memory compression.  The idea is to delay the need to swap hypervisor pages by compressing the memory pages managed by ESXi – if two pages can be compressed to use only one page of physical RAM, that’s one less page that needs to be swapped.  Because the compression/decompression process is so much faster than disk access, performance is preserved.

What about “Dynamic Memory”?

The days of ESX being the only hypervisor that could support memory oversubscription are over now that Citrix and Microsoft have added a feature they both call “Dynamic Memory” to their products. But how effective is Dynamic Memory compared to the multi-layered memory management technologies in vSphere?  In both XenServer and Hyper-V, Dynamic Memory relies on just a single method – guest ballooning – to reclaim memory from guests when the hypervisor needs to find more memory for other VMs once physical RAM is fully allocated.  As we found when engineering ESX, ballooning is important, but it isn’t fast enough to cope with rapid spikes in memory needs or times when all your VMs are using their full memory allotment.  That’s why we designed ESX and ESXi with all four levels of memory reclamation technology, including the last resort safety net of hypervisor swapping, which is essential to avoid guest freezes when guests are all working hard and memory is under pressure.

Ballooning alone isn’t sufficient.  The Taneja Group took a look at XenServer 5.6 and its performance under conditions of memory oversubscription in their “Hypervisor Shootout” study of VM density. The results were striking.  XenServer 5.6 with Dynamic Memory was able to keep up with vSphere 4.1 VM density, but only when the guests were running a light load.  When the workload in the guests was increased, XenServer suffered, “a performance penalty that was surprisingly high and unfortunately consistent.”


Will the new “Dynamic Memory” feature Microsoft has added to Hyper-V R2 SP1 fare any better when memory is overcommitted and VMs are busy?  According to Microsoft, Dynamic Memory improves their VM density by 40% over the previous version that maxed out before memory could be overcommitted.  It’s an improvement, but it won’t be enough to catch up to vSphere 4.1, which showed in Taneja Group’s testing, “an overall VM density advantage of two-to-one.”  As seen in XenServer 5.6, reliance only on ballooning hits a wall when your VMs actually need all the virtual RAM you’ve allocated to them.  You can see signs of that effect in the painfully slow 2 second max response times in the VDI test Microsoft ran to justify their 40% improvement claim.  Apparently, entire Hyper-V R2
SP1 hosts can also find themselves frozen by memory shortages when Dynamic Memory is enabled.  Just as Microsoft has reluctantly come around to adopt the VMware way of doing things such as live migration and ballooning, perhaps we’ll see them grudgingly admit that hypervisor swapping and memory compression are essential to handle those times when memory pressure hits a peak.

Other Dynamic Memory Curiosities.  To permit many VMs to be packed onto a Hyper-V host, Microsoft has come up with a scheme where VMs are powered on with a low “Startup RAM” setting (512MB is recommended) and memory is then hot added as the guest needs it.  You can probably think of lots of reasons why this is a bad idea.  How about:

  • Hot-adding RAM to VMs will play havoc with system monitoring tools that will now see the memory allocations of VMs changing randomly – administrators can expect a barrage of “Low Memory” alarms.
  • Hyper-V Dynamic Memory will now only work with the latest versions of Windows that support hot add RAM.  Running older Windows 2003 service packs or Windows XP?  You’re out of luck.  Linux support isn’t even mentioned.
  • Since Microsoft hasn’t yet invented “hot remove” of RAM, Dynamic Memory can only ratchet up the RAM allocation of a VM.  Getting back to the “Startup RAM” setting will require a reboot of the VM.
  • Applications requiring more than the “Startup RAM” amount that perform a pre-check will fail immediately.  Microsoft’s own software installers run into this problem.  Microsoft’s workaround is to trick Hyper-V into hot adding memory to the VM by running MS Paint before starting the installation – I’m not kidding.
  • Microsoft has now added a whole new range of complications for software vendors to test and certify now that their apps must cope with suddenly changing amounts of memory.  Don’t expect your software vendor to support their product in a Hyper-V VM with Dynamic Memory enabled until they’ve done lots of recoding and testing.

With VMware vSphere, VMs look and act just like physical machines .  Any guest OSs and any apps or monitoring tools in the VMs see a consistent, fixed amount of installed RAM.  We thought it would be easier for everyone if we put the memory management intelligence into our hypervisor rather than relying on the guest OS to perform some unnatural acts to enable good VM density.

There’s no need to disable Windows security to get great VM density with vSphere

Lastly, I’d like to clear up some confusion on Microsoft’s part.  They’re claiming that VMware is telling customers to disable a security feature in Windows guests called Address Space Layout Randomization (ASLR) to improve VM density.  ASLR is intended to make things harder on malware writers by scrambling the locations of Windows processes in memory.  [Whether it’s meeting that goal is up for debate as Microsoft’s own security team has acknowledged that the bad guys have figured out how to bypass ASLR.]  ASLR makes it somewhat more difficult for the ESXi Transparent Page Sharing feature to de-duplicate memory, so you’d expect that turning it off would boost VM density in vSphere.  The Project Virtual Reality Check team confirmed in their September 2010 report that disabling ASLR produced a modest 16% improvement, but the authors clearly warned, “that Project VRC does not blindly recommend disabling ASLR. This is an important security feature.”  I thought the Project VRC findings were interesting and I mentioned them in a session I gave at VMworld 2010 Europe called, “Getting the best VM density from your virtualization platform.”  I also cautioned that disabling ASRL would reduce security and was best reserved for isolated testbeds where absolute maximum VM density was desired.  Here’s exactly how my slide put it:


Note the “lessens security” warning.  Somehow the folks at Microsoft read that as a recommendation by VMware that our customers put themselves at risk to use our product.  Maybe they’re also the sort of people that see hot coffee as an unacceptable danger to society.  However, I think most would agree that the indignant posturing is just a way to distract users from the less than dynamic qualities of “Dynamic Memory.”

2010 – Another Great Year for VMware and Our Customers

The film industry awards season is just reaching its peak, but the software industry’s own Oscars for 2010 have been handed out and VMware finished up with another great year.  Here’s a sampling of the recognition VMware received from the IT trade press in 2010:

Those same members of the technical press had some very nice things to say about VMware and our products in 2010:

  • “Microsoft won’t come anywhere close to leapfrogging VMware in 2010” – Redmondmag.com
  • “VMware is in fact leaps and bounds ahead of competitors” – InfoWorld
  • “…the ESX feature set is substantially more mature…VMware remains the champ.” – InformationWeek

and my favorite…

  • “VMware is the king of virtualization. There is none higher.” – TechTarget

The analysts also directed much praise our way in 2010.  Gartner came out with their eagerly anticipated Magic Quadrant for x86 server virtualization and one look at the upper right corner of the chart would warm the heart of any IT manager who selected vSphere to be the foundation of his company’s virtualization and cloud efforts.  In December, I was in attendance at Gartner’s annual Data Center Conference where audience polls showed 87% of attendees were using VMware as their primary x86 virtual machine solution in 2010 – only 2% were using the nearest competitor.  Even better, when asked which virtualization provider they would be using in 2015, the Gartner attendees picked VMware by a 5:1 margin over the nearest alternative.

One thing our customers can find especially satisfying from this recognition is it points out how they made the right choice with VMware.  Whenever a customer invests in enterprise software, they want assurance that they will see steady improvements – big and small – in their chosen product.  In the case of VMware vSphere, our users have really won the lottery.  For almost ten years, each update and major release of our vSphere product family has rewarded our customers with improvements in performance and scalability as well as a steady stream of new capabilities – things like Storage vMotion, Fault Tolerance and Distributed Switches.  I imagine our early adopters who selected ESX or VMware Infrastructure are feeling pretty good about how their decision was rewarded.

But 2010 is behind us and we’re now focused on investments in upcoming waves of VMware products.  Our current customers and customers-to-be can feel assured that we’re working to keep up our pace of innovation (and recognition!) on into 2011 and beyond.