Home > Blogs > Virtual Reality


Hypervisor Memory Management Done Right

Memory management in VMware vSphere

Sophisticated and effective memory management has always been a key strength of the ESX (and now, ESXi) hypervisor that powers VMware vSphere.  Back in 2001, when ESX first came out, 2GB was a lot of RAM in an x86 server, so it was essential for a hypervisor to economize on memory to deliver the server consolidation benefits customers were so eager to realize.  Back then, the big attraction of server virtualization was running several lightweight utility and test/dev servers in VMs on a single host and memory was usually the resource that limited consolidation ratios.  Today, x86 machines are pushing 1TB of RAM, but because our customers have now virtualized their most memory-hungry production database, messaging and application servers, memory resources are just as critical as ever.

ESX and ESXi have been able to keep up with these huge increases in memory demands because our hypervisor has always employed a multi-layered approach to memory management that delivers the most efficient memory management, best VM densities and lowest costs to our users.  The methods employed are detailed in an excellent VMware technical paper and I’ve summarize them below.

Let users provide critical VMs with guaranteed memory.  The memory “Shares” and “Reservation” settings let you prioritize memory allocated to each VM and reserve enough host RAM for the active working memory of each guest OS.

Efficiently reclaim memory from VMs when needed.  Simply carving off a VM’s full memory allocation from the host’s physical RAM is tremendously wasteful since VMs rarely use all their virtual RAM.  Instead, ESXi borrows or “reclaims” memory from less active VMs when it is needed by other VMs.  We use four techniques for memory reclamation:

  1. Transparent Page Sharing.  Think of it as de-duplication for your memory.  During periods of idle CPU activity, ESXi scans memory pages loaded by each VM to find matching pages that can be shared. The memory savings can be substantial, especially when the same OS or apps are loaded in multiple guests, as is the case with VDI.  Transparent Page Sharing has a negligible effect on performance (sometimes it evens improves guest performance) and users can tune ESXi parameters speed up scanning if desired.  Also, despite claims by our competitors, Transparent Page Sharing will in fact work with large memory pages in guests by breaking those pages into smaller sizes to enable page sharing when the host is under memory pressure.
  2. Guest Ballooning.  This is where ESXi achieves most of its memory reclamation.  When the ESXi hypervisor needs to provide more memory for VMs that are just powering on or getting busy, it asks the guest operating systems in other VMs to provide memory to a balloon process that runs in the guest as part of the VMware Tools.  ESXi can then loan that “ballooned” memory to the busy VMs.  The beauty of ballooning is that it’s the guest OS, not ESXi, that decides which processes or cache pages to swap out to free up memory for the balloon.  The guest, whether it’s Windows or Linux, is in a much better position than the ESXi hypervisor to decide which memory regions it can give up without impacting performance of key processes running in the VM.
  3. Hypervisor Swapping.  Any hypervisor that permits memory oversubscription must have a method to cope with periods of extreme pressure on memory resources.  Ballooning is the preferred way to reclaim memory from guests, but in the time it takes for guests to perform the in-guest swapping involved, other guests short on memory would experience freezes, so ESXi employs hypervisor swapping as a fast-acting method of last resort.  With this technique, ESXi swaps its memory pages containing mapped regions of VM memory to disk to free host memory.  Reaching the point where Hypervisor swapping is necessary will impact performance, but vSphere supports swapping to increasingly common solid state disks, which testing shows can cut the performance impact of swapping by a factor of five.
  4. Memory Compression.  To reduce the impact of hypervisor swapping, vSphere 4.1 introduced memory compression.  The idea is to delay the need to swap hypervisor pages by compressing the memory pages managed by ESXi – if two pages can be compressed to use only one page of physical RAM, that’s one less page that needs to be swapped.  Because the compression/decompression process is so much faster than disk access, performance is preserved.

What about “Dynamic Memory”?

The days of ESX being the only hypervisor that could support memory oversubscription are over now that Citrix and Microsoft have added a feature they both call “Dynamic Memory” to their products. But how effective is Dynamic Memory compared to the multi-layered memory management technologies in vSphere?  In both XenServer and Hyper-V, Dynamic Memory relies on just a single method – guest ballooning – to reclaim memory from guests when the hypervisor needs to find more memory for other VMs once physical RAM is fully allocated.  As we found when engineering ESX, ballooning is important, but it isn’t fast enough to cope with rapid spikes in memory needs or times when all your VMs are using their full memory allotment.  That’s why we designed ESX and ESXi with all four levels of memory reclamation technology, including the last resort safety net of hypervisor swapping, which is essential to avoid guest freezes when guests are all working hard and memory is under pressure.

Ballooning alone isn’t sufficient.  The Taneja Group took a look at XenServer 5.6 and its performance under conditions of memory oversubscription in their “Hypervisor Shootout” study of VM density. The results were striking.  XenServer 5.6 with Dynamic Memory was able to keep up with vSphere 4.1 VM density, but only when the guests were running a light load.  When the workload in the guests was increased, XenServer suffered, “a performance penalty that was surprisingly high and unfortunately consistent.”

Taneja_fig_10

Will the new “Dynamic Memory” feature Microsoft has added to Hyper-V R2 SP1 fare any better when memory is overcommitted and VMs are busy?  According to Microsoft, Dynamic Memory improves their VM density by 40% over the previous version that maxed out before memory could be overcommitted.  It’s an improvement, but it won’t be enough to catch up to vSphere 4.1, which showed in Taneja Group’s testing, “an overall VM density advantage of two-to-one.”  As seen in XenServer 5.6, reliance only on ballooning hits a wall when your VMs actually need all the virtual RAM you’ve allocated to them.  You can see signs of that effect in the painfully slow 2 second max response times in the VDI test Microsoft ran to justify their 40% improvement claim.  Apparently, entire Hyper-V R2
SP1 hosts can also find themselves frozen by memory shortages when Dynamic Memory is enabled.  Just as Microsoft has reluctantly come around to adopt the VMware way of doing things such as live migration and ballooning, perhaps we’ll see them grudgingly admit that hypervisor swapping and memory compression are essential to handle those times when memory pressure hits a peak.

Other Dynamic Memory Curiosities.  To permit many VMs to be packed onto a Hyper-V host, Microsoft has come up with a scheme where VMs are powered on with a low “Startup RAM” setting (512MB is recommended) and memory is then hot added as the guest needs it.  You can probably think of lots of reasons why this is a bad idea.  How about:

  • Hot-adding RAM to VMs will play havoc with system monitoring tools that will now see the memory allocations of VMs changing randomly – administrators can expect a barrage of “Low Memory” alarms.
  • Hyper-V Dynamic Memory will now only work with the latest versions of Windows that support hot add RAM.  Running older Windows 2003 service packs or Windows XP?  You’re out of luck.  Linux support isn’t even mentioned.
  • Since Microsoft hasn’t yet invented “hot remove” of RAM, Dynamic Memory can only ratchet up the RAM allocation of a VM.  Getting back to the “Startup RAM” setting will require a reboot of the VM.
  • Applications requiring more than the “Startup RAM” amount that perform a pre-check will fail immediately.  Microsoft’s own software installers run into this problem.  Microsoft’s workaround is to trick Hyper-V into hot adding memory to the VM by running MS Paint before starting the installation – I’m not kidding.
  • Microsoft has now added a whole new range of complications for software vendors to test and certify now that their apps must cope with suddenly changing amounts of memory.  Don’t expect your software vendor to support their product in a Hyper-V VM with Dynamic Memory enabled until they’ve done lots of recoding and testing.

With VMware vSphere, VMs look and act just like physical machines .  Any guest OSs and any apps or monitoring tools in the VMs see a consistent, fixed amount of installed RAM.  We thought it would be easier for everyone if we put the memory management intelligence into our hypervisor rather than relying on the guest OS to perform some unnatural acts to enable good VM density.

There’s no need to disable Windows security to get great VM density with vSphere

Lastly, I’d like to clear up some confusion on Microsoft’s part.  They’re claiming that VMware is telling customers to disable a security feature in Windows guests called Address Space Layout Randomization (ASLR) to improve VM density.  ASLR is intended to make things harder on malware writers by scrambling the locations of Windows processes in memory.  [Whether it’s meeting that goal is up for debate as Microsoft’s own security team has acknowledged that the bad guys have figured out how to bypass ASLR.]  ASLR makes it somewhat more difficult for the ESXi Transparent Page Sharing feature to de-duplicate memory, so you’d expect that turning it off would boost VM density in vSphere.  The Project Virtual Reality Check team confirmed in their September 2010 report that disabling ASLR produced a modest 16% improvement, but the authors clearly warned, “that Project VRC does not blindly recommend disabling ASLR. This is an important security feature.”  I thought the Project VRC findings were interesting and I mentioned them in a session I gave at VMworld 2010 Europe called, “Getting the best VM density from your virtualization platform.”  I also cautioned that disabling ASRL would reduce security and was best reserved for isolated testbeds where absolute maximum VM density was desired.  Here’s exactly how my slide put it:

ASLR

Note the “lessens security” warning.  Somehow the folks at Microsoft read that as a recommendation by VMware that our customers put themselves at risk to use our product.  Maybe they’re also the sort of people that see hot coffee as an unacceptable danger to society.  However, I think most would agree that the indignant posturing is just a way to distract users from the less than dynamic qualities of “Dynamic Memory.”

7 thoughts on “Hypervisor Memory Management Done Right

  1. J. Perkins

    Great article and information is absolutely on point. I can’t see how “R2 Sp1″ Hyper-V dynamic memory would be acceptable in a large-scale production environments. Monitoring tools and applications will not understand how to deal with the dynamic memory allocations.

  2. Sean Clark

    Well played, Eric. Well played.
    The “restart to reclaim” memory trick that Hyper-V would leverage is actually a good idea that has some worth inside VMware environments as well, and therefore allow the best of both worlds. Customers could leverage all of VMware’s memory “Kung Fu” but still leverage automated right-sizing processes to reduce the reliance on VMware memory fu. That is, reduce the need to balloon, swap or compress memory. It’s analogous to reducing the spending limits on your credit card before heading to Vegas.
    This is nothing you probably don’t know, but felt it might be good for readers to think about. Especially since Hyper-V zealots will be quick to point out oversized VMs as a “fault” of VMware. When in reality, you can right-size VMware VMs just the same as Hyper-V VMs can be.
    As Mr. Miyagi says, “Best defense – not be there.”
    Cheers!
    Sean Clark – @vSeanClark

  3. Allan Robertson

    Nice post – always liked TPS.
    It is worth noting that with newer CPU, like Nehalem, HW assisted memory management means TPS will likely not be used on these systems.
    However what you lose in higher memory usage, you can get back in performance with fewer page table entries thanks to the use of large pages (2M vs the 4k pages scanned by TPS)

  4. Alexander Thoma

    “It is worth noting that with newer CPU, like Nehalem, HW assisted memory management means TPS will likely not be used on these systems.”
    This is not correct! Even on hw mmu systems, ESX(i) will prehash all 4KB pages contained in the 2 MB pages; breaking them apart and do TPS across them when going into memory pressure status.
    Just as a head up :-)

  5. Adam Knowles (@Pharkie)

    Having read much tonight about Oracle VirtualBox vs VMWare ESXi vs WinServer2008r2 Hyper-V: this is the post that has points and evidence substantial enough to convince me that VMWare is the first I should try.
    Thanks for the article!

  6. Leon

    Alex,
    Oracle Virtual is different from the other products you mentioned. Virtualbox is a desktop type 2 hypervisor – not really an enterprise solution for hosting. For server virtualization, you may need to consider other Oracle products.
    Also, look at the Xen Hypervisor.

Comments are closed.