Home > Blogs > Virtual Reality > Monthly Archives: March 2008

Monthly Archives: March 2008

Memory Overcommitment in the Real World

There has been a lot of talk on this blog and others about memory overcommit. Several of the blogs have tried to discredit the power of memory overcommit by saying it’s never used by customers in the real world and can simply be overcome by adding more memory to the server (see the long discussions in previous posts on this blog). One VMware customer has already commented about how he architects his environment with memory overcommit in mind. I received a private response from another VMware customer who is getting ready to implement a very large VDI environment with some interesting numbers that I’ll walk through below. The customer is a large US bank and to protect the security of their environment I did black out some of the names in the screenshots.

About the Environment

This customer configured their standard Windows XP environment for their call centers to run in a virtual machine. Each virtual machine is granted 512 MB of memory and 1 virtual CPU. Each VM runs a series of applications including Marimba, Microsoft Office, a call recording application, a customer database application, and a BPO (business process off-shoring) application.

The host used is an IBM x3850 M2 quad socket, quad core system (total of 16 cores). The server is configured with 64 GB of physical RAM. There are 32 memory sockets in the server.

At the time the system was actively being used by call center employees reflected by this CPU utilization graph.

cpu chart.jpg

Running the Numbers

Below is a screenshot of their environment showing a total of 178 VMs running on the system. You can also see in the screenshot that less than 20 GB of RAM out of the total 64 GB of RAM is being used on the system. With a total of 178 VMs configured for 512 MB of RAM each they are currently allocating 89 GB of memory to running VMs which means they are oversubscribed on the host.


Below are screenshots of the price for an identically configured host coming from IBM’s website on March 18, 2008.

VDI Configuration Summary.png

VDI Configuration Memory.png

Total cost for the server is $24,623.00. Remember that this box is overcommitted on memory. In order to run the same setup with a competitive solution we would need to have a server configured with at least 89 GB of RAM – the total allocated to all of the running virtual machines. Going back to IBM’s website and reconfiguring the server with more memory gives us the following screenshots. The closest configuration that supported at least 89 GB of RAM was 92 GB of RAM because of the memory configurations allowed in the server.

VDI Configuration - Competitor.png

VDI Confiuration Memory - Competitor.png

The new cost of the server is $36,423.00. Compared to the original configuration this is a difference of $11,800.00 just to add more memory to the server to support a solution that does not have memory overcommit. The cost of a 4 socket VMware VI3 Enterprise license is $11,500.00 list price. As you can see the cost of a VMware license is actually $300 less than the cost of adding more memory. Not much of an advantage on the cost side but it still drives home the point that the VMware solution is not more than the competitive "free" solution. What’s more is now you get all of the enhanced functionality of the VMware solution that the competitive solutions are lacking. But enough of the marketing pitch, let’s go back to the numbers and this customer case study.

We’ve shown you a real world scenario from a real customer where adding more memory to the server is still more expensive than the VMware license cost. I want to take this one step further and show what designing the architecture to more closely match the true working set size of memory could do. Remember that’s we’re using less than 20 GB of RAM to run this environment of 178 VMs. We went back to the IBM website and configured the starting point for a VMware solution to have at least 20 GB of RAM. The closest we could get is a host with 24 GB of RAM as seen below.

VDI Cheaper Starting Configuration.png

VDI Cheaper Starting Memory.png

As you can see, by truly engineering the solution to what you actually need we’ve reduced the hardware cost to $20,023.00. If we compare that to the competitive solution’s hardware cost we get a difference of $16,400.00 for the extra memory. Subtracting out the cost of the VMware license the VMware solution comes in at $4,900.00 cheaper than the competitive solution.

It’s All for Charity

I would love for us to give all of that $4,900.00 to charity but the customer wouldn’t go for that. Instead they’re using it to pay for some of the thin clients they’ll use in this solution. After all the $4,900.00 they saved by going with a VMware solution over a competitive "free" solution will buy them 33 thin clients at their current price.

The good news from all of this is James O’Neill from Microsoft was kind enough to volunteer $270 of his own money if we could show a real customer running with memory overcommit where adding memory to the server wouldn’t make the Microsoft solution still cheaper than VMware. I think we’ve adequately met that goal here. So, James, the charity of choice is One Laptop Per Child. And just in case you believe that we’ve cherry picked a use case we’ll be more than happy to connect you directly via phone to any one of the numerous customers we have leveraging memory overcommitment in their environment today.

More on VMware Memory Overcommit, for Those Who Don’t Trust the Numbers

James O’Neill of Microsoft had a pretty sharp reaction to my last post. He accused me of cooking the numbers to exaggerate the cost saving benefits of VMware’s memory overcommit feature. To help James understand just how memory overcommit works, I’ve taken the numbers out of the argument and have used simple algebra to compare the TCO per VM for a host with either VI3 or Microsoft hypervisors. Let’s do the math…


CH Cost of server hardware
CM Cost of memory per GB
CVMW Cost of VMware virtualization software
CMS Cost of Microsoft virtualization software
COS Cost of operating system software
MH Physical server memory, GB
MV Memory per VM, GB
r Memory overcommit ratio



Total system cost

Total system cost



Number of VMs with overcommit

Number of VMs without overcommit



VMware Cost per VM

Microsoft Cost per VM



Let’s simplify a little…



Now let’s make the assumption that the Microsoft virtualization software is free (CMS=0), and let’s go to extremes and tilt the numbers in Microsoft’s favor by assuming that the hardware and memory is also free (CH=0, CM=0).



Now let’s plug in some real numbers for a 2-socket server. We’ll use a conservative VMware memory overcommit ratio of 2, the list price of VMware Infrastructure Enterprise ($5750) and the list price for Windows Server Data Center Edition ($5998).



Finally, we get total cost per VM for both products



So, you can see in this extreme case where hardware is free, the VMware Infrastructure system still beats Microsoft Hyper-V, or any other hypervisor, in total cost per VM in a Windows environment. Plugging in any realistic costs for hardware and memory just tilts the balance further in VMware’s favor. I’m also using our most feature-rich VI3 Enterprise product in the example – had I used one of our products like ESX Server 3i or VI3 Foundation that is a little closer to the barebones capabilities of Hyper-V, the numbers would look even worse for Microsoft.

James also criticized my failure to include software support costs in my example. First, this analysis considers up front acquisition costs and not factors like support, maintenance and depreciation, which are spread over the lifetime of the products. Second, I also excluded Microsoft’s Windows Server support costs, which are comparable to those of VI3.

James also seems to think our customers aren’t using memory overcommit in the real world. The fact is most VI3 users are running with some level of overcommit and a few have kindly posted responses to that effect to my previous post. James may also want to have a chat with his VP at Microsoft. Here’s what Bob Muglia had to say about shared memory in a recent interview:

[Bink] We talked about Vmware ESX and its features like shared memory between VMs, [Muglia] "we definitely need to put that in our product"

James may not be a believer in the savings from memory overcommitment made possible by VMware’s exclusive memory sharing technologies, but apparently others at Microsoft are.

Cheap Hypervisors: A Fine Idea — If You Can Afford Them

Virtualization customers should focus on cost per VM more than upfront license costs when choosing a hypervisor.  VMware Infrastructure’s exclusive ability to overcommit memory gives it an advantage in cost per VM the others can’t match.

Our competition and a few industry observers have lately taken up the sport of bashing VMware Infrastructure as overpriced. Microsoft is playing up their plans to bundle Hyper-V with Windows Server 2008 as a way undercut VI3 pricing and jump-start their late entry in the virtualization market. One of our Xen-based competitors has even adopted a marketing tag line of “one-fifth the cost of comparable alternatives,” clearly referring to us.

VMware Infrastructure customers and prospective users should not be misled by those accusations of inflated prices. Our rivals are simply trying to compensate for limitations in their products with realistic pricing. In defense of our pricing, I could go into details about the powerful virtual infrastructure features you get with VMware Infrastructure 3 Enterprise that the competition is still far from matching. I could also describe the great bargains we offer with our VMware Infrastructure Acceleration Kits. I could explain that our competition is prone to apples-to-oranges comparisons and their offerings should really be weighed against our small business VMware Infrastructure Foundation bundle or the VMware Infrastructure Standard bundle that adds high availability. I could steer those of you looking for the absolute lowest-cost enterprise bare-metal hypervisor to VMware ESX Server 3i for $495 – the thinnest technology available. VMware also has a great TCO/ROI calculator to help you decide, but if all that seems like too much work, let me propose a simpler metric for comparing hypervisors – cost per virtual machine.

Cost per VM is not that hard to measure: just add up the costs for the server, the virtualization software, the operating systems and the application software; then start adding VMs running your workloads until they can no longer meet your required service levels. We’ve actually done that work for you and you might find the results surprising.

We took a common dual socket server with 4GB of RAM and tried the test with ESX Server 3, Citrix XenServer v4 and Microsoft Hyper-V beta. We created and powered on 512MB Windows XP VMs running a light workload and kept adding them until the server couldn’t take any more. Our Hyper-V and XenServer tests topped out at six and seven VMs respectively, which was expected. You see, both those products subtract the full amount of memory allocated to each running VM from the host’s physical RAM. When you factor in the additional memory required by the hypervisor and the management OS, there’s room left for at most seven VMs. In fact, XenServer and Hyper-V will flat out refuse to let you power on an additional VM with a warning that memory resources have been exhausted, as shown in the screen shots below. XenServer and Hyper-V can’t do what we call “overcommiting” memory and that should strike you as tremendously wasteful when most data center VMs are lightly utilized.


Citrix XenServer v4 does not support memory overcommit, so a 4GB server is only able to support seven 512MB VMs.


Microsoft Hyper-V beta is also missing memory overcommit support and only handles six running VMs.

So how did ESX Server fare in the same test? Before I get to the results, I should explain two very important memory management features built into ESX Server. The first is called Transparent Page Sharing and I’ve always considered it one of the most clever features we have. Transparent Page Sharing takes advantage of the fact that VMs will tend to redundantly load the same contents into memory pages if they are running similar operating systems. If you’re running 10 Windows Server 2003 VMs, you’d expect identical chunks of the Windows OS to be in memory. Transparent Page Sharing finds those matching chunks across all the VMs and keeps just a single copy of each. If one VM makes changes to a shared page, ESX Server stores and tracks those differences separately. It’s not quite as trivial as I make it sound; there’s a lot of careful optimization built-in to do the scans for similar pages at times when the VMs are idle and make decisions on how similar two memory pages need to be before they’re shared. The effort we put into developing Transparent Page Sharing pays off big for our users with dramatic reductions in per VM memory consumption and minimal performance impact.


VMware ESX Server uses exclusive Transparent Page Sharing technology to save a single copy of similar guest OS memory pages.

Our other memory technology is called the balloon driver and it’s part of the VMware Tools you load in each VM. The balloon driver process (vmmemctl) recognizes when a VM is idle and exerts artificial pressure on the guest OS causing it to swap out its memory to disk. The freed up memory is then reclaimed for use by other active VMs.


The VMware guest balloon driver frees memory in idle VMs for use by active VMs.

Working together, Transparent Page Sharing and the balloon driver let ESX Server comfortably support memory overcommitment. You can learn more about our memory management technologies in this white paper. Now, getting back to our VM density test, how did ESX Server do? Here’s the screen shot:


VMware Infrastructure 3 with memory overcommitment supports 40 concurrent VMs!

Those 40 VMs have more than 20GB of total RAM allocated and they are running fine on a server with 4GB of physical RAM – a 5:1 memory overcommit ratio. Our exclusive ability to efficiently overcommit memory lets VMware Infrastructure support more than five times as many VMs on the same hardware as our competition! We repeated the test using Windows 2000 Server VMs running SQLIOSim to see how we fared with heavily loaded VMs. Hyper-V and XenServer both topped out at six and seven VMs again when they hit their memory limits, but the ESX Server platform ran fine with 14 VMs – twice as much as the other hypervisors!

Now, let’s get back to the cost per VM comparison to see which hypervisors provide the most bang for the buck. In the table below, we add up the costs for a basic hypervisor deployment. We’ll assume a 2-way, 4GB server costs us $6,000. Next, we add the costs to run Windows in each VM. For that, we’ll take advantage of Microsoft’s policy that lets us run an unlimited number of Windows VMs on a host licensed with Windows Server Data Center Edition (and yes, that policy also applies to VMware and Xen hosts.) Licensing Windows Server Data Center Edition costs us $5998 for two sockets. After that, we plug in the cost of the VMware Infrastructure 3 licenses, and to make things interesting, we’ll assume the competing hypervisor is absolutely free.

The next row in the table shows how many concurrent 512MB VMs each hypervisor can support. For VI3, we’re assuming a conservative 2:1 memory overcommit ratio based on our heavy workload test, which lets us run 14 VMs. For our hypothetical free hypervisor, we’re stuck at seven VMs because memory overcommit isn’t an option. That’s right, no other hypervisor technology allows memory overcommitment – it’s a VMware exclusive.


Finally, we do the division and find that even our high-end VI3 Enterprise bundle beats a free hypervisor in cost per VM! Going with any other hypervisor means you’ll need more hardware, network and storage connections, switch ports, floor space, power and cooling to support a given population of VMs. That should make your decision easy if all you’re doing is simple server consolidation, but there’s more to consider. VI3 Enterprise includes a powerful array of virtual infrastructure services like VMotion, DRS, HA and more that let you automate, optimize and protect your operations, and those features put us far ahead of the offerings from the Xen vendors and Microsoft.

If you’re ready to get started consolidating your servers, don’t be lured by seemingly low cost hypervisors into a decision that will limit your VM density and lock you into spending more on hardware. Instead, put memory overcommitment at the top of your list of hypervisor feature requirements. You’ll spend less on the project by stretching your hardware further and, since only VMware has memory overcommitment, you’ll get the proven reliability and advanced virtualization features of VMware Infrastructure thrown in for free. Beware the high cost of a “free” hypervisor.

[Update: More on VMware Memory Overcommit, for Those Who Don’t Trust the Numbers]