Home > Blogs > Virtual Reality


Cheap Hypervisors: A Fine Idea — If You Can Afford Them


Virtualization customers should focus on cost per VM more than upfront license costs when choosing a hypervisor.  VMware Infrastructure’s exclusive ability to overcommit memory gives it an advantage in cost per VM the others can’t match.

Our competition and a few industry observers have lately taken up the sport of bashing VMware Infrastructure as overpriced. Microsoft is playing up their plans to bundle Hyper-V with Windows Server 2008 as a way undercut VI3 pricing and jump-start their late entry in the virtualization market. One of our Xen-based competitors has even adopted a marketing tag line of “one-fifth the cost of comparable alternatives,” clearly referring to us.

VMware Infrastructure customers and prospective users should not be misled by those accusations of inflated prices. Our rivals are simply trying to compensate for limitations in their products with realistic pricing. In defense of our pricing, I could go into details about the powerful virtual infrastructure features you get with VMware Infrastructure 3 Enterprise that the competition is still far from matching. I could also describe the great bargains we offer with our VMware Infrastructure Acceleration Kits. I could explain that our competition is prone to apples-to-oranges comparisons and their offerings should really be weighed against our small business VMware Infrastructure Foundation bundle or the VMware Infrastructure Standard bundle that adds high availability. I could steer those of you looking for the absolute lowest-cost enterprise bare-metal hypervisor to VMware ESX Server 3i for $495 – the thinnest technology available. VMware also has a great TCO/ROI calculator to help you decide, but if all that seems like too much work, let me propose a simpler metric for comparing hypervisors – cost per virtual machine.

Cost per VM is not that hard to measure: just add up the costs for the server, the virtualization software, the operating systems and the application software; then start adding VMs running your workloads until they can no longer meet your required service levels. We’ve actually done that work for you and you might find the results surprising.

We took a common dual socket server with 4GB of RAM and tried the test with ESX Server 3, Citrix XenServer v4 and Microsoft Hyper-V beta. We created and powered on 512MB Windows XP VMs running a light workload and kept adding them until the server couldn’t take any more. Our Hyper-V and XenServer tests topped out at six and seven VMs respectively, which was expected. You see, both those products subtract the full amount of memory allocated to each running VM from the host’s physical RAM. When you factor in the additional memory required by the hypervisor and the management OS, there’s room left for at most seven VMs. In fact, XenServer and Hyper-V will flat out refuse to let you power on an additional VM with a warning that memory resources have been exhausted, as shown in the screen shots below. XenServer and Hyper-V can’t do what we call “overcommiting” memory and that should strike you as tremendously wasteful when most data center VMs are lightly utilized.

XS_error_startingVM_caption

Citrix XenServer v4 does not support memory overcommit, so a 4GB server is only able to support seven 512MB VMs.

MS_win2k8_overcommit_error_final_caption1

Microsoft Hyper-V beta is also missing memory overcommit support and only handles six running VMs.

So how did ESX Server fare in the same test? Before I get to the results, I should explain two very important memory management features built into ESX Server. The first is called Transparent Page Sharing and I’ve always considered it one of the most clever features we have. Transparent Page Sharing takes advantage of the fact that VMs will tend to redundantly load the same contents into memory pages if they are running similar operating systems. If you’re running 10 Windows Server 2003 VMs, you’d expect identical chunks of the Windows OS to be in memory. Transparent Page Sharing finds those matching chunks across all the VMs and keeps just a single copy of each. If one VM makes changes to a shared page, ESX Server stores and tracks those differences separately. It’s not quite as trivial as I make it sound; there’s a lot of careful optimization built-in to do the scans for similar pages at times when the VMs are idle and make decisions on how similar two memory pages need to be before they’re shared. The effort we put into developing Transparent Page Sharing pays off big for our users with dramatic reductions in per VM memory consumption and minimal performance impact.

sld06a

VMware ESX Server uses exclusive Transparent Page Sharing technology to save a single copy of similar guest OS memory pages.

Our other memory technology is called the balloon driver and it’s part of the VMware Tools you load in each VM. The balloon driver process (vmmemctl) recognizes when a VM is idle and exerts artificial pressure on the guest OS causing it to swap out its memory to disk. The freed up memory is then reclaimed for use by other active VMs.

sld07a

The VMware guest balloon driver frees memory in idle VMs for use by active VMs.

Working together, Transparent Page Sharing and the balloon driver let ESX Server comfortably support memory overcommitment. You can learn more about our memory management technologies in this white paper. Now, getting back to our VM density test, how did ESX Server do? Here’s the screen shot:

VC_40VMs_started

VMware Infrastructure 3 with memory overcommitment supports 40 concurrent VMs!

Those 40 VMs have more than 20GB of total RAM allocated and they are running fine on a server with 4GB of physical RAM – a 5:1 memory overcommit ratio. Our exclusive ability to efficiently overcommit memory lets VMware Infrastructure support more than five times as many VMs on the same hardware as our competition! We repeated the test using Windows 2000 Server VMs running SQLIOSim to see how we fared with heavily loaded VMs. Hyper-V and XenServer both topped out at six and seven VMs again when they hit their memory limits, but the ESX Server platform ran fine with 14 VMs – twice as much as the other hypervisors!

Now, let’s get back to the cost per VM comparison to see which hypervisors provide the most bang for the buck. In the table below, we add up the costs for a basic hypervisor deployment. We’ll assume a 2-way, 4GB server costs us $6,000. Next, we add the costs to run Windows in each VM. For that, we’ll take advantage of Microsoft’s policy that lets us run an unlimited number of Windows VMs on a host licensed with Windows Server Data Center Edition (and yes, that policy also applies to VMware and Xen hosts.) Licensing Windows Server Data Center Edition costs us $5998 for two sockets. After that, we plug in the cost of the VMware Infrastructure 3 licenses, and to make things interesting, we’ll assume the competing hypervisor is absolutely free.

The next row in the table shows how many concurrent 512MB VMs each hypervisor can support. For VI3, we’re assuming a conservative 2:1 memory overcommit ratio based on our heavy workload test, which lets us run 14 VMs. For our hypothetical free hypervisor, we’re stuck at seven VMs because memory overcommit isn’t an option. That’s right, no other hypervisor technology allows memory overcommitment – it’s a VMware exclusive.

Vmware_vs_free_hypervisor_598x309_4 

Finally, we do the division and find that even our high-end VI3 Enterprise bundle beats a free hypervisor in cost per VM! Going with any other hypervisor means you’ll need more hardware, network and storage connections, switch ports, floor space, power and cooling to support a given population of VMs. That should make your decision easy if all you’re doing is simple server consolidation, but there’s more to consider. VI3 Enterprise includes a powerful array of virtual infrastructure services like VMotion, DRS, HA and more that let you automate, optimize and protect your operations, and those features put us far ahead of the offerings from the Xen vendors and Microsoft.

If you’re ready to get started consolidating your servers, don’t be lured by seemingly low cost hypervisors into a decision that will limit your VM density and lock you into spending more on hardware. Instead, put memory overcommitment at the top of your list of hypervisor feature requirements. You’ll spend less on the project by stretching your hardware further and, since only VMware has memory overcommitment, you’ll get the proven reliability and advanced virtualization features of VMware Infrastructure thrown in for free. Beware the high cost of a “free” hypervisor.

[Update: More on VMware Memory Overcommit, for Those Who Don't Trust the Numbers]

29 thoughts on “Cheap Hypervisors: A Fine Idea — If You Can Afford Them

  1. W. Craig Trader

    Lars writes “Wish you had included VMware Server too in your test”.
    Lars, effectively they did. VMware Server is going to have the same response pattern as MS Hyper-V (or MS Virtual Server), since it doesn’t support the balloon driver or TPS. The details will be slightly different, but the overall performance will be roughly equivalent, and it will have the same pricing model as the Hyper-V solution.
    Background: In 2006, my company began our virtualization efforts around MS VirtualPC and MS VirtualServer, due to price. In 2007, we migrated from MS Virtual Server to VMware Server because we needed better performance while still keeping our up-front costs low. In 2008 we’re migrating the same servers from VMware Server to VMware ESX to get better usability and manageability out of the same hardware.

  2. Richard Priest

    How do you reconcile the advice in the VI performance guide: “Avoid high memory overcomittment. Make sure the host has more memory than
    the total amount of memory that will be used by ESX plus the sum of the working
    set sizes that will be used by all the virtual machines.”

  3. Paul

    You left out management costs, which would actually improve your competitive argument. Microsoft is now licensing managment tools by Operating System Environment rather than by device. So while one copy of Operations Manager used to manage every VM on a server, you now need a management server CAL for each VM on a server. Or, you can buy the System Center Server Management Suite Enterprise, for $1,290, which licenses every VM on the device–but in your example, that still works out to another $200 per VM on a server running only 6 VMs. On the other hand, on the same device running 40 VMs, it works out to $32 per VM.

  4. Scott Drummonds

    I see how the recommendations in that document could be construed and conflicting with this blog. But I assure you they are not.
    The paper warns our customers against “high memory over-commitment”. As the term “high” is subjective, the paper then goes on to say precisely what the danger to performance is: when the sum of the working set sizes of all VMs exceeds the amount of memory on the host, ESX Server will be forced to swap which greatly reduces performance.
    The key here is a term familiar to operating system experts: working set. A working set is the set of memory that is frequently in use by an application or operating system. When an OS is installed on a system with 1G of RAM, a running application may only be actively using 100M of RAM. This 100M is its working set.
    What we find in the great majority of deployments is that VMs’ working sets are much, much smaller than the amount of memory provided to the VM. This is a good thing, because you want to size your VMs not just for the average working set but for the largest possible working set that might ever occur on the VM. But its a bad thing for hypervisors that can’t share memory, balloon, and/or swap because so much of the memory is often unused. By providing more memory to the VMs than is needed on average (over-committing memory resources) but verifying that the working sets don’t need more memory than the host provides (memory over-utilization), you can achieve industry-leading consolidation ratios at no impact to performance.

  5. Martin Jansen

    Having used this technology on a few standard esx2.5 server this really works. It is also advisable because of this feature to host as many of same type machine on a single host (where policy permits of course) then you will have the biggest advantage. What i also noticed myself that this even works better with linux guests as they tend not to swallow all the memory they can get.
    Also a fun thing to mention (for me) is that a windows 2003 installation on esx 2.5 have a far smaller memory footprint then a physical one. (VM ~97MB Physical ~ 140 – 200MB) I have no data for other virtual hosts as i have not actively used them

  6. bluvg

    Cool technologies, but why on earth would you limit a $6000 server to 4 GB of RAM? RAM is cheap. I think the argument here is based on a false premise.

  7. Rob

    Are you mad? Most of what you have written doesnt apply to any production environment. Also there are loads of virtualisation technologies in Linux and Windows offering overcommit on memory, CPU and various other things before VMw even thought of the idea.

  8. AP

    This looks more and more like VMWare trying to justify the high cost of thier software. VMWare keeps on talking about the Hypervisor in ESX like it is going to solve world hunger. The reality is that Hypervisors will become commodity and the magic is in the management of both the physical and the virtual layer.

  9. Jason Boche

    To address memory overcommit (no pun intended), I would like to offer up a quick case study where memory overcommit works in my company of 160,000 employees. A few weeks ago I begain posting preliminary analysis of running production Citrix Presentation Servers inside VMs on ESX 3.x (see http://communities.vmware.com/message/863920). My preliminary findings show that the production Citrix VMs are taking advantage of Content Based Page Sharing (for a detailed explanation of VMware’s Content Based Page Sharing, see Carl Waldspurger’s whitepaper on Memory Resource Management in VMware ESX Server at http://www.waldspurger.org/carl/papers/esx-mem-osdi02.pdf). One virtualized Citrix server is handling 50-85 sessions and it’s not full yet. Each of the sessions is running one of three published applications that all share the same base PowerBuilder code and .DLLs (about an 80MB memory footprint for each session). Because each of the 50-85 sessions shares the same code, the VMware’s Content Based Page Sharing consolidates many of the identical pages into single read only reference points and discards the redundant copies. The net result is significant ESX host memory savings. As an example, what I’m seeing inside the Citrix VM is nearly 4GB of RAM being used, but from the ESX host perspective, 1GB or less physical RAM is being utilized, leaving the additional 3GB of physical RAM for another VM in the cluster to use. Now multiply this memory savings by the number of virtualized Citrix servers and the memory savings adds up quickly.
    In the blog banter, a point is made by the opposition that it’s going to be rare for a number of VMs on the same host to all be identical such that there would be a significant savings in memory page sharing. My example is proof that you don’t even need multipe VMs to take advantage of VMware’s page sharing and memory overcommit. The fact is that VMware’s Content Based Page Sharing works both inter-VM (across VMs) and intra-VM (within a single VM). I verified this with Carl Waldspurger himself a few weeks ago. Summary: Citrix VMs running silo’d or partitioned application sets take advantage of intra-VM Content Based Page Sharing.
    On the inter-VM Content Based Page Sharing front, an assumption is made by the opposition that rarely will VMs share the same code to amount to any sort of memory savings. I don’t have hard numbers yet but I will say that in our VDI deployment that we are rolling out, all Windows XP images are rolled out from the same template which contains all of the tools each VDI user needs to do business. While there is no guarantee each VDI user will concurrently be running the same applications within their VDI, it’s a fact that at a minimum they will each be running the same base operating system (Windows XP), Microsoft Outlook plus all of the MS Office code that loads in the background and at startup. Most will probably have at least 1 IE browser open as well, although the IE footprint is fairly minimal in the grand scheme of things. Depending on the number of virtual desktops we deploy on each host, I think we’ll see generous memory savings. This is not to say we will intentionally overcommit memory. Ballooning and VMKernel swap allows memory overcommit but I’m not that desperate to take the performance hit that comes with it, although I’m not sure whether or not our VDI users would be able to tell the difference or not.
    Jas

  10. Jason Boche

    @Eric Horschman:
    Sorry to be anal here.
    Transparent Page Sharing was introduced by Disco and is a completely different technology than what VMware uses which is accurately labeled Content Based Page Sharing.
    Disco’s Transparent Page Sharing requires several guest OS modifications to identify redundant memory page copies. A special network interface was also rquired to share data communication between VMs.
    VMware’s Content Based Page Sharing is a completely different approach to page sharing and is oblivious to guest OS code. This means no modification to the guest VM.
    See http://www.waldspurger.org/carl/papers/esx-mem-osdi02.pdf pages 4-5.
    Jas

  11. Lars

    W. Craig Trader writes:
    > VMware Server is going to have the same response pattern as MS Hyper-V
    Not true. VMware Server has both transparent page sharing and memory stripping enabled by default. You’re however correct that VMware Server doesn’t have vmmemctl/ballooning.
    Lars

  12. Bradley Watt

    I have had some basic test environments for all 3 major VM platforms (VMware, Virtual Iron, and XenSource), and just about to try MS Hyper-V. At this point in time they seem to offer different value to different customers. I do feel that VMware is presently the best solution if you can afford VI3.
    BUT the testing done in this blog is so skewed, it is worthless.
    a) Memory for a server is relatively cheap, so why you would buy a dual socket server with 4GB of Ram is beyond me.
    b) With most new servers now Quad-Core, I personally would be purchasing, at a minimum 2GB per core (if not 4GB).
    c) Although most servers in a Data Center are normally run at considerably under capacity, one of the reasons to implement virtual technology is to more fully utilize these servers, with different workloads, different operating systems, and different applications. This considerably reduces the opportunity for VMware to “overcommit” memory resources.
    It is like saying that if you put a Ferrari in 6th and keep the engine at 1000 RPM, you will get a very high MPG rating. Who the hell cares? If you are going to buy a Ferrari, you are more interested in 0-60, 0-100, 0-200 etc, not irrelevant, never going to do, test data.

  13. DEBO Jurgen

    I known VMware very well. I suggested so many things which are realized by this company which finally became Thin application development, VMware Infrastructure etc. I understand completely, implementation and commercialization is a big deal in the workplay. However, no idea, no egg which make a chicken. My ROInvention was small. Sad.
    So I will not tell the solution to improve the products anymore. Voila.

  14. Joannah

    I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.
    Joannah
    http://linuxmemory.net

  15. walk on video

    there will always be bashing in any market as there will always be competition. If you stick to your guns and deliver what you promise your clients and new clients will always stick with you

  16. convert pdf to epub

    This looks more and more like VMWare trying to justify the high cost of thier software. VMWare keeps on talking about the Hypervisor in ESX like it is going to solve world hunger. The reality is that Hypervisors will become commodity and the magic is in the management of both the physical and the virtual layer.
    http://www.convertpdftoepub.net/

  17. vob converter

    I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.

Comments are closed.