VMware: Virtual Reality

We Reviewed Microsoft's ROI/TCO Model

Some of you may have seen Microsoft’s recently released virtualization ROI/TCO calculator. Briefly, the model purports to offer an accurate cost/benefit comparison between Microsoft’s Hyper-V offering and a “Competitive Server Virtualization Solution” – gee I wonder who the competitive solution is…?  Microsoft is beginning to advertise the calculator broadly in its partner newsletter and other email blasts – and we’ve even had customers bring it to our attention. Mainly, the VMware customers that have alerted us to the Microsoft ROI/TCO calculator were confused by many of the model’s assumptions and by the generated results - they wanted our opinion. So, we took a look.

Unfortunately We Had to Give It A Failing Grade

Of course the results were all hypothetical, because Hyper-V is not yet available, but what we found when running a realistic scenario through the model and then from reading the report’s fine print, is that like most Microsoft version 1.0 products, the initial release of this calculator has numerous errors, contains critical design mistakes, and completely misses its mark. Any results generated from this model are so unrealistic as to be completely worthless for accurately comparing costs and benefits of alternate virtualization solutions. (Maybe we all need to wait for the SP1?)

In Sum:   ROI/TCO Analysis = Good Idea ; Inaccurate Model = Bad Idea

But of course that’s what one would expect VMware to say right? Out comes a model that could show that a competitor’s product is less expensive and right away VMware is going to question its accuracy. Well I’ll soon list out the numerous issues we found with the Microsoft model – and those issues will articulate clearly why Microsoft’s model is inaccurate and why our questioning of the model is justified, but first just let me state that we at VMware fully support an accurate and realistic ROI/TCO analysis of our solutions. In fact, VMware launched an ROI/TCO calculator way back in April of 2007. Since then, our customers have generated over 30,000 reports that quantify the enormous benefits they are receiving from VMware installations. (Feel free to try it for yourself here.) However, we also believe, and I am sure you’d agree, that any ROI/TCO model must be based on 1) accurate pricing/licensing information, 2) fair product comparisons, and 3) defendable assumptions for both associated costs and realized benefits. Our analysis of the Microsoft supported model showed that it had none of those traits.

Play Along at Home

Following are some of the many mistakes and gross assumptions we found in the Microsoft model. So that you can follow along at home – please download a copy of the report we generated. I encourage you to evaluate the Microsoft calculator yourself – let us know what else you find! Also, as you read further, and as you consider all the errors in the model, keep in mind the following question:

Why Did MSFT Release Such a Misleading ROI/TCO Model?

A) Microsoft did a sloppy and hasty job with the calculator

B) Microsoft is deliberately fudging the facts

C) Both A and B

Now I don’t want to imply that Microsoft is trying to deliberately mislead anyone, but it does make me question Microsoft’s overall capacity for rigorous product testing. I hope that the same team that QA’d the Microsoft TCO model is not the same group that is currently testing the V1.0 release of Hyper-V!

Final Thought

Perhaps we at VMware could work with Microsoft and a third party to develop a standardized model for an ROI/TCO analysis of a virtualization solution? Thoughts?

Production & Dev/Test Server Virtualization – Competitive Cost Comparison
(Appendix B – page 18)

Why is a discounted Microsoft license cost being compared to VMware List Pricing??
The tool assumes the following pricing for Windows Server 2008 Editions: $719 for Standard, $2334 for Enterprise, and $2381 for Datacenter. That doesn’t match up to the list pricing that Microsoft publishes on its website, so the prices must be some sort of volume/discounted pricing. So the tool uses discounted pricing for Microsoft and list pricing for VMware ($5750 for VI3 Enterprise)? Not exactly an apples-to-apples comparison. One might argue that Microsoft does not have access to VMware volume pricing. But then again, the average reader doesn’t have access to Microsoft’s volume pricing either.

Where are the Microsoft System Center Server License Costs?
Under the “Virtualization Management Software” row, the VMware column includes the server license cost for VirtualCenter $5000. That’s fine, but to the best of our knowledge, an organization using the MSFT solution would also need server licenses, for 1) Microsoft System Center Operations Manager, 2) System Center Configuration Manager, and 3) System Center Data Protection Manager. Each component is $573 (list price) according to Microsoft’s own website. That $573 doesn’t include the SQL Server license – that’s extra. Currently, no additional server license cost is required for System Center Virtual Machine Manager 2007 - TBD whether that will remain true for Virtual Machine Manager 2008. System Center Operations Manager is required to do any performance tracking, Configuration Manager to do updates and patches, and Data Protection Manager to do backups. Most of that functionality is an integrated part of VMware VirtualCenter.

Since When Did All VMware Customers Decide to Run System Center?
The tool assumes that customers will use Microsoft System Center to manage their VMware Infrastructure deployment. Again, this little detail is buried deep in the appendices (Appendix B – “System Center Enterprise” row). Microsoft assumes that everyone who still chooses to run VMware will naturally want to manage it using Microsoft System Center. This seems like a pretty presumptuous assumption. Most companies we talk to have already invested in an enterprise management framework and want VMware to integrate with what they’ve already got. Unless they are already running Microsoft System Center, they really don’t want to rip out their existing investment and put in System Center. VMware VirtualCenter APIs are readily available through the VirtualCenter SDK making it possible for VMware partners like IBM, HP, CA, BMC, Symantec, Quest, and NetIQ (to name a few) to integrate seamless with VMware VirtualCenter. Customers can keep using what they’ve already invested in.

The Model Incorrectly Calculates Microsoft Licensing Costs
In the scenario we ran, Microsoft’s tool assumed 71 Windows Server Standard Edition licenses for 414 virtual machines running on 71 hosts. Since each Standard Edition license grants rights to run 1 VM, the model’s results leave 343 VMs in our hypothetical datacenter running out of compliance. Microsoft may claim that the TCO/ROI calculator is not a licensing calculator, but how can it calculate accurate TCO estimates using inaccurate licensing assumptions? We did find a one-line disclaimer buried in the 66-page document: “Warning – Check pricing advice and rules as the automated recommendations here may not reflect all licensing rules.” Come on, guys - licensing is such a basic component for accurate TCO estimates. The disclaimer feels pretty weak.

Microsoft Cost Savings are HUGE… if You Believe the Assumptions
We found many overly aggressive assumptions in the Microsoft calculator, which resulted in overly aggressive cost savings for the Microsoft’s solution. For instance, on page 26 “Production Server Operations and Administration Efficiency Savings”, Microsoft’s default assumption is an 83% reduction in person-hours-per-year after virtualizing with Microsoft’s solution. That translates into a VM-to-admin ratio of 250:1 after virtualization. While VMware agrees that virtualization absolutely increases administrative efficiency, 250:1 is overly aggressive.

Why are 4-way Servers the Model’s Default Assumption? Oh yeah… It Increases VMware’s Perceived Cost.
The tool assumes 4-way servers as the default configuration. Granted, 4-way server volume is growing, but IDC’s Quarterly Server Tracker published on February 26, 2008, shows only 4.27% of all servers shipped in 2007 were 4-ways, making them nowhere close to being the majority of servers virtualized today. This assumption does double the number of VMware Infrastructure 3 (VI3) Enterprise Edition licenses required since VI3 is licensed in 2-socket increments, thereby making VMware’s total deployment cost appear much higher. Most readers won’t see this point buried deep in the appendices; perhaps that was the goal? Oh, with quad-cores today and hexa/octo-cores in the near future, 2-way servers are still projected to be the majority volume virtualized server platform in 2008 and 2009.

VM Density Per Host Matters for TCO – a Fact that Microsoft Ignores
Not all virtualization platforms are created equal – some can run more virtual machines per host than others, thereby resulting in lower total cost of ownership, due to reduced hardware and related costs. In one of our recent blogs, we highlighted the lack of memory sharing technologies in Microsoft Hyper-V. This deficiency results in fewer virtual machines per host (due to inefficient memory usage), thereby increasing the number of required physical servers, thereby increasing cost. Microsoft’s tool assumes that Hyper-V will run as many VMs as VMware VI3 and deliver the same performance – we can’t wait until Hyper-V ships and prove this wrong.

VMware VI3 Enterprise Functionality is Far Greater than Hyper-V
Microsoft compares Hyper-V to VMware VI3 Enterprise Edition even though Hyper-V lacks key enabling capabilities that are prerequisites for a truly dynamic datacenter. Capabilities missing from Hyper-V include live VM migration (VMware VMotion and Storage VMotion), automated/dynamic load balancing (VMware DRS), dynamic power management (VMware DPM), integrated offline VM patching (VMware Update Manager), a next-generation ultra-thin hypervisor architecture (VMware ESXi), a clustered file system (VMware VMFS), and the ability to make VMs highly available without the storage management headache of assigning 1 LUN for each VM (VMware HA). For the basic server consolidation scenarios that Microsoft Hyper-V enables, VMware offers VI3 Foundation and VMware ESXi at very competitive price points – check out VMware’s complete server virtualization product offerings here.

Desktop and Application Virtualization – Competitive Cost Comparison

(Appendix B – page 19)

Microsoft Tool Assumes Wrong Costs for VMware VDI
To virtualize 6000 clients, Microsoft has incorrectly assumed 240 VI3 Enterprise licenses ($1,380,000) plus “Virtualization Management Software and Host Server” cost ($7000) plus 6000 connection broker client licenses ($300,000). The actual license cost of VMware VDI is 6000 x $150 = $900,000 plus virtualization management costs of $1250. This VMware VDI pricing has been in effect since Feb 2008 so the Microsoft calculator does not use up-to-date, publicly available information.

Readers Must Figure Out that Microsoft Compares Two Completely Different Technologies
By looking at the summary table on page 6, most readers would reasonably assume that the comparison is between two comparable products addressing the same business need. That’s not the case. Microsoft is comparing the cost of Terminal Services + MDOP (including Microsoft Application Virtualization) to the cost of a VMware VDI solution. VMware VDI offers full desktop VM virtualization, best in class application compatibility and true user isolation. Comparing terminal services + application virtualization to VDI is an apples-to-oranges comparison.

Microsoft Assumes Low Consolidation Ratios for VMware VDI
The consolidation ratios assumed by Microsoft for VDI are significantly lower than the consolidation ratios observed by VMware customers. On a 2-socket dual core server, VMware customers typically observe average consolidation ratios of 32-36 VMs depending upon the workloads. Microsoft only assumes 25 VMs per host, which skews VMware’s cost by another 20-30%. Of course, Microsoft assumes a higher consolidation ratio (30:1) for its own solution.

Questionable Assumptions for How Microsoft’s Desktop Solution Lowers Cost
We found some questionable assumptions on how Microsoft got some of the cost-savings estimates. For instance on page 39 “Desktop Virtualization Client Software License Cost Avoidance“, Microsoft claims that companies can reduce the number of “unused / not properly allocated licenses” by 70% using their desktop solution. Apparently, overbuying of Microsoft software licenses is a common issue. Another example is on page 37 “Desktop Virtualization Branch Office Server Cost Avoidance.” Microsoft claims that some branch office server infrastructure may be consolidated / retired and replaced with centralized virtualized servers for branch office applications. While this is one viable way to reduce IT capital cost, what about the additional IT capital costs to upgrade to low latency, highly available network connectivity to the branch offices? This network cost needs to be factored in.

Long-deferred vendor visions of agile data centers are finally coming true now that VMware virtualization with VMotion live migration has severed the ties that kept services fixed to x86 hardware.  Unfortunately, some vendors are trying to stage a revival with an inferior substitute for live migration.  Most notably, Microsoft is claiming that their "Quick Migration" feature is comparable to VMware VMotion and adequate for enterprise data centers, even though Quick Migration is not true live migration. We've even heard Microsoft tell audiences that our customers don't trust VMotion enough for production use. Don't fall for it -- VMotion is ready, proven and in heavy use today by VMware customers who are bringing true flexibility and agility to their IT operations.

Do you remember the many grand visions for IT that were trotted out by the vendors and analysts during the dot com boom times? Adaptive Enterprise Computing, Next Generation Data Centers, Organic IT, On-demand Computing, Utility Computing and more were relentlessly pitched to CIOs with PowerPoint promises of continuously available services effortlessly floating on pools of servers and storage, finding the resources they needed all by themselves and magically recovering from any faults and disasters that should arise.  CIOs put up with the daydreaming until the vendors were finally shamed into backing off on the hard sell by their noticeable inability to deliver on the promises. The technology, especially in the x86 world, just could not break the bonds that kept applications and services firmly welded to their physical hosts.

The phenomenal growth of virtualization is now reviving some of those grand IT visions. With a virtualization layer that includes live migration, x86 workloads can float free of the fixed servers and storage hardware that enterprises have in place. And, thanks to tools like VMware VMotion that live migrates servers between hosts and VMware Storage VMotion that allows transparent relocation of a VM's storage, those workloads finally can accomplish that floating without the slightest interruption to users and services. It’s not just VMware that is enabling this revolution-in-waiting; the Xen vendors are also starting to roll out their own live migration support.

It should not be surprising then, that Microsoft is using its entry into the virtualization market to bring its own grand architecture – the dormant “Dynamic Systems Initiative” – out of hibernation. Now apparently renamed as “Dynamic IT,” their vision was featured in Bob Muglia's January 21 V-day missive to hundreds of thousand of Microsoft customers and partners.  In laying out the benefits of virtualization and live migration, we couldn’t have said it better ourselves:

"In the data center, virtualization not only supports server consolidation, but it enables workloads to be added and moved automatically to precisely match real-time computing needs as demand changes. This provides greater agility, better business continuity, and more efficient use of resources."

That “moved automatically” part sounds pretty compelling. Of course, you’d only want workloads to get up and move themselves if they could do so without the inconvenience of planned maintenance windows and application downtime. That’s exactly what VMware users have been doing with VMotion since we introduced it in 2003. VMotion delivers true live migration – users and services see no interruptions when a virtual machine is moved from one host to another. VMotion has proven so liberating and reliable that 59% of VMware customers use it regularly in production; some have accumulated hundreds of thousands of perfectly transparent migrations as VMs are automatically load balanced across host clusters with DRS. You don't need to build a large-scale virtual infrastructure to benefit from VMotion. We see over and over how customers that adopt VMware Infrastructure for basic server consolidation projects quickly come to rely on the agility and freedom of VMotion as an essential element of their IT operations.  Here's what Qualcomm had to say about the flexibility provided by VMotion:

"We’ve utilized VMotion extremely heavily. It offers so many benefits: being able to deal with downtimes, being able to do maintenance on the hardware supporting ESX Server hosts, and being able to balance resources. VMotion is a must-have capability for anyone seriously thinking of deploying virtual infrastructure."

While we’re gratified to see virtualization taking the lead in reviving Microsoft's DSI story, its own virtualization tools are missing the crucial live migration support needed to pull it off. It’s important to know that Microsoft dropped plans for live migration in Hyper-V and is relying on a “not quite live” migration method it calls, “Quick Migration.” Microsoft Quick Migration works very differently than the iterative live memory transfer method used by VMware VMotion. Quick Migration fully suspends a VM, copies its memory image to disk, and then reloads and resumes the VM on a new host. That suspend/resume migration technique is far from live. In fact, Microsoft has documented (slide 47) that, even in ideal conditions, Quick Migration interrupts VMs between eight seconds and two minutes when using Gigabit speed networked storage, depending on VM memory size.

Unfortunately, that kind of downtime is more than most networked applications can tolerate. Just a few seconds of unresponsiveness will trigger TCP timeouts and application errors. We tried Quick Migration with the Hyper-V beta using Gigabit iSCSI storage connections and the results weren’t pretty, as you can see in this screen capture video:

(Clicking the screen icon switches to full screen mode, which will make the window text legible. If that doesn't work, you can go directly to this movie at blip.tv)

The Quick Migration downtime caused file copies to fail, VM console connections were severed, and database clients had to be restarted. Scheduling planned maintenance downtime and telling users their apps will be down does not fit anyone’s definition of “Dynamic IT.” In contrast, migrating the same VM with VMotion on a VMware Infrastructure platform didn’t cause even a blip in the network sessions as this video shows:

(Clicking the screen icon switches to full screen mode, which will make the window text legible. If that doesn't work, you can go directly to this movie at blip.tv)

In anticipation of any concerns that we stacked the deck in this demo to cause Quick Migration failures, we were careful to configure our Hyper-V setup exactly as documented in this Microsoft TechNet article (it was the only documentation on Quick Migration configuration we could find.)  We also used the latest Windows Server 2008 RTM and Hyper-V RC0 releases.  We've been repeating this Quick Migration test all the way back to the Viridian CTP release and with Virtual Server 2005 R2 before that, and we've always seen the same network session failures every time. Here's a diagram of our Win2008/Hyper-V setup shown on the right.

I encourage any readers who've tried their own Quick Migration tests to share their experiences.

If you're wondering why Quick Migration of a VM exhibits these network failures, but a normal Microsoft Cluster Service failover keeps network sessions alive, take a look at Mike DiPetrillo's excellent explanation.  It's due to the fact that during the time it takes to suspend then resume a quick migrating VM, there's no network stack available to respond to its IP address.  Mike also explains how VMotion preserves network connections and shows how VMotion has become an indispensable money-saver for VMware customers.

While Microsoft may not be ready to deliver the true live migration needed for Dynamic IT in Hyper-V, their customers don’t need to defer their dreams of automated workload migrations and resource balancing. Microsoft operating systems and applications run great in VMware virtual machines and users can take full advantage of powerful virtualization services like VMotion, DRS, HA, Consolidated Backup, Storage VMotion and Update Manager. Live migration is finally letting Dynamic IT and all the other grand architectures of IT live up to their promises of data center agility and hardware independence.  Customers just need to ensure they choose a virtualization technology that supports true live migration like VMware VMotion and not get trapped with an inadequate substitute.

VMware met with Yankee Group to provide our feedback on the factual inaccuracies in the report "Virtualization Price War: VMware's Little Big Horn?". We did this around the same time we put up our blog posting summarizing the issues point-by-point. Yankee Group understood our concerns, agreed to do a revision to address the issues, and promptly took down the original report from its website several weeks ago.

Because of the factual inaccuracies in the report, Yankee Group also committed to requesting that Microsoft, who had purchased distribution rights to the report, cease distributing the original report. However, three weeks after our discussion with Yankee Group and three weeks after Yankee Group pulled the report from its website, Microsoft still continues distributing the report, such as in the Microsoft Virtualization Newsletter that just went out on 4/2/2008.

I wonder why Microsoft would continue distributing a report as marketing material, when the author, Yankee Group has rescinded the report...

There has been a lot of talk on this blog and others about memory overcommit. Several of the blogs have tried to discredit the power of memory overcommit by saying it's never used by customers in the real world and can simply be overcome by adding more memory to the server (see the long discussions in previous posts on this blog). One VMware customer has already commented about how he architects his environment with memory overcommit in mind. I received a private response from another VMware customer who is getting ready to implement a very large VDI environment with some interesting numbers that I'll walk through below. The customer is a large US bank and to protect the security of their environment I did black out some of the names in the screenshots.

About the Environment

This customer configured their standard Windows XP environment for their call centers to run in a virtual machine. Each virtual machine is granted 512 MB of memory and 1 virtual CPU. Each VM runs a series of applications including Marimba, Microsoft Office, a call recording application, a customer database application, and a BPO (business process off-shoring) application.

The host used is an IBM x3850 M2 quad socket, quad core system (total of 16 cores). The server is configured with 64 GB of physical RAM. There are 32 memory sockets in the server.

At the time the system was actively being used by call center employees reflected by this CPU utilization graph.

Running the Numbers

Below is a screenshot of their environment showing a total of 178 VMs running on the system. You can also see in the screenshot that less than 20 GB of RAM out of the total 64 GB of RAM is being used on the system. With a total of 178 VMs configured for 512 MB of RAM each they are currently allocating 89 GB of memory to running VMs which means they are oversubscribed on the host.

Below are screenshots of the price for an identically configured host coming from IBM's website on March 18, 2008.

Total cost for the server is $24,623.00. Remember that this box is overcommitted on memory. In order to run the same setup with a competitive solution we would need to have a server configured with at least 89 GB of RAM - the total allocated to all of the running virtual machines. Going back to IBM's website and reconfiguring the server with more memory gives us the following screenshots. The closest configuration that supported at least 89 GB of RAM was 92 GB of RAM because of the memory configurations allowed in the server.

The new cost of the server is $36,423.00. Compared to the original configuration this is a difference of $11,800.00 just to add more memory to the server to support a solution that does not have memory overcommit. The cost of a 4 socket VMware VI3 Enterprise license is $11,500.00 list price. As you can see the cost of a VMware license is actually $300 less than the cost of adding more memory. Not much of an advantage on the cost side but it still drives home the point that the VMware solution is not more than the competitive "free" solution. What's more is now you get all of the enhanced functionality of the VMware solution that the competitive solutions are lacking. But enough of the marketing pitch, let's go back to the numbers and this customer case study.

We've shown you a real world scenario from a real customer where adding more memory to the server is still more expensive than the VMware license cost. I want to take this one step further and show what designing the architecture to more closely match the true working set size of memory could do. Remember that's we're using less than 20 GB of RAM to run this environment of 178 VMs. We went back to the IBM website and configured the starting point for a VMware solution to have at least 20 GB of RAM. The closest we could get is a host with 24 GB of RAM as seen below.

As you can see, by truly engineering the solution to what you actually need we've reduced the hardware cost to $20,023.00. If we compare that to the competitive solution's hardware cost we get a difference of $16,400.00 for the extra memory. Subtracting out the cost of the VMware license the VMware solution comes in at $4,900.00 cheaper than the competitive solution.

It's All for Charity

I would love for us to give all of that $4,900.00 to charity but the customer wouldn't go for that. Instead they're using it to pay for some of the thin clients they'll use in this solution. After all the $4,900.00 they saved by going with a VMware solution over a competitive "free" solution will buy them 33 thin clients at their current price.

The good news from all of this is James O'Neill from Microsoft was kind enough to volunteer $270 of his own money if we could show a real customer running with memory overcommit where adding memory to the server wouldn't make the Microsoft solution still cheaper than VMware. I think we've adequately met that goal here. So, James, the charity of choice is One Laptop Per Child. And just in case you believe that we've cherry picked a use case we'll be more than happy to connect you directly via phone to any one of the numerous customers we have leveraging memory overcommitment in their environment today.

James O’Neill of Microsoft had a pretty sharp reaction to my last post. He accused me of cooking the numbers to exaggerate the cost saving benefits of VMware’s memory overcommit feature. To help James understand just how memory overcommit works, I’ve taken the numbers out of the argument and have used simple algebra to compare the TCO per VM for a host with either VI3 or Microsoft hypervisors. Let’s do the math…

Variables

So, you can see in this extreme case where hardware is free, the VMware Infrastructure system still beats Microsoft Hyper-V, or any other hypervisor, in total cost per VM in a Windows environment. Plugging in any realistic costs for hardware and memory just tilts the balance further in VMware’s favor. I’m also using our most feature-rich VI3 Enterprise product in the example – had I used one of our products like ESX Server 3i or VI3 Foundation that is a little closer to the barebones capabilities of Hyper-V, the numbers would look even worse for Microsoft.

James also criticized my failure to include software support costs in my example. First, this analysis considers up front acquisition costs and not factors like support, maintenance and depreciation, which are spread over the lifetime of the products. Second, I also excluded Microsoft’s Windows Server support costs, which are comparable to those of VI3.

James also seems to think our customers aren’t using memory overcommit in the real world. The fact is most VI3 users are running with some level of overcommit and a few have kindly posted responses to that effect to my previous post. James may also want to have a chat with his VP at Microsoft. Here’s what Bob Muglia had to say about shared memory in a recent interview:

[Bink] We talked about Vmware ESX and its features like shared memory between VMs, [Muglia] "we definitely need to put that in our product"

James may not be a believer in the savings from memory overcommitment made possible by VMware’s exclusive memory sharing technologies, but apparently others at Microsoft are.

Virtualization customers should focus on cost per VM more than upfront license costs when choosing a hypervisor.  VMware Infrastructure's exclusive ability to overcommit memory gives it an advantage in cost per VM the others can't match.

Our competition and a few industry observers have lately taken up the sport of bashing VMware Infrastructure as overpriced. Microsoft is playing up their plans to bundle Hyper-V with Windows Server 2008 as a way undercut VI3 pricing and jump-start their late entry in the virtualization market. One of our Xen-based competitors has even adopted a marketing tag line of “one-fifth the cost of comparable alternatives,” clearly referring to us.

VMware Infrastructure customers and prospective users should not be misled by those accusations of inflated prices. Our rivals are simply trying to compensate for limitations in their products with realistic pricing. In defense of our pricing, I could go into details about the powerful virtual infrastructure features you get with VMware Infrastructure 3 Enterprise that the competition is still far from matching. I could also describe the great bargains we offer with our VMware Infrastructure Acceleration Kits. I could explain that our competition is prone to apples-to-oranges comparisons and their offerings should really be weighed against our small business VMware Infrastructure Foundation bundle or the VMware Infrastructure Standard bundle that adds high availability. I could steer those of you looking for the absolute lowest-cost enterprise bare-metal hypervisor to VMware ESX Server 3i for $495 – the thinnest technology available. VMware also has a great TCO/ROI calculator to help you decide, but if all that seems like too much work, let me propose a simpler metric for comparing hypervisors – cost per virtual machine.

Cost per VM is not that hard to measure: just add up the costs for the server, the virtualization software, the operating systems and the application software; then start adding VMs running your workloads until they can no longer meet your required service levels. We’ve actually done that work for you and you might find the results surprising.

We took a common dual socket server with 4GB of RAM and tried the test with ESX Server 3, Citrix XenServer v4 and Microsoft Hyper-V beta. We created and powered on 512MB Windows XP VMs running a light workload and kept adding them until the server couldn’t take any more. Our Hyper-V and XenServer tests topped out at six and seven VMs respectively, which was expected. You see, both those products subtract the full amount of memory allocated to each running VM from the host’s physical RAM. When you factor in the additional memory required by the hypervisor and the management OS, there’s room left for at most seven VMs. In fact, XenServer and Hyper-V will flat out refuse to let you power on an additional VM with a warning that memory resources have been exhausted, as shown in the screen shots below. XenServer and Hyper-V can’t do what we call “overcommiting” memory and that should strike you as tremendously wasteful when most data center VMs are lightly utilized.

Citrix XenServer v4 does not support memory overcommit, so a 4GB server is only able to support seven 512MB VMs.

Microsoft Hyper-V beta is also missing memory overcommit support and only handles six running VMs.

So how did ESX Server fare in the same test? Before I get to the results, I should explain two very important memory management features built into ESX Server. The first is called Transparent Page Sharing and I’ve always considered it one of the most clever features we have. Transparent Page Sharing takes advantage of the fact that VMs will tend to redundantly load the same contents into memory pages if they are running similar operating systems. If you’re running 10 Windows Server 2003 VMs, you’d expect identical chunks of the Windows OS to be in memory. Transparent Page Sharing finds those matching chunks across all the VMs and keeps just a single copy of each. If one VM makes changes to a shared page, ESX Server stores and tracks those differences separately. It’s not quite as trivial as I make it sound; there’s a lot of careful optimization built-in to do the scans for similar pages at times when the VMs are idle and make decisions on how similar two memory pages need to be before they’re shared. The effort we put into developing Transparent Page Sharing pays off big for our users with dramatic reductions in per VM memory consumption and minimal performance impact.

VMware ESX Server uses exclusive Transparent Page Sharing technology to save a single copy of similar guest OS memory pages.

Our other memory technology is called the balloon driver and it’s part of the VMware Tools you load in each VM. The balloon driver process (vmmemctl) recognizes when a VM is idle and exerts artificial pressure on the guest OS causing it to swap out its memory to disk. The freed up memory is then reclaimed for use by other active VMs.

The VMware guest balloon driver frees memory in idle VMs for use by active VMs.

Working together, Transparent Page Sharing and the balloon driver let ESX Server comfortably support memory overcommitment. You can learn more about our memory management technologies in this white paper. Now, getting back to our VM density test, how did ESX Server do? Here’s the screen shot:

VMware Infrastructure 3 with memory overcommitment supports 40 concurrent VMs!

Those 40 VMs have more than 20GB of total RAM allocated and they are running fine on a server with 4GB of physical RAM – a 5:1 memory overcommit ratio. Our exclusive ability to efficiently overcommit memory lets VMware Infrastructure support more than five times as many VMs on the same hardware as our competition! We repeated the test using Windows 2000 Server VMs running SQLIOSim to see how we fared with heavily loaded VMs. Hyper-V and XenServer both topped out at six and seven VMs again when they hit their memory limits, but the ESX Server platform ran fine with 14 VMs – twice as much as the other hypervisors!

Now, let’s get back to the cost per VM comparison to see which hypervisors provide the most bang for the buck. In the table below, we add up the costs for a basic hypervisor deployment. We’ll assume a 2-way, 4GB server costs us $6,000. Next, we add the costs to run Windows in each VM. For that, we’ll take advantage of Microsoft’s policy that lets us run an unlimited number of Windows VMs on a host licensed with Windows Server Data Center Edition (and yes, that policy also applies to VMware and Xen hosts.) Licensing Windows Server Data Center Edition costs us $5998 for two sockets. After that, we plug in the cost of the VMware Infrastructure 3 licenses, and to make things interesting, we’ll assume the competing hypervisor is absolutely free.

The next row in the table shows how many concurrent 512MB VMs each hypervisor can support. For VI3, we’re assuming a conservative 2:1 memory overcommit ratio based on our heavy workload test, which lets us run 14 VMs. For our hypothetical free hypervisor, we’re stuck at seven VMs because memory overcommit isn’t an option. That’s right, no other hypervisor technology allows memory overcommitment – it’s a VMware exclusive.

Finally, we do the division and find that even our high-end VI3 Enterprise bundle beats a free hypervisor in cost per VM! Going with any other hypervisor means you’ll need more hardware, network and storage connections, switch ports, floor space, power and cooling to support a given population of VMs. That should make your decision easy if all you’re doing is simple server consolidation, but there’s more to consider. VI3 Enterprise includes a powerful array of virtual infrastructure services like VMotion, DRS, HA and more that let you automate, optimize and protect your operations, and those features put us far ahead of the offerings from the Xen vendors and Microsoft.

If you’re ready to get started consolidating your servers, don’t be lured by seemingly low cost hypervisors into a decision that will limit your VM density and lock you into spending more on hardware. Instead, put memory overcommitment at the top of your list of hypervisor feature requirements. You’ll spend less on the project by stretching your hardware further and, since only VMware has memory overcommitment, you’ll get the proven reliability and advanced virtualization features of VMware Infrastructure thrown in for free. Beware the high cost of a “free” hypervisor.

[Update: More on VMware Memory Overcommit, for Those Who Don't Trust the Numbers]

Yankee Group recently came out with a report “Virtualization Price War: VMware’s Little Big Horn?” that unfortunately contained quite a few factual errors, inaccuracies, and unsupported claims. The report was not reviewed with VMware prior to publication (as some have claimed). Therefore, VMware is listing the factual errors and inaccuracies in the report because they distort the true cost and value of VMware's solution. Happy reading.

1. Exhibit 2: Double counting, omissions, and other issues

Exhibit 2 indiscriminately mixes VMware’s bundle and ala carte pricing, making it very difficult for readers to get an actual apples-to-apples cost comparison. If the table is meant to be a cumulative account of Microsoft and VMware pricing (i.e. add up all the rows), then there is significant double-counting in the VMware column. If the rows in the table are meant to be interpreted individually, then there are gross inconsistencies between how VMware and Microsoft’s costs are shown. Most casual readers will assume the table represents cumulative cost for both companies since they see the word “Included” throughout Microsoft’s column. The points below are written from the cumulative cost standpoint.

1. Double counting on cost of VMware: Row 1 of Exhibit 2 calls out the list price for VMware Infrastructure 3 Enterprise Edition ($5750) in the VMware column. However, the table continues by calling out the ala carte pricing for VMware individual capabilities that are ALREADY included in the price of VI3 Enterprise Edition (i.e. VMotion, DRS, HA, vSMP). This table effectively double-counts the cost for VMware. VMware offers ala carte pricing for customers who wish to purchase one of VMware’s other editions and add features individually. No one would buy Enterprise Edition plus many of the ala carte options in Exhibit 2 because most are included in the Enterprise Edition. There is a reference to a Mr. David Dodge on page 9 of the report that points out this issue but the feedback was apparently ignored.
2. Double counting on OS costs for VMware: The cost of a Windows Server Enterprise Edition License is counted twice for VMware – once under “Guest Licensing” and again under “Windows Server”. In this configuration, VMware products only require 1 Windows license (just like Microsoft), not 2. Again, Mr. David Dodge calls this out on page 9 of the report – “Dodge also argued that the VMware ESX3 solution can install on bare-metal and does not need an operating system to boot.” Therefore, the “Windows Server” for VMware should be $0 (the cost under “Guest Licensing” already covers it).
3. Wrong VMware product for a “single server configuration” comparison: The top of page 7 states that Exhibit 2 is a comparison of cost for a “single server configuration.” If that is the case, then why wasn’t VMware Server (further discussion in section 2 below) or the VMware Infrastructure Foundation Acceleration Kit used? The analyst is clearly aware of the acceleration kit because it is cited on page 6 of the report. Such bundles from VMware are specifically targeted at this segment. Also, if this is a single server configuration, when why would a customer want VMotion, DRS, HA which is only useful in a multi-server configuration?
4. Lack of clarity on management costs: Exhibit 2 fails to point out that a Microsoft System Center Enterprise Management Suite License (ESML) is $860 PER managed HOST (not including the mandatory cost of 2-years of Microsoft SA) while a VMware VirtualCenter Management Server license is a perpetual license independent of the number of managed hosts (VirtualCenter agents come with the VI3 license). By purchasing 1 VirtualCenter Management Server license, a customer can manage 5, 10, 15, 200, etc. number of ESX hosts. Also, where is meant by “Additional Charge $3,750 for Standard” (in the Management row)?
5. Error – Microsoft Virtual Server does NOT support vSMP: Microsoft Virtual Server does not support virtual SMP. That’s one of the key differentiators that Microsoft calls out between Virtual Server and Hyper-V.
6. Microsoft features not on-par with VMware features: Microsoft Virtual Server + SCVMM R1 offers nothing like DRS. SCVMM R1 can only recommend a physical server for initial placement when a VM is first provisioned. It has NO ability to dynamically move the VM (based on customer defined resource requirements) within a cluster of hosts to maximize resources. Also, Microsoft’s self-service provisioning is much less capable compared to VMware Lab Manager. For instance, Lab Manager handles self provisioning and management of multi-tier environments. SCVMM R1 offers nothing comparable.
7. Omission of key features in VMware VI3 Enterprise Edition: The report does not call out many capabilities that customers get if they purchase VI3 Enterprise Edition today. If the report is going to compare VI3 Enterprise Edition to Microsoft Virtual Server, then it should show all of the core capabilities of Enterprise Edition to provide readers an accurate picture of what they are getting. Omitted VI3 Enterprise Edition features include:
   • Storage VMotion (no MS equivalent)
   • Update Manager (no MS equivalent)
   • Guided Consolidation (no MS equivalent)
   • 64-bit guest OS support (no support in MS Virtual Server)

2. Why wasn’t VMware Server compared to Microsoft Virtual Server?

On page 7, the report states the following for why VMware Server was not used in this comparison.

"Customers do have the option of deploying VMware’s free hypervisor technology to even out the cost differential between VMware and competing server operating system-based hypervisors. However, according to the Yankee Group 2006 and 2007 Global Virtualization Surveys, which polled 800 end users worldwide, only 10% of corporations are deploying the free version of VMware for production networks." (Page 7)

This rationale is a double standard – what percentage of Microsoft Virtual Server deployments are currently in “production networks?” VMware’s internal research and anecdotal evidence suggests that MS Virtual Servers’s use in product environments is low. Certainly, it is not as high as the >85% of VMware customer deploying VI3 in production. At a bare minimum, a VMware Server vs. Microsoft Virtual Server comparison should have been included for comprehensiveness sake (since this is supposed to be an “unbias” report showing readers various cost options).

Also, comparing a Type 2 hypervisor (virtualization layer running on top of a host operating system), like Microsoft Virtual Server, with a Type 1 hypervisor (hypervisor running directly on top of the hardware), like VMware ESX Server, is a flawed comparison. Microsoft’s own plan to move away from Virtual Server and replace it with Hyper-V is evidence of the gap in the products. They know that a Type 2 hypervisor is architecturally dependent on the host operating system which impacts its performance and scalability. For instance, Microsoft’s customer references for Virtual Server (see Microsoft slides from TechEd 2007 as an example) consistently show a 3:1 to 5:1 consolidation ratio. On the other hand, many of VMware’s public customer references for ESX Server (see www.vmware.com/customer) achieve 10:1 and even >20:1 consolidation ratios.

3. Inaccurate reporting in the Security section

Inaccurate reporting of security bulletin: The two sentences below, as written, are misleading. The vulnerability to break out of the guest system to the host machine is only an issue on VMware’s hosted products (ex. Workstation, VMware Server). The Yankee Group report misleads readers to believe that the vulnerability affects VMware ESX Server.

"In late September, the online security alert service Bugtraq reported that VMware issued fixes to patch 20 security vulnerabilities that affected almost every product in its portfolio, including the VMware ESX Server, the VMware Server, VMware Workstation, VMware ACE and VMware Player. These vulnerabilities could have allowed hackers to break out of the guest system in the virtual machine and to halt processes on host machines." (Page 12)

Another inaccuracy in reporting: The report makes another inaccurate claim about a recent security bulletin (see quote below). First, the 20 security flaws in sentence 1 apply to both VMware hosted and ESX products. Of the 20, 5 applied to VMware’s hosted products. Therefore, the 18 attributed to ESX Server in sentence 2 is wrong. Second, only 3 (out of the 15 flaws that are applicable to VMware’s datacenter product ESX Server) were rated as high severity or critical. The majority of the others are low criticality. This fact would have been readily apparent by clicking through to view the details of the Secunia report. (Secunia summary bulletins only show the worst threat level among all the patches.)

"VMware released a patch to correct more than 20 security flaws in its various products. According to an alert issued by Secunia, a Denmark-based security service provider that tracks security flaws in software and operating systems, 18 were described as highly critical flaws that could impact just about every version of the ESX Server." (Page 12, emphasis added)

Apples-to-oranges: Comparing a hosted to a bare-metal product: Comparing vulnerabilities in Microsoft Virtual Server 2005, which is a hosted product, to all of ESX, which is a bare-metal (hypervisor) product, is a totally apples-to-oranges. If Yankee Group wants to compare to Microsoft’s hosted product, then it should compare bugs to VMware Workstation or VMware Server. If Yankee Group is going to compare a hosted product in the datacenter to a bare-metal product, it should include all the vulnerabilities in the computing stack. For Microsoft Virtual Server 2005, it needs to run on Windows Server 2003 and uses IIS. So how many critical Windows Server 2003/IIS vulnerabilities have there been since Jan 2007?

4. Other inaccuracies and unsupported claims

Have Citrix XenServer list prices really gone down?

"And in response to the increasing competition, Citrix in January announced new pricing for its offerings that represent a 20% cut." (Page 12)

As best as we’ve been able to track, it appears that Citrix XenServer list prices have actually gone up slightly, not down. At best, they’ve stayed about the same. Here’s our understanding of how Citrix XenServer list prices have changed.

Prices below are for two-socket hosts.

Confusion between Microsoft Virtual Server and Hyper-V

"Microsoft leveraged its greater economies of scale by embedding Virtual Server into the Windows operating system free of charge." (Page 7)

Microsoft Virtual Server R2 SP1 is not embedded into the Windows operating system. It requires a separate download and installation. This quote confused the architectures of Microsoft Virtual Server and the upcoming Microsoft Hyper-V.

Does this pricing comparison with Red Hat make sense?

"[Red Hat] retail price tags are 35% to 70% less than the same competing products from VMware." (Page 13)

Red Hat is sold only on a subscription basis and VI3 is sold in perpetual per-socket licenses, so a comparison like this is invalid unless the users plan to expire the machines in 12 months.

Who really has broader hardware support…

"’We’re way ahead of VMware in our support for I/O virtualization, native 64-bit HA as well as the total number of CPUs supported and we support a much broader set of hardware,’ Crosby claimed." (Page 16, emphasis added)

A check of VMware’s HCL and Citrix XenServer’s HCL shows that Simon Crosby’s claim about broader hardware support is unsupported. For example, VMware 3.0.2 HCL supports over 400 server platforms. XenServer’s HCL only shows 43 servers directly tested by Citrix. Even if the server systems tested by vendors are added, the Citrix HCL grows to 55 servers. Mr. Crosby's other 2 claims about I/O virtualization and 64-bit HA are pretty unsubstantiated as well.

Out-of-date information on VMware VI3 capabilities

"The Infrastructure Enterprise software also includes some new features… DRS… HA… VCB." (Page 15)

VMware has had DRS, HA, and VCB since June 2006. On the other hand, nowhere does the report mention the truly new features of VI3 (version 3.5) that were announced in Sept 2007 and GA’ed in Dec 2007. Readers are not getting current information from this report.

VMware VirtualCenter has had intelligent placement since VI3 in June 2006.

"He added that the functionality in VMware’s corresponding virtualization management offering falls far short of the functionality in Virtual Machine Manager. ‘VMware’s Lab Manager development tool focuses on desktops, not servers and even their new Virtual Data Center tool does not incorporate the intelligent placement feature of the Microsoft product,’ he said." (Page 11, emphasis added)

VMware VirtualCenter has had intelligent placement capabilities (for when a new VM is provisioned) since June 2006 with the launch of Virtual Infrastructure 3.

ZDNet posted a blog reviewing Microsoft Hyper-V beta “Review: Microsoft’s Hyper-V puts VMWare and Linux on notice” (February 14, 2008). The posting had some factual errors and other claims about VMware ESX Server – see below for a summary. The author of the blog has acknowledged the errors in the comments section of the blog but since most readers won't read through all the comments, here is a summary of the correct statements about VMware ESX Server as well as counterpoints to various claims.

Claim: VMware ESX Server does not support SATA disks.

"… SATA disk drives, which are now commonplace on commodity x86 server machines, are not currently supported in VMWare ESX 3."

Correction: VMware ESX Server 3.5 supports SATA disks. The author has acknowledged this in the comments section.

Claim: VMware ESX Server always requires VMFS and a dedicated SAN-based LUN to store virtual machine images.

“ESX Server also requires a special networked clustered file system known as VMFS to store the virtual machine images, and you have to dedicate a SAN-based LUN to it.”

Correction: ESX Server provides four options for VM storage: 1) Fiber channel (FC) attached, 2) iSCSI attached, 3) NFS, and 4) local storage. Only the FC and iSCSI options require a SAN-based LUN. Also, the NFS option does not use VMFS. Again, the author has acknowledged this error in the comments section. Note that Citrix XenServer and Microsoft Hyper-V also require shared storage to perform VM migration and other high availability functions.

Claim: VMware is a closed system. Therefore, VMware development goes at a much slower pace. VMware’s lack of SATA disk support is an example of this slower pace.

“VMWare keeps its ESX hypervisor source code very close to the vest, so development goes at a much slower pace – SATA disk drives, which are now commonplace on commodity x86 server machines, are not currently supported in VMWare ESX 3.”

Counterpoint: VMware’s purported lack of SATA disk support has already been refuted (see above). In addition, VMware has delivered more industry-breaking innovations around x86 virtualization compared to anyone else. As examples, VMware was the first to deliver virtual SMP, live migration (VMotion), centralized VM management (VC), dynamic load balancing (DRS), automatic VM restart after host failure (HA), next-generation thin embedded hypervisor (ESX Server 3i with a 32MB disk footprint), and Storage VMotion. The majority of other vendors still have not delivered these capabilities. VMware has also provided ESX Server source code to partners as part of the VMware Community Source Program. One good example of the output from the Community Source Program is the Mellanox Infiniband support.

Claim: VMware ESX Server’s hardware compatibility list (HCL) severely limits a customer’s hardware choices while Microsoft and Xen do not have this issue.

"However, [ESX Server] has a much tighter environment as to what kind of hardware it can run on – the hypervisor has a limited device driver compatibility list…"

Counterpoint: Both Microsoft Hyper-V beta and Xen-based products (ex. Citrix XenServer, Virtual Iron) have HCLs as well. Click here to see Microsoft’s list of hardware platforms “suitable” for Hyper-V beta. Click here to see Citrix XenServer’s HCL. Click here to see Virtual Iron's HCL. The Microsoft, XenServer, and Virtual Iron HCLs are actually more limited than VMware’s HCL. For instance, Microsoft has directly tested Hyper-V beta on 8 servers (TBD what the hardware requirements will be for Hyper-V GA). Citrix has directly tested 45 servers. Virtual Iron has directly certified 29 servers (for version 4). In contrast, VMware has over 400 servers on the ESX Server 3.0.2 HCL. Click here to see VMware's HCL.

Claim: VMware ESX Server is “pure software based virtualization”

"VMWare’s ESX differs from Hyper-V and Xen in that it currently uses pure software based virtualization…”

Counterpoint: This statement can potentially mislead readers to think that ESX Server is based on software emulation technology. This conclusion would be false. For the bulk of x86 CPU instructions, ESX Server does direct execution on the physical CPU. Readers may also interpret this statement to say that VMware is somehow philosophically opposed to using CPU hardware-assist. Again, this would be a false conclusion. VMware takes a pragmatic approach and uses whichever technology delivers the best performance. Currently VMware binary translation has advantages over today’s generation of CPU hardware-assist. Click here to read a VMware whitepaper on this topic. In regards to the future, VMware is already working closely with Intel and AMD on their next-generation of CPU hardware-assist technology.

I have seen presentations and documents, articles and blogs all talking about how these other virtualization technologies have the same features as we do. Citrix says their version of Resource Pools are comparable to ours. Microsoft says our VMotion is comparable to their "Quick" migration technology (where you incur downtime to the application). They do this to try and get that all important check mark (√) on the table to make them look equivalent to VMware Infrastructure 3. If you dig down into it, and what the check mark actually means, you will see they can't honestly place that mark in there.

For example, let's look at Resource Management of your Virtual Infrastructure. Citrix claims that because they have this capability called Resource Pools they are equivalent to VMware Infrastructure 3, but to be honest, their Resource Pools are nothing more than common configuration management as stated in their product overview:

Pool-based configuration - Common settings can be set and applied automatically on a pool-wide basis, simplifying reconfiguration.

Resource Pooling is the ability to aggregate the entire resources (CPU and  Memory) of a cluster of servers. From there, you can divvy these resources between different child pools while continuously optimizing the virtual machines' utilization of these resources across physical hosts. Our Resource Pools allow you to guarantee say 70% of the processing power to the production VMs, but allow it to grab the additional 30% of the cycles if necessary while guaranteeing that your staging environment (also hosted in the same cluster) never takes more than the 30% you have given it.

On the Microsoft front, they imply that their "Intelligent Placement" is resource management, or the equivalent of our Distributed Resource Scheduling as detailed here:

Intelligent Placement -Selecting the appropriate virtual machine host for a given workload is the key to maximizing the utilization of physical assets, whether the organization’s goal is to balance loads among existing hosts or to maximize resource usage on each host. In Virtual Machine Manager, this process is called “Intelligent Placement.”

But Distributed Resource Scheduling (DRS) is MUCH more than deciding where to place a virtual machine when you create it. What happens after that virtual machine has been running? The load on the system changes, perhaps this virtual machine, or another needs more resources, with DRS, we will redistribute this workload across all physical systems in the cluster on the fly without ANY downtime to the application

So as you can see, they are trying to marginalize our advances by saying their minimal features are equivalent to our feature rich and fully baked products. Take a look at the table below and see if their capabilities warrant a check.

To be honest, though, check marks aren't enough. If we were simply selling "Speeds and Feeds" you would see that we blow the competition away, but at VMware, we aren't creating features, we are providing solutions. We are helping customers  solve their real world problems that go way beyond server consolidation. We are leading the way to simplified Hardware and Software Maintenance, to more efficient Storage and Image and Power Management, to lessening the effects of Downtime and Availability, and to allowing for better Business Continuity. We provide tried and true solutions for Virtual Desktop, VM Lifecycle Management and Lab Self-Servicing.

You tell me, isn't that what you really want?

Today was my first day in the VMware booth at MacWorld 2008. Man was it crowded! What a show so far with Jobs announcing 4 great new technologies (my Apple TV thanks you). What really peaked my interest was all of the people coming by and asking what made VMware Fusion so much better than Parallels for Mac. Ed Baig from USA Today even got me on film talking about this very subject. So here's my personal (non corporate marketing) rundown on why I think we're better than Parallels. For what it's worth I used Parallels for a long time since VMware didn't have anything on the Mac. I loaded up Fusion since I work for VMware and decided to give it a try. The points below are what got me to switch. Yeah, I know, you say I would have switched anyways since I work for the company. Not true. I'm a technologist and use what works better. Anyhow, here's the list.