We Reviewed Microsoft’s ROI/TCO Model
Some of you may have seen Microsoft’s recently released virtualization ROI/TCO calculator. Briefly, the model purports to offer an accurate cost/benefit comparison between Microsoft’s Hyper-V offering and a “Competitive Server Virtualization Solution” – gee I wonder who the competitive solution is…? Microsoft is beginning to advertise the calculator broadly in its partner newsletter and other email blasts – and we’ve even had customers bring it to our attention. Mainly, the VMware customers that have alerted us to the Microsoft ROI/TCO calculator were confused by many of the model’s assumptions and by the generated results – they wanted our opinion. So, we took a look.
Unfortunately We Had to Give It A Failing Grade
Of course the results were all hypothetical, because Hyper-V is not yet available, but what we found when running a realistic scenario through the model and then from reading the report’s fine print, is that like most Microsoft version 1.0 products, the initial release of this calculator has numerous errors, contains critical design mistakes, and completely misses its mark. Any results generated from this model are so unrealistic as to be completely worthless for accurately comparing costs and benefits of alternate virtualization solutions. (Maybe we all need to wait for the SP1?)
In Sum: ROI/TCO Analysis = Good Idea ; Inaccurate Model = Bad Idea
But of course that’s what one would expect VMware to say right? Out comes a model that could show that a competitor’s product is less expensive and right away VMware is going to question its accuracy. Well I’ll soon list out the numerous issues we found with the Microsoft model – and those issues will articulate clearly why Microsoft’s model is inaccurate and why our questioning of the model is justified, but first just let me state that we at VMware fully support an accurate and realistic ROI/TCO analysis of our solutions. In fact, VMware launched an ROI/TCO calculator way back in April of 2007. Since then, our customers have generated over 30,000 reports that quantify the enormous benefits they are receiving from VMware installations. (Feel free to try it for yourself here.) However, we also believe, and I am sure you’d agree, that any ROI/TCO model must be based on 1) accurate pricing/licensing information, 2) fair product comparisons, and 3) defendable assumptions for both associated costs and realized benefits. Our analysis of the Microsoft supported model showed that it had none of those traits.
Play Along at Home
Following are some of the many mistakes and gross assumptions we found in the Microsoft model. So that you can follow along at home – please download a copy of the report we generated. I encourage you to evaluate the Microsoft calculator yourself – let us know what else you find! Also, as you read further, and as you consider all the errors in the model, keep in mind the following question:
Why Did MSFT Release Such a Misleading ROI/TCO Model?
A) Microsoft did a sloppy and hasty job with the calculator
B) Microsoft is deliberately fudging the facts
C) Both A and B
Now I don’t want to imply that Microsoft is trying to deliberately mislead anyone, but it does make me question Microsoft’s overall capacity for rigorous product testing. I hope that the same team that QA’d the Microsoft TCO model is not the same group that is currently testing the V1.0 release of Hyper-V!
Perhaps we at VMware could work with Microsoft and a third party to develop a standardized model for an ROI/TCO analysis of a virtualization solution? Thoughts?
Production & Dev/Test Server Virtualization – Competitive Cost Comparison
(Appendix B – page 18)
Why is a discounted Microsoft license cost being compared to VMware List Pricing??
The tool assumes the following pricing for Windows Server 2008 Editions: $719 for Standard, $2334 for Enterprise, and $2381 for Datacenter. That doesn’t match up to the list pricing that Microsoft publishes on its website, so the prices must be some sort of volume/discounted pricing. So the tool uses discounted pricing for Microsoft and list pricing for VMware ($5750 for VI3 Enterprise)? Not exactly an apples-to-apples comparison. One might argue that Microsoft does not have access to VMware volume pricing. But then again, the average reader doesn’t have access to Microsoft’s volume pricing either.
Where are the Microsoft System Center Server License Costs?
Under the “Virtualization Management Software” row, the VMware column includes the server license cost for VirtualCenter $5000. That’s fine, but to the best of our knowledge, an organization using the MSFT solution would also need server licenses, for 1) Microsoft System Center Operations Manager, 2) System Center Configuration Manager, and 3) System Center Data Protection Manager. Each component is $573 (list price) according to Microsoft’s own website. That $573 doesn’t include the SQL Server license – that’s extra. Currently, no additional server license cost is required for System Center Virtual Machine Manager 2007 – TBD whether that will remain true for Virtual Machine Manager 2008. System Center Operations Manager is required to do any performance tracking, Configuration Manager to do updates and patches, and Data Protection Manager to do backups. Most of that functionality is an integrated part of VMware VirtualCenter.
Since When Did All VMware Customers Decide to Run System Center?
The tool assumes that customers will use Microsoft System Center to manage their VMware Infrastructure deployment. Again, this little detail is buried deep in the appendices (Appendix B – “System Center Enterprise” row). Microsoft assumes that everyone who still chooses to run VMware will naturally want to manage it using Microsoft System Center. This seems like a pretty presumptuous assumption. Most companies we talk to have already invested in an enterprise management framework and want VMware to integrate with what they’ve already got. Unless they are already running Microsoft System Center, they really don’t want to rip out their existing investment and put in System Center. VMware VirtualCenter APIs are readily available through the VirtualCenter SDK making it possible for VMware partners like IBM, HP, CA, BMC, Symantec, Quest, and NetIQ (to name a few) to integrate seamless with VMware VirtualCenter. Customers can keep using what they’ve already invested in.
The Model Incorrectly Calculates Microsoft Licensing Costs
In the scenario we ran, Microsoft’s tool assumed 71 Windows Server Standard Edition licenses for 414 virtual machines running on 71 hosts. Since each Standard Edition license grants rights to run 1 VM, the model’s results leave 343 VMs in our hypothetical datacenter running out of compliance. Microsoft may claim that the TCO/ROI calculator is not a licensing calculator, but how can it calculate accurate TCO estimates using inaccurate licensing assumptions? We did find a one-line disclaimer buried in the 66-page document: “Warning – Check pricing advice and rules as the automated recommendations here may not reflect all licensing rules.” Come on, guys – licensing is such a basic component for accurate TCO estimates. The disclaimer feels pretty weak.
Microsoft Cost Savings are HUGE… if You Believe the Assumptions
We found many overly aggressive assumptions in the Microsoft calculator, which resulted in overly aggressive cost savings for the Microsoft’s solution. For instance, on page 26 “Production Server Operations and Administration Efficiency Savings”, Microsoft’s default assumption is an 83% reduction in person-hours-per-year after virtualizing with Microsoft’s solution. That translates into a VM-to-admin ratio of 250:1 after virtualization. While VMware agrees that virtualization absolutely increases administrative efficiency, 250:1 is overly aggressive.
Why are 4-way Servers the Model’s Default Assumption? Oh yeah… It Increases VMware’s Perceived Cost.
The tool assumes 4-way servers as the default configuration. Granted, 4-way server volume is growing, but IDC’s Quarterly Server Tracker published on February 26, 2008, shows only 4.27% of all servers shipped in 2007 were 4-ways, making them nowhere close to being the majority of servers virtualized today. This assumption does double the number of VMware Infrastructure 3 (VI3) Enterprise Edition licenses required since VI3 is licensed in 2-socket increments, thereby making VMware’s total deployment cost appear much higher. Most readers won’t see this point buried deep in the appendices; perhaps that was the goal? Oh, with quad-cores today and hexa/octo-cores in the near future, 2-way servers are still projected to be the majority volume virtualized server platform in 2008 and 2009.
VM Density Per Host Matters for TCO – a Fact that Microsoft Ignores
Not all virtualization platforms are created equal – some can run more virtual machines per host than others, thereby resulting in lower total cost of ownership, due to reduced hardware and related costs. In one of our recent blogs, we highlighted the lack of memory sharing technologies in Microsoft Hyper-V. This deficiency results in fewer virtual machines per host (due to inefficient memory usage), thereby increasing the number of required physical servers, thereby increasing cost. Microsoft’s tool assumes that Hyper-V will run as many VMs as VMware VI3 and deliver the same performance – we can’t wait until Hyper-V ships and prove this wrong.
VMware VI3 Enterprise Functionality is Far Greater than Hyper-V
Microsoft compares Hyper-V to VMware VI3 Enterprise Edition even though Hyper-V lacks key enabling capabilities that are prerequisites for a truly dynamic datacenter. Capabilities missing from Hyper-V include live VM migration (VMware VMotion and Storage VMotion), automated/dynamic load balancing (VMware DRS), dynamic power management (VMware DPM), integrated offline VM patching (VMware Update Manager), a next-generation ultra-thin hypervisor architecture (VMware ESXi), a clustered file system (VMware VMFS), and the ability to make VMs highly available without the storage management headache of assigning 1 LUN for each VM (VMware HA). For the basic server consolidation scenarios that Microsoft Hyper-V enables, VMware offers VI3 Foundation and VMware ESXi at very competitive price points – check out VMware’s complete server virtualization product offerings here.
Desktop and Application Virtualization – Competitive Cost Comparison
(Appendix B – page 19)
Microsoft Tool Assumes Wrong Costs for VMware VDI
To virtualize 6000 clients, Microsoft has incorrectly assumed 240 VI3 Enterprise licenses ($1,380,000) plus “Virtualization Management Software and Host Server” cost ($7000) plus 6000 connection broker client licenses ($300,000). The actual license cost of VMware VDI is 6000 x $150 = $900,000 plus virtualization management costs of $1250. This VMware VDI pricing has been in effect since Feb 2008 so the Microsoft calculator does not use up-to-date, publicly available information.
Readers Must Figure Out that Microsoft Compares Two Completely Different Technologies
By looking at the summary table on page 6, most readers would reasonably assume that the comparison is between two comparable products addressing the same business need. That’s not the case. Microsoft is comparing the cost of Terminal Services + MDOP (including Microsoft Application Virtualization) to the cost of a VMware VDI solution. VMware VDI offers full desktop VM virtualization, best in class application compatibility and true user isolation. Comparing terminal services + application virtualization to VDI is an apples-to-oranges comparison.
Microsoft Assumes Low Consolidation Ratios for VMware VDI
The consolidation ratios assumed by Microsoft for VDI are significantly lower than the consolidation ratios observed by VMware customers. On a 2-socket dual core server, VMware customers typically observe average consolidation ratios of 32-36 VMs depending upon the workloads. Microsoft only assumes 25 VMs per host, which skews VMware’s cost by another 20-30%. Of course, Microsoft assumes a higher consolidation ratio (30:1) for its own solution.
Questionable Assumptions for How Microsoft’s Desktop Solution Lowers Cost
We found some questionable assumptions on how Microsoft got some of the cost-savings estimates. For instance on page 39 “Desktop Virtualization Client Software License Cost Avoidance“, Microsoft claims that companies can reduce the number of “unused / not properly allocated licenses” by 70% using their desktop solution. Apparently, overbuying of Microsoft software licenses is a common issue. Another example is on page 37 “Desktop Virtualization Branch Office Server Cost Avoidance.” Microsoft claims that some branch office server infrastructure may be consolidated / retired and replaced with centralized virtualized servers for branch office applications. While this is one viable way to reduce IT capital cost, what about the additional IT capital costs to upgrade to low latency, highly available network connectivity to the branch offices? This network cost needs to be factored in.