VMware: Virtual Reality

If you haven't already done so, now might be a good time to surf over to the Project Virtual Reality Check site (registration required).  No, it's not a web page dedicated to proofreading this blog.  It's run by a team of virtualization consultants that have been publishing comparisons of VDI and Terminal Services performance in various hypervisors.  They made a big splash back in January with their first set of reports comparing ESX 3.5, XenServer 5.0 and Hyper-V R1.  Our exclusive ability to overcommit memory made ESX the clear winner in the VDI tests with support for more than twice as many VMs as the others.  On the Terminal Services tests, it was XenServer that came out ahead in the number of user sessions it could handle.  It wasn't good news for VMware, but the story doesn't end there.

Nothing gets us more worked up than finishing second in a product comparison, so our performance team started digging into the Project VRC benchmark to see what might be going on.  Right off the bat, we saw that ESX wasn't configured to use the hardware assist for memory management (AMD RVI) that was present on Project VRC's servers.  ESX 3.5 was the first hypervisor to support hardware assist for memory management and we certainly wanted ESX to look its best.  The Project VRC guys updated their ESX test report in March and the findings showed that ESX narrowed the XenServer lead in Terminal Services user sessions by half once RVI was enabled.  It was an improvement, but we still weren't ready to accept second place.

The Project VRC results prompted us to do some rigorous testing of ESX performance with XenApp -- a workload based closely on Terminal Services -- and our findings convinced us that we could support more user sessions, so what was going on with the Project VRC tests?

It turns out that the Project VRC team was also taking a second look at their benchmark.  Our performance team (and also the experts at Citrix) collaborated with Project VRC's test architects and all groups came to the conclusion that timing issues were skewing the results.  Use of in-guest timers, one of the classic demons in virtualization benchmarking, was identified as a problem area.  You can see our take on it here.

The Project VRC team dug in and developed a completely new Terminal Services benchmark that reduced use of in-guest timing, when compared to the first version.  The new workload created by Project VRC still depends on in-guest timing, but is a big improvement over the first version and we're grateful for Project VRC's efforts and flexibility.  VMware was also hard at work since the first Project VRC tests improving performance in vSphere.  Our own testing demonstrated a 30% improvement in XenApp throughput between ESX 3.5 and 4.0, so we expected a strong showing in new test runs.

When the first Project VRC comparisons came out, we saw Citrix promote them heavily, especially with customers that were deciding on a virtualization platform for their XenApp servers.  XenApp and Presentation Server customers have been rapidly moving those servers into VMware virtual machines and it's understandable that Citrix wanted some independent test results that might slow down what they saw as defections and keep those XenApp servers on their hypervisor platform.

If you're a XenApp user making a virtualization platform choice you need to read the new Project VRC paper that came out last week NOW.  The paper is titled, "VRC, VSI and Clocks Reviewed".  The results have shifted dramatically since the January findings.  Instead of trailing, ESX 4.0 now has a 4% advantage over XenServer 5.5 in the number of Terminal Services users it can support on a server. 

We attribute the turnaround for vSphere in the latest tests to the Terminal Services/XenApp performance enhancements we made in ESX 4.0 and improvements made by Project VRC to their benchmark.  These new results will come as big relief to XenApp users that felt pressured to virtualize on XenServer after seeing the old Project VRC numbers.  When presented with test findings that seemed to show XenServer could support more users per host, it was harder to choose the proven reliability and richer feature set provided by VMware vSphere.  We certainly hope no customers were lured into making the wrong choice based on test results now known to be flawed and outdated, but those of you yet to make that decision will be happy to know that the latest Project VRC comparisons make the selection of vSphere a no-brainer.