Virtualizing XenApp on XenServer 5.0 and ESX 3.5
There has always been interest in running Citrix XenApp (formerly Citrix Presentation Server) workloads on the VMware Virtual Infrastructure platform. With the advent of multi-core systems, purchasing decisions are driven towards systems with 4-16 cores. However, using this hardware effectively is difficult due to limited scaling of the XenApp application environment. In addition to the usual benefits of virtualization, these scaling issues make running XenApp environments on ESX even more compelling.
We
recently ran some performance tests to understand what can be expected
in terms of performance for a virtualized XenApp workload. The results
show that ESX runs common desktop applications on XenApp with
reasonable overhead compared to a native installation, and with
significantly better performance than XenServer. We hope this data will
help provide guidance when XenApp environments are transitioned from physical hardware to a virtualized environment.
Together with partners, we have been developing a desktop workload for over a year. The workload has been tested extensively on virtual desktop infrastructure (VDI) environments with one user per virtual machine (VM). VDI results have been presented and published in numerous locations (e.g., http://www.vmware.com/resources/techresources/1085, VMworld 2008 presentation VD2505 with Dell-EqualLogic). Great attention was paid to selecting the most relevant applications as well as to specifying the right types and amount of work each should do. Many other Terminal Services-style benchmarks fail to be representative of actual desktop users. Porting the workload from a VDI environment to the XenApp environment was straightforward.
XenApp was run in a single 14 GB 2-vCPU Virtual Machine (VM) booted with Windows Server 2003 x64. The hypervisors used were ESX 3.5 U3 and XenServer 5. The VMs for both had the appropriate tools/drivers installed. The XenServer VM had the Citrix XenApp optimization enabled. For comparison, the tests were run natively with the OS restricted to the same hardware resources. The hardware is a HP DL585 with 4 quad-core 2210 MHz “Barcelona” processors and 64 GB memory. Rapid Virtualization Indexing (RVI) was enabled.
The test consists of 22 operations, always executed in the following order:
IE_OPEN_2 Open Internet Explorer IE_ALBUM Browse photos in IE EXCEL_OPEN_2 Open Excel file EXCEL_FORMULA Evaluate formula in Excel EXCEL_SAVE_2 Save Excel file FIREFOX_OPEN Open Firefox FIREFOX_CLOSE Close Firefox ACROBAT_OPEN_1 Open PDF file ACROBAT_BROWSE_1 Browse PDF file PPT_OPEN Open PowerPoint file PPT_SLIDESHOW Slideshow in PowerPoint PPT_EDIT Edit PowerPoint file PPT_APPEND Append to PowerPoint file PPT_SAVE Save PowerPoint file WORD_OPEN_1 Open Word file WORD_MODIFY_1 Modify Word file WORD_SAVE_1 Save Word file IE_OPEN_1 Open Internet Explorer IE_APACHE Browse Apache doc in IE EXCEL_OPEN_1 Open Excel file EXCEL_SORT Sort column in Excel EXCEL_SAVE_1 Save Excel file
The two horizontal lines labeled “QoS” denote the Native latency for 35 and 38 users. Either of these may be considered as a reasonable maximum Quality of Service for latency. They correspond to somewhat less or more, respectively, of half of the available CPU resources (see the CPU figure below), which is a commonly used target for XenApp. At higher utilizations not only does the latency increase rapidly but operations may start to fail. We required that all operations succeed (just like a real user expects!) for a test to be deemed successful. The points where the QoS lines cross the ESX and XenServer curves gives the number of users that can be supported with the same total latency. Normalizing with the number of Native users (35 or 38) gives the fraction of Native users each virtualization product can support at the given total latency:
ESX consistently supports about 86% of the native number of users, while XenServer supports about 77%. Shown below is the average CPU utilization during the second to fifth iteration of the last user, given as a percentage of a single core. Perfmon was used for Native, esxtop for ESX, and xentop for XenServer. ESX uses less CPU than XenServer no matter how the comparison is made: for a given number of users, or for a given total latency:
XenApp and other products that virtualize applications are prime candidates to be run in a VM. These results show that ESX can do so efficiently compared to using a physical machine. This was shown with a benchmark that: represents a real desktop workload, uses a metric that includes latencies of all operations, and requires that all operations complete successfully. Furthermore, ESX supports about 13% more users than XenServer at a given latency while using less CPU.
Posted by Jeff Buell on January 30, 2009 | Permalink
Just curious, what about memory? How did XenApp behave vs ESX in terms of TPS.
Posted by: Duncan | January 31, 2009 at 04:37 AM
euuhm I mean XenServer of course.
Posted by: Duncan | January 31, 2009 at 04:38 AM
Playing the devil's advocate here clearly native works far better than ESX or XenServer. Why on earth don't we just leave the Citrix Farms alone? Same may apply to Oracle [RAC] and other I/O intensive apps.
Why are we virtualizing anyways, just push them all in the Cloud and let Grid, Clusters, and all other concoctions handle the workload. Its all about the workload and SLA, right?
Disclaimer: I love and fully propose virtualization to firms that are madly buying hardware because some moron architects and other vested political interests who want to spend the tax dollars of citizens [we're talking about Gov installations I've come across in the past 8 years of mad consulting in Virtualization in across the planet] because "the budget for this year must be finished no matter how irresponsible I will be with those tax dollars".
Having said that I prefer to also hold the discussions with those who just don't get it and want to go ahead with the apps as they are. I've been in the industry for almost two decades and have been in the middleware before we started the virtualization game.
Posted by: Tarry Singh | January 31, 2009 at 08:17 AM
disclaimer, I am an employee of VMware, but the opinions in this comment are my own.
Tarry, I think the reason people are starting to move to virtualizing their XenApp and Oracle and other applications is due to the fact that today the hardware is beginning to outpace the software. Meaning, I don't get much major benefit going from say a 4 core system to an 8 core system (I certainly don't get two times the performance). So as I compare like configured boxes (WRT to hardware specs), I do have some performance overhead with a virtualized solution, but on a current architecture box, I can get more VMs that can do different things on the same box and am not wasting resources.
Posted by: Kenon Owens | January 31, 2009 at 10:34 AM
Duncan,
TPS? The test used only about 7 GB, so none of the platforms were under memory pressure.
Tarry,
Kenon is right. Many people don't even use more than 2 CPUs for XenApp. So even with some virtualization overhead, you'll come out way ahead by running several XenApp VMs on a single box.
Posted by: Jeff Buell | January 31, 2009 at 12:04 PM
And let's not forget that virtualization is not just about consolidation. Virtualization is a valuable mechanism to increase availability, perform workload leveling, enable rapid scale out, and simplify disaster recovery. I've seen customers virtualizing workloads that utilize an entire physical host for just these reasons.
Posted by: Bryan Salek | February 01, 2009 at 06:14 AM
Although VMWare still holds overall a better performance than XenServer and HyperV when speaking about regular server virtualization, this is not true concerning XenApp servers. All the papers I have read so far from independant vendors/companies demonstrated that. You can check for example the papers written by www.virtualrealitycheck.com. We've made two tests with customers of ours and each time we managed to put more users on the XenServer machine than on the VMWare one.
Posted by: Jack | February 02, 2009 at 01:27 AM
I don't think these results are very objective.
The are:
http://www.virtualrealitycheck.net/
And these results are slightly different...
Posted by: Nico | February 02, 2009 at 05:42 AM
I can tell you that these results are exactly opposite of what we are seeing in production environments. XenServer far outperforms ESX, even for MSOffice users.
Posted by: Robert | February 02, 2009 at 06:05 AM
I think there is a lot of improvment for vmware since they support the RVT (AMD) feature for huge memory applications.
As everyone know XenApp and W$ are using lots of memory calls ...
qed
Posted by: Pulporange | February 02, 2009 at 08:25 AM
I confirm Nico's comment - we did production testing with users simultaneously accessing XenApp as they do in the real-world (i.e. not scripting one test after one completes as in your scenario) showed that XenServer *far* outpaced ESX... it wasn't even close even after patching/tuning. I don't deny you captured these results but the methodology is flawed or purposely devised. I'm a VMW biggot but our testing came to be from collegues at two different organizations who confirm similar results. We still like ESX and I don't see it going away anytime soon but XenServer is, simply put, the better choice for XenApp.
Posted by: Michael | February 02, 2009 at 10:40 AM
Wrong reference.. meant Robert.
Posted by: Michael | February 02, 2009 at 03:28 PM
http://community.citrix.com/blogs/citrite/simoncr/2009/02/02/VMware+Wins!+(Bad+Science+Required)
Need I say more?
Posted by: Alex | February 03, 2009 at 12:11 PM
We have done our own testing similar to this just with ESX vs. native and our results were very different. In our test the physical server (native) far far far out performed the single VM. To me this data showing a single VM and native being so close is laughable.
Posted by: Kevin | February 04, 2009 at 09:26 AM
Considering the fact that VMware insist on reviewing test methodologies before anyone publishes test results for ESX performance; did the testers consult with Citrix and Microsoft before releasing these results?
Thought not.
Posted by: Simon Bramfitt | February 04, 2009 at 09:36 AM
Thanks for all your comments. I'm going to turn off comments here and invite everyone to continue the discussion over at VMware Communities http://communities.vmware.com/thread/193473
Posted by: John Troyer | February 09, 2009 at 07:56 PM