VMTN Blog

Two new articles from VROOM!, the VMware performance team blog:

Getting the best performance from Workstation 6.0

On top of all the goodness Workstation 6.0 has to offer, we now have a revised performance document as well. You can find it here. We've tried our best to provide tips not only about things that you could tune with Workstation and its features, but things you could tweak on your host operating system and within the guest, and even what to look out for on the hardware level. The reorganization of the document makes it easy to find what you need to achieve the best possible experience.

Windows Vista Performance in VMware Workstation 6.0

We ran a set of workloads to measure the CPU, memory, disk and network performance of the VM. Vista host performance is on par with XP, except that Vista itself consumes more memory than XP. This means that Vista leaves less memory for the use of VM's than XP. ...

While Vista guest performance is on par with XP in most of our workloads, we did find a few cases that perform worse on the Vista VM than on the XP VM. To understand why Vista was slower in those particular cases, we conducted the same measurements on native physical systems, rather than on virtual machines. We found that Vista is slower than XP on native hardware almost to the same degree as on virtual hardware. This made it clear that VMware Workstation 6.0 wasn't introducing any Vista-specific overheads, and that the relative performance on Vista is as good as on XP.

From VMware's Eric Horschman, The benchmarks are here, but don't lose sight of the big picture:

If you’re considering virtualization, make intelligent use of the performance data that VMware, our partners and our competitors are providing, but don’t get overly enamored with the benchmarks.  The bigger picture that includes the reliability, management scalability and security of your virtual infrastructure is always going to be more important than a single attribute like performance.  John Humphreys of IDC has made the same point, “While the focus in the vendor community has been on performance (i.e., should a customer chose a paravirtualized or native virtualized path), I would suggest the efforts and energy applied to wringing out a few extra performance percentage points would be better placed in demonstrating the underlying platform stability and reliability.” VMware has taken this approach from the beginning and we’re delivering the most stable hypervisor available -– one so solid that many of our customers proudly report uptimes for their VMs exceeding 1000 days.

A recent virtualization performance test claimed a high performance overhead for a virtualized web application compared to a physical system. Unfortunately, the test was not an apples-to-apples comparison. Here VMware points out some of the problems with this test configuration and gives some guidance how the benchmark might be improved.

Response to the article "Load Testing a Virtual Web Application"

This document presents VMware’s response to the recently published article "Load Testing a Virtual Web Application" by Web Performance Inc. The article has also been picked up by Slashdot.

1. All test results are obtained using VMware Server, our freely available hosted product. VMware Server uses a hosted architecture, as opposed to the bare-metal hypervisor architecture used by VMware ESX Server. VMware expects the performance results for the same test using VMware Infrastructure 3 would be better than those published in this article.

2. The test results are another example of "apples to oranges" comparison, and the reason VMware requires a benchmark review to ensure that benchmark test methodology is correct. Here are more details about how the physical and virtual configurations differ:

   1. CPU: In this case, the physical environment consists of dual Intel Xeon processors with hyperthreading enabled i.e. there are 4 logical CPUs in the physical environment. The virtual environment details are not provided, but assuming default values, we imagine the virtual machine is using a single virtual CPU. So in essence the test is comparing results from a 4-processor physical environment to a 1-processor virtual configuration. This can have a huge impact on multi-threaded apps such as this .Net application.

   2. Memory: The physical environment used 2GB memory available to the machine. In the virtual environment, the VM was also assigned 2GB (the article implies that the physical machine has more memory). While all the details are not available, this memory configuration may result in swapping since the host operating system and VMware Server have their own memory requirements.

3. The article does provide one data point that validates that the physical-to-virtual comparison is not correct.

   When the tests were performed with hyperthreading disabled, they saw 481 maximum users on physical versus 403 on virtual. This is a 16% decrease in performance (versus the 43% claim). Performance within 16% of native is considered to be relatively good (especially for a product that uses hosted architecture).

   To elaborate further, the test with hyperthreading disabled compares the performance of a 2-physical CPU machine to a single virtual CPU machine. While this data point still does not reflect a completely fair comparison, it is a very good indication that the virtualization overheads for a web application are much lower.

4. So what would you change with the test methodology to make this a fair comparison? There are several ways to do this and here are just a few:

   1. Boot the physical environment as single CPU with hyperthreading disabled, and then compare with 1-VCPU virtual machine.
   2. Or, configure the virtual machine with two virtual processors. VMware Server supports dual processor SMP virtual machines.
   3. Run multiple virtual machines so that the number of virtual and physical processors is equal.
   4. Make sure that the memory, disk and network configurations for both physical and virtual environments match.

    

5. Another very important point to notice is that the test measures the performance of a single virtual machine. The best way to achieve performance scaling in a virtual environment is to use multiple virtual machines. A good test would have been to measure the performance of 1, 2, 3, 4 and 5 single CPU virtual machines versus the performance of a physical box. Not only would this give more data points for fair comparison, it will also illustrate how customers are using virtualization technology in real life to get around some application problems and achieve better scalability using existing hardware investments

6. VMware has an externally published benchmark for a J2EE application based on IBM Websphere. Furthermore, there are several white papers that use another web application, the open source DVD Store application, at Dell Power Solutions.

   Please refer to VMware’s performance page for more details. We have posted a benchmark approval process; benchmark approval requests should go to benchmark@vmware.com.

[Updated: see below]

There are some good reasons that VMware wants pre-publishing review of benchmarks using our software. There are also good reasons not to like this policy, and my personal view is that it probably should be phased out at some point. However, we see problems with virtual performance testing over and over again. Virtualization benchmarking is hard, and virtualization is still so new that people love to take ancedotes and generalize to the usefulness of virtualization technologies, for all time.

Case in point: this recent benchmark got picked up by Slashdot today: Load Testing a Virtual Web Application. It really isn't a good test: they use VMware Server 1.0.1 (we don't recommend using Server for high-throughput production uses!) instead of ESX Server, they don't tune anything, etc.

The Slashdot users, however, give a good picture of real-world usage in the comments.  Read some excerpts after the jump...

As always, talk to people who are virtualizing their infrastructure today to get the real scoop on the the limitations and the benefits.

--jtroyer

[Update: See also Comparing ESX Server and VMware Server using VMmark. Thank you, VMware performance team!]

[Update 2: While my main point was that the Slashdotters recognized that the benchmark wasn't correct and didn't reflect the value they see every day in their virtual infrastructure in their own data centers, the commenters do exhibit some common misconceptions about what can be done with today's virtualization technologies: for instance, that you can't run a production database in virtual infrastructure. See the graph on page 11 of this paper on DB2 scalability or just ask for examples on the VMTN Forums to see what people are doing in the real world right now.]

Two comments on the recent back and forth on hypervisor performance. Both bloggers use VMware Infrastructure in their jobs every day, just so you know their biases going in.

Geert Baeke:

The results are interesting because performance is very similar. But is this enough to conclude that the XenSource product is a valid alternative to the VI3 offering? We think not and this for several reasons:

1. It is important for our customers to use a product that has established itself in the market. The XenSource product is not there yet and will need some time to mature.
2. The management tools have to be top-notch. Our experience shows that most customers virtualize Windows systems and are primarily running Windows in their environment. They do not want to mess with Linux configuration and management so it is important that the management products allow this. VI Client with VI3 comes very close to this need although some actions do require some command line knowledge.
3. Hardware compatibility and support. ESX 3.0.1 supports a wide array of hardware and storage devices. Installation and configuration is very straightforward. XenSource's products are not at that level yet.

(Note Geert is now trying out XenEnterprise (part 1, part 2) to compare for himself.)

Kent A aka Bowulf:

The trick is not virtualization of a system or even virtualization at near native performance — no matter how difficult current Microsoft products make it look. There are or will be many products that will do this in due time. If all you want is lowest cost of obtaining virtualization and only have 10-15 servers, perhaps that is all that matters. However, there are a number of other “must haves” — such as VMotion. If I have to take a physical host down — do I really want to affect 20-30 VMs as well. As Geert Baeke correctly points out, the management capabilities in Virtual Center far outstrip the other competitors shipping products. Enterprise data centers need that level of granularity of control and capabilities — in fact they often need more as evidenced by the throngs of companies buying 3rd party add-ons like ESX Charter and P2V applications, like Leostream.

Where is the 3rd party industry surrounding Xen products at this point? I am not saying this to discount the Xensource product only to highlight the differences. Scalability of solutions is just one area that this lack of features shows. (Although VMware’s products also face scalability on the extreme end as well– >100 ESX hosts & 1500 VMs) Where are the Data Centers using 1500 VMs running upon Xen outside the hosting world?

Bruce Herndon of VMware's performance team uses the VMmark benchmark to take a look at how ESX Server utilizes NUMA (non-uniform memory access) while scaling to high numbers of virtual machines. Link: Studying NUMA with VMmark.

In a NUMA system, the processors are divided into sets, also known as nodes. Each node has direct, local access to a portion of the overall system memory and must communicate via a network interconnect to access the remaining, remote memory at other NUMA nodes. The memory interconnect adds latency to remote memory accesses, making them slower than local ones. Applications that heavily utilize the faster local memory tend to perform better than those that don't. VMware ESX Server is fully NUMA-aware and exploits local memory to improve performance. ...

All in all, I am quite pleased with the results. They tell us that we need not worry about overstressing NUMA systems even as vendors make quad-core processors ubiquitous. In fact, I would say that virtual environments are a great match for commodity NUMA-based multi-core systems due to the encapsulation of memory requests within a virtual machine, which creates a largely local access pattern and limits stress on the memory subsystem. Of equal importance, these results show that the ESX scheduler exploits these types of systems well, which is good to see given how much work I know our kernel team has put into it. This type of exercise is just another area where a robust, stable, and representative virtualization benchmark like VMmark can prove invaluable.

From Jeff Buell at the performance team blog, VROOM!:

At VMworld last November I had the opportunity to talk to many ESX users and to discover for myself what performance issues were most on their minds. As it turned out, this endeavor was not very successful; everybody was generally happy with ESX performance. On the other hand the performance and best practice talks were among the most popular, indicating that users were very interested in learning new ways of getting the most out of ESX. VMworld was just the wrong audience to reach people who had concerns about performance. I was preaching to the choir, instead of the non-virtualized souls out there. At the same time aggresive marketing by other virtualization companies creates confusion about ESX performance. So it was decided that we needed to make a better effort at clearing misconceptions and providing real performance data, especially to enterprises just starting to consider their virtualization options.

Let us first have Tarry Singh remind us that the real value of benchmarking is in configuring and managing your systems better, not about raw competition in non-real-world speed races. From TarryBlogging:

Benchmarking is not about VMware is better than Parallels is better that Xen is better than whatever. It is about telling the customer to be careful when applying and deploying certain scenarios. VMware is doing a pretty good job at it.

Also, as VMware is explaining in excruciating detail, a virtual infrastructure solution is about much more than a hypervisor -- you need to be managing storage, networking, and other devices; you need resource pool management solutions like DRS, HA, and VCB; you need management automation tools; you then need to combine all these things with other tools to take care of your business in areas like business continuity, virtual desktop, and software lifecycle. The complete solution is a heckuva lot more than just a hypervisor.

All that being said, let's take a look at hypervisor performance: VMware ESX Server 3.0.1 vs. Xen 3.0.3 running Microsoft Windows Server 2003 here: A Performance Comparison of Hypervisors. Yes, VMware ESX Server is faster.

Single Virtual CPU Tests

For both SPECcpu2000 and Passmark –CPU tests, the Xen hypervisor showed on average twice the overhead compared to VMware ESX Server. For enterprise applications that are sensitive to CPU resources, this means that the Xen hypervisor can deliver much lower throughput than VMware ESX Server for the same CPU utilization. Furthermore, most enterprises start implementing virtualization to consolidate underutilized servers. These results imply that VMware ESX Server can support many more virtual machines per core compared to the Xen hypervisor.

The Netperf tests performed extremely poorly for the Xen hypervisor compared to VMware ESX Server. We believe that this happened because the Xen hypervisor lacks an open-source paravirtualized network driver for Windows, similar to the paravirtualized vmxnet driver provided by VMware ESX Server. The commercial versions of Xen are expected to offer paravirtualized network drivers similar to the vmxnet driver. However, such proprietary guest drivers will further add to forking of the open Xen source code and make it difficult for datacenter customers to migrate between various flavors of Xen. Furthermore, unlike the Xen hypervisor, these commercial and supported versions will not be free, and hence will change ROI and TCO calculations that were based on an open-source free offering.

Virtual SMP Tests

The tests for both virtual SMP configuration as well as virtual machine scalability could not be run due to issues with the Xen virtualization hypervisor. A two virtual CPU Windows guest could not be booted using the Xen hypervisor.

The virtual machine scalability tests could not be run because more than two uniprocessor Windows guests could not be booted using the Xen hypervisor. At this time, it is not known when this issue will be fixed and the tests can be tried again.
While Xen claims to support virtual SMP and virtual machine scalability, the results from these experiments demonstrate that enterprise customers should run their own tests to make sure such configurations actually work.

Linux tests are coming -- remember that VMware has shown we can run paravirtualized guests as well, so I would be very surprised if those results come out differently.

But remember, let's end where we started. The hypervisor is not the full solution. To make this technology anything but a partitioning toy, you need a full virtual infrastructure solution stack surrounding it. Benchmarks mean little compared to your own workloads running in your own infrastructure. Talk to other VMware customers to get a real sense of what can happen when you virtualize.

Bruce Herndon does some scaling tests with VMmark on a HP DL380G5 server with the new Intel Woodcrest dual-core processors. The different workloads (mail server, java server, database, web server, and file server)  have different scaling characteristics, depending on their need for CPU, memory, disk, and network capacity. He also looks at the scaling of realistic workloads looks two different configurations: six disks in a single LUN, and two three-disk LUNs.

Link: VROOM! Trying VMmark on Some New Hardware

These two different disk configurations highlight some interesting tradeoffs and tuning opportunities exposed by VMmark. The single LUN configuration utilizing six disks has the benefit of providing high disk throughput for one VM at the expense of scalability if multiple disk-intensive VMs are running. On the other hand, creating multiple LUNs provides both good predictability and excellent scaling but limits the total throughput of any single VM by providing only a subset of the hardware resources to each one. From a benchmarking perspective, the multi-LUN approach is clearly better since it results in a higher overall score. In practice, the proper approach depends upon the needs and goals of each user. I am excited by the ability VMmark gives us to study these types of performance tuning tradeoffs in a representative multi-VM environment. I feel that building performance tuning expertise in these complex situations and getting that information to our customers along with the ability to evaluate hardware and software platforms for virtualization should make VMmark an extremely valuable tool. Please stay tuned as we work to make that a reality.

Despite some claims to the contrary, VMmark is available to beta testers now (email vmmark-info@vmware.com), and will be released for general availability.

From Aravind Pavuluri on the VMware performance blog, VROOM!

Link: VROOM!: Performance Tuning Best Practices for ESX Server 3

Over time a number of customers have been asking us for a single comprehensive ESX performance tuning guide that would encompass its CPU, memory, storage, networking, resource management and DRS, component optimizations. Finally we have the Performance Tuning Best Practices for ESX Server 3  guide. ...

Some customers will want to carefully benchmark their ESX installations, as a way to validate their configurations and determine their sizing requirements. In order to help such customers with a systematic benchmarking methodology for their virtualized workloads, we've added a section in the paper called "Benchmarking Best Practices". It covers the precautions that have to be taken and things to be kept in mind during such benchmarking. ...

The strength of the paper is that it succinctly (in 22 pages) captures the performance best practices and benchmarking tips associated with key components.