Response to ‘Load Testing a Virtual Web Application’

A recent virtualization performance test claimed a high performance overhead for a virtualized web application compared to a physical system. Unfortunately, the test was not an apples-to-apples comparison. Here VMware points out some of the problems with this test configuration and gives some guidance how the benchmark might be improved.

Response to the article "Load Testing a Virtual Web Application"

This document presents VMware’s
response to the recently published article "Load
Testing a Virtual Web Application" by Web Performance Inc. The article
has also been picked up by Slashdot.

1. All test results
   are obtained using VMware Server, our freely available hosted product. VMware Server
   uses a hosted architecture, as opposed to the bare-metal hypervisor architecture
   used by VMware ESX Server. VMware expects the performance results for the same
   test using VMware Infrastructure 3 would be better than those published in this
   article.

2. The test
   results are another example of "apples to oranges" comparison, and
   the reason VMware requires a benchmark review to ensure that benchmark test
   methodology is correct. Here are more details about how the physical and
   virtual configurations differ:

   1. CPU: In this case, the physical environment consists of
      dual Intel Xeon processors with hyperthreading enabled i.e. there are 4 logical
      CPUs in the physical environment. The virtual environment details are not
      provided, but assuming default values, we imagine the virtual machine is using a
      single virtual CPU. So in essence the test is comparing results from a 4-processor
      physical environment to a 1-processor virtual configuration. This can have a
      huge impact on multi-threaded apps such as this .Net application.

   2. Memory: The physical environment used 2GB memory available
      to the machine. In the virtual environment, the VM was also assigned 2GB (the
      article implies that the physical machine has more memory). While all the
      details are not available, this memory configuration may result in swapping
      since the host operating system and VMware Server have their own memory
      requirements.

3. The article does
   provide one data point that validates that the physical-to-virtual comparison is
   not correct.

   When the tests were performed with hyperthreading
   disabled, they saw 481 maximum users on physical versus 403 on virtual. This is a 16% decrease in performance (versus
   the 43% claim). Performance within 16% of native is considered to be relatively
   good (especially for a product that uses hosted architecture).

   To
   elaborate further, the test with hyperthreading disabled compares the
   performance of a 2-physical CPU machine to a single virtual CPU machine. While
   this data point still does not reflect a completely fair comparison, it is a very
   good indication that the virtualization overheads for a web application are
   much lower.

4. So what would
   you change with the test methodology to make this a fair comparison? There are
   several ways to do this and here are just a few:

   1. Boot the physical environment as single CPU with hyperthreading disabled, and then compare with 1-VCPU virtual machine.
   2. Or, configure the virtual machine with two virtual processors. VMware Server supports dual processor SMP virtual machines.
   3. Run multiple virtual machines so that the number of virtual and physical processors is equal.
   4. Make sure that the memory, disk and network
      configurations for both physical and virtual environments match.

    

5. Another very
   important point to notice is that the test measures the performance of a single
   virtual machine. The best way to achieve performance scaling in a virtual
   environment is to use multiple virtual machines. A good test would have been to
   measure the performance of 1, 2, 3, 4 and 5 single CPU virtual machines versus the
   performance of a physical box. Not only would this give more data points for
   fair comparison, it will also illustrate how customers are using virtualization
   technology in real life to get around some application problems and achieve
   better scalability using existing hardware investments

6. VMware has an
   externally published benchmark for a J2EE application based on IBM Websphere.
   Furthermore, there are several white papers that use another web application,
   the open source DVD Store application,
   at Dell Power Solutions.

   Please refer to
   VMware’s performance page for
   more details. We have posted a benchmark approval process; benchmark approval requests should go to [email protected].