posted

1 Comment

A recent virtualization performance test claimed a high performance overhead for a virtualized web application compared to a physical system. Unfortunately, the test was not an apples-to-apples comparison. Here VMware points out some of the problems with this test configuration and gives some guidance how the benchmark might be improved.

Response to the article "Load Testing a Virtual Web Application"

This document presents VMware’s
response to the recently published article "Load
Testing a Virtual Web Application
" by Web Performance Inc. The article
has also been picked up by Slashdot.

  1. All test results
    are obtained using VMware Server, our freely available hosted product. VMware Server
    uses a hosted architecture, as opposed to the bare-metal hypervisor architecture
    used by VMware ESX Server. VMware expects the performance results for the same
    test using VMware Infrastructure 3 would be better than those published in this
    article.

  2. The test
    results are another example of "apples to oranges" comparison, and
    the reason VMware requires a benchmark review to ensure that benchmark test
    methodology is correct. Here are more details about how the physical and
    virtual configurations differ:

    1. CPU: In this case, the physical environment consists of
      dual Intel Xeon processors with hyperthreading enabled i.e. there are 4 logical
      CPUs in the physical environment. The virtual environment details are not
      provided, but assuming default values, we imagine the virtual machine is using a
      single virtual CPU. So in essence the test is comparing results from a 4-processor
      physical environment to a 1-processor virtual configuration. This can have a
      huge impact on multi-threaded apps such as this .Net application.

    2. Memory: The physical environment used 2GB memory available
      to the machine. In the virtual environment, the VM was also assigned 2GB (the
      article implies that the physical machine has more memory). While all the
      details are not available, this memory configuration may result in swapping
      since the host operating system and VMware Server have their own memory
      requirements.

  3. The article does
    provide one data point that validates that the physical-to-virtual comparison is
    not correct.

    When the tests were performed with hyperthreading
    disabled, they saw 481 maximum users on physical versus 403 on virtual. This is a 16% decrease in performance (versus
    the 43% claim). Performance within 16% of native is considered to be relatively
    good (especially for a product that uses hosted architecture).

    To
    elaborate further, the test with hyperthreading disabled compares the
    performance of a 2-physical CPU machine to a single virtual CPU machine. While
    this data point still does not reflect a completely fair comparison, it is a very
    good indication that the virtualization overheads for a web application are
    much lower.

  4. So what would
    you change with the test methodology to make this a fair comparison? There are
    several ways to do this and here are just a few:

    1. Boot the physical environment as single CPU with hyperthreading disabled, and then compare with 1-VCPU virtual machine.
    2. Or, configure the virtual machine with two virtual processors. VMware Server supports dual processor SMP virtual machines.
    3. Run multiple virtual machines so that the number of virtual and physical processors is equal.
    4. Make sure that the memory, disk and network
      configurations for both physical and virtual environments match.

     

  5. Another very
    important point to notice is that the test measures the performance of a single
    virtual machine. The best way to achieve performance scaling in a virtual
    environment is to use multiple virtual machines. A good test would have been to
    measure the performance of 1, 2, 3, 4 and 5 single CPU virtual machines versus the
    performance of a physical box. Not only would this give more data points for
    fair comparison, it will also illustrate how customers are using virtualization
    technology in real life to get around some application problems and achieve
    better scalability using existing hardware investments

  6. VMware has an
    externally published benchmark for a J2EE application based on IBM Websphere.
    Furthermore, there are several white papers that use another web application,
    the open source DVD Store application,
    at Dell Power Solutions.

    Please refer to
    VMware’s performance page
    for
    more details. We have posted a benchmark approval process; benchmark approval requests should go to benchmark@vmware.com.