VROOM!

It is no secret that vSphere 4.0 delivers excellent performance and provides the capability to virtualize the beefiest of workloads. Several impressive performance studies using ESX 4.0 have been already been presented. (My favorite is this database performance whitepaper.) However, I continue to hear questions about the scheduling overhead of larger VMs within a heavily-utilized, mixed-workload environment. We put together a study using simple variations of VMware’s mixed-workload consolidation benchmark VMmark to help answer this question.

For this study we chose two of the VMmark workloads, database and web server, as the vCPU-scalability targets. These VMs represent workloads that typically show the greatest range of load in production environments so they are natural choices for a scalability assessment. We varied the number of vCPUs in these two VMs between one and four and measured throughput scaling and CPU utilization of each configuration by increasing the number of benchmark tiles up to and beyond system saturation.

The standard VMmark workload levels were used and were held constant for all tests. Given that the workload is constant, we are measuring the cost of SMP VMs and their impact on the scheduler . This approach places increasing stress the hypervisor as the vCPU allocations increase and creates a worst-case scenario for the scheduler. The vCPU allocations for the three configurations are shown in the table below:

Config2 uses the standard VMmark vCPU allocation of 10 vCPUs per tile. Config1 contains 20% fewer vCPUs than the standard while Config3 contains 40% more than the standard.

We also used Windows Server 2008 instead of Windows Server 2003 where possible to characterize its behavior in anticipation of using Server 2008 in a next-generation benchmark. As a result, we increased the memory in the Javaserver VMs from 1GB to 1.4 GB to insure sufficient memory space for the JVM. The table below provides a summary of each VM’s configuration:

Below is a basic summary of the hardware used:

• Dell PowerEdge R905 with 4 x 2.6GHz Quad Core AMD Opteron 8382
• Firmware version 3.0.2 (latest available).
• 128GB DDR2 Memory.
• 2 x Intel E1000 dual-port NIC
• 2 x Qlogic 2462 dual-port 4Gb
• 2 x EMC CX3-80 Storage Arrays.
• 15 x HP DL360 client systems.

Experimental Results

Figure 1 below shows both the CPU utilization and the throughput scaling normalized to the single-tile throughput of Config1. Both throughput and CPU utilization remain roughly equal for all three configurations at load levels of 1, 3, and 6 tiles (6, 18, and 36 VMs, respectively). The cost of using SMP VMs is negligible here. The throughputs remain roughly equal while the CPU utilization curves begin to diverge as the load increases to 9, 10, and 11 tiles (54, 60, and 66 VMs, respectively). Furthermore, all three configurations achieve roughly linear scaling up to 11 tiles (66 VMs). CPU utilization when running 11 tiles was 85%, 90%, and 93% for Config1, Config2, and Config3, respectively. Considering that few customers are comfortable running at overall system utilizations above 85%, this result shows remarkable scheduler performance and limited SMP co-scheduling overhead within a typical operating regime.

Figure 2 below shows the same normalized throughput of Figure 1 as well as the total number of running vCPUs to illustrate the additional stresses put on the hypervisor by the progressively larger SMP configurations. For instance, the throughput scaling at nine tiles is equivalent despite the fact that Config1 requires only 72 vCPUs while Config3 uses 126 vCPUs. As expected, Config3, with its heavier resource demands, is the first to transition into system saturation. This occurs at a load of 12 tiles (72 VMs). At 12 tiles, there are 168 vCPUs active – 48 more vCPUs than used by Config2 at 12 tiles. Nevertheless, Config3 scaling only lags Config2 by 9% and Config1 by 8%. Config2 reaches system saturation at 14 tiles (84 VMs), where it lags Config1 by 5%. Finally Config1 hits the saturation point at 15 tiles (90 VMs).

Overall, these results show that ESX 4.0 effectively and fairly manages VMs of all shapes and sizes in a mixed-workload environment. ESX 4.0 also exhibits excellent throughput parity and minimal CPU differences between the three configurations throughout the typical operating envelope. ESX continues to demonstrate first-class enterprise stability, robustness, and predictability in all cases. Considering how well ESX 4.0 handles a tough situation like this, users can have confidence when virtualizing their larger workloads within larger VMs.

(*) The spartan memory and disk allocations for the Windows Server 2008 VMs might cause readers to question if the virtual machines were adequately provisioned. Since our internal testing covers a wide array of virtualization platforms, reducing the memory of the Standby Server enables us to measure the peak performance of the server before encountering memory bottlenecks on virtualization platforms where physical memory is limited and sophisticated memory overcommit techniques are unavailable. Likewise, we want to configure our tests so that the storage capacity doesn’t induce an artificial bottleneck. Neither the Standby Server nor the Javaserver place significant demands on their virtual disks, allowing us to optimize storage usage. We carefully compared this spartan Windows Server 2008 configuration against a richly configured Windows Server 2008 tile and found no measurable difference in stability or performance. Of course, I would not encourage this type of configuration in a live production setting. On the other hand, if a VM gets configured in this way, vSphere users can sleep well knowing that ESX won’t let them down.