It is my great pleasure to announce that VMmark 3.1 is generally available as of February 7, 2019!
What’s New?
This release adds support for persistent memory, improves workload scalability, and better reflects secure customer environments by increasing side-channel vulnerability mitigation requirements.
Visit our main VMmark HTML page for more information.
Please note that VMmark 3.0 will end of life on March 15th, 2019.
To learn more about VMmark3 see the introductory blog article here
The recently described Meltdown/Spectre vulnerabilities have implications throughout the tech industry, and the VMmark virtualization benchmark is no exception. In deciding how to approach the issue, the VMmark team’s goal was to address the impact of the these vulnerabilities while maintaining the value and integrity of the benchmark.
Applying the full set of currently available Meltdown/Spectre mitigations is likely to have a significant impact on VMmark scores. Because the mitigations are expected to continue evolving for some time, that impact might even change. If the VMmark team were to require the full set of mitigations in order for a submission to be compliant, that might make new submissions non-competitive with older ones, and also introduce more “noise” into VMmark scores as the mitigations evolve. While our intention for the future is that eventually all new VMmark results will be obtained on virtualization platforms that have the full set of Meltdown/Spectre mitigations, we have chosen to take a gradual approach.
Beginning May 8, 2018, all newly-published VMmark results must comply with a number of new requirements related to the Meltdown and Spectre vulnerabilities. These requirements are detailed in Appendix C of the latest edition of the VMmark User’s Guide.
Before performing any VMmark benchmark runs intended for publication, check the VMmark download page to make sure you’re using the latest edition of the VMmark User’s Guide. If you have questions, you can reach the VMmark team at vmmark-info@vmware.com.
VMmark 3.0, VMware’s multi-host virtualization benchmark is generally available here. VMmark3 is a free cluster-level benchmark that measures the performance, scalability, and power of virtualization platforms.
VMmark3 leverages much of previous VMmark generations’ technologies and design. It continues to utilize a unique tile-based heterogeneous workload application design. It also deploys the platform-level workloads found in VMmark2 such as vMotion, Storage vMotion, and Clone & Deploy. In addition to incorporating new and updated application workloads and infrastructure operations, VMmark3 also introduces a new fully automated provisioning service that greatly reduces deployment complexity and time.
Cloud-based deployments continue to be a hot topic in many of today’s corporations. Often the discussion revolves around workload portability, ease of migration, and service pricing differences. In an effort to bring performance into the discussion we decided to leverage VMware’s new benchmark, Weathervane. As a follow-on to Harold Rosenberg’s introductory Weathervane post we decided to showcase some of the flexibility and scalability of our new large-scale benchmark. Previously, Harold presented some initial scalability data running on three local vSphere 6 hosts. For this article, we decided to extend this further by demonstrating Weathervane’s ability to run within a non-VMware cloud environment and scaling up the number of app servers.
Weathervane is a new web-application benchmark architected to simulate modern-day web applications. It consists of a benchmark application and a workload driver. Combined, they simulate the behavior of everyday users attending a real-time auction. For more details on Weathervane I encourage you to review the introductory post.
Environment Configuration:
Cloud Environment: Amazon AWS, US West.
Instance Types: M3.XLarge, M3.Large, C3.Large.
Instance Notes: Database instances utilized an additional 300GB io1 tier data disk.
Instance Operating System: Centos 6.5 x64.
Application: Weathervane Internal Build 084.
Testing Methodology:
All instances were run within the same cloud environment to reduce network-induced latencies. We started with a base configuration consisting of eight instances. We then scaled out the number of workload drivers and application servers in an effort to identify how a cloud environment scaled as application workload needs increased. We used Weathervane’s FindMax functionality which runs a series of tests to determine the maximum number of users the configuration can sustain while still meeting QoS requirements. It should be noted that the early experimentation allowed us to identify the maximum needs for the other services beyond the workload drivers and application servers to reduce the likelihood of bottlenecks in these services. Below is a block diagram of the configurations used for the scaled-out Weathervane deployment.
Results:
For our analysis of Weathervane cloud scaling we ran multiple iterations for each scale load level and selected the average. We automated the process to ensure consistency. Our results show both the number of users sustained as well as the http requests per second as reported by the benchmark harness.
As you can see in the above graph, for our cloud environment running Weathervane, scaling the number of applications servers yielded nearly linear scaling up to five application servers. The delta in scaling between the number of users and the http requests per second sustained was less than 1%. Due to time constraints we were unable to test beyond five application servers but we expect that the scaling would have continued upwards well beyond the load levels presented.
Although just a small sample of what Weathervane and cloud environments can scale to, this brief article highlights both the benchmark and cloud environment scaling. Though Weathervane hasn’t been released publicly yet, it’s easy to see how this type of controlled, scalable benchmark will assist in performance evaluations of a diverse set of environments. Look for more Weathervane based cloud performance analysis in the future.
Datacenters continue to grow as the use of both public and private clouds becomes more prevalent. A comprehensive review of density, power, and performance is becoming more crucial to understanding the tradeoffs when considering new storage technologies as a replacement for legacy solutions. Expanding on previous articles around comparing storage technologies and the IOPS performance available when using flash-based storage, in this article we are comparing the density, power, and performance differences between traditional hard disk drive (HDDs) and flash-based storage. As might be expected, we found that the flash-based storage performed very well in comparison to the traditional hard disk drives. This article quantifies our findings.
In addition to VMmark’s previous performance measurement capability, VMmark 2.5 adds the ability to collect power measurements on servers and storage under test. VMmark 2.5 is a multi-host virtualization consolidation benchmark that utilizes a combination of application workloads and infrastructure operations running simultaneously to model the performance of a cluster. For more information on VMmark 2.5, see this overview.
Environment Configuration:
Hypervisor: VMware vSphere 5.1
Servers: Two x Dell PowerEdge R720
BIOS settings: High Performance Profile Enabled
CPU: Two x 2.9GHz Intel Xeon CPU-E5-2690
Memory: 192GB
HBAs: Two x 16Gb QLE2672 per system under test
Storage:
– HDD-Configuration: EMC CX3-80, 120 disks, 8 Trays, 1 SPE, 30U
– Flash-Based-Configuration: Violin Memory 6616, 64 VIMMs, 3U
Workload: VMware VMmark 2.5.1
Testing Methodology:
For this experimentation we set up a vSphere 5.1 DRS-enabled cluster consisting of two identically configured Dell PowerEdge R720 servers. A series of VMmark 2.5 tests were then conducted on the cluster with the same VMs being moved to the storage configuration under test, progressively increasing the number of tiles until the cluster reached saturation. Saturation was defined as the point where the cluster was unable to meet the VMmark 2.5 quality-of-service (QoS) requirements. We selected the EMC CX3-80 and the Violin Memory 6616 as representatives of the previous generation of traditional HDD-based and flash based storage, respectively. We would expect comparable arrays in these generations to have characteristics similar to what we measured in these tests. In addition to the VMmark 2.5 results, esxtop data was collected to provide further statistics. The HDD configuration running a single tile was used as the baseline and all VMmark 2.5 results in this article (excluding raw Watts metrics, %CPU, and Latency) were normalized to that result.
Average Watts and VMmark 2.5 Performance Per Kilowatt Comparison:
For our comparison of the two technologies, the first point of evaluation was reviewing both the average watts required by the storage arrays and the corresponding VMmark 2.5 Performance Per Kilowatt (PPKW) score. Note that the HDD configuration reached saturation at 7 tiles. In contrast, the Flash-based configuration was able to support a total of 9 tiles, while still meeting the quality of service requirements for VMmark 2.5.
As can be seen from the above graphs, the difference between the two technologies is extremely obvious. The average watts drawn by the Flash-based configuration was nearly 50% less than the HDD configuration across all tiles tested. Additionally, the PPKW score of the Flash-based configuration was on average 3.4 times higher than the HDD configuration, across all runs.
Application Score Comparison:
Due to the very large difference in PPKW, we decided to dig deeper into the potential root causes, beyond just the discrepancy in power consumed. Because the application workloads exhibit random access patterns, as opposed to the sequential nature of infrastructure operations, we focused on the differences in application scores between the two configurations, as this is where we would expect to see the majority of the gains provided by the Flash-based configuration.
The difference between the scaling of the application workloads is quite obvious. Although running the same number of tiles, and thus attempting the same amount of work, the flash-based configuration was able to produce application workload scores that were 1.9 times higher than the HDD configuration across 7 tiles.
CPU and Latency Comparison:
After exploring the power consumption and various areas of performance difference, we decided to look into two additional key components behind the performance improvements: CPU utilization and storage latency.
In our final round of data assessment we found that the CPU utilization of the flash-based storage was on average 1.53 times higher than the HDD configuration, across all 7 tiles. Higher CPU utilization might appear to be sub-optimal, however we determined that the systems were waiting less time for I/O to complete and were thus getting more work done. This is especially visible when reviewing the storage latencies of the two configurations. The flash-based configuration showed extremely flat latencies, and had on average less than one tenth of the HDD configuration’s latencies.
Finally, when comparing the physical space requirements of the two configurations, the flash-based storage was effectively 92% denser than the traditional HDD configurations (achieving 9 tiles in 3U versus 7 tiles 30U). In addition to physical density advancements, the flash-based storage allowed for a 29% increase in the number of VMs run on the same server hardware, while maintaining QoS requirements of VMmark 2.5.
The flash-based storage showed wins across the board for power and performance. The flash-based storage consumed half the power while achieving over three times the performance. Although the initial costs of flash-based storage can be somewhat daunting when compared to traditional HDD storage, the reduction in power, increased density, and superior performance of the flash-based storage certainly seems to provide a strong argument for integrating the technology into future datacenters. VMmark 2.5 gives us the ability to look at the larger picture, making an informed decision across a wide variety of today’s concerns.
At VMworld 2012 we demonstrated a single eight-way VM running on vSphere 5.1 exceeding one million IOPS. This testing illustrated the high end IOPS performance of vSphere 5.1.
In a new series of tests we have completed some additional characterization of high I/O performance using a very similar environment. The only difference between the 1 million IOPS test environment and the one used for these tests is that the number of Violin Memory Arrays was reduced from two to one (one of the arrays was a short term loan).
Configuration:
Hypervisor: vSphere 5.1
Server: HP DL380 Gen8
CPU: Two Intel Xeon E5-2690, HyperThreading disabled
Memory: 256GB
HBAs: Five QLogic QLE2562
Storage: One Violin Memory 6616 Flash Memory Array
VM: Windows Server 2008 R2, 8 vCPUs and 48GB.
Iometer Configuration: Random, 4KB I/O size with 16 workers
We continued to characterize the performance of vSphere 5.1 and the Violin array across a wider range of configurations and workload conditions.
Based on the types of questions that we often get from customers, we focused on RDM versus VMFS5 comparisons and the usage of various I/O sizes. In the first series of experiments we compared RDM versus VMFS5 backed datastores using 100% read workload mix while ramping up the I/O size.
click to enlarge
As you can see from the above graph, VMFS5 yielded roughly equivalent performance to that of RDM backed datastores. Comparing the average of the deltas across all data points showed performance within 1% of RDM for both IOPS and MB/s. As expected, the number of IOPS decreased after we exceed the default array block size of 4KB, but the throughput continued to scale, approaching 4500 MB/s at both 8KB and 16KB sizes.
For our second series of experiments, we continued to compare RDM versus VMFS5 backed datastores through a progression of block sizes, but this time we altered the workload mix to include 60% reads and 40% writes.
click to enlarge
Violin Memory arrays use a 4KB sector size and perform at their optimal level when managing 4KB blocks. This is very visible in the above IOPS results at the 4KB block size. In the above graph, comparing RDM and VMFS5 IOPS, you can see that VMFS5 performs very well with a 60% read, 40% write mix. Throughputs continued to scale in a similar fashion as the read-only experimentation and VMFS5 performance for both IOPS and MB/s were within .01% of RDM performance when comparing the average of the deltas across all data points.
The amount of I/O, with just one eight-way VM running on one Violin storage array, is both considerable and sustainable at many I/O sizes. It’s also noteworthy to point out that running a 60% read and 40% write I/O mix still generated substantial IOPs and bandwidth. While in most cases a single VM won’t need to drive nearly this much I/O traffic, these experiments show that vSphere 5.1 is more than capable of handling it.
Last year at VMworld 2011 we presented one million I/O operations per second (IOPS) on a single vSphere 5 host (link). The intent was to demonstrate vSphere 5’s performance by using mutilple VMs to drive an aggregate load of one million IOPS through a single server. There has recently been some interest in driving similar I/O load through a single VM. We used a pair of Violin Memory 6616 flash memory arrays, which we connected to a two-socket HP DL380 server, for some quick experiments prior to VMworld. vSphere 5.1 was able to demonstrate high performance and I/O efficiency by exceeding one million IOPS, doing so with only a modest eight-way VM. A brief description of our configuration and results is given below.
Configuration:
Hypervisor: vSphere 5.1
Server: HP DL380 Gen8
CPU: 2 x Intel Xeon E5-2690, HyperThreading disabled
Memory: 256GB
HBAs: 5 x QLE2562
Storage: 2 x Violin Memory 6616 Flash Memory Arrays
VM: Windows Server 2008 R2, 8 vCPUs and 48GB.
Iometer Config: 4K IO size w/ 16 workers
Results:
Using the above configuration we achieved 1055896 total sustained IOPS. Check out the following short video clip from one of our latest runs.
Look out for a more thorough write-up after VMworld.
Previous blog entries utilizing VMmark 2.1 introduced the benchmark, showed the effects of generational scaling, and evaluated the scale-out performance of vSphere clusters. This article analyzes the performance impact of the type of storage infrastructure used, specifically when comparing the effects of Enterprise Flash Drives (EFDs; often referred to as SSDs) versus traditional SCSI HDDs. There is a general perception, both in the consumer and business space, that EFDs are better than HDDs. Less clear, however, is how much better and whether the performance benefits of the typically more expensive EFDs are observed in today’s more complex datacenters.
VMmark 2 Overview:
Once again we used VMmark2.1 to model the performance characteristics of a multi-host heterogeneous virtualization environment. VMmark 2.1 is a combination of application workloads and infrastructure operations running simultaneously. In general, the infrastructure operations increase with the number of hosts in an N/2 fashion, where N is the number of hosts. To calculate the score for VMmark 2.1, final results are generated from a weighted average of the two kinds of workloads; hence scores will not increase linearly as workload tiles are added. For more general information on VMmark 2.1, including the application and infrastructure workload details, take a look at the expanded overview in my previous blog post or the VMmark 2.1 release notification written by Bruce Herndon.
Environment Configuration:
Testing Methodology:
To analyze the comparative performance of EFDs versus HDDs with VMmark 2.1, a vSphere DRS enabled cluster consisting of two identically-configured HP ProLiant DL380 servers was connected to the two EMC storage arrays. A series of tests were then conducted against the cluster with the same VMs being moved to the storage array under test, increasing the number of tiles until the cluster approached saturation. Saturation was defined as the point where the cluster was unable to meet the minimum quality-of-service (QoS) requirements for VMmark 2.1. The minimum configuration for VMmark 2.1 is a two-host cluster running a single tile. The result from this minimal configuration on the HDD storage array was used as the baseline, and all VMmark 2.1 data in this article were normalized to that result. In addition to the standard VMmark 2.1 results, esxtop data was also collected during the measurement phase of the benchmark to provide additional statistics.
Results:
In a top-down approach to reviewing the two storage technologies, it seems natural that the first point of comparison would be the overall performance of VMmark 2.1. By comparing the normalized scores, it’s possible to immediately see the impact of running our cluster on EFDs versus traditional HDDs at a variety of load levels.
Click to Enlarge
The improvement in score is apparent at every point of utilization, from the lowest-loaded 1-tile configuration out to the saturation point of 6 tiles. Overall, the average improvement in score for the EFD configuration was 25.4%. And while the HDD configuration was unable to meet the QoS requirements at 6 tiles, the EFD configuration not only met the requirements, but also improved the overall VMmark 2.1 score, even when the cluster was completely saturated (as seen in the graph below). VMmark 2.1 can drive a considerable amount of I/O, up to many thousands of IOPS for large numbers of tiles. Digging deeper into the root cause of such dramatic improvement for EFDs led me to investigate the overall throughputs for each of the configurations.
Click to Enlarge
It’s apparent from the above graph that there was significant improvement in the total bandwidth, represented by Total MB/s, in the EFD configurations. Compared to the HDD configuration, the EFD configuration’s total throughput improved (8%, 9.2%, 9.5%, 6.5%, and 14.5%, respectively). The amount of improvement actually increased as the I/O demands on the cluster increased. Another interesting detail that arose from reviewing the data over numerous points of utilization was that %CPU used on the EFD configuration was typically higher than its HDD counterpart at the same load. Although slightly counter-intuitive at first, it makes sense that if the system is waiting less for I/Os to complete, it can spend more time doing actual work as demonstrated by the higher VMmark 2.1 scores. This observation leads to another interesting comparison. Disk latency characteristics are often used to predict hardware performance. This can be useful, but what can be unclear is how this translates to real-world disk latencies running a diverse set of workloads.
Lower is Better:Click to Enlarge
Above is a series of graphs that display the average latency reported per write and read I/Os (note that lower latency is better). In looking at each of the key latency counters we can get a better sense for where the additional performance is derived. There’s a generalization that EFDs have poor write speeds by comparison to today’s HDDs. The results here show that the generalization doesn’t always apply. In fact, when looking at the average write latency for the tested EFDs across all data points, it was within 1% of the average write latency for the tested HDDs. Additionally, reviewing the read latency comparison data showed massive reductions in latency across all workload levels, 76% on average. Depending on the workload being run, this in itself could be all the justification needed to move to the newer technology.
It isn’t surprising that EFDs outperformed HDDs. What is somewhat unexpected is the amount of performance, and the ability for EFDs to show immediate advantages even on the most lightly loaded clusters. With an average VMmark 2.1 score improvement of 25.4%, an average bandwidth increase of 9.6%, and a combined average read latency reduction of 76%, it’s easy to imagine there are a great many environments that might benefit from the real-world performance of EFDs.
The steady march of technological improvements is nothing new. As companies either expand or refresh their datacenters it often becomes a non-trivial task to quantify the returns on hardware investments. This difficulty can be further compounded when it’s no longer sufficient to answer how well one new server will perform in relation to its predecessor, but rather how well the new cluster will perform in comparison to the previous one. With this in mind, we set off to explore the generational scaling performance of two clusters made up of very similar hardware using the newly released VMmark 2.1. VMmark 2.1 is a next-generation, multi-host virtualization benchmark that models not only application performance but also the effects of common infrastructure operations. For more general information on VMmark 2.1, including the application and infrastructure workload details, take a look at the expanded overview in one of my previous blog posts.
Environment Configuration:
Testing Methodology:
To measure the generational improvement of the two clusters under test every attempt was made to set up and configure the servers identically. The minimum configuration for VMmark 2.1 is a two-host cluster running a single tile. The result from this minimal configuration on the older cluster, or cluster #1, was used as the baseline and all VMmark 2.1 scalability data in this article were normalized to that score. A series of tests were then conducted on each of the clusters in isolation, increasing the number of tiles being run until the cluster approached saturation. Saturation was defined as the point where the cluster was unable to meet the minimum quality-of-service (QoS) requirements for VMmark 2.1. Results that were unable to meet minimum QoS for VMmark 2.1 were not plotted.
Results:
The primary component of change between the two clusters, making up the predominant factor in the generational scaling, is the change in processors. The AMD Opteron™ 2427 processors provide six cores per socket for a total of twelve logical processors per server, whereas the newer AMD Opteron™ 6174 processors have twelve cores per socket for a total of twenty-four logical processors per server. Factor in a doubling of the L3 cache per socket, as well as a doubling of the systems’ memory speeds, and the change in server characteristics is quite significant.
As shown in the above graph, the generational scaling between the two clusters under test is significant. In the one-tile case, both clusters were able to perform the work requested without the presence of resource constraints. The performance improvement of the newer cluster became more apparent once we started scaling up the number of tiles and significantly increasing the level of CPU over commitment and utilization. It’s important to note that while adding tiles does effectively linearly increase the application workload requests being made, the workload caused by infrastructure operations does not scale in the same way, and was a constant across all tests. Cluster #1 scaled to three tiles, at which point it was saturated and unable to support additional tiles while continuing to the meet the minimum quality-of-service (QoS) requirements of the benchmark. For comparison, Cluster #2 was able to achieve an increase of normalized VMmark 2.1 scores of 1%, 14% and 9% for the one-tile, two-tile, and three-tile configurations, respectively. Cluster #2, was then scaled to seven tiles, beyond which point it was unable to meet the minimum QoS requirements.
The newer generation cluster, with two Dell PowerEdge R815 AMD Opteron™ 6174 based hosts running vSphere 4.1, exhibited excellent scaling as the load was increased up to seven tiles, more than doubling the previous generation cluster’s performance and work accomplished. Because VMmark 2.1 not only utilizes heterogeneous applications across a diverse computing environment, but also measures the impact of the common place infrastructure operations, it provided valuable insight on the generational scaling of the two cluster generations. VMmark 2.1 proved itself an able benchmark for acquiring the answers for previously difficult datacenter questions.
The first article in our VMmark 2 series gave an in-depth introduction to the benchmark while also presenting results on the scaling performance of a cluster based on a matched pair of systems. The goal of this article is to continue to characterize larger and more diverse cloud configurations by testing scale-out performance of an expanding vSphere cluster. This blog explores an enterprise-class cluster’s performance as more servers are added and subsequently the amount of work being requested is increased. Determining the impact of adding hosts to a cluster is important because it enables the measurement of the total work being done as cluster capacity and workload demand increases within a controlled environment. It also assists in identifying the efficiency with which a vSphere managed cluster can utilize an increasing number of hosts.
VMmark 2 Overview:
VMmark 2 is a next-generation, multi-host virtualization benchmark that models not only application performance but also the effects of common infrastructure operations. VMmark 2 is a combination of the application workloads and the infrastructure operations running simultaneously. Although the application workload levels are scaled up by the addition of tiles, the infrastructure operations scale as the cluster size increases. In general, the infrastructure operations increase with the number of hosts in an N/2 fashion, where N is the number of hosts. To calculate the score for VMmark 2, final results are generated from a weighted average of the two kinds of workloads; hence scores will not increase linearly as tiles are added. For more general information on VMmark 2, including the application and infrastructure workload details, take a look at the expanded overview in my previous blog post.
Environment Configuration:
Testing Methodology:
To test scale out performance with VMmark 2, five identically-configured HP ProLiant DL380 servers were connected to an EMC Clarion CX3-80 storage array. The minimum configuration for VMmark 2 is a two-host cluster running one tile. The result from this minimal configuration was the baseline used, and all VMmark 2 scalability data in this article were normalized to that score. A series of tests were then conducted on this two-host configuration, increasing the number of tiles being run until the cluster approached saturation. As shown in the series’ first article, our two-host cluster approached saturation at four tiles but failed QoS requirements when running five tiles. Starting with a common workload level of four tiles, the three-host, four-host, and five-host configurations were tested in a similar fashion, increasing the number of tiles until each configuration approached saturation. Saturation was defined to be the point where the cluster was unable to meet the minimum quality-of-service requirements for VMmark 2. For all testing, we recorded both the VMmark 2 score and the average cluster CPU utilization during the run phase.
Results:
Organizations often outgrow existing hardware capacity, and it can become necessary to add one or more hosts in order to relieve performance bottlenecks and meet increasing demands. VMmark 2 was used to measure such a scenario by keeping the load constant as new hosts were incrementally added to the cluster. The starting point for the experiments was four tiles. At this load level the two hosts had approached saturation, with nearly 90% CPU utilization The test then determined the impact on cluster CPU utilization and performance by adding identical hosts to the available cluster resources.
As expected, scoring gains were easily achieved by adding hosts until the environment was generating approximately the maximum scores for the four tile load level, as CPU resources become more plentiful. In comparison to the two-host configuration, the normalized scores increased 6%, 12%, and 12% for the three-host, four-host, and five-host configurations, respectively. The configurations with additional hosts were able to generate more throughput while also reducing the average cluster CPU utilization as the requested work was spread over more systems. This highlights the additional CPU capacity held in reserve by the cluster at each data point. By charting two or more points at the same load level, it is much easier to approximate the expected average CPU utilization after adding new hosts into the cluster. This data, combined with established CPU usage thresholds, can make additional purchasing or system allocation decisions more straight-forward.
The above analysis looks at scale out performance for an expanding cluster with a fixed amount of work. To get the whole picture of performance it’s necessary to measure performance and available capacity as the load and the number of hosts increases. Specifically, as we progress through each of the configurations, does the reduction in cluster CPU utilization and improved performance measured in the previous experiment hold true for varied amounts of load and hosts?
As shown in the above graphs, the vSphere based cloud effortlessly integrated new hosts into our testing environment and delivered consistent returns on our physical server investments. It’s important to note that in both the two-host and three-host configurations, the test failed at least one of the quality-of-service (QoS) requirements when the cluster reached saturation. Also important, the five-host configuration was not run out to saturation due to a lack of additional client hardware. During our testing the addition of each host showed expected results with respect to scaling of VMmark 2 scores. As we went through each of the configurations, the normalized scores increased an average of 13%, 13%, and 16%, for the three-host, four-host, and five-host configurations, respectively. Each of the configurations exhibited nearly linear scaling of CPU utilization as load was increased. Based on these results, the VMware vSphere managed cluster was able to generate significant performance scaling while also utilizing the additional capacity of newly-provisioned hosts quite efficiently.
Thus far all VMmark 2 studies have involved homogenous clusters of identical servers. Stay tuned for experimentation utilizing varying storage and/or networking solutions as well as heterogeneous clusters…