VMmark was originally developed to fill the need for a server consolidation benchmark for a rapidly changing datacenter that was becoming increasingly dominated by virtualization. The design of VMmark, which is a collection of workloads, gives us the ability to quickly change workload parameters to modify the behavior of the entire benchmark. This allows us to use VMmark to exercise technologies that were not available at the time the benchmark was designed. The VMmark 3 run rules provide for academic or research results publication using a modified version of the benchmark.
VMmark 3 was designed in 2015 when the memory size of a typical high-end 2 socket server was 768 GB. Each VMmark 3 tile was configured to use 156 GB of memory, allowing multiple tiles to be run on each server. A new technology, Intel Optane DC Persistent Memory, now allows up to 3 TB of memory in a 2 socket server, with plans to increase that even further. Testing the performance of this technology with an unmodified version of VMmark 3 wouldn’t be easy as we’d saturate CPU resources long before we could fully exercise this large amount of memory. Thankfully the flexible nature of VMmark allows us to modify it to consume significantly more memory with minimal changes in CPU usage.
The two primary VMmark workloads are Weathervane and DVD Store. Each can be modified to consume more memory. Weathervane, as configured for VMmark 3, uses 14 VMs. Thus while it would be possible to modify this application, doing so would be a time-consuming process. We therefore decided to look at DVD Store, which uses only four VMs. Most of the work is done in the DVD Store database VM which was our target for modification.
Determining the best configuration for DVD Store to utilize a larger amount of memory required multiple iterations of testing. We modified one test parameter of the DVD Store workload, and then examined the results to determine the effect on the VMmark tile. We were looking for larger memory usage with a minimal increase in CPU usage so that we could exercise the larger memory configuration without requiring additional CPUs. The following table lists the default configuration and the variables we changed:
| Parameter | Default | Configurations Tried |
| VM Memory Size | 32 GB | 128, 250 and 385 GB |
| Think Time | 1 second | 0.5, 0.9, 1.25, and 1.5 seconds |
| Number of Threads | 24 | 36 and 48 |
| Number of Searches | 3 | 5, 7, and 9 |
| Batch Search Size | 3 | 5, 7, and 9 |
| Database Size | 100 GB | 300 and 500 GB |
The final configuration that we determined to have the most increased memory usage while keeping the CPU usage moderate was 250 GB DS3DB VM memory size, 1.5 seconds think time, and 300 GB database size. All other parameters were kept at the default.
The following table lists the CPU and memory utilization of the default configuration and the “increased memory” configuration.
| Configuration | CPU Utilization | Memory Utilization |
| Default | 26.3 | 126 GB |
| Increased Memory | 24.1 | 350 GB |
We were able to almost triple the memory consumption of a single VMmark tile without increasing the CPU usage. Using this “increased memory” configuration for VMmark we can now see the effect of the additional memory provided by Intel Optane DC Persistent Memory in Memory Mode.
More detailed information about this configuration and the methodology used to refine it can be found in the Intel Optane DC Persistent Memory whitepaper. Detailed instructions to configure VMmark 3 to increase the memory footprint can be obtained by emailing the VMmark team at [email protected]. We encourage you to experiment with VMmark under academic rules for your own studies and to let us know if you have any questions.