Our new benchmark, VMmark, had its first Beta release on December 21st. Now we are busy supporting the Beta users as well as trying to address some of the feedback we received during the earlier VMmark technology preview program with some of our hardware partners. We heard from almost everyone that the memory footprint of 7GB per tile should be reduced. Looking at the trends in the mid-range space, the feedback makes sense. Many current two-socket, 4-core systems have only 8 DIMM slots. One would have to break the bank buying 4GB DIMMs to get 8GB/core and 4-core chips are arriving. Ultimately, I hope the hardware vendors add more memory slots to address this looming imbalance. But for now, if we are going to measure these types of systems, we’ll need to reduce the memory usage of VMmark.
Three of the workloads in a VMmark tile, the web, file, and standby servers, together consume only 1GB of memory. They are already pretty lean, so squeezing memory from them would have limited benefits. The remaining three workloads, the database, mail, and java servers, use 2GB each. Databases tend to like a large, well-tuned buffer cache. I’d rather leave that one alone since it is a fairly typical database size. That leaves the java and mail server VMs as candidates. If we cut both of those VMs down to 1GB each, the total memory footprint drops to 5GB. In this configuration, 3 tiles will fit into 16GB, which should max out a current 2-socket, dual-core system using the cheaper 2GB DIMMs while leaving plenty of headroom for quad-core with 4GB DIMMs.
I first made the necessary changes for the mail and java server VMs to use 1GB. I then ran them each in isolation to get an idea of the performance impacts. To my surprise, the java server exhibited only a 2-3% reduction in throughput while mail server showed no discernable difference. Looking back, I suspect that this is due to the workload throttling we implemented in VMmark to insure that the workloads run at less than full utilization as they would in a datacenter consolidation scenario. Given that we initially sized our VMs based upon various industry and customer surveys, I am led to wonder if there aren’t lots of servers over-configured with not only CPU but also memory. As a final series of tests, I reran the newly modified VMmark on several systems for which I already had data for the existing 7GB tile size. Overall I saw very little effect on the benchmark scores. It looks like the 5GB VMmark tile is a go.