Exchange 2010 Disk I/O on vSphere

In the first part of this series on Exchange 2010 on vSphere the focus was scale-up performance of a single Mailbox Server VM.  The results showed performance was great across all points tested, ranging up to 8000 users.  This article will take a look at disk I/O which is an important aspect of Exchange performance.

Enhancements made to Exchange 2010 have resulted in a reduction in disk I/Os per second (IOPS) in comparison to previous versions.  This reduction is reported by Microsoft to be as big as 70% in some cases.  Exchange 2010 makes more efficient use of memory as a cache which results in fewer reads and writes to disk than before.  However sufficient memory is still very important for optimal performance.  Exchange 2010 also uses larger I/O and more sequential I/O to improve performance.  Tests were conducted in the VMware labs to measure how beneficial additional RAM could be to an Exchange 2010 Mailbox Server VM running on vSphere 4. 


The same test configuration was used as in the previous blog.  To summarize, a Dell PowerEdge R710 with dual Xeon X5570 quad-core processors and 96 GB of RAM was installed with a development version of vSphere 4 (build 235768). Three Exchange 2010 VMs were created and used for testing: Mailbox Server, Client Access Server (CAS), and Hub Transport Server.   

An EMC CLARiiON CX4-960 provided Fibre Channel based storage with two nine-disk RAID-5 LUNs for data and an eight-disk RAID 1/0 LUN for logs.  Normally, RAID 1/0 would have been used for both data and logs because it provides the highest performance with fault tolerance.  Due to the improvement in Exchange 2010 IOPS performance, it was decided to try RAID-5 for these tests to see if this lower performing RAID type would provide acceptable performance.


Exchange 2010 can use RAM as a cache for disk I/O.  This reduces the amount of IOPS on the physical disks because more requests for information are satisfied in memory.  If the workload is kept constant, then a VM that is assigned more RAM will have lower IOPS.  As more requests are satisfied by information in memory, the average response time will also decrease. 

The mailbox VM was configured with 4 vCPUs and a range of memory sizes from 16 to 64 GBs.  LoadGen 2010 Beta was used with the Very Heavy Outlook 2007 Online profile with 100 MB mailboxes to simulate 8000 users.  Increasing RAM reduced both IOPS and Average SendMail response time.  The charts below show the results of these tests.


The total IOPS is low for 8000 users even at the highest recorded level of 1921 IOPS with 16 GB of RAM.  Increasing RAM to 24 and 32 GB showed a significant decrease in IOPS, but additional gains are much less as more RAM is added.  The performance improvements as shown by the SendMail response time follow the same curve as IOPS with big improvements at 24 and 32 GB.  The reduction in disk I/O is directly related to the improvement in performance.

The next chart shows CPU utilization broken out for the Mailbox Server, Hub Transport Server, and CAS VMs during these same tests.  The only change between tests is the amount of RAM assigned to the Mailbox server VM. 


CPU utilization of all the roles remained essentially the same across all tests.  The amount of IOPS reduces as RAM is increased, but the amount of users remains the same.  The VMs are doing the same amount of work but with a better response time.  In this case, CPU is not affected by the change and does not reflect any change in performance.  


Assigning more RAM to an Exchange 2010 Mailbox Server VM can result in decreased IOPS and faster response time.  This makes disk requirements for Exchange something that can be changed based on each environment.   The strategy could be to support more users with the same number of disks or use fewer disks to support the same number of users.  Improvements in performance with Exchange 2010 and the lower cost per GB of RAM in current servers can make it a worthwhile exercise.  The flexibility of assigning granular sizes of RAM to VMs makes it easy for an Exchange 2010 VM to be tuned to the correct amount of RAM. 

The next Exchange 2010 on vSphere blog will be on some scale-out performance with multiple Mailbox Server VMs.


8 comments have been added so far

  1. While you make a valid point about RAM and response times, which seems to be the point of the article, I struggle to see how this a valid from a Disk I/O viewpoint.
    Have you completed testing to validate this using a typical storage design for an Enterprise?
    a) Have you tested Exchange 2010 using SATA disk? FC disk is very expensive compared to DAS/JBOD/SATA which is supported by 2010 now. Unless you have CX/FC storage available, there are arguably more appropriate storage solutions for Exchange 2010 now.
    b) Why use a user profile of just 100MB mailboxes? Exchange supports mailboxes of many GB now, mostly due to Enterprise data/retention demands.
    If you repeated this testing using 2GB and 5GB mailboxes on SATA/SAS storage then this would validate running Exchange 2010 on vSphere for me.

  2. I did struggle a bit with the title for this post, and maybe I could have done better. In trying to keep it brief it may not fully describe the RAM and IOPS interaction I was trying to explore. I’m glad that you decided to comment and now we can have some discussion.
    It seems that we agree that Fibre Channel is generally the best performing storage solution. It has also been used by many large enterprises for Exchange historically, which is why I used it for these tests. I did use RAID 5 LUNs which is a lower performance option (as compared to RAID 1/0) that provide more useable space.
    I have not tested with lower performing disks, but reported these results with the raw IOPS numbers. Regardless of the speed of the disk, the number of IOPS can be reduced by adding RAM. This reduction in IOPS could make it possible to use less or lower performing disks.
    I am planning to do some tests with larger size mailboxes to see how this affects performance. I do not expect it to significantly change the performance in terms of response time, but I won’t know until I do the tests. I will not be able to test up to the same 8000 users due to size capacity limits, and will have to user a lower number.
    Thanks – Todd

  3. You state that a separate VM was created for the Mailbox, CAS, and Hub Transport Exchange 2010 server roles which I can see as an option. To test the other use case of a remote or satellite office, I would be very interested in testing the 2gb/5gb size mailboxes when you have all three exchange roles on the same guest using SAS level drives. No doubt the common practice in the large enterprise is to separate the roles, for the remote sites the desire is to minimize the server\guest footprint.
    Regards – Jeremy

  4. Was any of this testing done using DAG for the Mailbox servers; or were all these stand alone exhange mailbox servers?

Leave a Reply

Your email address will not be published.