Microsoft Exchange is a very common critical application managed by enterprises. Exchange 2013 has brought forth many changes to the performance and IO profile of its infrastructure.
Exchange Server 2013 is an even greater candidate for virtualization than its predecessors. Architectural changes and improvements to the core of Exchange Server, along with advancements in server hardware, make vSphere the default choice for Exchange 2013.
The Exchange information store (the Managed Store) has been rewritten to further optimize resource consumption. This update to the Managed Store has also led to further reduction in storage I/O requirements.
Advances in server hardware such as multicore processors, higher memory density, and advances in storage technology are far outpacing the performance requirements for applications, including Exchange 2013. Virtualization becomes an effective way to leverage the full power of these systems.
The advances in Exchange Server 2013 and server hardware technology have coincided with advances in vSphere. Virtual machines support up to 1TB RAM and 64 vCPUs and are capable of running even the largest Exchange Mailbox servers.
The goal of this exercise was to see if the IO performance required for 100,000 Exchange 2013 mailboxes can be supported by the test environment. The main goal of the testing is storage performance and not capacity or other parameters. Virtual machines and storage will be sized for a Exchange mailbox performance test.
Infrastructure Components:
The following components were used for testing OLTP performance on an all Flash storage array:
Dell R910 server with 40 (2.4 GHZ )cores and 256 GB RAM
Pure Storage FA-420 FlashArray with two shelves that included 44 238 GB Flash drives and 8.2 TB usable capacity.
8 X Windows 2012 Virtual machines with 4 vCPU & 16 GB RAM, Four 250 GB Data Drives and two 100 GB Database Log drives each.
SW ISCSI on two dedicated 10GBPS ports.
Exchange 2013 drivers and Microsoft JetStress Utility.
Sizing of Virtual Machine:
The virtual machine used for testing was a Windows 2012 Server R2 with Exchange 2013 Drivers and the Microsoft JetStress utility. Storage configuration of virtual machines were:
Boot Disk: C Drive and swap space on shared LUN with other virtual machines
VM had 4 PVSCSI adapters with disks evenly distributed across them.
4 Data Disks across separate LUNS on the Pure Storage and 2 Log Disks across separate LUNS on the Pure Storage.
Disk Capacity and configuration of each Exchange Server:
Preparing the Database:
Each Exchange server has four databases and four log files distributed across four data LUNS and two Log LUNS. Four Exchange databases were created for each Exchange server and placed across multiple data drives as shown below.
Exchange Database Creation:
Exchange Jetstress configuration and Virtual Machines used for testing:
Eight identical virtual machines were used to generate Exchange load equivalent to a total of 100000 Exchange Mailboxes with an average daily mail flow of 300 messages. Each virtual machine had 4 vCPU & 16 GB RAM, Four 250 GB Data Drives and two 100 GB Database Log drives each.
Exchange JetStress Server Optimizations:
MS Exchange server best practices relating to using multiple SCSI adapters with virtual disks distributes across all adapters were implemented. Four Data disks and two log disks each on individual LUNS were configured on the Pure Storage Array and provisioned to the Exchange virtual machine.
IOPS Sizing for Jetstress testing:
The Technet Blog Exchange Sizing Guide was used to come up with the required IOPS per user (0.2) for an average mail volume of 300 per day.
IOPS requirement by the number of messages per day per mailbox:
Testing for 80,000 mailboxes: (2000 IOPS per node, 8 nodes)
Jetstress was run in parallel across eight virtual machines with four databases in each server. The target IOPS for each virtual machine was 2000. All the eight nodes passed the test validating the solution for 80000 mailboxes.
Testing for 100,000 mailboxes: (2500 IOPS per node, 8 nodes)
Initial Testing:
Jetstress was run in parallel across eight virtual machines with four databases in each server. The target IOPS for each virtual machine was 2500. Majority of the eight nodes failed the test with the IOPS required not being achieved.
During the analysis it was seen that the maximum throughput across the two 10 GBPS ISCSI adapters was 650 MBPS and there were no problems in any component of the infrastructure. ESXTOP analysis showed that the ISCSI QUEUE Depth could be a limiting factor for the performance.
Final Testing:
After changing the ISCSI QUEUE Depth for the ESX host, the test was then repeated. All eight virtual machines passed the test with IOPS exceeding 2600. It was also observed that the throughput of the ISCSI links was much higher with a peak approaching 950 MBPS.
Network Utilization during successful Jetstress for 100000 users:
During the entire test it was observed that the PURE storage array performed well with less than 1ms latency and throughput sometimes reaching 900 MBPS.
Storage performance during successful Jetstress for 100000 users:
Results:
The successful result of the Jetstress testing for one of the Exchange host is shown below.
Conclusion:
The results show that vSphere is an excellent platform for Exchange 2013 workloads. From a single server with 40 modern cores attached to a high performance Pure Storage All Flash array, it was shown that the IO performance requirements of 100000 mailboxes can be met with multiple virtual machines operating in parallel.The Pure Storage All Flash storage array, which is a critical component in the infrastructure performed with sub millisecond latencies throughout the test. The Pure storage array also provided and average data reduction in the order of 7:1 for the Exchange workloads.
