At VMworld, together with IBM we announced a new capacity record for the number of Microsoft Exchange mailboxes on a single physical server. Microsoft Exchange Server doesn’t like a huge number of mailboxes on any one instance, so administrators "scale out" by racking up stacks of servers in a clustered configuration. (For the purposes of today’s discussion, we’ll ignore server roles.) Instead, our team set up 8 virtual machines, each managing 2,000 users and taking up 2 cores and 14GB of memory — all running on ESX Server 3i on a single IBM physical server.
Check out the details from Kaushik Banerjee on our VROOM! performance blog: Link: 16,000 Exchange Mailboxes, 1 Server – VMware VROOM!.
We recently finished a large Exchange 2007 capacity test on VMware
ESX Server 3.5. How large? Well, larger than anything ever done before on a
single server. And we did it from start to finish in about two weeks.We did this test because we have felt for a while that
advances in processor and server technology were about to leave another
widely-used and important application unable to fully utilize the hardware that
vendors were offering. Microsoft has guidelines on what environment works well
with Exchange, and a system with more than eight CPUs and/or 32GB of RAM is beyond
the recommended maximums.
Gabrie van Zanten was at the VMworld session where we talked about this and fleshes out a little of the narrative. Link: VMworld Europe 2008 – AP03 – Virtualization of Microsoft Exchange Server ( the 16.000 mailboxes story).
Normally when scaling an Exchange Server, the MS recommendation is to
not go beyond 8 cores and 32GB of RAM per server. When using these
figures, a physical Exchange 2007 server can only go to a max of 8000
mailboxes. Although there are very few stories about physical machines
running this number of mailboxes, there is some reference about 6000
mailboxes per host. David and Scott decided to take it even further.
Using the VM building block they created, the managed to put 4 VMs on
this physical server without problems, latency times (key measurement
factor for the Exchange admin) remained very low. When adding VM number
6, latency went up a little but was still well below the limits. VM
number 7 showed there was trouble on the road ahead, latency doubled to
400ms and VM number 8 turned out to be the limit with 16.000 mailboxes
!!!! and 900 ms latency. Even a 900 ms latency is below the MS limits
of what is acceptable, but it was obvious that adding a ninth VM would
go over this limit of 1000ms. All VMs together were now using 140Gb
RAM, made possible by VMware transparent page sharing.
As Rich Brambley notes, tests like this, besides being pretty cool, also give great insight into how to virtualize smaller Exchange installations. Link: VM /ETC » How to run 16,000 Exchange mailboxes on ESX.
16,000 Exchange Mailboxes, 1 Server not only offers insights on how to configure Exchange 2007 VMs to
support large numbers of mailboxes, but it shows that ESX 3.5 and ESX
3i allow applications to utilize hardware resources that exceed the
vendor’s recommended maximums in a physical deployment. Although this
test was able to squeeze the Exchange 2007 implementation on a single
ESX host without degrading the user experience, the technical details
of how it was done provides administrators a blueprint to spread the
Exchange VMs across multiple ESX hosts and fully leverage ESX
Enterprise features. …Many companies currently run large Exchange mailbox servers in a
multi-node clustered configuration, and they are reluctant to migrate
to VI. This test from VMware helps illustrate that breaking the
clusters and migrating back to multiple Exchange Server VMs has
performance and capacity advantages. Leveraging DRS and VMotion,
Exchange VMs maintain the ability to provide business continuity and
high availability when in a virtual environment with many ESX hosts in
a VMware cluster.
Scott Wilson over at CIO Weblog has some choice words for the state of application deployment today, including ones like ‘travesty’ and ‘shame,’ but I’ll just include these. Link: The CIO Weblog: Another VMware win.
While this may not seem like a terribly big deal (other than, of course, demonstrating their product’s ability to utilize hardware more efficiently), to me it speaks to the inherent limitations of Microsoft’s server system and the inefficiencies of Windows in general.
Here are a few other resources on virtualization Microsoft Exchange to get you started:
- Virtualization Solutions for Exchange Server 2007. This is VMware’s portal on virtualizing your Exchange Server. New resources will be added here — worth a bookmark.
- Exchange and VMware at the Dell Enterprise Technology Center wiki
- Virtualizing Exchange Server 2007 on ESX 3 – Whitepaper
- VMotion and HA with Exchange 2007 on ESX 3 – Whitepaper
- Virtualizing Exchange 2007: The Final Frontier – Presentation
- Exchange Server 2003 Performance on ESX 3 – Whitepaper
- A new whitepaper with Dell and EMC that’s not quite on our website yet: Virtual Solution for Microsoft® Exchange Server 2007 Using VMware Infrastructure 3 Software, Dell PowerEdge Servers and EMC CLARiiON CX3-20 iSCSI Storage. See Chuck’s Blog for some other interesting new tech papers that got released at VMworld, including one on SQLserver.
- A quick session note from Gabrie on virtualizing Exchange 2007 vs Exchange 2003:
First they made it clear that virtualizing Exchange 2007 is much easier
then virtualizing Exchange 2003. Exchange 2003 is a heavy hitter in the
storage area because of 4k disk block size, where Exchange 2007 uses 8k
blocks and write coalescing. With Exchange 2007 previous I/O challenges
have been addressed.
Finally, you may be asking yourself, "OK, John, fine, but what about support from Microsoft?" We’ll save that for another day, but for now let me reassure that Exchange gets virtualized in production every day. Link: Microsoft Exchange Virtualized by VMware Virtualization @ VIRTUALIZATION JOURNAL.
For example, Adrian Jane, Infrastructure &
Operations Manager at The University of Plymouth, who is responsible for
running approximately 50,000 Microsoft Exchange mailboxes across four virtual
machines running VMware Infrastructure 3, said, “Our entire Microsoft Exchange
deployment is virtualized on VMware Infrastructure 3, and we are extremely
pleased with the performance we’ve seen. Furthermore, VMware also provides us with
a high availability solution that has advantages over traditional clustering
options. When it comes to managing production applications, VMware is a
strategy, not just a product.”
