[Updated: see below]
There are some good reasons that VMware wants pre-publishing review of benchmarks using our software. There are also good reasons not to like this policy, and my personal view is that it probably should be phased out at some point. However, we see problems with virtual performance testing over and over again. Virtualization benchmarking is hard, and virtualization is still so new that people love to take ancedotes and generalize to the usefulness of virtualization technologies, for all time.
Case in point: this recent benchmark got picked up by Slashdot today: Load Testing a Virtual Web Application. It really isn’t a good test: they use VMware Server 1.0.1 (we don’t recommend using Server for high-throughput production uses!) instead of ESX Server, they don’t tune anything, etc.
The Slashdot users, however, give a good picture of real-world usage in the comments. Read some excerpts after the jump…
As always, talk to people who are virtualizing their infrastructure today to get the real scoop on the the limitations and the benefits.
–jtroyer
[Update: See also Comparing ESX Server and VMware Server using VMmark. Thank you, VMware performance team!]
[Update 2: While my main point was that the Slashdotters recognized that the benchmark wasn’t correct and didn’t reflect the value they see every day in their virtual infrastructure in their own data centers, the commenters do exhibit some common misconceptions about what can be done with today’s virtualization technologies: for instance, that you can’t run a production database in virtual infrastructure. See the graph on page 11 of this paper on DB2 scalability or just ask for examples on the VMTN Forums to see what people are doing in the real world right now.]
From Slashdot:
To us, the whole point of virtualization is that we have several
servers which are mostly idle at all times and completely idle at most
times, and for support reasons we’re not supposed to be running
anything else on the same copy of windows. So we can replace five or
six systems with a four-core 1U box with just a few gigabytes of
memory, which will recover rack space and reduce power consumption. …One of the other things we prototyped and deployed was ‘site services
packages’ – get GSX (now VMWare Server), stick it on a pair of 2U
servers, and attach a storage array to both of them. Then create your
‘template’ fileserver, DHCP server, print server, proxy, that kind of
thing and deploy them to this package. It worked very well indeed – you
get a whole new order of magnitude on stability (although to be fair
that’s in part because we through away the crappy workstations that
were doing the ‘low intensity’ stuff) and was extremely managable, and
trivially replacable in the event of a hardware failure. …As an experienced sysadmin, my reaction is screw performance.
Let’s start with reliable and scalable, and then performance just
naturally follows, as does a really high grade service. …Another
useful guideline: various types of services are impacted differently by
being virtualized. Generally, the best candidates for virtualization
are ones that spend a lot of time idle. This is actually more common
than you might think – people need a server set up for something, can’t
put it on a pre-existing system for security/compatibility reasons, so
they go out and buy a new system which is ten times more powerful than
they need. You can put a lot of these kinds of systems on a single,
reasonably powerful ESX server. …The
best solution is Virtual Infrastructure (used to be ESX). That product
tackles most of the failings of VMWare server and fixes them. The only
exception is that I still wouldn’t run anything I/O heavy on VI. SQL’s
a no-no. [Actually, lots of people run SQL Server, Oracle, and other databases in virtual infrastructure, and have found that their “I/O heavy” applications run just fine. -jt] Also, if you’re not getting the performance from a single web
server that you expect, you can easily throw up more web servers. Now,
obviously you might get into M$ licensing issues, but that’s why you
run your web services on Apache 😀 …We’ve done testing with many tools, and VMware ESX is the fastest true virtualization suite that we’ve tested.
