VMware

VMworld 2007 Call for Presentations | Main | VDI roundup

March 29, 2007

Bad benchmark reality-checked by Slashdot readers

[Updated: see below]

There are some good reasons that VMware wants pre-publishing review of benchmarks using our software. There are also good reasons not to like this policy, and my personal view is that it probably should be phased out at some point. However, we see problems with virtual performance testing over and over again. Virtualization benchmarking is hard, and virtualization is still so new that people love to take ancedotes and generalize to the usefulness of virtualization technologies, for all time.

Case in point: this recent benchmark got picked up by Slashdot today: Load Testing a Virtual Web Application. It really isn't a good test: they use VMware Server 1.0.1 (we don't recommend using Server for high-throughput production uses!) instead of ESX Server, they don't tune anything, etc.

The Slashdot users, however, give a good picture of real-world usage in the comments.  Read some excerpts after the jump...

As always, talk to people who are virtualizing their infrastructure today to get the real scoop on the the limitations and the benefits.

--jtroyer

[Update: See also Comparing ESX Server and VMware Server using VMmark. Thank you, VMware performance team!]

[Update 2: While my main point was that the Slashdotters recognized that the benchmark wasn't correct and didn't reflect the value they see every day in their virtual infrastructure in their own data centers, the commenters do exhibit some common misconceptions about what can be done with today's virtualization technologies: for instance, that you can't run a production database in virtual infrastructure. See the graph on page 11 of this paper on DB2 scalability or just ask for examples on the VMTN Forums to see what people are doing in the real world right now.]

From Slashdot:

To us, the whole point of virtualization is that we have several servers which are mostly idle at all times and completely idle at most times, and for support reasons we're not supposed to be running anything else on the same copy of windows. So we can replace five or six systems with a four-core 1U box with just a few gigabytes of memory, which will recover rack space and reduce power consumption.  ...

One of the other things we prototyped and deployed was 'site services packages' - get GSX (now VMWare Server), stick it on a pair of 2U servers, and attach a storage array to both of them. Then create your 'template' fileserver, DHCP server, print server, proxy, that kind of thing and deploy them to this package. It worked very well indeed - you get a whole new order of magnitude on stability (although to be fair that's in part because we through away the crappy workstations that were doing the 'low intensity' stuff) and was extremely managable, and trivially replacable in the event of a hardware failure. ...

As an experienced sysadmin, my reaction is screw performance. Let's start with reliable and scalable, and then performance just naturally follows, as does a really high grade service.  ...

Another useful guideline: various types of services are impacted differently by being virtualized. Generally, the best candidates for virtualization are ones that spend a lot of time idle. This is actually more common than you might think - people need a server set up for something, can't put it on a pre-existing system for security/compatibility reasons, so they go out and buy a new system which is ten times more powerful than they need. You can put a lot of these kinds of systems on a single, reasonably powerful ESX server.  ...

The best solution is Virtual Infrastructure (used to be ESX). That product tackles most of the failings of VMWare server and fixes them. The only exception is that I still wouldn't run anything I/O heavy on VI. SQL's a no-no. [Actually, lots of people run SQL Server, Oracle, and other databases in virtual infrastructure, and have found that their “I/O heavy” applications run just fine. -jt] Also, if you're not getting the performance from a single web server that you expect, you can easily throw up more web servers. Now, obviously you might get into M$ licensing issues, but that's why you run your web services on Apache :D ...

We've done testing with many tools, and VMware ESX is the fastest true virtualization suite that we've tested.

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/t/trackback/1051344/17318218

Listed below are links to weblogs that reference Bad benchmark reality-checked by Slashdot readers:

Comments

In Reference to the statement:
"[Actually, lots of people run SQL Server, Oracle, and other databases in virtual infrastructure, and have found that their “I/O heavy”applications run just fine. -jt]"

Can you elaborate on this? I can't seem to find any info about running vldb or other I/O intensive sql server application in a virtual environment.

Everything I find deals with tiny dbs (a few gb at best)...and we have a client that wants to force VMWare on us.

Our db will start off at at least 100GB of online/active data, several hundred normal users and several to many BI users.

Our db (because the nature of our business) will be a hybrid OLTP/OLAPish product...

I need proof of performance info with db statistics....any help would be greatly appreciated.

Thanks...JR

Post a comment

If you have a TypeKey or TypePad account, please Sign In