Crackpot Ideas

Timothy Prickett Morgan over at The Register has penned an article entitled, “Fake server beats real server in web test”.  The gist of the article is that VMWare has released results showing that virtual Linux servers running on VMWare’s ESX hypervisor have garnered the highest single-server performance for a 16-core machine, and significantly beat out a “comparably configured” non-virtualized server.  How is that possible?

Well, Morgan points out, it’s not a completely apples-to-apples comparison. The article says:

“As it turns out, that ESX Server-based setup beat out a real DL580 server that HP did some tests on using Intel’s quad-core “Harpertown” processors running at 2.93 GHz. The DL580 is a quad-socket box, and in this case, HP set it up with 64 GB of main memory, eight internal disks plus two MSA 70 disk arrays, with a total of 2.1 TB of disk capacity. Now, right here, before we go any further, that’s half the memory and one-seventh the disk capacity of the ESX Server setup on similar processing iron. This is a pretty big piece of cash.”

Except I can’t find the result that Morgan is talking about.  I can’t find that config described, at least in 2009 or the second half of 2008.  Here’s the link to the VMware SPEC result:

http://www.spec.org/osg/web2005/results/res2009q1/web2005-20090128-00126.html

And here’s the link to the closest comparable result I could find, which is also a DL585, not a DL580, and also uses the Opteron chip:

http://www.spec.org/osg/web2005/results/res2008q3/web2005-20080618-00112.html

So – there are a few differences here.  The VMware result is using Opterons that are 300MHz faster than the stock HP results,  and the VMware server does have twice the RAM, 128GB.  However, they have basically the same number of disks – 110 in the VMware example and 100 in the HP, and the same chip architecture.  What was the difference in score?  The VMware result received a score of 44,000, while the stock HP received 43,854.

It’s indeed hard to see how doubling the RAM and increasing the processor speed by 10% wouldn’t increase the overall speed by more than a few percentage points.  I agree with Morgan that it would have been much nicer to see a true exact-hardware duplication of the results – virtual vs. non-virtual.

However VMware, like everyone else, is in the “lies, damn lies, and stastistics” world.  There’s still a strong perception out there that VMware and other virtualization solutions are “slow” – a statement that totally ignores potential areas where a performance tradeoff could yield huge benefits.  People are simply stuck on “speeds-and-feeds” concepts.  As long as they are, VMware will need these kinds of statistics to “prove” they’re “fast enough”.

(Here’s a link to the VMware press release)