VMmark virtual machine performance benchmark tool in hardware decision making

By Bridget Botelho, News Writer 22 Aug 2008 | SearchServerVirtualization.com

Server virtualization news and opinions Digg This!     StumbleUpon     Del.icio.us

For more on VM benchmarking: VMware VMmark fills virtual benchmark void Measuring virtualization performance: Current option

Traditional benchmarks like those provided by the standards body Standard Performance Evaluation Corp. (SPEC) measure a single workload on a server but don't measure the performance of multiple virtual machines (VMs) or the ability of a server to support multiple simultaneous workloads on a single server.

VMware is working with SPEC's virtualization subcommittee to develop standard methods of comparing virtualization performance for servers, but until one is developed, VMmark can be used to measure VM performance. Besides VMware, VMmark can also be applied to other virtualization platforms.

The benchmarking begins with two-CPU servers using workloads VMware calls "tiles." Each tile includes a mail server, Java server, and idle VM reflecting a standby server, Web server, database server, and file server, said Jennifer Anderson, the senior director of research and development at VMware.

Vendors run tiles to test performance, and if they choose to, they can submit the VMmark benchmark results to VMware for listing on its website.

"We provide the benchmarks and template VMs for some workloads … the run rules and reporting rules. The vendors run it themselves, and if they need help with tuning, our engineers can help," Anderson said. "When a vendor is ready to submit a result, they send VMware a results package. … We review it to make sure they used a proper and sound methodology and we post it if it is appropriate."

"The intention [of VMmark] is that customers can look at the results and make decisions based on what they see. It isn't just about the fastest server; it's about making system comparisons; between blades and rackmounts or a two-core or four-core system. Someone can see how much more performance they get from upgrading to four core processors, for instance."

Mark Bowker, an analyst who focuses on virtualization at the Milford, Mass-based Enterprise Strategy Group, said that VMmark is useful for OEMs that want to know VM performance overhead.

"There is a need for this benchmark because we can consolidate so many servers onto virtual machines and have no real way to tell how much overhead there actually is," Bowker said.

Of course, users and OEMs never see poor performance results, because vendors aren't required to submit any of their results to VMware for posting, Anderson said. The same voluntary scenario applies to SPEC benchmarks.

That said, vendors have begun to use the results as product selling points. For instance, Hewlett-Packard Co. sent an email to press showing that its ProLiant DL785 G5 server achieved the x86 industry's top virtualization performance result on the VMmark benchmark. The DL785 eight-socket server achieved a score of 21.88 at 16 tiles (or 96 VMs), which is the highest virtualization performance and largest number of VMs on a x86 server in VMmark to-date.

That's great, but the DL785 G5 is also the only system in the 32-core category, so it won the top spot by competing against nothing.

HP also holds the highest VMmark result with its four-socket, 16-core ProLiant DL585 G5, where there are a total of nine systems being compared. Dell's PowerEdge M905 holds the No. 2 spot in the category, followed by the PowerEdge R905 and then the R900. IBM System x3850 M2 is fifth on VMmark's performance list for 16 core systems.

In the eight-core category, Dell's PowerEdge 2950 received the highest performance marks in the list of eight systems – six of which are from Dell. The other two are from Sun Microsystems Inc. x8440 servers.

Let us know what you think about the story; email Bridget Botelho, News Writer. And check out our; Server Virtualization blog.

Tags: Virtual machine performance management,  Virtual machine monitoring, troubleshooting and alerting,  VMware virtualization,  VIEW ALL TAGS

Digg This!     StumbleUpon     Del.icio.us