"Today, people primitively measure performance on virtual machines by looking at overhead the way they would with a physical machine, or they ask us about VM density," said Andrea Eubanks, VMware senior director of enterprise and technical marketing. "VMmark is a tool to answer those workload questions."
To develop the benchmark, the Palo Alto, Calif.-based virtualization software provider polled a number of its 20,000 users over the past two years on the types of workloads, configurations and the applications they use most widely on virtual machines.
VMmark measures six kinds of workloads -- file server, Microsoft Exchange Server, database server, standby server, a Java application, and OLTP (online transaction processing) -- with half running on Windows, the other half on Linux, Eubanks said.
"We chose workloads and configurations based on what users said were most important, and that could certainly change over time," Eubanks said. "Right away, we test 32-bit systems, and we anticipate 64-bit testing next."
Vendors can publish a VMmark score that provides scaling information about the workloads each product supports and the overall performance level of virtual machines running on a server.
Filling the virtual benchmark void
Mark Bowker, an analyst who focuses on virtualization at the Milford, Mass-based Enterprise Strategy Group, said the tool will be useful for OEMs in spelling out the overhead of virtual machines.
"There is a need for this benchmark," Bowker said, "because we can consolidate so many servers onto virtual machines and have no real way to tell how much overhead there actually is."
A vendor-provided benchmark automatically assumes bias, but VMware worked with partners to design and implement VMmark, and results can be compared among different virtualization platforms. For example, someone using Virtual Iron software can still download the benchmark from VMware's Web site and use it.
"[Benchmarking] has to start somewhere, so why not with the market leader?" Bowker said. "It isn't beneficial for them to create a benchmark for VMware alone; you can also apply it to other virtualization software and different hardware."
VMware is also working with SPEC to develop an industry-standard benchmark system for virtualization; but given the lengthy testing, design, documentation and approvals required, it will be about two years before the system is available, Eubanks said.
In the meantime, in October 2006, SPEC formed a working group called the Open Systems Group to focus on virtualization. The group includes Advanced Micro Devices Inc., Dell Inc., Fujitsu Siemens Computers, IBM Corp., Intel Corp., Hewlett-Packard Co., Microsoft, Sun Microsystems Inc., SWsoft, Trigence Corp., Unisys Corp. and VMware.
"The SPEC benchmark will be a bit different than VMmark -- probably not as comprehensive -- but that has yet to be seen," Eubanks said.
VMmark can be downloaded free of charge from the VMware Web site.
Let us know what you think about the story; email Bridget Botelho, News Writer.