Another vendor, SWsoft, has joined the Standard Performance Evaluation Corp.'s (SPEC) working group for virtualization, but don't expect the company's presence to hasten the benchmark authoring process.
SWsoft, which makes the Virtuozzo virtualization platform, and shepherds the open source OpenVZ, has joined existing working group members AMD, Dell, Fujitsu Siemens, Hewlett-Packard, Intel, IBM, Sun Microsystems, and VMware, which chairs the group. The working group was announced this past November in the hopes of assessing "heterogeneous workloads that are spread across multiple virtual machines on a single server," the press release read.
But according to Ezequiel Steiner, SWsoft director of business development, the working group is still in its infancy. "They've had a few meetings," said Steiner, who participates in the group, but they're "still trying to define several things" – things as fundamental as "what the benchmark measures."
Unlike other benchmarks such as SPEC's own SPECweb2005 or the Transaction Performance Processing Council's TPC-C, which measure a single workload on a single system, the difficulty of the virtualization benchmark is that "there's not one application or one workload [that you are trying to measure], but a lot of them on a single machine," Steiner said.
Valid questions the benchmark's authors are still asking include whether the benchmark should compare the virtualization layer's performance to a physical machine, or to another virtualization platform? Likewise, should it expose how many lightly loaded virtual machines it can run, or how far you can push a few VMs?
Apples and oranges
SWsoft's Virtuozzo, it must be said, is very different architecturally than VMware, the only other pure-play virtualization software vendor in the working group, and thus, has different performance characteristics. As an operating system virtualization platform, SWsoft's Virtuozzo makes multiple highly efficient virtual instances of the same baseline operating system instance. VMware, as a hardware virtualization play, allows users to run any x86-compatible OS, but probably with a bit more overhead.
"Hardware virtualization enables diverse sets of workloads, and those kinds of combinations are not possible in an OS virtualization environment," said Raghu Raghuram, VMware vice president of products and solutions marketing.
VMware, for its part, has stepped in with a temporary solution and has released its own virtualization benchmark "VMmark" to partners partaking in a private beta. The benchmark measure seven different application workloads and groups them in to what VMware dubs "tiles." The company plans to publish VMmark this year, Raghuram said, and will use it until which point SPEC has published a satisfactory replacement.
When will that happen? "The sooner the better," said Raghuram, but at the same time, the benchmark must be "rigorous, so that will take a bit more time."
How much more time? Probably years, said Richard Fichera, Hewlett-Packard director of blade system strategies, who in previous role as an analyst, worked with vendors attempting to develop the benchmark. "It's such a quagmire," he said. "It's going to be a long time before you see a 'TPC-v'."
Let us know what you think about the story; email: Alex Barrett, News Director