Earlier this month, Sunnyvale, Calif.-based Advanced Micro Devices Inc. (AMD) caused a stir when it demonstrated live migration of virtual machines (VMs) between one system running its new Shanghai quad-core Opteron processor and another running on an Intel Xeon processor. The feat was performed with the live-migration capabilities in Red Hat's (Xen) virtualization suite. But neither VMware Inc. nor Intel Corp sanctioned the migration, and their mutual skepticism puts the prospect of cross-platform live migration in doubt.
In an ideal world, people could live-migrate virtual machines (VMs) between any system, but "from VMware's perspective, the issues associated with testing and qualification increase greatly as you expand supported configurations," said Gordon Haff, an analyst at Illuminata Inc. As for Intel, it has dominated the market with its quad-core Xeon CPUs (thanks in part to AMD's lateness in releasing its first quad-core processor, Barcelona) and, therefore, "doesn't have a lot to gain by making it easy to live-migrate to AMD processors," Haff said.
And while VMware supports live migration (which it calls VMotion) across processors in the same family (i.e., from the same vendor), it does not support it across different types of processors because it could hinder system stability.
"VMware currently provides full support for Enhanced VMotion Compatibility, which allows live migration of enterprise workloads across different processor families within the same CPU vendor. This technology provides customers the flexibility to move these workloads across different processor iterations in a stable and reliable way," said Richard Brunner, the chief platform architect at VMware and a former Intel CPU architect.But "attempting to make cross-vendor x86 instruction sets and features compatible in a VM for live migration puts this stability at risk, and so we have not pursued it," he added.
Not surprisingly, Intel agreed. Jake Smith of Intel's Server Product Group said that live migration across different vendor platforms is not viable today because AMD and Intel technologies are simply not designed to interact.
"It is the same reason Chrysler parts don't work in GM cars. Our architects don't consult AMD, and vice versa," Smith said.Technological hurdles
Cynics might argue that VMware's stance on cross-platform live migration has less to do with technological roadblocks and more to do with Intel's 2007 investment in VMware the tune of $218.5 million . But both VMware and Intel insist that they haven't pursuing cross-platform live migration because of significant technological barriers.
In fact, VMware goes "to great lengths to prevent people from doing cross-vendor VMotions … because they are not safe," wrote a VMware engineer who has worked on CPU compatibility issues for VMotion.
"It's unpredictable. We assume that users care a lot about stability of their VMs, so we invest a lot of time to prevent people from doing things that endanger the stability," the engineer wrote. "Our stance from a technical side is better be safe than sorry. We want a VM to run on a rock-solid platform."
One of the reasons cross-platform live migration isn't considered stable is that AMD and Intel CPUs have "certain extensions -- new instruction sets like SSE3, SSSE3, SSE4.1 [on the Intel side] and various other instructions on the AMD side. Any VM that uses any of these instructions that are not present on the target of a hot migration will crash after the migration," the engineer wrote.
The Extended Migration technologies in both Intel and AMD processors "hide" the presence of such extensions from VMs to avoid crashes during live migration, the engineer said.
Each vendor also has dozens of special registers in its CPUs that operating systems use to control low-level functions of the hardware, and OSes adjust themselves depending on whether they run on an Intel or AMD chip, the VMware engineer said.
"They typically do this once when they start up and then behave the same way until you shut down the OS. A hot migration to a different vendor [CPU] then could mean that the OS is doing the wrong thing once it migrates," the engineer said.
Cross-platform live migration would also mean less sophisticated chips, Intel argued. Iddo Kadim, the director of virtualization technologies for Intel's Server Platforms Group, said in order to live-migrate between different vendor CPUs without posing reliability risks, all of the years of instruction sets and technology add-ons that make Intel and AMD processors unique would have to be peeled away.
"There is a tradeoff between flexibility and performance. To get to the point where you can live-migrate VMs between the two vendors, you have to dumb down the processors. … You would have to eliminate instructions down to the point where the instructions of the two vendors are the same, and this would take us decades backward," Kadim said.AMD, user see the need
For its part, AMD is unapologetic about its plans to develop cross-platform live migration, and dismissed VMware as "being very conservative and pragmatic about this," Mueting said.
"We understand there are challenges to making this work on a broad scale, but our engineers are committed to it," said Tim Mueting, AMD's virtualization expert. "There are certain situations where [cross-platform live migration] is not the best solution, but there enough workloads out there that are suitable to make it worthwhile, and the customers we talk to think it is worthwhile."
One such potential customer is Bob Plankers, the lead Linux and VMware systems engineer at the University of Wisconsin, who said the flexibility to live-migrate between vendor platforms would be a tremendous benefit to users by eliminating CPU vendor lock-in. "[Today], when you want to grow your virtual infrastructure, you have to do it with the CPU vendor you already have, regardless of price, performance or feature considerations," Plankers said.
AMD has a history of pushing the envelope with live migration, Mueting added. VMware customers used to be much more constrained about which the they could VMotion between, but we brought them technology to migrate across different generations of processors [with Extended Migration].We were very aggressive in finding a solution, and this is in the same vein. We want to go down this path," Mueting said.
And AMD dismisses the argument about performance tradeoffs to make cross-vendor live migration a reality. "We don't want to hold back technology advances, which happen at a rapid pace, but as virtualization becomes pervasive, we want to make sure we offer the best chips for that space," Mueting said. "We understand there are issues to be solved, and those who discount this technology can come up with plenty of reasons to discount this demo, but our objective was to start the conversation about it," he said. "Good ideas will come out of it. We generated interest in this within the ecosystem."