News Stay informed about the latest enterprise technology news and product updates.

VMware details future Enhanced VMotion capabilities

At VMworld 2007, VMware announced details on Enhanced VMotion, which will enable live migration of virtual machines between multiple CPU types.

SAN FRANCISCO -- Moving virtual machines (VMs) across different types of CPUs won't be a problem for much longer, according to VMware Inc., whose executives gave a session on day two of VMworld 2007 titled "Enhanced VMotion."

For more on live migration:  
AMD pushes VMware, et al. to broaden live migration

VMware VMotion constraints may fade, Intel and AMD say

With support from Intel Corp. and Advanced Micro Devices Inc. (AMD), VMware is working to enable live migrations of virtual machines without issue, according to Matthias Hausner, manager of software reliability for VMware Inc. For now, VMotion requires compatibility between source and destination to guarantee stability of the virtual machine after migrations.

The information about Enhanced VMotion was revealed at a session at VMworld in San Francisco on Tuesday, Sept. 11, titled "VMotion between apples and oranges: Understanding CPU compatibility constraints for VMware VMotion."

Intel and AMD keep coming up with more features … and the magnitude of incompatibility grows with each addition.
Richard Brunner,
VMware Inc.

Intel's live migration technology Flex Migration allows live migration of virtual machines across its recent Xeon processor family (i.e., Pentium 4, AMD K7), and beyond. AMD has a similar offering, AMD-V, which allows users to perform live migrations across its Opteron processors and to future generations.

The problem is that users cannot move virtual machines across enemy lines from an Intel-based server to an AMD-based server, or vice versa, because they have different model-specific registers.

When moving between companies, even cold migrations can be problematic because of different kernel configurations, Hausner said.

Augmenting the problem, "Intel and AMD keep coming up with more features … and the magnitude of incompatibility grows with each addition," said Richard Brunner of VMware. "VMware believes it cannot solve these incompatibility scenarios without hardware support, so we have worked with CPU vendors to create a mechanism currently called Enhanced VMotion."

Enhanced VMotion will hide CPUID instruction-feature bits but doesn't disable associated instruction feature operation codes, Brunner said. It will hide, for example, SSE4 CPUID feature codes, but not operation codes.

Thus with Enhanced VMotion, the CPUs need not be an exact match.

"When a virtual machine boots, we will assign a feature flag image by data center policy or explicit override. … So when we are not using AMD-V or Intel's Flex Migration technologies, you can use the built-in CPUID Enhanced VMotion [feature]."

Brunner provided no information on when Enhanced VMotion will be available, adding that it could be included in the next version of ESX Server.

It's all about the CPU
For now, figuring out which CPU type, family and properties are built into each server is required for doing live migrations, and that can be tricky, Hausner said.

"Server-model compatibilities is where the difficulty comes in, because you have to figure out which CPU is installed -- and with which features -- to see if they are compatible," Hausner said. "In some cases, there can be a server with the same numbers, as with Dell [PowerEdge] 1950 which has an [Intel] Pentium 4 in one version, and another has a [Intel] Woodcrest chip in it."

Another bump in the road is the level of 64-bit support; users cannot run a VM with Windows XP Professional x64 Edition on a host that does not have a 64-bit CPU, nor can they migrate a VM with a 64-bit guest OS to a host that does not have a 64-bit CPU.

When it comes to support for NX/XD (the ability to mark memory pages as nonexecutable) -- and what SSE level the server is equipped with --VMotion capabilities are also stumped, according to Hausner. The next-generation Enhanced VMotion strives to make these problems a thing of the past.

Live migration best practices
Live migration requires that the physical server hosts are in the same data center and connected on the same Gbit network and on the same storage system. The destination host also requires sufficient resources to handle the virtual machine. Hausner recommends having a dedicated Gbit network for VMotion.

"CPUs are used by hypervisors as pass-through devices, and when you are moving a VM between noncompatible CPUs, you will likely end up with a crash," Hausner said. "It isn't an issue just with VMware; every hypervisor out there fights with this issue right now."

Hausner suggests that users check the CPUID instructions, which indicate the vendor, family, model and features that are supported. CPUID values and requirements can be changed in the Advanced section of the CPU Identification Mask dialog box -- as long as the virtual machine is not running, he said.

Users can bend the rules with some complicated CPU manipulation. If the NX bit is hidden from the guest OS, it's possible to do a live migration between CPUs with and without NX. In VI3, the NX CPU feature can be masked to hide it from the guest, a trick fully supported by VMware, according to Hausner.

Some operating systems that use NX if present include Windows XP SP2, Windows Server 2003, Windows Vista, Red Hat Enterprise Linux (RHEL) 3 and 4, Upd3, SUSE Linux 9.2, 10, and Solaris 10.

Let us know what you think about the story; email Bridget Botelho, News Writer.

Also, check out our news blog at

Dig Deeper on VMware virtualization

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.