This content is part of the Essential Guide: Fine-tune your virtualization performance management skills

How can testing improve nested hypervisor performance?

Not all hypervisors and OSes can play nice with others. The best method for avoiding complications down the road is to thoroughly test your nested virtualization deployment.

Are there any limitations or issues with nested virtualization?

Perhaps the most noteworthy limitation of nested virtualization is the compatibility of certain hypervisors nested within others. Not all hypervisors and operating system versions can nest successfully or reliably within all other hypervisors. For example, if Xen 4.5.0-rc4 is installed as the host hypervisor (L0) and VMware ESX is installed as a guest hypervisor (L1 or L2) Windows 7 or Windows 8 operating systems reportedly will not start causing a BSOD, while VMware-Workstation will cause startup problems when x86 or x64 RHEL versions are used.

For older hypervisors that don't virtualize processor enhancements or support extended page tables (EPT) or rapid virtualization indexing, you'll only be able to run 32-bit (x86) guest instances; processor extensions like VT-x or AMD-V must still be present for the L0 hypervisor.

Some nesting combinations have trouble with extended page tables. For example, Xen supports L2 EPT or VM control structure (VMCS) shadowing on L1 EPT, but Xen won't handle L2 EPT or VMCS Shadowing atop L1 Shadowing. That feature isn't yet supported. In other cases, using guest paging in an L1 hypervisor for an L2 guest can potentially disrupt and lock up the L0 hypervisor.

Generally, it's important to see that a nested hypervisor can produce a range of complex interdependency issues, so changing a hypervisor version at one level may cause instability problems for nested hypervisors and OSes above it. Not all virtualization hardware enhancements are fully supported at every nesting level. This can lead to performance problems, stability problems or outright VM startup failures. Test and benchmark everything before deploying changes -- don't assume a later version is better. Always refer to the latest hypervisor documentation and evaluate any vendor test results in nested hypervisor environments -- they may have already discovered incompatibilities -- and perhaps even found workarounds -- for certain hypervisor combinations.

Nested virtualization is still considered largely experimental and not quite ready for large-scale production. This is primarily because not all versions of all hypervisors will work at peak performance when nested under other hypervisor products -- the compatibility matrix is still evolving. It's important to do a bit of due diligence when planning a nested virtualization deployment and check each hypervisor you plan to use in order to understand preexisting vendor support for specific L0, L1 or even L2 hierarchies. Test and benchmark the combinations yourself to verify performance and determine where additional server hardware or software upgrades may enhance the performance of a nested hypervisor.

Next Steps

CPU requirements for virtualization

Prevent antivirus software from affecting hypervisor performance

Nested virtualization creates new cloud disaster recovery opportunities

Dig Deeper on Virtual machine performance management