Are there any limitations or issues with nested virtualization?
By submitting your personal information, you agree that TechTarget and its partners may contact you regarding relevant content, products and special offers.
Perhaps the most noteworthy limitation of nested virtualization is the compatibility of certain hypervisors nested within others. Not all hypervisors and operating system versions can nest successfully or reliably within all other hypervisors. For example, if Xen 4.5.0-rc4 is installed as the host hypervisor (L0) and VMware ESX is installed as a guest hypervisor (L1 or L2) Windows 7 or Windows 8 operating systems reportedly will not start causing a BSOD, while VMware-Workstation will cause startup problems when x86 or x64 RHEL versions are used.
For older hypervisors that don't virtualize processor enhancements or support extended page tables (EPT) or rapid virtualization indexing, you'll only be able to run 32-bit (x86) guest instances; processor extensions like VT-x or AMD-V must still be present for the L0 hypervisor.
Some nesting combinations have trouble with extended page tables. For example, Xen supports L2 EPT or VM control structure (VMCS) shadowing on L1 EPT, but Xen won't handle L2 EPT or VMCS Shadowing atop L1 Shadowing. That feature isn't yet supported. In other cases, using guest paging in an L1 hypervisor for an L2 guest can potentially disrupt and lock up the L0 hypervisor.
Generally, it's important to see that a nested hypervisor can produce a range of complex interdependency issues, so changing a hypervisor version at one level may cause instability problems for nested hypervisors and OSes above it. Not all virtualization hardware enhancements are fully supported at every nesting level. This can lead to performance problems, stability problems or outright VM startup failures. Test and benchmark everything before deploying changes -- don't assume a later version is better. Always refer to the latest hypervisor documentation and evaluate any vendor test results in nested hypervisor environments -- they may have already discovered incompatibilities -- and perhaps even found workarounds -- for certain hypervisor combinations.
Nested virtualization is still considered largely experimental and not quite ready for large-scale production. This is primarily because not all versions of all hypervisors will work at peak performance when nested under other hypervisor products -- the compatibility matrix is still evolving. It's important to do a bit of due diligence when planning a nested virtualization deployment and check each hypervisor you plan to use in order to understand preexisting vendor support for specific L0, L1 or even L2 hierarchies. Test and benchmark the combinations yourself to verify performance and determine where additional server hardware or software upgrades may enhance the performance of a nested hypervisor.
CPU requirements for virtualization
Prevent antivirus software from affecting hypervisor performance
Nested virtualization creates new cloud disaster recovery opportunities
Related Q&A from Stephen J. Bigelow
Photon OS optimizes VMware Photon platform deployment, not only in vSphere but in GCE, EC2 and more. Follow these steps to learn how to run Photon OS...continue reading
Performance problems can be caused by a number of things, including overprovisioning and poor vCPU selection and assignment to VMs. Use these ...continue reading
Think about what types of workloads are running on a VM before assigning compute resources, and consider using vCPUs from different cores for ...continue reading
Have a question for an expert?
Please add a title for your question
Get answers from a TechTarget expert on whatever's puzzling you.