You can't rely on traditional performance monitoring tools for monitoring a virtual infrastructure. Using the same tools that you would in a physical environment often results in inaccurate and incomplete metrics.
For the most accurate method, you need to know which performance monitoring tools to use and what to monitor in your infrastructure.
Physical performance monitoring tools: Why not?
Performance monitoring tools that are designed for a physical infrastructure use an operating system to measure and monitor the performance statistics of servers' physical hardware. In a virtual infrastructure, the guest OS can't see the physical hardware, only the virtual hardware that has been emulated by the hypervisor. Because an OS can't directly access most of the host server's physical hardware, traditional performance monitoring tools won't function properly in a virtual infrastructure.
Further, a hypervisor provides the mapping between a virtual machine's hardware and the host's physical hardware. Because a guest OS is not aware of the virtualization layer, it believes it has the host hardware all to itself. As a result, an OS cannot detect or measure underlying resource bottlenecks in the virtualization layer. So performance monitoring tools from the physical world can produce inaccurate results.
Monitoring CPU usage
To gather the most accurate performance metrics, you should monitor performance at the virtualization layer instead of at the guest OS layer.
The hypervisor handles all VM hardware requests, but it uses different methods for different types of hardware. A virtual machine (VM) sees CPU that is not emulated, and the VM sees the same CPU type that is in a host server. But the VM does not see all the CPUs or cores that are on a host. Instead, it's been tricked into thinking that it has only the amount of CPUs that are assigned to it in the hardware configuration.
When a VM uses its CPU resources, it must go through the hypervisor, which schedules all requests for CPU time. When the host is busy, it can create a delay as a VM waits for a CPU request to be scheduled on a physical CPU.
This delay is not measurable from inside the guest OS. Only the host is aware of it, so only performance monitoring tools that are designed for virtualization can detect this delay. The measurement is important because it indicates whether there's a CPU resource bottleneck.
Monitoring virtual memory
Performance monitoring tools also keep track of memory resources. The hypervisor manages physical memory access and provides the mapping from the memory pages that a VM sees to the physical memory pages on the host server.
For a busy host, memory is often in short supply, so the host tries to conserve memory. The host consolidates duplicate memory pages -- known as page sharing -- and uses memory ballooning inside the guest OS. But VMs are unaware of the host's behind-the-scenes techniques, so physical performance monitoring tools would not detect this behavior.
The host may extra overhead memory because of emulated virtual hardware and the translation methods used. VMs are not aware of this extra memory usage. In fact, it's possible to overcommit memory on a host -- meaning that a host allocates more memory to VMs than it has physically available.
If all the physical memory has been used up, or the host places memory limits on a VM, the VM's memory is swapped to disk. Memory that's mapped to a disk-swap file is much slower than physical host memory, and VM performance can suffer. Again, a VM is not aware of this low-memory situation, so performance monitoring tools that monitor inside a guest OS will provide inaccurate results.
Page sharing, ballooning and overhead memory don't always affect a VMs performance, but it is important to be aware of these measurements to get the big picture on a host's memory usage.
Monitoring the network
When it comes to networking, performance monitoring tools are also critical. VMs use virtual network interface cards (NICs) that connect to virtual switches (vSwitches). These vSwitches have physical NICs, called uplinks, assigned to them.
Usually a host has a limited amount of physical NICs, many more vNICs than physical NICs are assigned to a vSwitch. As a result, VMs on the same vSwitch always share the host's physical NICs with other VMs. The hypervisor handles all the virtual-to-physical NIC translation and load balancing on the vSwitch, but the competition for a limited amount of physical cards can affect network performance.
Because virtual machines are not aware of what's happening inside the vSwitch, performance monitoring tools functioning inside the guest OS can produce flawed results.
Monitoring I/O resources
A VM's virtual disk resides on a local or shared host data store, which is shared with many other VMs' virtual disks. VMs have virtual disk controllers that connect to their virtual disk files, and the host's physical disk/network controllers connect to the physical device that stores the virtual disk. As a result, multiple VMs are competing for disk I/O on the host.
Disk is often the slowest resource, so bottlenecks can easily occur. With virtualization, it is common for more than one host to access shared data stores for advanced features, such as live migration. When multiple hosts access the same shared data stores for their VMs' virtual disks, bottlenecks can strike the back-end storage device.
A storage subsystem has many components, so as the host queues VM disk requests, bottlenecks can occur in different areas. I/O measurements inside the VM can provide an accurate measure of the disk throughput, but performance monitoring tools at the VM-level won't tell the whole story.
Virtual machine monitoring
At the virtualization layer, virtual machine monitoring is necessary; physical performance monitoring tools cannot see the layer between the guest OS and the physical hardware. When it comes to resource usage on a host, monitoring performance at the virtualization layer offers more accurate results. Performance monitoring tools also provide key measurements that are not apparent from looking inside a guest OS and that can be used to prevent and resolve resource bottlenecks.
But don't ignore performance monitoring inside VMs either. Inside the guest OS, there are performance measurements that you can't get by monitoring at the virtualization layer. For the most complete reporting possible, use performance monitoring tools that monitor the guest OS layer and the virtualization layer.
Applications such as VMware Tools, which provide an interface between the guest OS and the hypervisor, improve the accuracy of performance monitoring inside the guest OS. For VMware infrastructures, there are specific Windows Perfmon counters that provide reporting statistics at the virtualization layer.
Accurate performance monitoring and reporting is critical in a virtual infrastructure, where many VMs compete for limited host resources. Many enterprise performance monitoring tools are now designed to work with both physical servers and VMs, so a single-vendor solution is indeed possible. Still, tools specifically designed to monitor a virtual infrastructure typically provide the best information and results.
About the expert
Eric Siebert is a 25-year IT veteran with experience in programming, networking, telecom and systems administration. He is a guru-status moderator on the VMware community VMTN forums and maintains vSphere-land.com, a VMware information site.
This was first published in December 2010