Hyper-V and PerfMon: Not your usual processor counters

If you want to monitor virtual machine performance on a Hyper-V platform, you'll need to learn about the differences of Hyper-V's PerfMon counters. An expert explains their distinguishing features in this tip.

So you've virtualized a few servers to run atop Microsoft's Hyper-V. Now you want to gain an understanding of their performance behaviors. You want to view how many resources those virtual machines (VMs) are consuming on your virtual host. In an all-physical environment, accomplishing this involves looking at a few counters within PerfMon. In this tip, I'll explain how to use PerfMon's counters in Hyper-V in order to effectively monitor the performance of your VMs.

PerfMon counters in Hyper-V
Installing Hyper-V to a new server enables more than one computer to run on top of a physical host, which means that more than one set of PerfMon counters apply. At the same time, the ways in which Hyper-V's hypervisor drives those residing virtual machines changes how performance is measured. If you plan to keep an eye on the performance of your VMs, you're going to have to learn Hyper-V's new PerfMon counters to be effective.

A physical processor can run only a single instruction at a time. Each computer, physical or virtual, assumes that they have unrestricted and sole access to onboard physical processors at all times. This means that a major role of Hyper-V's hypervisor is in scheduling processor access to virtual machines as they need it. With multiple VMs on a single host, this also means that virtual machines that aren't actively serviced by the hypervisor actually enter a type of wait state until their next turn.

Each individual period of time a virtual machine gains processor attention is so small and swapped so fast that collocated VMs appear to run simultaneously. But, in fact, they're not. To illustrate this, if you look at the % Processor Time counter on a fully-loaded virtual machine from the PerfMon instance inside that virtual machine, you might see 100% utilization. However, if at the same time you look at that counter from the host itself, you'll see something entirely different and, more than likely, far less utilization. The hypervisor's scheduling of processor resources in combination with any reservations and/or limits you apply in the Hyper-V Manager console affect the virtual machine's true resource usage.

Using PerfMon counters to monitor VMs
To gain an honest vision into your virtual machine's processor utilization, the correct counter to watch is not Processor\% Processor Time. That counter will give you the total processor utilization across all activities being done on the system. For a specific virtual machine, use the counter Hyper-V Hypervisor Virtual Processor\% Guest Run Time ({instanceName}). The value of {instanceName} will equal the computer name along with the virtual processor number associated with that virtual machine. This metric will display the processor utilization associated with the running of just that virtual machine. If you want to add to this number the hypervisor processing overhead associated with that virtual machine, use the counter Hyper-V Hypervisor Virtual Processor\% Total Run Time ({instanceName}).

One benefit of using this new counter is in determining individual processor use across multiple virtual machines at once. If you add multiple counters to your PerfMon graph, one for each VM instance, you'll be able to identify which virtual machines are consuming certain amounts of processor attention.

You'll see that another set of counters is also available called Hyper-V Hypervisor Logical Processor. Whereas Hyper-V identifies a Virtual Processor as one that is assigned in a virtual machine's configuration, a Logical Processor is equivalent to the count of actual processors actually on the system. In effect, a Virtual Processor can run atop any Logical Processor in accomplishing its mission.

This counter is referred to as Logical Processor as opposed to physical processor because it takes into account the number of cores per socket as well as hyperthreading CPUs. Thus, a dual-core, quad-socket computer with hyperthreading enabled will show 16 logical processors. This added counter gives you a second perspective of processor utilization on your Hyper-V system. While the earlier counter identifies processor use by a specific virtual machine, this secondary counter gives you utilization by individual processor device. Both can come in handy during performance monitoring to determine how well Hyper-V is making use of its available logical processors.

Another counter that is the inverse of what we've discussed to this point is Hyper-V Hypervisor Logical Processor\% Idle Time. This counter can be considered functionally equivalent to Processor\% Idle Time on a server without Hyper-V installed. This counter – which can be selected in total or by Logical Processor – identifies the percentage of time that the server's processors are waiting to perform useful work. As a general rule of thumb, you can consider this counter to represent the amount of resources that remain available on that server. Measuring this counter over time will give you a general understanding of whether your server can handle additional virtual machines or not.

A final processor counter to keep an eye on is Hyper-V Logical Processor\Context Switches/sec. Since any particular processor can only run one instruction at a time, and since the job of Hyper-V's hypervisor is to swap out running instructions to give the illusion of simultaneous VM processing, this counter identifies the rate in which virtual machines are swapped out of any Logical Processor.

A high rate of Context Switches means that a server is spending a large amount of time switching between needy processes, and as a result may be overloaded with virtual machines. To that end, offloading one or more virtual machines may improve its overall performance. The best way to watch this counter is in comparison with others in your environment. If you see one server that has a high rate for this counter in comparison with others, consider re-balancing your running virtual machines across your available hosts.

ABOUT THE AUTHOR:Greg Shields, MVP, is a co-founder and IT guru with Concentrated Technology (www.concentratedtechnology.com) with nearly 15 years of IT architecture and enterprise administration experience. He is an IT trainer and speaker on such IT topics as Microsoft administration, systems management and monitoring, and virtualization. His recent book Windows Server 2008: What's New/What's Changed is available from SAPIEN Press.

Dig Deeper on Microsoft Hyper-V and Virtual Server