Once you've run benchmarks in your storage subsystem and assessed their results, you're ready to tackle them head-on....
The best place to begin is by looking for configuration problems in your storage infrastructure and removing tenants that monopolize your storage I/O bandwidth.
Solving configuration problems
In these days of flash drives and flash-rich storage arrays, it is possible for the storage infrastructure to be perfectly adequate for the task. Configuration problems, however, restrict VMs from achieving top performance. Just as there are many ways in which storage infrastructure can be a bottleneck, there are many ways in which configurations can gum up the works. Some of these are counterintuitive.
The most important bit of VM-specific advice is to install the hypervisor's virtualization tools into the guest OS. These tools contain drivers for the various drive types, and they can greatly speed up an individual VM's ability to make use of the available storage resources.
See if your hypervisor offers a paravirtual disk controller. This is a better choice than any of the others. Usually, however, this requires you to use either a special OS install disk with the drivers preloaded, or to change the controller type after the OS has been installed and the hypervisor virtualization tools have been loaded.
Often forgotten by many virtual administrators is the humble pass-through disk. If your storage system provides storage in the form of logical unit numbers, or you have some other form of dedicated storage device you would like a VM to use, passing that storage device through to the VM directly can greatly increase performance.
Use hypervisor-aware, antimalware software. There is no longer a need to run a bunch of uncoordinated antimalware tools inside each VM. Today's hypervisors are capable of coordinating with antimalware products to scan VMs in a joined-up fashion that places a lot less stress on the storage.
Also, you might need to align your partitions. This isn't an issue with all storage systems, and it's less of a problem with today's OSes than it was in the past. Still, it's worth investigating.
Handling extreme storage
One thing to bear in mind about virtualization platforms is that none of them were really designed with the idea that you would have a single virtual disk on a single VM using 10 million IOPS. When the core of these platforms was created, 10 million IOPS was a lunatic's mad fantasy. Even today, a storage system that can provide that will be costly.
It is, in fact, possible to buy a storage system that provides 10 million IOPS. You can even get hyper-converged clusters capable of a cumulative 10 million or more IOPS. So how do you make use of it?
First, to make 10 million IOPS available to a single VM, you're going to want to involve your hypervisor vendor. You simply aren't going to get there without working directly with the vendor's engineering team -- no matter what storage underpins the cluster.
If, however, you have 10 million IOPS available and need to figure out how to use them efficiently, look at storage limits in the hypervisor management software. Some workloads will gobble up every resource they can, making them what are called noisy neighbors.
Noisy neighbors cause all sorts of problems, and these are only magnified as the resources available multiply. Consider conducting an analysis for each of the workloads deployed, and locking down each of those workloads with CPU and disk limits, if not network limits as well.
Queue depth matters, especially on today's fast storage systems. An abundance of flash in the storage system, after all, won't do much good if you can't get the information to the flash drives.
Non-Volatile Memory Express is one example of a storage protocol invented to solve this problem. Flash drives are capable of such high performance in large part because they have low queue depths. Controllers -- both physical and virtual -- that group physical drives together and distribute data across them can also be subject to low queue depths. You can test for this by running benchmarks with a low number of queues and threads, and ramping them up incrementally to see if performance either flattens or -- more likely -- falls off a cliff as the I/O queue depth increases. New drivers, firmware or controller software can help solve these problems.
Storage is complicated. This is part of why there are so many vendors with so many products in the storage market. Problems can exist in many different locations, and the answer isn't always as simple as adding speed.
The number one thing a virtualization administrator can do to solve storage issues is to stop guessing and start testing. Remove uncertainty and blind stabs in the dark from the equation. Gather evidence, analyze the evidence, devise a hypothesis and test possible solutions.
Remember the interconnected nature of storage, especially when dealing with spinning disk. The more reads, the fewer writes and vice versa.
Know your storage. Different systems have different penalties for writes. Moving a write-intensive workload from a system that executes one read for every write to one that executes two reads for every write can have dramatic repercussions on other workloads.
When in doubt, ask for help. The storage and virtualization communities are huge, and chances are someone else has faced that same difficulty. Reach out to VMware vExperts, Microsoft MVPs, Cisco Champions and the EMC Elect. Asking in a relevant forum or poking the right people on Twitter can lead to quick answers or at least to narrowing down which vendor you should contact to get the issue sorted out.
Eliminate storage configuration problems to boost VM performance
Investigating Flash technology outliers
What to consider when purchasing an all-flash storage array