Maintaining a virtual environment is no small task. Even in small deployments, there's a lot going on behind the scenes that administrators need to be keep track of. Without proper planning and diligent attention to changes, minor problems can spiral out of control quickly.
This month, we're asking our Advisory Board experts about the most common mistakes they see and how you can prevent those mistakes in your virtual environment.
Jason Helmick, Concentrated Technology LLC
You're making a mistake if you don't have a plan for configuration management and drift.
Public cloud or private, from a handful of servers to thousands, you must have a plan in place to manage your virtualization environment's server configurations and control drift. Talk to any Linux expert and they will spend countless hours describing the benefits of Puppet or Chef, and they are correct. Talk to a knowledgeable Windows administrator and they will do the same about Desired State Configuration (DSC).
You need to be able to quickly and easily deploy servers with the exact configuration you need without long complicated scripts and lengthy platform testing. You also need to control what happens to the servers over time. Is there an admin making changes and causing the servers to drift from your perfect configuration?
When something – anything – goes wrong in your environment, the first question you ask is: "What's changed?" It's rare that hardware just quits working. Most of our outages come from someone making a change, intentional or not. Prevent those outages and get better control over your environment with proper configuration management. Pick your weapon: Puppet, Chef or DSC. No matter your choice, get a handle on configuration management before it handles you.
Brian Kirsch, Milwaukee Area Technical College
With today’s virtual environments, software has removed much of the challenge of setup, configuration and maintenance. Virtualization is slowly moving down a path of setup wizards and preconfigured appliances, allowing anyone from the most skilled to the amateur to create and maintain a virtual environment. This is where the true problem can arise. It is not a technical issue, but a case of complacency. Even things that appear simple on the outside can be large and complex underneath. I don’t have to mention icebergs and ships, do I?
While a virtual environment seems simple, it is an incredibly complex system to install, configure and manage. In a perfect world, we never see beyond that magic curtain because nothing ever goes wrong. Last I checked, we are not in a perfect world and things do go wrong, which means someone has to fix it. The technicians have to work with consultants and the vendor to find out what happened, a process where the more you know about what is behind the scenes, the better. It does not mean you have to be able to fix the issue yourself, but the ability to properly communicate the issue can become one of the key steps in getting your systems back online.
A business can avoid many of these headaches by moving virtualization and key production applications to online or cloud-based services. This removes the possibility for mistakes and maintenance issues. There are dozens of possible cloud services to provide an organization the modern data center it needs without the headaches. Well, unless you count the outages that Amazon Web Services, Dropbox, Microsoft Office 365, Adobe Creative Cloud, Google Drive and several mail services such as Gmail, Outlook and Yahoo had in the last 18 months. While it is somewhat rare for these services to be offline, waiting on hold along with hundreds of thousands of other folks makes me think twice about trusting core production to the cloud.
As your organization starts down the path of virtualization, whether inside your own data center or the cloud, ensure that your staff is well trained. They don’t have to be experts in every aspect, but they should not be scared either. Virtualization can offer an amazing advantage for your company, but it is something that needs to be handled with a level of control and understanding or it can turn into an unruly monster.
Rob McShinsky, Dartmouth-Hitchcock Medical Center
Neglecting your virtual infrastructure is the biggest mistake you can make that can come back to haunt you. With the servers of yesteryear, you could get away with old firmware, infrequent patching and poor storage or network management because a single server hosted one application. With server virtualization, disregard of these basic tasks can cost you performance, and at worst, create downtime for hundreds of VMs. All hypervisors, servers and storage are constantly evolving, and bugs are found. VM workloads that run on top of this infrastructure are also expanding exponentially. With resource levels so high on these virtual hosts and the infrastructure behind it, it is no wonder that these resources can bend or break causing downtime or bad performance.
The lesson here is, know your stack from top to bottom. Know how updating one piece of this infrastructure warrants upgrades in another. Be aware of performance thresholds as well. If nothing has changed in the environment and you suddenly are seeing instability, look to how taxed any one of your resource layers might be. To succeed now, you need to be a student of your environment, and select a monitoring tool that allows for both diagnostic and trending across your private cloud stack. The public cloud entities are already excelling at this resource and standardization design. Keep an eye on where they start to fit to lessen your constant learning curve of today.
Dave Sobel, Level Platforms Inc.
The biggest mistake I find when managing virtual environments is the absence of a system to manage and maintain inventory and information about the environment. Often times, systems proliferate organically, and small to medium sized organizations will not deploy systems to retain information on system configuration, usage or physical location of the hosts. In the event of a disaster scenario, this information is critical for rapid response. Unfortunately, this is often overlooked during organic growth.