Published: 10 Dec 2008
If you've recently consolidated and virtualized your physical servers, you may think that the worst is behind you and that your data center is vastly more efficient. In a perfect world, this would be the case. However, there are things that demand attention upon completion, such as backup and prevention of VM sprawl, to ensure that your virtual environment remains healthy and secure. In this tip, I'll explain eight specific tasks that should be undertaken directly following the completion of a virtualization project in your infrastructure.
Eight must-do tasks after implementing virtualization
1. First off, you should dispose of all the old physical servers that you no longer need. Chances are the reason you virtualized in the first place was to reduce power and cooling costs, so why keep the root of those problems lying around? Resist the temptation to use them for other things, as that defeats the purpose of virtualization. Unplug them, take them out of the racks and call someone to haul them off.
That being said, you might want to keep some of the newer servers around and run free virtualization products on them to complement your new virtual hosts.
2. Devise a regular schedule to apply operating system (OS) patches to your virtual hosts. The importance of this cannot be emphasized enough. Virtual hosts have many guests running on them and if they are compromised due to an un-patched vulnerability, the guests can suffer the same fate no matter how secure and well-patched they are.
Patching host servers can be more cumbersome than patching guests because normally the guests need to be updated (i.e. VMware tools) and the host needs to be restarted. To ease the process, I implement automated patching scripts or use third-party applications.
Also, be aware of patches as they are released. Not all vendors release patches on a set schedule. Subscribe to email updates so you can be notified when patches are released and patch your hosts in a timely manner.
3. Stay vigilant against VM sprawl and look for ways to prevent it in your environment. This is no easy task, as it tends to happen over a period of time. As virtual machines do not have a physical presence, they can be easily created by several administrators. Many of these admins perceive virtual machines as "free" servers that can be created as needed.
Unfortunately, VMs are not free. Every virtual machine consumes host resources, and the more VMs you have, the more your host's resources are strained. If VM sprawl isn't controlled, it can result in a lack of host resources and bottlenecks that may severely impact the performance of all of your VMs. As a result, you may end up having to purchase more host servers and add additional shared storage, which costs money.
There are several options out there to prevent sprawl. One option is to implement a chargeback system like Vkernel's capacity and chargeback appliance or Vizioncore's vFoglight products for VMware that create reports on resource usage. Microsoft Hyper-V has some chargeback metrics available but lacks the tools to make effective use of them.
Another option is to limit who can create virtual machines. Devising a formal process for requesting new VMs is a more effective method than allowing every admin in the data center the access to create virtual machines at will. You should require justification for new VM requests and have an approval process in place so users can consider whether or not they really need to create a new VM.
4. Backing up virtual environments with traditional methods is often inefficient and time consuming. Installing a backup agent on each VM and backing the server up through the guest OS is resource intensive on a host server and can negatively affect all the VMs running on that host. I suggest implementing third-party backup products that are designed to work in virtual environments, such as VMware's Consolidated Backup, which acts as a backup proxy server to indirectly back up the VMs on a host. There are also alternate methods for backing up virtual machines, such as copying or replicating their virtual disk files to other servers or disk storage devices.
5. Traditional server management tools and products may not work as effectively in virtual environments because they are not aware of the underlying hypervisor layer of the virtual host. As a result, the performance metrics of your VMs may not be accurate because of products that are not designed to work in virtual environments.
This is an issue that comes from the basic nature of virtualization. The guest is fooled into thinking it's a physical server and that it has exclusive access to all of its resources. For example, a guest with 2 GB of RAM simply sees 2 GB of RAM, but in reality the host server is managing the memory and may be using advanced techniques such as memory page sharing and memory ballooning to make more efficient use of it. Additionally, if a host runs out of physical memory it may start using virtual swap files to provide the necessary memory to its VMs.
As a result, it's important to know what's happening in the virtual layer, which traditional performance tools are not aware of. I suggest using any built-in virtual performance and management tools, or you could implement one of the many third-party applications that are designed to manage virtual environments. This also applies to networking, as virtual networks frequently cannot be managed with traditional network management tools. There are many third-party applications available that are designed to work with virtual networks that can monitor and protect virtual switches in addition to the guests that are connected to them.
6. Security is of the utmost importance in virtual environments to prevent un-authorized access to your virtual hosts. It is an ongoing chore that requires appropriate measures to ensure that your environment is protected. This includes periodically scanning your hosts with security tools to look for configuration settings and open ports that could weaken your host's security.
There are a number of free security scanners that can be used to scan Linux and Windows hosts. In addition, there are many virtualization-specific security applications like Configuresoft's Compliance Checker and Tripwire's Config Check for VMware ESX hosts. You should also monitor your host server's log files to look for any security-related events that could indicate un-authorized access attempts.
7. Most virtual host system administrators are typically Windows or Linux systems admins who take on the additional responsibility of running the virtual host servers. Virtual hosts are different than traditional OSes and require a specific skill set to administer them properly. Therefore, you should make sure that your virtualization admins are properly trained to handle the job.
You may find that the installation and setup of the virtual environment was fairly straightforward, but the first time you encounter a critical problem it may turn out that your admins do not have the proper knowledge or skills to quickly correct the problem. Problems in virtual environments are amplified because they can affect several VMs and require that admins quickly pinpoint and resolve the problem. Proper training is a big part of this, so make sure that your admins have it and continue to get it since virtualization software tends to change rapidly as new versions and features are released.
8. Finally, now that you have virtualized you should continue to look for more ways to leverage your investment. Look for opportunities to virtualize additional physical servers, as virtual servers have a number of advantages (snapshots, for example). Make sure that your virtual hosts are as fully utilized as is possible, while at the same time reserving the necessary capacity to protect against a host failure. You should aim to use at least 70% of the capacity of your virtual hosts. Anything less and you're defeating the purpose of virtualization, which is to use all of the resources available on a server and minimize waste.
If you find that you are over-utilizing one resource while sparingly using others, consider adding capacity to that particular resource. For example, if your CPU utilization is less than 50% on your hosts but the memory consumption is at 90%, add more memory if possible so you can fully utilize the other resources.
As you can see, the work doesn't stop after you implement virtualization in your environment. By performing the tasks discussed in this tip, you can ensure that you have a healthy and secure virtual environment, while at the same time maximizing the return on your investment in virtualization.
About the author:
Eric Siebert is a 25-year IT veteran with experience in programming, networking, telecom and systems administration. He is a guru-status moderator on the VMware community VMTN forums and maintains VMware-land.com, a VI3 information site.