After plenty of research, you've finally made the switch to a virtualized environment.
You've installed all of your hosts, created hundreds or thousands of virtual machines, but there is no time to rest. The initial virtualization rush has slowed down, you're now at that goal of 80% or higher of a virtualized environment. However the virtualization journey has not ended, there is still a part that is often overlooked. As with any project, once the heavy lifting is done there is always some cleanup that needs to be done. These odds and ends are not miscellaneous tasks to be done by the team intern. They are the final pieces that complete your virtualization efforts and ensure that it will continue to be successful in the years to come.
Baselines and performance
One of the first indications of knowing if something is broken is when it does not function as expected. But how do you know something is broken if you don't know how it is supposed to function in the first place? Performance data gives you a current or past view of your servers or infrastructure, but that still doesn't tell you if something is wrong. For you to know if something is wrong, the performance data needs to be trended over time to help establish a working baseline of that server. Is the server busy at night or during the day? Is there a rush with month end processing of data? All of these spikes and lulls will help you to establish a pattern of usage for that server.
Once you have established a pattern of activity, then you have a working baseline for how that server performs. Any behavior outside of this baseline can be a reason for investigation. A sudden jump in server resource use after patching or a software update could require investigation into a possible issue with the update. This gap between where you were and where you are today can help to prevent unwanted outages or resource hogs in your virtual infrastructure. While some variance will always occur, it's the big jumps that are the cause for concern; and the only way you can tell it's a big jump is knowing where it was supposed to be.
Documentation can sometimes be as elusive as Bigfoot but it's not for the reasons that one might suspect. Management talks about how important it is and that it must be done. Although administrators would rather be working with the technology than creating documents about what they already did, it doesn't mean they won't do it. One could say management gives the administrator plenty of time to do documentation after a project, and would never reassign them to the next hot project without that cleanup time, right? So, the truth is somewhere between the administrator only partly wanting to do it and management giving them almost no time to do it.
Documentation doesn't have to be a dirty word in IT. There are ways to make documentation successful without sidelining your administrator for weeks to create it. Believe it or not, most administrators enjoy documentation. Now, before anyone riots, let me explain. If you look at most administrators' cubicles or work areas, there is almost always a common element found -- a whiteboard. IT administrators use this to design the next infrastructure, but why stop there? Often times these boards are photographed and referred to over the course of the project. Instead of leaving it on the board, bring it into the computer where it can be used as a foundation for your documentation. Visio has been a product many have used to create in-depth pictures of infrastructure and, as we all know, a picture is worth a thousand words.
While documentation can be done at the end, if it's started at the beginning it becomes a living document that no longer requires those weeks at the end of a project to create. So here are a few tips to help you create along the way:
As you start to layout your hardware design, put it on paper. It doesn't matter if it is blades or rack mounts. Creating both a logical and physical layout helps to show data center location and IP addresses of ILO or DRAC.
- If this is a new cluster, continue your logical design that already shows names and IP addresses, and add with master and slave configurations for HA. The logical cluster layout should also show network switch connections, VLANs and SAN/NAS connections for the cluster.
- Start a new tab and continue to build the infrastructure and the logical document of your cluster will become even more important. Add the VMware vCenter you are connected to and then expand it by adding all of the DRS and HA rules.
- With another new tab, add the resources pools, limits and shares.
With these steps you are no longer creating a document at the end of a project, but a living document that walks with you hand in hand as you make the changes. What you will find is, at the end of a project, this living document has 90% of all of the "documentation" your management will be looking for, without the burden of trying to rush it all at the end. And when you combine the documentation you have created with the performance and baselines data, you now have a complete picture of your virtual infrastructure.