We use VMware's vSphere to protect some of our business' most critical servers and applications, so wouldn't it make sense to keep vCenter under the same protection? In fact, VMware's decision to release a vCenter virtual appliance encourages administrators to take a virtual-only approach. VMware has clearly put a lot of planning into this with tools like High Availability, vCenter Server Heartbeat and Distributed Resource Scheduler. All of these features are designed to protect a virtual vCenter Server in the event of an issue, but can we rely on them?
All of these features are designed to protect vCenter in the event of an issue, but can we rely on them?
While I might be labeled as a heretic for saying this, I prefer to keep my vCenter physical and outside of a VMware infrastructure. Before the eye rolling starts, let's look at an example. When I leave my house, I lock it, but don't always take my house key with me. Instead, I rely on my garage door opener so I can keep my house secure (and the key secure inside). Everything works smooth until a power issue occurs and my garage door opener no longer works. Placing vCenter in the same virtual environment that it manages is OK, as long as you never have any issues that VMware High Availability (HA), vCenter Heartbeat or Distributed Resource Scheduling can't handle -- and what are the chances of that?
A few years ago, I worked for a very large company that experienced a complete data center outage. This outage encompassed every device on the raised floor, including switches, storage area networks, servers and AC units. The outage was caused by a person who accidently ran into a safety power shunt. Once power was restored, our ESX hosts started to automatically power back up and begin the process of restarting virtual machines (VMs). However, the storage frames were not back online yet, so HA could not function properly and was left in an unstable state because the hosts came online without storage available. We needed vCenter to be up and running for us to correct the unstable state and bring the hosts and VMs online after the storage frames came back online.
In this example, vCenter was on a physical server outside of the virtual environment. By having access to this critical data center resource, we were able to bring up the virtual infrastructure in about an hour. If vCenter was virtualized, we would have needed to locate it on the correct storage LUN and then try to bring up a single host and manually add it so we could gain access to our central management console and start to repair HA before we could bring up the rest of the infrastructure. In reality, it may have taken an hour to even locate the LUN on which vCenter might reside before we could even begin the process of trying to bring it back online. This event happened with vSphere 4.1 and, of course, there have been improvements made that may now prevent this type of problem. However, I haven't found anyone willing to lend me a data center to see if vSphere 5.5 would solve it.
Other opinions on virtualizing vCenter
Is it a good idea to virtualize vCenter server?
Faceoff: Is virtualizing vCenter worth the effort
Comparing vCenter virtual appliance and physical install
Even if improvements to vSphere 5.5 and HA could prevent this management nightmare, we still have to consider how a virtual vCenter Server will restart. In the event of a recovery, HA will restart vCenter, but will also restart all VMs marked with a "high" restart priority. In a large environment, this could mean hundreds of VMs. In this case, all you can do is wait for vCenter to respond because you have no way of knowing the progress or if it has encountered a problem, since you don't have access to your main administration point. Giving status updates to customers or executives when you can't check the progress is a very uncomfortable situation.
As VMware releases new versions and pushes a vCenter virtual appliance format, it's likely that we will eventually be forced to keep a virtual vCenter Server in the same environment that it manages -- unless we think outside the box. Since ESXi is free, with no memory limitations, it is possible to install vCenter on a single host with local storage and backup, or use vCenter Heartbeat to ensure we have local hardware failure protection. With no clusters or HA, we know exactly where vCenter is and we have a clear starting point to begin recovering our virtual data center. In a crisis, having a clear starting point is one of the most important steps to recovering.