Disaster recovery, failover and high availability for virtual servers
Staging a disaster recovery VM
By Chris Wolf
A key benefit of running production servers inside virtual machines is the portability of the VM's virtual hardware emulation. If one of your production Dell servers that hosts a VM crashes, for example, the VM can be redeployed on an HP server in order to resume operations. The VM's host system hardware is different, but the emulated hardware seen by the virtual machine would remain the same.
Of course, this is assuming that you are currently running a virtualization engine that provides full hardware emulation. VMware and Microsoft Virtual Server both offer this level of portability. Hypervisor-based virtual machines improve VM performance by allowing virtual machines to run on the bare metal. This improves performance but presents additional challenges for restoring VMs to a host with a different hardware platform.
I'll get to those issues later. First, let's look at the general steps involved in prepping a generic disaster recovery VM.
I look at disaster recover (DR) preparedness as having two viable approaches: one-to-many and one-to-one.
One-to-many DR VM staging
For organizations with limited resources, the one-to-many approach is the most efficient method for staging a DR virtual machine. With this method, you stage a single DR VM for each unique OS that you manage. For example, an organization that has virtualized Windows NT 4.0, Windows Server 2003, Red Hat Enterprise Linux 4.0 and a NetWare 6.5 will need to stage four unique virtual machine instances.
In having a virtual machine ready for each OS that you manage, your primary concern following a disaster or major system failure will be in restoring data from the latest backup. With the pre-staged OS, you will not have to worry about installing an operating system in order to recover from backup.
To create the one-to-many DR VM, you should follow these general steps:
- Create the virtual machine using your preferred virtualization software and assign it hardware resources that are equal to the resources of the strongest production VM. In other words, if your top production VM is using 1GB of RAM and has four 20GB virtual SCSI hard disks, the DR VM should be created with the same configuration. It's easier to remove or roll back resources in a pinch then to add them.
- Install the operating system. Note that installing any service packs or patches is not necessary since these will be added when the data is restored from backup.
- Copy your backup software agent installation files to the VM.
If your backup software supports bare-metal restores (such as Veritas NetBackup), then a single VM shell without an OS would be all that is necessary for full recovery. Your particular backup software's capabilities should drive your ultimate virtual machine DR strategy.
Assuming that you have prepared a DR virtual machine as outlined in this section, you will just need to do the following to recover a failed system in the event of a disaster:
- Copy the DR VM to the new server and folder associated with the source VM.
- Boot up the DR VM, set its IP address and rename it to the name of the source VM.
- Install your backup agent software on the VM.
- Use your backup software at the DR site to recover the VM's data from your most recent backup.
Although this approach allows you to maintain one DR virtual machine for each OS type, as you can see, recovering several virtual machines can be a very lengthy process. Another alternative is to create a matching DR VM for every production VM.
One-to-one DR VM staging
In the one-to-one approach, you configure a dedicated replacement VM for each production VM. This is far and away the best recovery approach; however, it will require more storage space due to the number of DR VMs that you will need to save to online or near-line storage media.
If your local and DR sites are limited on available online drive space, saving backup VMs to removable media such as DVDs or to tape is another alternative.
To stage a DR virtual machine for each of your production VMs, you can create the DR VM using any of the following methods:
- Use a third party tool to duplicate the live VM
- Power-down the production VM and create an offline copy of its associated files
- Perform a clean OS install in the DR VM and use your backup software to restore the most recent production VM backup to the DR VM
Several third-party tools can help you stage and maintain DR replica VMs for each production VM. Amongst the most popular tools in this area are Vizioncore's esxRanger and esxReplicator and PlateSpin's PowerConvert. PlateSpin's solution is unique in that it supports multiple virtual infrastructures, including VMware, Microsoft and Virtual Iron.
For those without third party VM cloning and replication products, a common alternative is to power down the production VM and then make a complete copy of all of the VM"s associated files. If the VM is mounted to physical disks (instead of virtual hard disks), you will need to secure a backup copy of the VM's data on the physical disk as well.
One method for capturing all of the data associated with the VM's physical disk is to back up the disk's files via a backup agent on the VM and then restore the data to the standby VM at your DR site.
Another alternative for creating a matching DR VM is to perform a clean install or duplicate a previously sysprepped VM. Once this is done, you would then just need to restore a backup of the production server image to the new VM.
Making the DR VM will save you time in terms of recovery, but it is often advantageous to keep the DR VM synchronized as close as possible its production counterpart. Software-based replication programs such as CommVault's ContinuousDataReplicator provide one other alternative for maintaining the standby VM. Alternatively, file level replication programs such as rsync or Robocopy can allow you to synchronize data on two remote systems.
If your virtualization engine is hypervisor-based, such as with XenSource Enterprise, then ideally you would want to have a standby VM host server on-hand that offers the same hardware configuration as your live production server.
When this is not possible, such as when a true disaster strikes and systems have to be brought online at a new site, then recovery from backup may require more than just a simple restore operation. Because the restored virtual machines would see the new hardware on the recovery system, you may run into driver conflicts that could prevent the system from booting.
Again, with the right enterprise data management software, this issue might not be too severe. Veritas Bare Metal Restore for NetBackup, for example, supports restore operations to hosts with dissimilar hardware.
Plenty to chew on
Perhaps the most difficult aspect of discussing DR preparedness is that every organization's circumstances are different. Since many of the enterprise data protection software vendors have well documented solutions for virtual machine backup and recovery, your organization's preferred software vendors are often a good place to start.
If you're in a smaller IT shop on a limited budget, hopefully you'll find some of the methods described in this article useful to your environment. In future tips, I'll be outlining how to use open source and free OS tools to maintain stand-by virtual machines across both LANs and WANs.
Chris Wolf is a Microsoft MVP for Windows Server – File System/Storage and is a MCSE, MCT and CCNA. He's a Senior Analyst for Burton Group who specializes in the areas of virtualization solutions, high availability, enterprise storage, and network infrastructure management. Virtualization: From the Desktop to the Enterprise (Apress), Troubleshooting Microsoft Technologies (Addison Wesley) and a contributor to the Windows Server 2003 Deployment Kit (Microsoft Press).
Reach him at firstname.lastname@example.org
02 Jan 2007
Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.