The two data centers that serve the Ohio Department of Developmental Disabilities (DoDD) have gone from zero to 90% virtualized in the last six months. IT managers attribute the virtualization project's success to advanced planning that had virtual storage, data protection and disaster recovery ready before virtual servers were powered on.
The DoDD provides social services to 80,000 Ohioans with developmental disabilities and their families, in part through 10 developmental centers in the state. Prior to the virtualization project, the agency set out to offer its employees and external users better application availability, with the goal of zero downtime, and to prepare for future growth.
Information technology manager Kipp Bertke said his IT background at financial institutions influenced how the department approached the task -- first by articulating its vision for the project without involving specific technologies or vendors, and then building the infrastructure to meet the goal. Bertke said this planning took about nine months, beginning in mid-2009. "Before we looked at the tools, we did the architecture," he said.
Planning ahead for DR was a priority for Bertke, given his background in the financial sector. "I come from a banking background, and at my previous company we set up replication to a secondary site," said Bertke. "It took a lot of staff. It was complex, and it didn't always work."
Previously, DoDD had five physical servers connected to a Storage Center storage area network (SAN) from Compellent Technologies Inc. Brian Brothers, the DoDD network administrator manager said the SAN was chosen with virtualization in mind, and it was selected over competitors for its built-in replication capabilities. The agency also chose Veeam's Backup and Replication software for day-to-day data backup as it planned for virtualization.
VMware's Site Recovery Manager stitches together virtual disaster recovery
With replication to a secondary site in place, DoDD added VMware's Site Recovery Manager (SRM) to coordinate the provisioning of new virtual machines at that secondary site, as well as to automate failover between the two sites in case of a disaster. Bertke said SRM was recommended by a local channel partner, Netwave, which had worked with other state agencies on virtualization projects.
Previously, the agency's disaster recovery plan required servers to be manually rebuilt and re-provisioned. As they were testing SRM, Bertke and Brothers said it took an hour and half to failover 50 machines. "I don't think I could [manually] rebuild one server in an hour and a half," Brothers said.
Currently, Site Recovery Manager doesn't offer automated failback to the original site, but Brothers said he doesn't see much need for that feature. DoDD has SRM set up to require a manual kickoff by an administrator before failover begins, so "false positives" don't trigger it accidentally. SRM can do 'what if' testing without actually failing over, eliminating two situations in which rapid failback would be required. For DoDD, the plan is to fail over using SRM only if there is a complete disaster at the primary site, in which case, there would be ample time to manually fail back without using a single automated operation.
The end result: a rapid-scaling virtual environment
With disaster recovery, backup, and virtual storage already in place, DoDD then started adding virtual servers to the empty, preconfigured infrastructure on July 1, 2010. Since then, the organization has spun up about 200 virtual machines, representing more than 90% of its servers. The goal is to have a 98% virtualized data center by the end of this year. After that, DoDD aims to have a virtual desktop infrastructure (VDI) fully deployed by the end of next year's first quarter.
"To date, we've spent more time on the planning and documentation than on the migration [to a virtual environment]," Brothers said.
Beth Pariseau is a senior news writer for SearchServerVirtualization.com. Write to her at email@example.com.