The problem with disaster recovery plans, Greg Folsom says, is not only anticipating all the things that can go wrong, but also paying for protection against them. "There is not enough money in the world to enable a medium-sized company to devote that much to DR," said Folsom, senior vice president of IT at Arnold Worldwide Partners, a Boston-based advertising firm. Well, certainly not enough money in the roughly $2 million annual IT budget at Folsom's disposal.
But if Folsom's 12-person IT team cannot afford to cover every DR contingency, it does need to provide a strong safety net for Arnold, which does more than $100 million in revenue and employs more than 650 people in Boston and regional offices in New York and Washington, D.C.
The agency, owned by the French media company Havas SA, may be medium in size but it trades in big ideas for brand-name clients that run the gamut from Hershey's and Lee Jeans to Carnival Cruise Lines and McDonald's. (The "Gimme back that Filet-o-Fish" spot-turned-YouTube phenom is an Arnold ad.)
When Folsom decided to upgrade the firm's DR infrastructure a couple of years ago, he was also thinking big.
"We were hearing a lot of buzz around server virtualization and green data centers, or greener data centers. I wanted to get into all this stuff and I needed disaster recoverability," Folsom said. "I needed a flexible solution that could accommodate a variety of situations. I call it my Swiss Army knife of disaster recoverability."
He combined a server virtualization effort with a virtualized storage area network (SAN) for a solution that he said gives back a lot more than DR.
Folsom consolidated 45 servers in the Boston data center to four physical servers, using VMware ESX server software. The servers in New York are also virtualized. He installed a Compellent Storage Center SAN in Boston and a second Compellent SAN at the New York office.
And, yes, flexibility is a major virtue of the solution. If one of the physical servers goes down in Boston, its 10 virtual servers will transition to the other machines, "and it happens in an intelligent way," Folsom said, with the technology gauging where best to plunk the data. The Boston data is mirrored in New York and vice versa (asynchronous replication over IP) so he can bring up his Boston office in New York automatically or manually. Right now he's doing it manually, so he can control how to fix the problem if a machine dies.
"I can figure out if this is a real disaster and want to bring up the machine in New York and then have to deal with bringing the data back to Boston, or if it makes sense to get the piece of hardware fixed and have the virtual machines moved to other Boston servers," he said. The Data Instant Replay (snapshots) can instantly restore a database containing metadata for nearly all of the agency's creative work.
He kept the tape backups Arnold relied on before, but now he also uses disk-to-disk backup and disk data deduplication. "It's like having a belt and suspenders," he said. Backups are stored either off-site or on a separate floor in the Boston office. (Data deduplication, a "game changer" technology like virtualization, is another gadget on his DR Swiss Army knife, allowing him to cram "lots and lots of redundant data" in a smaller space. ) Compellent's Automated Tiered Storage moves inactive data off high-performance drives.
Even better, the solution is saving him money in heating and cooling costs in the data center, not to mention eliminating those former command performances in the data center when the HVAC went out. The SAN gives him the ability to easily grow disk drives and take daily snapshots.
What we have learned is that our disaster recovery line [the point-to-point data circuit] can lead two lives, which allows my company to save money.
Greg Folsom, CIO, Arnold Worldwide Partners
Paul Clifford, founder of Davenport Group , a SAN integrator and Compellent partner, is a big fan of employing virtualization in the service of disaster recovery. Companies need to realize they have to build an environment that can deal with change, he said, "because if there is a single word that describes IT today, it is change."
"I think where most CIOs make their mistakes is that they really don't build a flexible plan. They build a plan 'to bring it up' if something goes wrong. The problem is that you cannot foresee all of the things that can go wrong."
Close to 80% of system downtime, Clifford said, is due to the complexity of IT architecture, from planned downtime to software failure and pilot error. Natural disasters are only a piece of DR. The more CIOs can simplify their architectures and centralize the control of data, so that "you truly have a data-centric approach," the more flexible --and hence robust -- their disaster recovery plan. "You do that through virtualizing. If you simplify and centralize, then you get the most important element -- flexibility," he said.
Arnold's DR solution is not perfect, Folsom said. In an ideal world, Boston and New York and D.C. would be replicating data amongst themselves. But Compellent SANs and VMware solutions don't come cheap. Also, the Washington, D.C., office, with about 90 employees and one IT person, until recently did not have the Internet connectivity to support the solution. Plus, with only five physical servers there, virtualization will have to wait until those machines need replacing, he said.
But his flexible disaster recovery plan is still paying dividends, Folsom said.
"What we have learned is that our disaster recovery line [the point-to-point data circuit] can lead two lives, which allows my company to save money," Folsom said.
"By day, we utilize the DR line for our employees in the [New York] DR site to come across to my network and get out to the Web and come through some security devices I have here in Boston, so I do not need to duplicate devices in their office," he said. The 100 MB line is Verizon's Ethernet over Sonet.
The IT team still takes a couple of data snapshots during the day and replicates those across the line, "but we have enough bandwidth to support the replication and normal business use."
Let us know what you think about the story; email Linda Tucci, Senior News Writer.