News Stay informed about the latest enterprise technology news and product updates.

Katrina: IT lessons in disaster recovery

SCP Pool Corp. CIO Tim Babco is among those executives whose disaster recovery plan stayed one step ahead of Hurricane Katrina.

While Hurricane Katrina battered the Lousiana headquarters of SCP Pool Corp., just north of New Orleans, the company's disaster recovery and relocation plan kept its business and data safe. Some 500 miles away, the $1.4 billion distributor of swimming pool supplies was operating from a VeriCenter Inc. center in Dallas.

The plan was put into action around 4 p.m. Sunday of last week, once Katrina was rated a Category 5 hurricane, according to VeriCenter founder Michael Sullivan. About a dozen key personnel, including senior director of IT Tim Babco and some of the company's call center staff, are using VeriCenter's data recovery center as their temporary headquarters -- and, in some cases, as a hotel. As a result of the data switchover, "in less than 24 hours they had everything cut over and up and running in our Dallas facility," Sullivan said. "They're living in hotels, and our data center and the office space we allocated to them have become the headquarters for their IT operations, and now their business," Sullivan said.

SCP could soon have some company, Sullivan said, as calls pour into VeriCenter from other New Orleans-based businesses trying to salvage their operations and data. "We're going to let them stay as long as they need to," Sullivan said. "If that's three months -- I imagine it's somewhere between 30 and 90 days, just because of the situation in New Orleans, with the water needing to be pumped out -- it could be months before they get back to some semblance of ordinary business operations."

VeriCenter is now managing servers for SCP, along with Active Directory and e-mail. The systems SCP is managing in Dallas include back-office accounting systems to distribution and warehousing systems. SCP's 1-800 phone number has also been redirected to VeriCenter's Dallas facility, Sullivan said, which caused the biggest delay in the switch.

The ability of companies to salvage their IT operations and their data in the wake of Katrina depended on corporate resources, the quality of IT disaster recovery plans and the ability to move data and employees quickly. Even then, it's hard to imagine every scenario in advance, said Nancy Markle, a former CIO at Arthur Andersen-Americas and Home Savings of America. Markle joined Home Savings' corporate office in Woodland Hills, Calif., just when the 1994 earthquake hit. The company had a rigorous disaster recovery plan. "You have to, in banking," she said. In the event of an earthquake the company also had arranged that people could get hotel rooms nearby.

"But what happened was, people could not navigate the streets, and that's what we didn't think through. Once you were on campus, you couldn't get off. If you weren't on campus, you couldn't get on," Markle recalled. The company had alternate generators, so it had power, but didn't anticipate people having to sleep on campus. One of the women who worked there had an RV.

"I don't know how in the world she got it there, but she did. She lived next to the data center, which was a good thing. And having an RV gave people a place to shower and change their clothes and take a nap," Markle said.

Markle learned another lesson the hard way. The company tested its disaster recovery plan several times a year, making sure its tapes got to the right location and worked. "What you don't plan for and what you don't test is going backward. You've moved to that third-party site. Now you've got your site up and running. How do you get back, and can you do it as seamlessly as going forward? It is not complex, but if you don't plan for it, things happen you didn't anticipate. Now you're messed up big time and you end up operating in this third-party site longer than you expected."

At Home Savings, every employee got an earthquake kit that included military-like supplies of food and first aid. "We even had body bags," Markle said. She remembers a colleague from a Home Savings office in Florida commenting that the emergency packs were more suitable for Southern California than Florida because her employees would never need body bags in a hurricane. "Now you look at New Orleans and just shudder."

Business partners as lifelines

M. Victor Janulaitis, CEO of Janco Associates Inc., a Park City, Utah-based management consulting firm, had one New Orleans-based client that illustrated the best-case approach to disaster recovery. His client, a $250 million, publicly traded distribution company, put a disaster recovery plan into place several months ago. As soon as Katrina was rated a Category 5 hurricane, the company moved its data processing center to a backup site in New Jersey and began evacuating its fleet of trucks. Switching to the New Jersey data center meant the business continued operating with no loss of data. It also saved 60 trucks from being stranded.

"That company's losses are minimal, except what losses they have at their facility," Janulaitis said. "Anything within 10 feet of ground level is not functional. They're up and running. Just not running in New Orleans."

Meanwhile, Thomas Jarrett, CIO for the state of Delaware and president of the National Association of State CIOs, said he offered help, like replacement IT equipment, to Louisiana deputy chief information officer Allen Doescher. Doescher's reply was, "very short, curt and basically said, 'Thanks for the offer. We're doing OK,'" Jarrett said. He said disaster recovery plans aren't a hot topic when farflung state CIOs get together, because disaster recovery plans vary from state to state.

"Those of us that are in coastal states pay attention to hurricanes," Jarrett said. "Somebody in Utah sure doesn't. They have other issues. We don't talk about the weather. What we do talk about is enterprise approaches to system deployment that helps you with the whole approach to disaster recovery."

"In a lot of cases it's about financial resources," Jarrett said. "If you lose everything, you have to go out and start buying new equipment. I'm pretty sure the Feds aren't stockpiling IT equipment anywhere."

Forming close business partnerships is another aspect of disaster recovery, Jarrett highlighted. "That's who we'd probably go to more than anyone else [in the wake of a disaster]," Jarrett said. "Verizon, IBM -- our business partners -- going to them to replace mainframe equipment or whatever else I had to do."

SunGard Availability Services' crisis management center usually provides disaster recovery services to one or two clients at a time. In the wake of Katrina, SunGard is handling 25 clients, said Bob DiLossi, manager of its Philadelphia-based crisis management center. Prior regional outages, like the Northeast power outage of 2003, have brought in as many as 76 clients at one time, he said. The typical length of stay for a disaster recovery client is six to nine days, DiLossi said. But one client whose data center was flooded by Hurricane Ivan last year stayed from September to January, and DiLossi expects similar lengths of stay from Katrina victims.

"A lot of folks will be here a while," DiLossi said. "They're definitely out of their centers, out of their areas. They're ravaged. Some are getting destroyed. We're definitely going to have people here for long period of time."

Dig Deeper on Enterprise disaster recovery and business continuity planning

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.