A level 5 disaster recovery plan |
 |
By Stefanie McCann, Staff
24 Oct 2005 | SearchCIO.com |
|
|
For Aline Ward, CIO of Mississippi Power, Hurricane Katrina's force changed things forever. Ward lost her Bay Saint Louis, Miss., home and her personal possessions, except for some family silver, in the hurricane. She also learned an unforgettable lesson in disaster recovery.
Ward, like so many people, had to operate in emergency mode for many days, until Sept. 10, the day Mississippi Power could restore power to its customers. Her personal loss didn't stop her from dealing with the task at hand -- getting Mississippi Power's IT infrastructure back up while encouraging her IT employees to keep going after they lost so much.
Ward planned to defend against Katrina with a step-by-step disaster recovery plan -- detailed in a four-inch thick binder -- that began with emergency preparation. Step one: Wrap all PCs in plastic and move them to the center of each office. Step two: Top off all generators with fuel. Step three: Run all generators at full throttle to ensure they are working. Step four: Communicate with employees and inform them where to ride out the storm. In the end, Ward successfully fended off Katrina and kept her telecom room dry, although it came within two inches of being flooded.
She discovered what many CIOs with crisis experience already knew -- that people proved more valuable than processes.
Ward's IT employees found themselves working on projects that they never imagined. "Some of our IT guys literally were holding the door shut to our telecommunications room … while the wind was trying to blow it open. Then, as the water started to rise, they began to sweep out water from our telecom room," Ward said. "They put sandbags around the room; they continued until [the water] started to go down," Ward recounted. That's what saved us."
Due to Hurricane Katrina, Mississippi Power and Alabama Power had 930,000 customers without power. More than 10,000 utility poles were downed or damaged, and 3,450 miles of transmission lines were ruined.
Other folks on Ward's team carried four 250-lb. servers down seven flights of stairs and over a half mile to the makeshift headquarters. "All of our people did what they had to do to get us up and running," Ward said. "Everyone showed up for storm duty, some even walked from their homes. Once they arrived, they were willing to do what ever was needed."
Ward, CIO for 10 years at Mississippi Power, runs a shop with 40 IT staffers and a $40 million budget. Gulfport, Miss.-based Mississippi Power is an energy company that is a subsidiary of Atlanta-based Southern Company.
Before the storm hit, Ward grabbed that four-inch thick binder, nicknamed Crit Sit for its longer title, Southern Company Critical Situation Process. Crit sit contains predetermined disaster recovery procedures with flow charts, checklists and criteria for declaring a critical situation. As Katrina moved into the Gulf of Mexico, Ward referred to the binder to make the step-by-step decisions. Besides wrapping PCs in plastic and fueling the generators, there were some obvious steps, such as recharging laptop and cell phone batteries, but then there were the more obscure tasks that were crucial to communications immediately after the storm. One stood out as super important: satellite availability.
"[Satellite dishes] helped to get communications up and running," Ward said.
Using satellite phones, Mississippi Power was able to get its vendors -- suppliers of tents, food and fuel to Gulfport. Mississippi Power built a makeshift headquarters for the recovery efforts, which was key in restoring power to the company's customers.
For Ward, it's the images of her staff hard at work that remain the most vivid. "One thing that I have to say, it's about the people you hire. Our IT people were willing to do anything," she said. "They had endless energy and teamwork."
Ward lost her house as a result of Katrina. The Southern Company family services team at Mississippi Power went to her home to pick up what little valuables she still had. "They found one picture of my sister who had passed away several years back, and they dug some of my mother's silver out of the mud."
To get through every day since Katrina, Ward is fueled by the feedback she gets from her company and how proud they make her feel about her IT staff. "They ask me all the time, 'How do you pull rabbits out of a hat when you don't even have a hat?"
|
|
|
|
