Requires Free Membership to View
a few days, high water knocked out not only generating facilities and substations in Iowa, but also the unthinkable: its 21-story downtown central facility, including the data center and network operations center housed on the fifth floor. More than 800 Alliant Energy employees were displaced, and 35,000 customers were left without power.
|
In the heat of the crisis, Alliant Energy learned that its move toward IP telephony was smart. So was a recent decision to invest in an OC-48 broadband connection to replicate data between its Cedar Rapids and Madison data centers. Both of these decisions helped bring the company back to its feet in just a few days.
On the other hand, an IT disaster recovery plan that focused on its Madison headquarters wasn't much help in Cedar Rapids and clearly was insufficient for a territory that stretches 54,000 square miles across three states. The powerful power company also learned you can't go it alone in disaster recovery (DR) -- it takes a village of willing vendor/partners to get people back to work.
A VoIP deployment proves its mettle as DR component
When Debbie Runkel, manager of corporate support and access, and colleague Dee Dee Crowley, team leader of Alliant Energy integrated network services, recount the events of the first few days of the crisis, you can still hear the adrenalin in their voices.
"Our job in IT is to keep everything running," Crowley said. And indeed, the pressure was enormous to get Alliant Energy's displaced Cedar Rapids employees back to work, so they in turn could help get power restored to thousands of Iowans.
 |
||||
|
|
|||
|
||||
Meantime, the scramble was on to set up displaced company employees with phone and network connectivity. Alliant Energy began calling telco partners and drummed up eight temporary facilities provided by a local college and by vendors in and around Cedar Rapids.
When the floods hit, IT was about one-third of the way through replacing an aging Nortel fleet of about 70 private branch exchanges (PBXes) throughout Wisconsin and Iowa with Voice over Internet Protocol (VoIP). The plan called for replacing about 5,500 phones at 136 sites, allowing fax, email and voicemail all with one system. Such unified messaging turned out to be very useful when the company went under water and employees were left without communications. Moreover, the changeover was overdue: The old phones required too much maintenance and the PBXes were many software versions behind. The business case for the project was pretty airtight, said Runkel and Crowley.
IT had also argued that IP telephony would help in DR, by delocalizing control of the phone system. But ironically, it had not been touted as a big piece of the IT disaster recovery plan. "We lucked out," Runkel said. "The rollout allowed us to respond much more quickly."
When the evacuation order came in, Runkel and Crowley kicked the VoIP deployment into high gear, setting up 120 workers in space secured from telco vendor Paetec Holdings Corp. In a matter of hours, they pulled off what would have taken two months to plan and implement.
"In other locations, we had IP phones hanging off of cable modems. We put an ASA box (another form of a pix-firewall) and plugged phones into those, so that gave us a tunnel back to our main network, so people had phone and network connectivity," Crowley said.
A week later, cable modems were dropped into trailers hauled in for other displaced workers.
"We had people on networked phones and network connectivity, which in the old world would not have happened," Crowley said. The old world would have required them to put in analog lines with different phone numbers.
Displaced workers did not have Cedar Rapids phone numbers initially, because the data center was shut down, so they received Madison numbers. IT was able to add those numbers to the corporate directory, so the Cedar Rapids employees could get voicemail.
The unified messaging proved to be "huge," Crowley said. "They could still get voicemail through email, so once they had network connectivity they were up and running. It was so important to get these people back to their jobs so they could get power back on for Cedar Rapids and other areas in Iowa."
The best part? "Sometimes I think IT is not always appreciated," Runkel said. "People were very impressed."
The takeaways
Runkel said the company had concentrated a lot of its DR planning on Madison, a huge hub for the rest of its Wisconsin territory. All the servers are there. And indeed, the business continuity plan helped Alliant Energy avoid the 20% per share financial hit it had warned about early in the disaster, according to reports in the local media.
|  | ||||||||||||||||
| |||||||||||||||||
On the positive side, the company had recently walked away from an only partially successful contract with an outside vendor to replicate its environment, and invested instead in an OC-48 line between its Cedar Rapids and Madison data centers to do the replication itself. "I think it really paid back for us," Runkel said.
"We never thought we'd lose power to the downtown tower. We never thought we'd have to vacate that building. In hindsight, we probably should have started replicating the data earlier, instead of doing it in such a rush," she said.
Going forward, Alliant Energy is considering also switching its call centers to IP telephony, which will make them more robust in case of the next disaster. This deployment was not included in the original scope of the project, which incidentally is on track to finish on time and on budget in March.
The utility is also looking into its network design to make it self-healing, so when a hub goes down the substations attached to it are not on islands unto themselves.
Let us know what you think about the story; email: Linda Tucci, Senior News Writer
Join the conversationComment
Share
Comments
Results
Contribute to the conversation