"What happened was our building had lost a primary power transformer," Ridolfo said. The backup transformers had failed, too, and the fire department was investigating reports of smoke in the electrical room. The utility company was unsure when the power would be back on, and the fire department was restricting access to the building.
"If you take our disaster recovery backup facility out of the picture, Cambridge is the hub of all our corporate computer assets," Ridolfo said. "Everything hubs off this site." He said the Cambridge office even serves as a hub for the IP telephony service of the company's London and New Zealand branches.
Ridolfo had been developing a new disaster recovery (DR) plan for nearly 18 months, in response to the Aug. 14, 2003, blackout that left New York and much of the Northeast in the dark for days. He had a new backup data center in nearby Somerville, Mass., and his employees were all enabled to work remotely during a disaster.
"The majority of our professional staff travel, and they have always had some degree of ability to work remotely. It's really only been since the power outage in New York that we've consciously put in an effort to redesign and configure our systems to be accessible from anywhere. Not having power for three days in New York taught us what was at risk."
During this most recent go-around, a call went out to senior IT staff and senior executives to initiate a call tree that would inform the entire business of the situation. The power failure hit at 8:30 p.m. By midnight, everyone in the company had received a message.
Next he verified that the company's backup data center had taken over for Cambridge. "We ran into a few items that didn't work, so we had to make manual configuration changes."
By the next morning, the power was still out. One employee was sent into the building to collect any laptops and printed files still inside that were critical for employees. "All our [Cambridge] employees who were not traveling worked from home that day," Ridolfo said.
The risks of not planning for such a disruption to the business were made plain to Ridolfo as he watched the chaos unfold that morning as other tenants in his building struggled to deal with the situation.
"It was a huge mess," he said. "People were showing up for work with no clue. Us, we had one person show up who didn't get the message."
Ridolfo said there was minimal impact to his business. The Cambridge telephone system rolled all calls to the New York office, and the operator would reroute calls according to a list of alternative phone numbers for employees.
Ridolfo controlled this issue by explaining to employees that lower-level fixes would have to wait until the event was over. The help desk instead concentrated on access issues for employees who were working from home.
Ridolfo said the success of his disaster recovery plan was also tied directly to early vendor involvement. It's absolutely critical to include your vendors in the planning and testing process. SH&E relies heavily on a network management firm, Atrion Networking Corp. in Warwick, R.I. Throughout Ridolfo's DR planning and implementation, Atrion stepped up and ensured that the designs were sound and fully tested. Bringing vendors up to speed while in the middle of a crisis "can be a nightmare," he said. By making Atrion part of the whole planning process, Ridolfo was able to issue simple requests that the vendors were already prepared to execute.
"This freed my team up to focus on the highest-priority issues."
While Ridolfo had hurricanes, terrorist attacks and major blackouts in mind while he was designing his disaster recovery plan, he never thought something as small as a blown transformer could pose so much risk to his company.
"We always think of the big things that can impact a business," Ridolfo said. "We tend to forget that small things can trip us up quite a bit. The transformer in our building was put in by the power company 15 or 20 years ago. When it failed, they had trouble finding parts for it. That's really something you don't anticipate. What this illustrated was that there are all manners of problems that could happen to take us out of service. You can't see what it is, and you have to build around that. It's not just terrorism and natural disasters that you have to worry about. It's the tenant above you that might spill some cleaning fluids and force you to evacuate for a day."
Ridolfo said having his Cambridge office offline that day would have cost his company $45,000 without a successful disaster plan.
"The longer we're offline, however, the per-day cost to the business climbs dramatically," he said. "A single day or a few days' event can generally be ridden out by rescheduling meetings, working from documents stored on laptops, etc. After a few days, we run the risk of meeting client deliverables, processes like invoicing and vendor payment begin to slow or halt, selling new business becomes much harder as you focus on just keeping the existing clients happy. Missing a deadline for a client could have an impact of hundreds of thousands or a million dollars, depending on the work.
"Our CEO was thrilled with our response and the outcome was much better than he anticipated," he said. "You never now how well it's going to function until you have to go through it."
Let us know what you think about the story; email: Shamus McGillicuddy, News Writer