Prologue: The Forecast
When Michael Evanson left his office at Oreck Corp. in New Orleans on Friday, Aug. 26, he didn't know if he would be coming back to work on Monday. Out in the Atlantic Ocean, a Category 2 hurricane named Katrina was churning toward the southern coast of the U.S., but the National Hurricane Center predicted that the storm would miss New Orleans before its expected landfall on Monday. Late Friday the forecast changed: Katrina might hit the city. Evanson, the vice president of IT at Oreck, alerted IBM's disaster recovery unit that the privately held vacuum cleaner maker might be in for some heavy weather. In his two years at the company, Evanson had had to flee headquarters only once because of a storm, but even then he didn't have to switch to a backup data center. The company had done that only during tests.
Oreck's executives scheduled a conference call for 11 a.m. the next day to discuss what to do if Katrina continued to threaten New Orleans. Starting at 5 a.m. on Saturday morning, Evanson monitored the storm on TV and the Internet: Katrina was now a Category 3 hurricane -- and it was headed straight for New Orleans.
Evanson called Pat Eiermann, the company's AS/400 administrator, and told him to notify IBM that Oreck was going into emergency mode and would shut down its systems that day and switch its data operations to a Big Blue facility. Next, Evanson dialed in for a conference call with CEO Tom Oreck, who told his executives what most of them already knew: The company was going to evacuate.
Evanson drove to Home Depot to buy plywood and spent the rest of the day boarding up his house and calling the data center to check on the backup progress. Evanson pounded the last nail around 11 p.m. That same night, Tom Oreck flew his private plane with his family and the backup tapes to Houston.
At 5:30 on Sunday morning, Evanson, his wife and their three dogs piled into the family truck and joined the hundreds of thousands of people fleeing the city. That day the National Hurricane Center upgraded Katrina to a Category 5 storm, the top of the scale, and tracked winds gusting to 215 miles per hour. As Evanson drove toward Memphis, Tenn., the sky was clear and sunny, with only a slight breeze hinting that any sort of trouble might be on its way.
Evanson hoped that he might be back in his office on Tuesday. Instead, he would be gone six weeks.
Part 1: The Disaster
Shortly after 6 a.m. on Monday, Aug. 29, Katrina, now a Category 4 storm, made landfall in Louisiana, cutting a broad swath of torn roofs and felled trees. From his hotel room in Memphis, Evanson spoke with Tom Oreck in Houston. Initial reports indicated that corporate headquarters in New Orleans and the company's 375,000-square-foot factory in Long Beach, Miss., had suffered only minor storm damage. And the destruction in New Orleans itself was less extensive than many had feared.
Then CNN broadcast the awful news: The levees were failing. Around 11 a.m., the swollen waters of Lake Pontchartrain, on the city's north side, breached a berm, and New Orleans began to flood. Other sections of the levee gave way as well; and before long, 80% of the city was underwater, deeper than 20 feet in some neighborhoods.
That afternoon, Evanson's cell phone stopped working -- as did the phones for all the company's executives and everyone else in the 504 area code, which was overwhelmed by the emergency volume of calls.
The real disaster had begun. The mayor ordered any remaining residents to evacuate, the Louisiana Superdome filled with refugees, looting flared across dry patches of the city, and the entire infrastructure of a major American city collapsed. A week later, half of New Orleans was still submerged.
Katrina would turn out to be the costliest disaster in American history, walloping a three-state area of the Gulf Coast almost as large as the U.K., with a death toll estimated at more than 1,300 people and damages topping $125 billion.
"I've never seen anything like it," says Tom Oreck. "The level of destruction was mind-numbing."
Oreck's company, of course, had a disaster recovery plan. It was based on the assumption that either company headquarters or its factory located 76 miles away would remain usable. "We figured a storm might take out one or the other of our facilities, but not both," says Oreck.
He was wrong. The hurricane devastated the area surrounding the plant, sweeping homes off their foundations and tossing cars around like toys. The storm tore the Oreck sign off the factory, but the building remained largely intact. Worse, half of Oreck's 1,200 workers were displaced, with many of their homes destroyed.
Moreover, Katrina left the company's IT systems at both headquarters and the factory in tatters. By Wednesday, Oreck's core AS/400 systems -- finance, manufacturing, distribution, call center and order entry/customer service -- were running at IBM's facility in Boulder, Colo. But the data center could not establish communications with headquarters or the factory, where important local area network and PC-based files, such as payroll records, remained stored on servers. Nor could it communicate with vital business partners, such as credit card processors or UPS, or with Oreck's e-commerce or point-of-sale (POS) systems, which were hosted by a New Orleans firm that had been driven from the city and was unable to access its own data center.
All these crucial parts of the company's business were tied together by dedicated T1 lines from the corporate office in New Orleans. And Katrina snapped those lines like twigs. The outage also took down the 800 phone numbers that Oreck employees were supposed to call in an emergency, so the company had no way to communicate with its scattered workforce. (One of the few bright spots for Oreck was that when its call center in Long Beach was shut down by the storm, the company had an agreement to switch to a partner with dispersed offices whose operations weren't affected by Katrina.)
"Our communication plans fell apart," Evanson says. "We always assumed one or the other facility would survive. We didn't plan for cell phone service dying. We never thought about office space, housing needs. A lot of people lost homes. We had to attend to our employees' personal needs so they could focus on business. These things just don't show up in a typical disaster recovery plan."
On Tuesday, as the magnitude of the disaster became clear, Oreck executives suddenly faced something they had never planned for: extended exile from their key facilities. Oreck had two weeks of inventory at its 450 retail outlets across the country, but its biggest sale channel -- direct sales via the Internet -- was suddenly sucked into a black hole.
The company had to figure out how to keep its business alive.
Nothing Is Easy Now: Lessons From the Aftermath
| It may be years before life returns to normal in the Big Easy. Some 25% of New Orleans' housing stock (30,000 to 50,000 houses) faces demolition because of flood damage, according to some estimates, and half of the city's municipal employees have been laid off, including virtually all public school teachers. For midmarket companies already strapped for resources, the road back will be tough. And some won't come back at all.
Amid Katrina's wreckage, you can count a number of corporate disaster recovery plans. From companies that had no idea what to do when the cell phone network went down to those that had never contemplated a scenario involving a total breakdown of municipal infrastructure, Katrina offered abject lessons in just how disastrous a disaster recovery failure could be.
"You have to prepare for the impact of multiple disasters," says Michael Croy, director of business continuity solutions at Forsythe Technology Inc., a consulting firm based in Skokie, Ill. "After Katrina, companies need a different paradigm. The heart of American business is now IT. Business continuity needs to be part of daily operations. It's not an IT issue; it's a business issue, it's a corporate governance issue."
And while the magnitude of Katrina was difficult to anticipate, many of the lessons apply to disasters on a far lesser scale.
"We had great recovery plans for our clients but a miserable one for ourselves," says CommTech Industries President Darryl d'Aquin, whose IT management firm was driven from the city by the storm and, six weeks later, was still waiting to move back into its flood-damaged office. "The big issue was finding staff; they scattered. Text messaging [on cell phones] is how we found 90% of our staff."
Even the companies where things went according to plan learned lessons. Consider Stewart Enterprises. The company, which was founded in New Orleans in 1910 and went public in 1991, is one of the biggest funeral home and cemetery owners in the U.S., operating in 27 states and Puerto Rico.
Stewart evacuated its 500-person corporate office in New Orleans and ran operations from offices in Irving, Texas, and Orlando, Fla. In Orlando, a cemetery houses Stewart's help desk, and the company brought in two trailers of equipment to add 60 workstations for the IT department.
"When you're down for weeks, you have to bring everything back," says CIO Kent Alphonso, whose house was flooded and rendered uninhabitable by the storm. "You can have the best plans, but you need the people to pull it off. We never planned for a total office move. You plan for critical systems; then you realize you have to run everything."
Three of Stewart's funeral homes and five cemeteries in the New Orleans area were damaged by the storm, while the company's corporate office escaped with little damage. Stewart's national business wasn't directly impacted by the storm, although some of the company's sales systems were offline for a week, forcing funeral homes around the country to revert to paper for orders and invoices. And with the company's fiscal year ending Oct. 31, Stewart had to delay the reporting of its quarterly results to the Securities and Exchange Commission.
"We lost six weeks," Alphonso notes.
Stewart was also four months into an 18-month project to rewrite its aging client-server cemetery- and funeral-software management program in Microsoft's .NET -- a project that Alphonso now has to pick up again.
Even when the company was able to return to its office, half of its area employees were without homes, and the Stewart parking lot was filled with trailers from the Federal Emergency Management Agency that temporarily housed workers.
"We still have people in Orlando because they have nothing to come back to or no one to stay with," Alphonso says.
Some companies came to the same decision. Indeed, just days after Katrina, Ruth's Chris Steak House announced that it wasn't returning to its New Orleans headquarters where it was founded but would relocate to Orlando instead.
The uncertainty of when life in New Orleans will return to normal plays havoc with recovery plans.
"The biggest timing issue that has never been discussed in any disaster recovery meeting I've ever been in is the children," says d'Aquin. "They dictate where people work and when they return to work."
One of the overarching lessons of Katrina was the number of contingencies that DR plans can't encompass. D'Aquin, for example, received a text message on his cell phone from a client who was trapped in a building surrounded by flood waters and needed a boat.
"Finding a boat isn't normally on your list of disaster recovery steps," says d'Aquin.
Part 2: The Company
In 1963, David Oreck quit his job as a sales manager at Whirlpool in New York to start his own company. A World War II veteran who flew bombing missions over Japan, Oreck noticed that Whirlpool couldn't seem to give away its upright vacuum cleaner. So he made a deal with the company to redesign the machine and sell it himself -- and Oreck Corp. was born.
On a winter day that first year, Oreck visited a poorly performing RCA distributor in sunny New Orleans. He bought the distributor, turned around its performance and decided to make New Orleans the headquarters for his vacuum cleaner business as well.
Oreck built lightweight but powerful vacuums marketed to hotels, 50,000 of which the company says have purchased its models since they were first introduced. The units proved so popular with maids that they asked to buy them for their homes too. Today, Oreck's range of products includes a $299 starter model and a top-end $699 machine, which comes complete with a 21-year warranty. The privately held company won't disclose its revenues, although a published account in 2000 estimated that Oreck sales were about $200 million a year.
Smiling and bald, David Oreck became the public face of the company, singing on radio commercials and promising to stop if listeners called his 800 number. And they did. Even after he turned the business over to his son Tom in 1999, David Oreck continued to personify the company. He flew vintage planes at corporate events and appears as an animated character on the Oreck Web site, hoisting a vacuum cleaner (the latest model of which features an FM radio built into the handle) above his head.
Oreck Corp. wasn't quite as innovative in its use of back-office technology, as Michael Evanson learned when he joined the company as its first CIO in 2003.
The IT department had a backlog of 700 projects and growing, dating back several years. New e-commerce and POS systems were in early development. Neither initiative had a project manager or even a project plan. The credit card settlement system routinely crashed.
"Everything was wrong when I walked in here as CIO," Evanson recalls. "We ran mostly Windows 95. The new computers were Windows 98. The network didn't work well. Our systems aren't really adequate. E-mail didn't work. We needed everything."
Evanson canceled the Web and POS projects and started a search for off-the-shelf systems. He spent nearly a year selecting CRS Retail Systems for POS and SAP for everything else.
When Katrina hit, Oreck was four months into its 18-month SAP deployment. "This was the worst time for me," Evanson says. "SAP would have replaced all this crap. It was a frigging disaster."
Part 3: The Recovery
From his hotel room in Houston, Tom Oreck vowed that he wasn't going to let a little hurricane wipe out the company that his father built.
The company's first task was to find its employees. Oreck set up a new 800 phone number for its scattered staff to contact the company, and the 15-year-old son of the company's marketing vice president put up a static Web site available at the company's old URL with the information. The company also posted pleas for employees to call in on local bulletin boards such as Nola.com.
If you had a job at Oreck before the storm, Tom Oreck promised his employees, you would still have one.
On Friday, Sept. 2 -- a week after Oreck closed its New Orleans office -- the company moved into a new office space: 100 cubicles crammed into an IBM business center in Dallas. Oreck flew up from Houston, and Evanson drove from Memphis. At the same time, a small staff was working at the Boulder data center.
Oreck assigned teams to find generators and mobile homes to, respectively, restore power to the Long Beach plant and furnish temporary housing for displaced workers. When an executive found three generators in Florida, Oreck put a $60,000 down payment on his personal credit card.
The first generator didn't arrive in Long Beach until the following Tuesday. Later that week, a convoy of more than 40 trailers arrived at the plant to provide emergency shelter for workers, followed by trucks loaded with food and water. The impromptu trailer park housed 150 employees and was dubbed "Oreckville."
The company's retail outlets had almost two weeks of inventory -- which might give the firm enough time to restart shipping and production at the Long Beach plant.
Oreck's disaster plan called for replacing the destroyed T1 lines with ISDN connections, although it turned out that only one of the company's business partners had the capability to switch telephony systems. So instead, Evanson decided to go with a virtual private network -- a technology no one on his staff had any experience with. Nevertheless, the Boulder data center was soon communicating with the rest of the business. Evanson then dispatched a team to New Orleans to truck the company's servers to Dallas, where they were bridged into the data center, effectively recreating the company's network a week after Katrina struck.
"Bringing up a system doesn't do any good if no one can connect to it," Evanson says.
But Oreck was still severely handicapped. The flooding in New Orleans took out the local service provider hosting Oreck's e-commerce and POS systems -- as well as that company's backup facilities. Evanson hired another service provider to put up a simple Web site available at the old URL that directed customers to Oreck's 500 retail locations. Oreck stores reverted to paper, handwriting receipts for customers. Paragon, a small software firm, later furnished Oreck with a Web-based POS application.
"The least damaged channel was retail, so we pushed customers through there," Evanson says.
Getting vacuum cleaners to customers was another matter. Oreck soon made a deal with UPS to handle its distribution from UPS' Atlanta hub. UPS would truck food and water down to Long Beach and return to Atlanta with vacuums, which it would then distribute for the company. This meant connecting UPS to Oreck's data center in Boulder and training its personnel in how to navigate the company's arcane and laborious supply chain software: old and heavily customized versions of Assist Cornerstone and MacPro.
In those first two frantic weeks, Oreck had been able to reach only about half of its 500 factory workers. Paying them was another challenge, since work records were on the servers at the Long Beach plant. Evanson sent a team to bring the boxes back to Dallas, buying two seats on a plane for the servers. "We didn't miss payroll," Evanson says.
On Friday, Sept. 9, Tom Oreck flew his plane down to the plant to celebrate the reopening of the factory with a barbecue. Two out of five production lines went live that day.
A month after Katrina, Oreck's IT systems were mostly restored, and the company began to plan for moving back to New Orleans.
It was far from easy. Evanson looked into syncing up the New Orleans data center with Boulder over the network, but the volume of data and the system architecture ruled out that possibility. Turning off the system, running backup tapes and restarting again was the only practical option. That, too, was complicated. Getting the tapes to New Orleans proved difficult: Flights to the city were still limited, and cancellations common. A full backup of the AS/400 also takes eight hours, and twice that time for a second set of tapes. It would take another day to get everything back up and running. All of which meant that Oreck would have to go offline for three days.
"Three days," Evanson notes, "was a real problem for the business."
The factory was starting to catch up with production, but distribution was still lagging. Oreck had invested heavily in advertising, and its call center partner needed the systems to handle orders.
Must-Haves in a DR Plan
| Redundancy. Back up critical IT systems, but also have a plan for running all systems necessary to keep the business alive.
Communications. Devise a way to get in touch with employees if local cell phones don't work.
Operations. Have a plan for running the business from a new location for an extended period of time.
Families. Recognize that employees need housing and food before they can return to work and that their children need schools to attend
Infrastructure. Account for a worst-case scenario in which local government and municipal services shut down.
Evanson told his IT department to find a way to do it faster. "I gave my team the challenge to do it in one day," Evanson says. "We had one shot."
On Thursday, Sept. 29, the company committed to moving back over the weekend of Oct. 8. The company ordered a superfast $15,000 tape drive for next-day delivery. It would perform a backup in three hours and would be used to transfer data from backup tapes made in Boulder to the machines that returned to New Orleans.
On Friday, however, the package didn't show up.
"Emery [a delivery company] was more or less unresponsive, didn't know where the drive was and said they couldn't deliver it before next Friday," Evanson recalls. "That was obviously too late."
Luckily for Oreck, the company had a strong relationship with UPS, which had recently acquired Emery. UPS found the package in Emery's system and sent a team to pick it up in Shreveport, La. The drive wasn't there. UPS sent teams to walk Emery terminals around the country looking for the package. The drive was finally located -- on a train headed to Dallas. UPS yanked the box from the train and drove it to New Orleans, where it finally arrived at around 11 a.m. on Thursday, Oct. 6, just in time for the weekend's planned switchover.
The next morning, server administrator Pat Eiermann flew from Dallas to Boulder. That day the network servers in Dallas were turned off and trucked to New Orleans. At 6 p.m. on Saturday, Oreck shut down the AS/400 in Boulder and ran off two sets of backup tapes. At 6 a.m. on Sunday morning, Eiermann hopped aboard a $10,000 chartered plane with the tapes. He was in New Orleans by 11 a.m.
On Monday morning, Oct. 10, Oreck was back home.
Epilogue: The Return
Several months after Katrina, most of the water is gone, but vast portions of New Orleans resemble a ghost town. Blue plastic tarps patch roofs ripped by the storm, and heaps of debris are everywhere. Piles of abandoned refrigerators stand on corners, many bearing messages that bespeak pain and anger: "Katrina, kiss my ice," "FEMA, look inside for dinner" and "From the people of New Orleans to George Bush." Forsaken cars litter sidewalks, parks have been turned into huge, stinking garbage dumps, and every major street is festooned with small signs shouting business opportunities: "We gut," "Got mold?" and "Signs like these in 24 hours."
But Oreck headquarters hums with life, especially at lunchtime, when employees gather for a catered meal in a conference room because so many local restaurants are still closed.
Tom Oreck says that 250 employees lost their homes and 100 employees have never been heard from, but he returned to New Orleans to find that his own home suffered only minor storm damage and no flooding. "I was very lucky," he says.
So was Evanson, even though a 100-foot pine tree had crashed into his roof, coming to rest in the master bedroom closet where he stored his collection of thousands of vinyl records. A box of 45s was smashed, but the records inside were not damaged. Rainwater had run along the tree trunk and away from the records.
Down on the Gulf Coast near the factory, a number of families are still living in company-furnished trailers. An employee relief fund has raised nearly $1 million to help displaced workers. "Preparation and planning kept us from go-ing out of business," Oreck says. "You prepare and you plan, and then you aggressively improvise."
On the other side of the large plant, Evanson ponders the lessons from Katrina.
"It can happen to you," Evanson says. "The most important thing is the recognition that IT is only a tiny part of a disaster plan. The disaster plan needs to include the whole business. Think long and hard about how to operate remotely. Make sure you have a good disaster plan with your business partners. That's a gap in all the disaster plans I've done."
Between the time lost to the hurricane and the lapse until the company was able to pick up the project again, Katrina set the enterprise resource planning system replacement process back about eight months. So Evanson has been forced to start some patch-up projects on the legacy systems. New Orleans was not exactly a hotbed of IT talent before Katrina; now Evanson expects it will be harder than ever to hire skilled people. Before the storm, he was looking to add more than a dozen people; now he's down six employees, who quit to take other jobs during the chaos.
"I just cannot wait on SAP completion," he says. "The eight-month delay means I won't complete SAP before next hurricane season, so I have to put together a new disaster plan based on our current systems. I lost 25% of my IT staff, including a few really key people, and have had difficulty filling the open positions. When I restart SAP, I'm going to have to run the project from another city for a while. There just aren't 15 hotel rooms available in New Orleans to house the implementation team right now."
The company is also rewriting its disaster recovery plan, reviewing the details of what worked and what didn't.
"We're convinced we can put together a plan that will allow us to operate through a similar disaster with no downtime," he says. "We continually ask the question 'What could we have done to prevent this problem in the first place?' These meetings will continue until we complete and test a new disaster plan for the company. We will have one in place before next hurricane season."
Plans are fine, Evanson adds, but it's the people who implement them that really make the difference. "Disaster brings out the real character in people and companies as well," Evanson says. "We had critical people coming unglued. We also had a lot of heroes here."
Michael Ybarra was a senior features writer at CIO Decisions. Tom Kaneshige was a senior features editor at CIO Decisions. To comment on this story, email [email protected].