Four years ago OhioHealth, a Columbus-based medical care nonprofit, reconfigured its network, which cut off the emergency room at one of the organization's 17 hospitals from the data center for 45 minutes. IT had become so central to the hospital's functions that management was about to start redirecting patients to other emergency rooms when service resumed.
At the time, OhioHealth outsourced a hot site to provide failover DR coverage. But management realized that the organization needed to invest in a state-of-the-art BC program, so Patterson was hired to put together a five-year plan to improve OhioHealth's preparedness.
"It was a massive undertaking," CIO Michael Krouse said. "We got Bob for the sole purpose to develop a high-availability business continuity plan. After three years of effort we're in the wind-down phase."
OhioHealth found itself in the same place as many other upper-midmarket companies: with disaster recovery/business continuity plans that were good but not great. And it was time to make the leap up to an enterprise-class system.
Indeed, beefing up DR capability was the No. 2 priority for the more than 200 IT executives responding to Computer Economics Inc.'s IT Spending, Staffing and Technology Trends poll, conducted in the first quarter of 2008.
That's certainly true of Jim Honerkamp, CIO at The Hillman Group Inc., a Cincinnati-based midmarket distributor of fasteners and other hardware items. The company, which has grown through acquisitions to become a half-a-billion-dollar business, hired Honerkamp four years ago to reinvent the way it used IT. High on his list of priorities has been creating a robust DR/BC strategy.
"We didn't have the budget for DR before," he said. "We used to outsource the data center -- nothing was redundant. Now we're moving to a two-data center scenario, both active. I can't afford to have a data center sitting around, waiting for a disaster to happen."
The increasing importance of business continuity is underlined by a Gartner Inc. survey that found that the majority of executives tasked with the subject report directly to the CEO (22%), followed by those reporting to the CIO (16%).
Roberta Witty, an analyst at Stamford, Conn.-based Gartner, said that while many midmarket firms have adequate backup systems in place, not all companies involve the business as completely as necessary to ensure a truly effective strategy for continuity. She tells CIOs that they need to perform gap analysis to determine where plans need to be improved as well as expand the scenarios and outage time frames that might accompany regional disasters or even a pandemic breakout such as the SARS virus in China.
"Getting the business to buy in is still a fairly sizeable task for many IT execs," Witty said. "In many cases, organizations have good DR plans in place. But they're not getting the support from the business for recovery. It's a lot more than technology. It's having the right staff, a key group of people. Some organizations think as long as they have data backup that's good enough. That may be good enough for data center disruptions, but do they have work-area recovery plans in event of a regional disaster or plans on how to get in touch with people?"
That's precisely the approach OhioHealth took. Patterson started by cataloguing the hundreds of apps used by the organization, ranking them in terms of recovery time objectives (RTO) on a scale of zero to five, the first two being mission critical, the next two less important and the last not important.
"Every time we intro a new app we have to go through the process again," Patterson says. "Everyone thinks their apps needs to be high-availability until you go through the cost. We do a pretty good job figuring out which ones really matter."
One key early decision was whether to upgrade the hot-site capability to high availability or build a second dedicated data center nearby.
"For medical imaging the volumes of data were too high to handle over a long-distance link," Patterson said.
After three years the data center is live for mission-critical apps. During the next 18 months, the organization is working on setting up a quick ship program for vendors to supply equipment for less important apps, where RTOs are on the order of days or even weeks.
"The technical challenges were significant," Patterson said. "We did stuff that was not commonplace. We were ahead of our vendors. They clustered databases but not the user interfaces. That's not high availability for us. We had to lead the vendors through a lot of this stuff."
Krauss added: "We put a fair amount of pressure on our vendors. We have a definition of high availability, which means the entire system is duplicable. That definition has to be met by our vendors. That's a critical discussion item with vendors before we sign a contract."
The result, Krouse said, is the hospitals will still be running as long as they're standing.
"We plan for anything you could dream up," Krouse said. "Tornado, bombs. We're not too worried about tidal waves."
Michael Ybarra is a monthly columnist for SearchCIO-Midmarket.com and a former senior writer at CIO Decisions magazine. He is also the author of Washington Gone Crazy. Write to him at firstname.lastname@example.org.