A disaster recovery plan for branch offices: Five layers of redundancy

By Linda Tucci, Senior News Writer 19 Mar 2009 | SearchCIO.com

IT news and analysis for CIOs Digg This!     StumbleUpon     Del.icio.us

More disaster recovery resources When IT disaster recovery plan is put to the test, VoIP becomes savior How disaster recovery savings can pay for business continuity planning Seven steps to securing funding of your disaster recovery plan

"Our promise to the patients was that we could ensure all of the appropriate documentation for their care was available as they moved through our very complex and very speedy system of treatment. We literally operate faster than any other organization that I know of in health care," said Eckes, who was recruited in December 2005 to spearhead the digital makeover at the for-profit CTCA

CTCA ,with hospitals in suburban Chicago, Philadelphia, Tulsa, Okla. and suburban Phoenix, offers what is sometimes called destination medicine. The average patient travels 500 miles to be treated at CTCA and through the course of treatment might be propelled through multiple departments within a hospital or even receive care at more than one CTCA center, Eckes said. "The paper world couldn't keep up with movement of the patient," he said.

Going paperless, however, posed considerable risk without the technology infrastructure to support it. The electronic health records system could not go down. CTCA's reputation is predicated on its "Mother Standard" of care, a trademarked mission to treat patients as family. Electronic medical records needed to be reliably and securely managed, readily accessible for input and output of information and connected to the hospital's medical equipment.

"Those reasons are why we started going down the path of building a highly redundant infrastructure that was focused on disaster recovery," Eckes said.

A layered approach to disaster recovery for remote locations

CTCA needed to have centralized data centers because its widely dispersed hospitals share electronic health records. That requirement affected all three risks that must be managed for in a disaster recovery plan -- power, applications and data, and the network.

Cost and the model for the disaster recovery plan The Cancer Treatment Centers of America's (CTCA) disaster recovery project entailed a "full retrofit of the IT infrastructure, which is all new since 2005," said Chad Eckes, CIO. He declined to state how much IT spent on the DR plan, stating that the privately held CTCA does not disclose its financial information. But the IT department runs lean, said Eckes, who worked in financial services prior to becoming a health care CIO. CTCA's IT budget as a percentage of revenue is "multiples lower than the 6.5% of revenue" that is average for hospitals running electronic health records, he said. He advises health care CIOs to look at other industries, including financial services, for models on how to do disaster recovery. "I knew a lot of the equations [from being in financial services] for doing disaster recovery. You can imagine the reactions in a bank if their systems go down for more than a minute." -- L.T.

Managing power for remote locations is not so different from local sites, Eckes said. "You always want to make sure that you have dual power grids, you always want to make sure you have uninterrupted power supplies, you always want to make sure that you have generator backup to run all the systems," Eckes said.

CTCA takes that a step further for its Phoenix location, where a UPS system supports the entire hospital, versus just supporting IT to ensure against an outage on its medical equipment. Similarly, dealing with application and data redundancy, in concept at least, is no different for local or remote sites, Eckes said, but the risk impact for CTCA is heightened because of its centralized databases. "If an application goes down, that not only impacts the service offering at one hospital but across our four facilities," he said.

CTCA has built in four layers of redundancy for its systems data, and has a fifth layer of redundancy in the event of a worst-case scenario -- an approach Eckes touts as uncommon, if not unique, among health care organizations.

• Every one of the production systems is clustered, so if one part of a cluster fails, the systems remain up and running.
• Every piece of data from all sites is immediately mirrored to a second data center, so in the event of an outage, CTCA can shift processing to the redundant center with no data loss. The CTCA located its second center 59 miles from its primary center in Schaumburg, Ill., a strategic decision based on the speed of data transmission. "We chose to have it this close because we couldn't replicate without it being this close. Information can only travel so fast over the lines."
• Backups are stored to disk. "Disk is very fast to restore off of, and we have that immediately available in our data center. We keep seven days' worth of backups on disk." CTCA backs up approximately 4 terabytes nightly.
• Standard tape backups are stored off-site in a vault facility in downtown Chicago. "We can't keep the disk backup as long as we want to, from a cost standpoint. It wouldn't be prudent. We also want to protect against a situation where both data centers go down."
• Data is in PDF format. In the event that all other redundancies fail, CTCA "has written a massive dump of data that goes out to all the individual sites." The data, which includes all vital patient information needed for care, is pulled every four hours and stored in PDF format on a server in each of the hospitals. "In the worst-case scenario, folks at that hospital can go to the server, print it off and be taking care of our patients safely."

The upshot is that if CTCA loses its main data center today, every system can be up and running within two hours; Real-time replication guarantees zero data loss, Eckes said.

Taking control of the uncontrollable: Network redundancy, with two WANs

But probably the toughest aspect of building the infrastructure to fully support going paperless was achieving network redundancy, the rung of disaster recovery that is not fully under one's control, Eckes said. The easy part was the LAN networks within CTCA's facilities. "We had a strong partner in Cisco. We built high redundancy in everything we have done at the facilities."

Designing a structure at the metropolitan level and the wide area network level (WAN) proved more difficult.

"What we did -- and we're told we have one of the most complex designs in greater Chicagoland -- was to design two full-production, wide area network WANs," Eckes said. One WAN is with AT&T and one is with Qwest. The WANs run synchronously and "are sized at a point that allows us to run on either/or and still have plenty of bandwidth to run both of our facilities," he said. The WANs transmit 20 megabytes per second. In addition, all of the Cisco gear can shift processes automatically and immediately, if there is a problem with either one. Eckes also negotiated with the two telecom providers to make sure the CTCA networks are on independent fiber, to prevent a single point of failure.

The mission dictates the DR plan

More important than the nuts and bolts of disaster recovery, Eckes said, was aligning the plan with CTCA's mission. Eckes runs IT with a team of 84 people, who are hired as much or even more for their appreciation of the organization's mission to care for patients, than for technical skills, which "can be taught."

If an application goes down, that not only impacts the service offering at one hospital but across our four facilities. Chad Eckes CIO, Cancer Treatment Centers of America

Analyst Stephanie Balaouras, who covers disaster recovery and business continuity at Cambridge, Mass.-based Forrester Research Inc., said it's helpful for CIOs at any organization to step back and look at business impact before crafting a disaster recovery plan. "We in IT tend to focus on individual applications and lose sight of business processes," she said. In addition, the trend to consolidate remote office backup and recovery to a centralized model makes sense from both a technology and skills perspective.

Eckes said the business mission should always inform IT's DR strategy. "What that translates to, from an IT perspective, is the question I constantly ask my team: 'If your mother or father were being treated here, hooked up to medical equipment that is connected to our EHR, how redundant would you want this system?'"

Eckes agrees. "Quite honestly, we'll take that to the nth degree. That is what drove our goal, which is 100% system uptime," he said, acknowledging that many IT people would dismiss that as impossible. "But why would you target anything less? We'll keep on chasing the tail of redundancy until we achieve that standard."

Tags: Enterprise disaster recovery planning,  Health care industry,  Remote connectivity management,  VIEW ALL TAGS

Digg This!     StumbleUpon     Del.icio.us

RELATED CONTENT Enterprise disaster recovery planning Review these trends in IT disaster recovery planning and outsourcing Disaster recovery is dead; long live continuous business operations Data recovery solutions must address a range of concerns Swine flu preparedness: Business continuity during an H1N1 outbreak IT disaster recovery outsourcing: A planning guide for enterprise CIOs Recession squeezing IT disaster recovery budgets Seven tips to make the value case for disaster recovery Technology is changing IT disaster recovery outsourcing Swine flu -- not hurricanes -- leads disaster recovery agenda A disaster recovery plan meets cloud computing Health care industry Healthcare IT standards still not clear How to build an IT vendor management office or standards body Health care CIO tackles complex security, privacy mandates BPM gets kicked up a notch with business event processing Electronic medical records at risk of being hacked, report warns Vertical market guide for CIOs Disk management helps firm solve storage problems High-tech exec market picking up Virtualization eases health care company's server sprawl Business intelligence aids health center Remote connectivity management Creating a backup and disaster recovery plan for remote employees WAN optimization solves latency issue for growing law firm Networking for CIOs: A SearchCIO.com supercast Slow application response time has negative effect on remote workers Rural hospital tests VPN to meet state regs The best-laid VoIP plans SSL VPNs get a green light from more CIOs

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary