Home > CIO Briefings > Data lifecycle management tips for CIOs > Strategy: Planning your life together > Disaster planning: Bank builds data center for the long haul
Briefings: Data lifecycle management tips for CIOs:
EMAIL THIS
 START   STRATEGY   EXECUTION   TOOLS AND TECHNOLOGIES   
Strategy: Planning your life together

<< PREVIOUS | NEXT >>: Data center energy efficiency strategies for CIOs

Disaster planning: Bank builds data center for the long haul

By Linda Tucci, Senior News Writer
30 Apr 2008 | SearchCIO.com

IT news and analysis for CIOs
Digg This!    StumbleUpon Toolbar StumbleUpon    Bookmark with Delicious Del.icio.us    Add to Google

Some days are hard to forget. On Aug. 29, 2005, Hurricane Katrina ripped the roof off Hancock Bank's 15-story headquarters in Gulfport, Miss. Furniture from the tower's top floor was discovered blocks away. Winds blew out every window in the building. The basement filled with 31 inches of water. On the fifth floor, the location of the bank's data center, the storm dislodged a wall. Water swept over the racks of equipment, including the bank's nerve center, its IBM mainframe.

"The building was turned inside out," said Shane Loper, chief operations officer at Hancock Bank, which has assets under management of about $6 billion and operates 106 branches across Louisiana, Mississippi, Alabama and Florida.

More on disaster recovery
Disaster recovery and business continuity supercast

New Orleans CIO: VoIP, portals and Katrina
Within 24 hours, bank systems were up and being restored from tape. In 48 hours the systems were completely functional, Loper said. SunGard Availability Services, a division of SunGard Data Systems Inc., handled the recovery at its facility in Wood Dale, Ill. IT staff members from Hancock were dispatched to assist. Some showed up at SunGard, "with [only] the shirts on their backs and shoes on their feet," Loper recalled, having left behind slabs on the Gulf Coast where their houses used to be. "They said, 'Get me to Chicago, so I can help get the bank restarted,'" he said, getting choked up, even today.

$16 million fix

Loper, who began his 16-year career at Hancock in technology and rose to a senior operations position, was the bank's human resources director at the time. A major in the National Guard, with the military's penchant for orderly processes, Loper worked with Hancock personnel across four states to coordinate the recovery and build a makeshift data center in a bank training facility.

"On November 12, at 2:07 in the afternoon, we moved home from SunGard." Loper said. He was happy to have IT operations back but acutely aware that this was a short-term fix. "We worried every time it rained hard and the wind blew."

DR tips from Hancock Bank COO Shane Loper
1. Understand fully the impact of the TSP process on your communications network. Fire, police and other emergency providers have priority. "When you call the AT&Ts or whoever provides service, after a disaster, you will encounter roadblocks."

2. Make sure your systems are tested under a full load at your backup site provider. "We learned that some of the tests we did were adequate to test whether the systems were working, but inadequate under a full production load onto those systems."

3. Know where your people are and be able to ensure them that you can take care of them. "If your people have to worry about basic needs, that is when you lose effectiveness." Companies should have at the ready a list of suppliers that can provide your employees with building material, fuel and food. "If you call them, they can they launch, because it is already pre-planned."

When disaster hits, "a lot of things change," Loper said. One change, made at the highest levels of the bank, was how to house the IT assets that ran the business. Hancock wanted a data center as close to hurricane proof as possible, but structures built to bomb-shelter specifications are scarce.

Mindful of its role in reinvesting in the community, Hancock bought property 10 miles inland and 45 feet above sea level to build a data center from the ground up. The cost: $16 million, considerably less than the $50 million spent to rehab the bank's headquarters, but much more than the $1 million to $2 million spent on a bank branch.

Pleased with SunGard's performance in the wake of the storm --"They were the net beneath us, like their motto says, and more," Loper said -- Hancock hired the Wayne, Pa.-based provider to execute the move.

In January 2007, Mickey Zandi, managing partner of SunGard's data center consolidation and migration practice, was surveying the poured concrete floor and "imagining a data center." "They had an architect and builder. They really wanted a trusted adviser throughout the construction phase," Zandi said.

For a trusted adviser, SunGard put in a ton of grunt work.

First, Loper and Zandi agreed that SunGard would take responsibility for the layout of the facility, independent of the architecture and engineering firm. SunGard prides itself on its "proven methodology" for migrating data centers. This starts with discovery -- collecting information on all the hardware, services and applications that will be utilized on Day 1 and needed going forward.

Hancock's IT infrastructure revolves around multiple platforms, including IBM for the mainframe, Hewlett-Packard as one of the platforms for the ATM machines and so on. The bank was also in the early phases of virtualization, so Zandi's team had to understand which parts were not going to the new center.

'No formal CMBD'

To collect the information for the migration, SunGard uses a set of tools based on open systems and off-the-shelf technology. The tools are flexible enough to import and export data into any configuration management database (CMBD), Zandi said.

"In the case of Hancock, there was no formal CMBD," Zandi said. So SunGard interviewed Hancock's technical team and ran scripts, under the "very crowded" conditions of Hancock's temporary data center, and created one. The end product was a visual map showing the location and dependencies of all the IT assets. SunGard left the map behind for Hancock's use.

With the data collected, Zandi's team could begin on the layout of the center for maximum efficiency. But the facility was under construction while SunGard planned the move, adding another layer of complexity. "It was not as though you could walk over and say, "OK, this will go through this door, and this floor supports this amount of weight," said Lowell Grabel, senior engagement manager for the SunGard consulting organization.

'No options for error'

The Tier 3 data center was completed in seven months. Because of Hancock's interdependent system of applications and platforms, the migration could not be broken in chunks but had to be coordinated as a "big bang" move.

"We had one weekend for everything to be taken down and moved and brought up -- and no options for error," said Byron Atcheson, SunGard's account executive for Hancock. "We have a saying at SunGard, 'Moving is nothing more than a planned disaster.'"

That meant bringing down four platforms. It meant moving circuits. In a geography recovering from disaster, clean power and circuits can be rare commodities. "I personally participated in multiple escalation calls with the Ciscos and BellSouths of the world to ensure all elements were available," Zandi said.

We worried every time it rained hard and the wind blew.
Shane Loper
COO, Hancock Bank
Indeed, the government's role in telecom after a disaster caught Hancock unaware, Loper said. "When Homeland Security gets involved, they essentially freeze all the telecom assets and go into TSP mode," he said, referring to the
Telecommunications Service Priority process, which assigns entities priority in getting service. "If you do not have a TSP code, you will not get anything done."

The new data center is not only built to withstand hurricane conditions but is also self- sufficient, with its own water and sewer systems, dual 900 kW generators and a 25,000-gallon fuel tank. The tank can be filled with diesel or regular gas, and suppliers have been notified they can use it to top off their vehicles when gas shortages occur in a disaster, Loper said.

In the wake of the storm, the bank also learned a lot about fortifying its people, not just data, Loper said. When Katrina hit, the bank provided food and clothing; it even sent construction crews to put up temporary roofs on the houses of employees sent out of state to get the bank up and running. Since Katrina, Hancock has built a system that will help employees withstand disasters.

"Now, if we send a group of tech people to a disaster site, we send their family," Loper said. "If they have a dog, we are going to kennel their dog. If they have a grandmother, we'll let grandmother with them, so they are self-sufficient."

Let us know what you think about the story; email: Linda Tucci, Senior News Writer



Tags: Enterprise data storage managementOperating systems managementEnterprise server managementExecution: Realizing data center and desktop virtualization effortsStrategy: Planning your life togetherExecution: Recovering from disaster, focusing on business continuityExecution: Dodge risks in practiceEnterprise disaster recovery planningVIEW ALL TAGS

Digg This!    StumbleUpon Toolbar StumbleUpon    Bookmark with Delicious Del.icio.us    Add to Google


<< PREVIOUS | NEXT >>: Data center energy efficiency strategies for CIOs
VIEW ALL IN THIS CATEGORY


RELATED CONTENT
Enterprise data storage management
The real cost of cloud computing services
Managing IT spending cuts: Don't take the easy way out, CIO advises
Disaster recovery strategy shift reduces data loss, recovery time
Private cloud replaces antiquated IT infrastructure for $300K per year
Avoid these architecture mistakes in your disaster recovery planning
Software and services guides for CIOs
Systems management guides for CIOs
Economic downturn hits IT budgets
Virtual security tools fill gap in virtual data center
Giving thanks for a robust technology outlook

Operating systems management
Is Google Chrome OS a turning point or yawner?
Windows XP users weigh dwindling support vs. Windows 7 migration
Microsoft: Spatial computing the future
Virtual desktops? Brush up on Microsoft licensing
Microsoft Windows software must change, experts warn
Ten strategic technologies to watch in 2008
Startup gives virtualized twist to mobile computing
Microsoft to reach out to IBM, Cisco on interoperability
Dell to join forces with Microsoft, Novell on interoperability
CIO disillusioned with Windows PCs, tests Macintosh computers

Enterprise server management
Private cloud replaces antiquated IT infrastructure for $300K per year
Software and services guides for CIOs
The challenges and benefits of a private cloud
Beyond server virtualization: The private cloud
Economic downturn hits IT budgets
Virtual security tools fill gap in virtual data center
Technology innovation drives data center of the future
Gartner: 25 ways to cut IT costs
Networking technology key to data center efficiency
IT asset management focus drives Toyota Motorsport

RELATED RESOURCES
2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems
Search Bitpipe.com for the latest white papers and business webcasts
Whatis.com, the online computer dictionary



CIO solution center has news, research, and guides to assist the unique challenges of the CIO
About Us  |  Contact Us  |  For Advertisers  |  For Business Partners  |  Site Index  |  RSS
SEARCH 
TechTarget provides technology professionals with the information they need to perform their jobs - from developing strategy, to making cost-effective purchase decisions and managing their organizations' technology projects - with its network of technology-specific websites, events and online magazines.

TechTarget Corporate Web Site  |  Media Kits  |  Site Map




All Rights Reserved, Copyright 2007 - 2009, TechTarget | Read our Privacy Policy
  TechTarget - The IT Media ROI Experts