Some days are hard to forget. On Aug. 29, 2005, Hurricane Katrina ripped the roof off Hancock Bank's 15-story headquarters in Gulfport, Miss. Furniture from the tower's top floor was discovered blocks away. Winds blew out every window in the building. The basement filled with 31 inches of water. On the fifth floor, the location of the bank's data center, the storm dislodged a wall. Water swept over the racks of equipment, including...
the bank's nerve center, its IBM mainframe.
"The building was turned inside out," said Shane Loper, chief operations officer at Hancock Bank, which has assets under management of about $6 billion and operates 106 branches across Louisiana, Mississippi, Alabama and Florida.
Within 24 hours, bank systems were up and being restored from tape. In 48 hours the systems were completely functional, Loper said. SunGard Availability Services, a division of SunGard Data Systems Inc., handled the recovery at its facility in Wood Dale, Ill. IT staff members from Hancock were dispatched to assist. Some showed up at SunGard, "with [only] the shirts on their backs and shoes on their feet," Loper recalled, having left behind slabs on the Gulf Coast where their houses used to be. "They said, 'Get me to Chicago, so I can help get the bank restarted,'" he said, getting choked up, even today.
$16 million fix
Loper, who began his 16-year career at Hancock in technology and rose to a senior operations position, was the bank's human resources director at the time. A major in the National Guard, with the military's penchant for orderly processes, Loper worked with Hancock personnel across four states to coordinate the recovery and build a makeshift data center in a bank training facility.
"On November 12, at 2:07 in the afternoon, we moved home from SunGard." Loper said. He was happy to have IT operations back but acutely aware that this was a short-term fix. "We worried every time it rained hard and the wind blew."
When disaster hits, "a lot of things change," Loper said. One change, made at the highest levels of the bank, was how to house the IT assets that ran the business. Hancock wanted a data center as close to hurricane proof as possible, but structures built to bomb-shelter specifications are scarce.
Mindful of its role in reinvesting in the community, Hancock bought property 10 miles inland and 45 feet above sea level to build a data center from the ground up. The cost: $16 million, considerably less than the $50 million spent to rehab the bank's headquarters, but much more than the $1 million to $2 million spent on a bank branch.
Pleased with SunGard's performance in the wake of the storm --"They were the net beneath us, like their motto says, and more," Loper said -- Hancock hired the Wayne, Pa.-based provider to execute the move.
In January 2007, Mickey Zandi, managing partner of SunGard's data center consolidation and migration practice, was surveying the poured concrete floor and "imagining a data center." "They had an architect and builder. They really wanted a trusted adviser throughout the construction phase," Zandi said.
For a trusted adviser, SunGard put in a ton of grunt work.
First, Loper and Zandi agreed that SunGard would take responsibility for the layout of the facility, independent of the architecture and engineering firm. SunGard prides itself on its "proven methodology" for migrating data centers. This starts with discovery -- collecting information on all the hardware, services and applications that will be utilized on Day 1 and needed going forward.
Hancock's IT infrastructure revolves around multiple platforms, including IBM for the mainframe, Hewlett-Packard as one of the platforms for the ATM machines and so on. The bank was also in the early phases of virtualization, so Zandi's team had to understand which parts were not going to the new center.
'No formal CMBD'
To collect the information for the migration, SunGard uses a set of tools based on open systems and off-the-shelf technology. The tools are flexible enough to import and export data into any configuration management database (CMBD), Zandi said.
"In the case of Hancock, there was no formal CMBD," Zandi said. So SunGard interviewed Hancock's technical team and ran scripts, under the "very crowded" conditions of Hancock's temporary data center, and created one. The end product was a visual map showing the location and dependencies of all the IT assets. SunGard left the map behind for Hancock's use.
With the data collected, Zandi's team could begin on the layout of the center for maximum efficiency. But the facility was under construction while SunGard planned the move, adding another layer of complexity. "It was not as though you could walk over and say, "OK, this will go through this door, and this floor supports this amount of weight," said Lowell Grabel, senior engagement manager for the SunGard consulting organization.
'No options for error'
The Tier 3 data center was completed in seven months. Because of Hancock's interdependent system of applications and platforms, the migration could not be broken in chunks but had to be coordinated as a "big bang" move.
"We had one weekend for everything to be taken down and moved and brought up -- and no options for error," said Byron Atcheson, SunGard's account executive for Hancock. "We have a saying at SunGard, 'Moving is nothing more than a planned disaster.'"
That meant bringing down four platforms. It meant moving circuits. In a geography recovering from disaster, clean power and circuits can be rare commodities. "I personally participated in multiple escalation calls with the Ciscos and BellSouths of the world to ensure all elements were available," Zandi said.
Indeed, the government's role in telecom after a disaster caught Hancock unaware, Loper said. "When Homeland Security gets involved, they essentially freeze all the telecom assets and go into TSP mode," he said, referring to the Telecommunications Service Priority process, which assigns entities priority in getting service. "If you do not have a TSP code, you will not get anything done."
The new data center is not only built to withstand hurricane conditions but is also self- sufficient, with its own water and sewer systems, dual 900 kW generators and a 25,000-gallon fuel tank. The tank can be filled with diesel or regular gas, and suppliers have been notified they can use it to top off their vehicles when gas shortages occur in a disaster, Loper said.
In the wake of the storm, the bank also learned a lot about fortifying its people, not just data, Loper said. When Katrina hit, the bank provided food and clothing; it even sent construction crews to put up temporary roofs on the houses of employees sent out of state to get the bank up and running. Since Katrina, Hancock has built a system that will help employees withstand disasters.
"Now, if we send a group of tech people to a disaster site, we send their family," Loper said. "If they have a dog, we are going to kennel their dog. If they have a grandmother, we'll let grandmother with them, so they are self-sufficient."
Let us know what you think about the story; email Linda Tucci, Senior News Writer.