Modern Infrastructure

Disaster response and recovery: This is not a drill

Serg Nvns - Fotolia

Manage Learn to apply best practices and optimize your operations.

Handling disaster recovery during a bona fide crisis

Harvey Koeppel has been there, done that in terms of IT DR, but no disaster recovery and business continuity plan prepared him for the tragedy that struck when he was the CIO of a global bank on assignment overseas.

Much is written about disaster recovery (2.4 million Google results, 8.8 thousand books and other resources from Amazon) and even more about continuity of business (35.1 million Google results, 5.2 thousand from Amazon). As a veteran of the IT industry (with the requisite collection of Purple Hearts, Badges of Courage and Medals of Honor to affirm my veteran status), I feel as though I have read most of them. As might be expected, some disaster recovery and business continuity plans are broad in scope, some deep in detail, some are profound in their philosophical approach, and many are tactical operating playbooks. And still, reflecting upon the collective wisdom embodied in this great treasure trove of techno-business-pedagogy, I find it curious -- and perhaps even a bit unsettling -- that so little is written, taught or practiced around the psychological, emotional or spiritual truths and consequences associated with leading and managing through a bona fide disaster. The story I am about to relate is true; the names and places have been changed to respect the privacy of the people involved.

A few years ago, I was the consumer group CIO of a major global financial services institution on assignment in a country I will call "Eastasia" located in the Asia Pacific region where our bank had significant business presence. Prior to my arrival in Eastasia, the bank (with the encouragement of Eastasia's banking regulatory authority due to some allegedly questionable business practices) had closed down its Eastasia-based private banking business (which as CIO I did not support) and terminated much of the staff. This was not considered a good thing by the thousands of customers, staff and families that were adversely impacted, and certainly not by the bank's executives. At the same time, the consumer group operations team, over which I was the accountable executive, had experienced several incidents where magnetic tapes and data cartridges containing customer and financial account information were "lost in transit" between our data center and our satellite processing center in Eastasia. The Eastasia banking regulatory authority had been unhappy with our offshore processing model from the beginning, and their concerns around our repeated loss of customer data were mounting. No actual breaches or compromise of any information had occurred -- yet. My job was to fix the operations "challenges" and regain the confidence of the country's banking regulatory authority to avoid either having to move our operations onshore (very expensive) or, worse, be told we had to exit our consumer business from Eastasia -- the disaster scenario.

Data breach? Brand damage? Am I out of a job?

One evening around 10:30 p.m, I was lying in my hotel bed when the phone rang. My instinct was that my worst fears had come true -- millions of additional customer records were "lost in transit," the bank was breached and vulnerable to significant financial losses, the brand damage (on a global scale) was inevitable, and the country's bank regulatory authority was about to inform us to pack up, go home and to not even think about coming back. My next thought was about the thousands of additional customers, staff and their families that could be negatively impacted, some in unthinkable ways due to lost savings, unemployment or worse. What could I say or do to make this right again? Who could I appeal to? Somewhere on this list that flashed through my mind as the ringer on the phone seemed to be getting louder was a gripping panic that one of those unemployed people might be me.

I took a deep breath, picked up the phone and said, "Good evening, this is Harvey. How may I help you?" In the moments and hours that followed, things got worse -- much worse than anything I had imagined.

The local team was quite proud of how they handled that situation -- I was now gravely concerned that the bank was under attack.

The caller identified himself as Juan Carlos, country head of San Lorenzo -- one of our bank's largest Latin American markets. For an instant, I felt relieved my team and I were not being asked to leave the country, although that relief was extremely short-lived. Juan quickly informed me of the reason for his call: A bank guard had discovered a plain brown paper bag in the ATM lobby of one of our San Lorenzo branches. Assuming a customer had unintentionally left it behind, he went to pick it up. The bag exploded and the guard was killed instantly. Juan was calling me (11.5 thousand miles and 12 time zones away) to ask what he should do.

My fears about lost data and lost jobs vanished. A human life had been lost. Other lives were at stake and as completely unprepared as I felt, it was time to be the leader I was hired to be. My first question to Juan was, "Did the guard have a family?" Silence. After what felt like an eternal pause, he said he didn't know. I quickly asked him if anyone else was injured. He informed me that it was early in the morning and the branch was not crowded, so fortunately, there were no other injuries. I then asked about physical damage to the branch, which at that point, felt inconsequential by comparison. Broken window; that was all. I quickly moved on and instructed Juan to find out about the guard's family and call me back. In the interim, I shared the news with the appropriate bank executives and risk management teams in New York.

Forty-five long minutes later, Juan called me back. He had contacted the guard company and found out the guard was living with his girlfriend. They had tried to reach her, so far, without success. I asked if the guard had an ex-wife or any children and again listened to what seemed like a very long silence. Finally, he admitted he did not know, but said there was more going on that he needed to brief me about. I felt my heart literally stop.

In the 45 minutes between our calls, another bomb was found in a second bank branch in San Lorenzo a few miles down the road from the first. This time, the police and fire departments were called and the bomb was moved to a park across the street from the branch and detonated. A fireman sustained a minor injury from the flying debris, but no other injuries were sustained or damage done. The local team was quite proud of how they handled that situation -- I was now gravely concerned the bank was under attack. It was hard to be sure of much, but there was one fact I was quite sure of -- nothing I had learned in kindergarten had prepared me for this. I reminded Juan to find out if the guard had left behind an ex-wife and/or any children and asked him to call me back. Again, I called the sobering news into New York.

About an hour later, my phone rang again. Juan was calling to inform me the guard had left behind an ex-wife and a young son. They had been contacted. But, once again, he said there was more information he needed to brief me on. Amazingly, my thoughts went to how easy it would seem to meet with Eastasia's banking regulatory authority later that morning (even with no sleep) and deal with matters of lost data which, just a few hours ago, felt like a world-shattering problem. In the hour that had passed since our last conversation, a third bomb had been detected, this time in a branch of a local bank not related to our bank. I was simultaneously shocked and relieved -- shocked this nightmare was continuing, and relieved our bank was no longer (for now) the target.

With some time to catch our breath, Juan and I began to discuss the best way to handle company benefits that the bank would be extending to the guard's family. Without having to think twice, I instructed him to work with human resources to ensure the guard's family received the maximum allowable benefits our company could provide. When he pointed out that the guard was a sub-contractor and not an "actual employee" of the bank, which could limit the benefits his family might be entitled to, my response was even more swift. I did not care what the logo on his paycheck looked like -- he was working for, and, in fact, literally gave his life, in service of our company. We would do everything in our power to respect his contribution and honor his family, period.

We continued to talk about how to best monitor the still potentially volatile and extremely dangerous situation and agreed to remain in touch every four hours or more frequently as needed until the threat subsided. Later that morning, while I was meeting with the regulatory authority, my assistant passed me a note informing me that the San Lorenzo bomber had been captured and jailed by the local police, and all were confident the threat had been neutralized. As my assistant left the conference room, I sighed in deep relief as I realized that, despite being confronted with significant operational and global political challenges, there was nothing more important than human life.

Harvey KoeppelHarvey Koeppel

Thirty minutes later my assistant reappeared and, again my heart stopped as she passed me another note. I held my breath and, as I read the note, I felt a smile sneak across my face as I read, "… We just lost another tape."

Let me know what you think. Post a comment or drop me a note at Discuss, debate or even argue -- let's continue the conversation ...

Article 1 of 4

Next Steps

Read more about DR and BC planning

Need new reading material? Here are the top five DR and BC books

Disaster recovery and business continuity planning tips for SMBs

Best practices for DR and BC testing

Free DR plan templates

Dig Deeper on Enterprise disaster recovery and business continuity planning

Join the conversation

1 comment

Send me notifications when other members comment.

Please create a username to comment.

What was your worst disaster scenario and how did you handle it?

Get More Modern Infrastructure

Access to all of our back issues View All