News Stay informed about the latest enterprise technology news and product updates.

A Wells Fargo roadmap to sidestep technology integration risks

Learn how an air space analysis system helped Wells Fargo CIOs avoid technology integration disasters during the merger with Wachovia.

This is the second of a two-part Q&A with Wayne Mekjian, executive vice president and CIO of information...

services at Wells Fargo & Co., and Martin Davis, executive vice president and head of the company’s technology integration office, about the technology integration of Wells Fargo and Wachovia. In this interview, Mekjian and Davis share advice on avoiding integration pitfalls and explain how they created an “air space analysis” system and methodology to avert integration disasters. In “Wells Fargo and Wachovia: The technology integration of two giants,” Mekjian and Davis explained how they created a blended Wells Fargo/Wachovia technology model to begin converting 70 million banking customers while keeping service interruptions to a minimum.

Wayne Mekjian
Wayne Mekjian

The Wells Fargo and Wachovia merger creates a financial services organization with $1.3 trillion in assets and 280,000 employees. The technology integration encompasses 80 lines of business and 4,000 application bundles and involves more than a dozen CIOs, as well as integration leaders assigned to each line of business.

What challenges did you encounter during the technology integration process?
Davis: There are very complex conversions, like our brokerage system. In some cases, we took the heart of a Wachovia system and implanted it in a Wells Fargo system. Our brokerage systems had hundreds of interfaces that had to be unplugged from Wachovia and synchronously plugged back into Wells Fargo, and all those connections had to flow properly.

This was not unique to the brokerage conversion, but sometimes someone forgets to delete a test file and they put the test file in instead of the production file. We have steps and controls all the along the way to say, “Wait, that was a test file, delete that back up, restart with the production file.”

How else did you avoid technology integration hiccups?
Mekjian: We did a pilot with 30 to 40 customers and converted them onto a new system, and we could mess up and see it. We could see if our interfaces were working properly. Did a wire transfer go through when we made the application change? Did you actually buy the stock? Do your balances look right?

Davis: But every time we go through a conversion, we have a command center that handles any issues that come up to resolve it within 24 hours. If the customer is impacted by that problem, we’re going to make some type of atonement to them and make sure it doesn’t happen on the next conversion.

Since you both have been through mergers and technology conversions, what was your game plan coming into this one? Were there pitfalls that you had foreseen?
Davis: I structured my organization, as did Wayne, around potential pitfalls. I had one leader in my group whose entire job was looking at performance and scalability, because what has gone wrong in prior acquisitions is you bring more volume onto a system and a bottleneck occurs somewhere. So you need one individual working with all the development teams, all the infrastructure teams, looking for bottlenecks and challenges. A financial person reports to me because when there’s a technology integration checkbook, every line of business wants to add in all the things they’ve always wanted. The financial person makes sure that doesn’t happen.

Martin Davis
Martin Davis

Another thing we’ve run into in past integrations is that there will be an issue and teams won’t escalate the issue quickly enough, so we’ve structured the command centers with satellite command centers to make sure the information flows up.

Mekjian: I’ve done conversions where the technology drove [the integration] and the business paid the price. Other cases, the business did it strictly by itself and technology paid the price. In this case, it was a case of saying we all own it, and we spent a lot of time [discussing] that, too. We needed joint ownership on this because we didn’t want to hear later on that we didn’t do something or [the business] didn’t. So it was as much skin in the game in all activities on both sides. When you’re converting [customer] accounts, you’re not just moving data, you’re moving customers, so there has to be a reaction from the [bank] branch also to that flow of data.

What about obstacles that you encountered specific to banking?
Mekjian: Along the way we had problems that we never had before. We learned that if you want to do overdraft protection, you need to create a surrogate account to connect them together. The timing can be a little difficult to keep that customer transaction intact. Something else we just learned, and we fixed, was that you can [warn] the customer that their old [ATM] card won’t work on a given day, and the day of conversion comes and they’re still using their old card. So, we can try to convince them to use a new card, or find a way of not making them replace their card right away. That’s a way to make the customer happy, and we save money by not reissuing cards right away.

Did you work with an outside technology integration expert?
Mekjian: We did try that once before and it doesn’t work. They don’t know your organization. Anyone can create a cookbook. We could create one and sell it, but when someone went to use it it wouldn’t work because everyone’s different. Unless they’re exactly the same company, on the same systems and the same people, it won’t work.

What did the integration team leaders do to make sure that everyone kept the integration front and center?
Mekjian: We had tech summits where everyone talked about their project, the issues they were encountering and how they were going to approach it. There were hundreds of other people listening into their plans and strategies who said, “What about me? You can’t do that without me.”

Davis: Every line of business is represented by an integration leader. Once a quarter, all the integration leaders come together to share where they are, what they’re doing, what worked and lessons learned, so everyone can learn from everyone else. Every Friday we bring together every technology leader across the company to communicate the same thing. Everyone has to know the master plan, that’s the key takeaway.

Mekjian: The world also isn’t going to change just because we’re doing an integration. We also do flybys on the infrastructure suite, where we do roughly 10,000 changes a month. Those are 10,000 changes that everyone is looking at and thinking about the effect they will have on the integration. Should we do it and when should we do it? Timing is critical.

How do you manage the constant technology integration changes and continue to manage the day-to-day of 10,000 infrastructure changes?
Davis: We put together a report called the air space analysis. So for any integration event, you want to know what else is happening in your air space. So say we’re going live this weekend with X, Y, Z events. We ask, “What else is in the air space?” We have three different teams that look at that air space analysis report and say, “Hmm, I don’t know if we should be IPLing the mainframe at the same time we’re doing these other changes.” So we huddle up, push the mainframe IPL off to another week. That is just one example, but we are looking at thousands of changes and making those kinds of calls.

More on IT integration

The technology integration of two giants

Converged data center infrastructure promises to banish IT silos

Mekjian: And the key is to get it onto one change control system. So all changes going on are logged into that system, and it’s not just today or for that weekend -- it’s way out. Before we had multiple change management systems and technology groups out there and we learned quickly that if you’re not all on the same page, you’re dead. Because we have executive management tie-in, everyone’s following the rules. There’s been a lot of times where we had to look at that [air space analysis] report when the infrastructure group has said, “It’s critical; we have to do it.” And we have to say, “Well, we need to do this and if we do both, the risk is too high.”

What is the biggest risk you set out to avoid in this technology integration?
Mekjian: My biggest risk is doing too much and not being able to recover. You promise too much, too many players are involved; therefore if you do have a hiccup, you’re in trouble. We’ve been working through that and have found ways to mitigate it.

Understanding how much risk is in a weekend, so identifying what’s going on, how many people are involved, how many orders, how many interfaces, how many customer are involved … just understanding how to risk assess an event is extremely important. The air space analysis report, that was because we felt we had too much risks on certain weekends. So to mitigate that, we went through and said some of this stuff has got to give, we’re not going to do it. We’re not going to plan stuff to fail.

CIO Innovators

The CIO Innovators profile series highlights how CIOs use technology to meet both IT and business leadership objectives. To suggest a leader for a future CIO Innovator profile, email [email protected].

Let us know what you think about the story; email Christina Torode, News Director.

Dig Deeper on Enterprise application development, DevOps and software agility