General Electric faced a scenario typical for many IT organizations these days.
IT needed to tie its outage reporting system, which runs in the cloud, to its HR system, an on-premises legacy application. The IT team built an application programming interface (API) to integrate the two.
By submitting your personal information, you agree that TechTarget and its partners may contact you regarding relevant content, products and special offers.
"It's a system that made sense to invest developers in," said Chris Drumgoole, COO of IT at GE, referring to the on-premises HR system. The data that needed to flow between the two systems was well defined, and the data in the HR system doesn't undergo rapid change, he explained, so the manual development work to connect this hybrid environment wasn't too taxing.
"You can't do that for everything," he said. "There are thousands of systems that will require APIs, and we can't put that same muscle behind every system that's going to have to talk to other systems."
Indeed, as Fairfield, Conn.-based GE moves further into next-generation platforms, with more applications moving into the public cloud or running as software as a service (SaaS), Drumgoole said the integration work facing his IT staff will become increasingly complex.
GE isn't the only one staring down the challenge of integrating a hybrid infrastructure. Most IT organizations are working in a hybrid environment that includes some on-premises applications, some SaaS products, and still other applications running in either public or private clouds or both.
As a result, integration work has become increasingly more complex.
"When you put applications in more than one place, you have to synchronize data," said Phil Shelley, president of Newton Park Partners, a Chicago-area consulting firm. Getting that synchronization right isn't easy, he said. And the closer it gets to happening in real time, the more complex the challenge can become.
The challenges of a hybrid environment arise around several key areas: data, timing and networking needs, as well as resource provisioning -- that is, getting the time, money and personnel needed to do the integration work.
Tony Peleska, CIO at the Minnesota Housing Finance Agency, is contending with those pain points. He has both on-premises and cloud-based systems in his IT environment, and like many CIOs, he plans to move more into the cloud.
"Integrating these services is our core service," Peleska said. "People want that ability to not have to log into different systems, to not have to enter data multiple times, to have that single source of truth. And to have that single source of truth, you have to integrate these systems."
Like many other IT departments faced with this integration work, Peleska uses APIs. He noted that all but two of his software vendors provide APIs, but some don't meet his needs. When that happens he can hire some of those vendors to build APIs specifically for his needs, or he can use the tools provided by the vendors to have his own team develop APIs that suit his requirements. "I can create third-party APIs using the tools the vendors give me, and the place we're making strides is when they give us the ability to do that," Peleska said.
Chris DrumgooleCOO of IT, GE
However, he acknowledged that the agency isn't quite fully integrated. There still are systems that don't work together, requiring staff or clients to enter information into the system more than once.
But Peleska is addressing those pain points. He has brought on consultants to develop strategies around the initial integration work, as well as maintenance. He's training staff so they have the skills needed to do the work. And he's advancing his organization's work around data management, data governance, audit compliance and business process management so the required data can flow smoothly between systems once the technical integration work is done.
"It's a big task," Peleska said. "It's never as simple as building a piece of software. It has to meet the needs of what the business wants."
Doug Shoupp, a principal at Deloitte Consulting LLP and chief strategy officer for the U.S. technology consulting practice, said integration has always been a challenging task for IT folks, even when all the applications were on-premises. Indeed, integration has been a challenge ever since IT starting moving from monolithic systems containing all the components, he said. Cloud just upped the complexity.
"It is a more complicated world when you start moving components of your IT stack outside. There are obviously benefits to that, but it is a more complicated world. It gets harder when one side isn't in your company," he said.
Sometimes, an API may be all that's needed, Shoupp said, but that is rare.
The complexity of the integration work varies depending on the components involved, he and others said. Integrating a Salesforce app (a SaaS offering) with an on-premises system will likely require an approach that differs from the one used for integrating with an on-premises system with an application running in a public cloud. In that latter case, he noted: "It's less about the application and more about operations and security and risk and all the things that happen when the servers don't sit in your building anymore or you don't own the servers."
Further complicating the matter is the fact that CIOs and their IT team, who were once in complete control of their environment, often must work with cloud vendors and within the limits of the contract -- that's the price organizations pay for moving to the cloud. CIOs have to match their organization's business processes to the cloud-based system to make integration work. As Shoupp pointed out: "Integration isn't just technical, it's a process integration."
Hybrid environment: Start with the data
Greg Davidson, director of the information management services practice at the consultancy AlixPartners, said IT organizations need to start with data, ensuring that they have strong master data management so that the data (including the metadata) required to move between the systems syncs up.
Then there are the timing needs to consider, Shelley said.
"Real-time data integration is very difficult, so batch integration is how it's usually done -- hourly, daily, weekly, whatever. If you need to get close to real time, that requires different architecture -- physical and software -- and that may not exist in every company. And it's more costly," he said, explaining that some systems aren't designed for real time because they may lack key components such as the required security protections.
To be sure, not all systems need to be synchronized in real time, Shelley said. In cases, however, where on-premises systems do need to work with cloud-based ones -- say for sales, marketing, social engagement or transactions, where real-time synchronization is essential -- several often over-looked components must be in place.
Network capability is crucial, Shelley and other experts said.
Resources -- the time, the personnel and the budget -- required to integrate a hybrid environment, is another component sometimes left out of the planning process altogether as organizations plot their move to more cloud-based applications.
Today, most CIOs have some experience around cloud deployments and are getting better at incorporating all these details into their strategies, Shelley said. However, they still might be blindsided by a business unit that deploys a SaaS product or sets up a new app in a public cloud and at the last minute brings IT into the loop to integrate that with existing applications.
Even once the IT team conquers the initial work, IT still has plenty of integration-related work ahead. IT must contend with ongoing maintenance of a hybrid environment. As Deloitte's Shoupp explained, if there's an update or change on one side -- whether to the cloud-based app or the one on premises -- that has to be addressed on the other side, too; if it's missed, that initial integration can break down.
Shelley concurred, saying IT needs to be prepared to maintain these points of integration with skilled staff because, as techies know, "things do break. That's the nature of IT."
Hybrid infrastructure expertise
Given the requirements around setting up and maintaining integrated systems, it's not surprising that the Minnesota Housing Finance Agency's Peleska and others say that integration is now recognized as a core IT service.
"It's complicated, and it's a role that's changing and stretching the IT organization today," Shoupp agreed.
Help is emerging. Both new and improved integration tools, such as cloud integration platforms, are evolving to meet organizational needs, industry observers said. However, they also said that even these aren't plug-and-play solutions.
All this plays into what Shoupp and others said is the shifting role of IT. "It's less about apps and more about data today than ever," he said.
"As more and more systems you don't control are going to touch your enterprise data, you have to control your data," he said. "That's a major shift from a talent perspective. It's not about having the best engineers to custom-build components. It's about whether you have the right data components, the right integration professionals who can be proactive in APIs, and who can have flexibility in changing and adding APIs."
Not everyone, however, sees that in the future.
"I think many IT departments don't want to do integration. They want to get the business users to do the integration. It's happening now. There are more business users who made decisions around SaaS and they're using APIs to integrate," said Holger Mueller, vice president and principal analyst at Constellation Research Inc.
New tools are enabling that shift. In his "Inside the Nine Trends Every CxO Needs to Know in 2015," Mueller wrote: "The traditional enterprise integration vendors as well as the newer cloud integration vendors are stepping up their efforts to offer more diverse and advanced integration alternatives. Specifically, that means enterprises can now connect multiple on-premises 'islands of automation' while running their integration automation from the cloud."
Mueller said he sees integration work getting easier in the future, predicting that, one, evolving tools will decrease the integration work load now on IT and, two, cloud vendors will offer to take on more of the task.
In fact, he predicted that today's challenges around integration will be somewhat short-lived and that this topic will be obsolete within 10 years.
"The question," he said, "is, 'How fast will organizations get there?'"
About the author:
Mary K. Pratt, a freelance writer based in Massachusetts, writes frequently about business management and information technology. She can be reached at firstname.lastname@example.org.
Hybrid management strategy requires asking the right question
Learning how to navigate between the cloud and legacy systems
Solve your enterprise's hybrid cloud integration issues