Sooner or later, every growing midmarket company arrives at an agonizing choice: Do you build your own data center or outsource it?
In greater numbers than ever before, midmarket firms are sampling a burgeoning menu of opticoons to supply their most critical computing facilities. They range from outsourcing every stick of infrastructure to handpicking just a few key technologies for third-party hosting (also called managed services) to taking the traditional route of building it in the basement.
In outsourcing's corner, there's CIO Greg Schueman. Avoiding the financial burdens of a data center build-out actually figured into the business strategy behind Employers Direct Insurance Co. in Thousand Oaks, Calif. "Once you decide to be an owner, so much of your future is predicated on that," says Schueman. "Once you build it, you're locked in."
A 15-year IT career in large enterprises gave Schueman a lot of experience constructing enormous data centers. "Philosophically, it becomes a question of agility," he notes. His current 18-person IT staff has extensive expertise in the insurance business and project management of external service providers such as Salesforce.com and Avnet Inc. "Fundamentally, our strategy is to outsource or host everything we possibly can. We have the least troubled IT organization I've ever run," the CIO adds, "because literally, things don't go bump in the night."
At only 2 years old, $165-million Employers Direct is California's only investor-owned, direct writer of workers' compensation policies -- meaning it uses no agents or brokers. "The structure here is a pretty radical departure from what you'll usually find in the commercial insurance segment," explains CEO James Little. "The vision of not bringing [IT] in-house has really been the key to the success our company."
In the other corner, there's Barry Brunetto, vice president of information systems for Blount Inc., a $750-million, Portland, Ore.-based manufacturer of outdoor products and industrial cutting tools. "Most companies that sell outsourcing services are trying to make the CEOs and CFOs believe that technology is a utility. I do not believe that," says Brunetto. "If you outsource your strategic enabler, you are risking outsourcing your future."
Just last month, he fired his managed services vendor (IBM), brought a major production application (SAP) back in-house, and installed his own Dell and EMC hardware at an AT&T data center hosting facility in Mesa, Ariz. (see "Meet the Terminator").
After subtracting first-year costs of $850,000 for new hardware, Brunetto expects eventually to save between $700,000 and $900,000 annually by bringing his SAP application back in-house and providing his own hardware to the hosting facility in Arizona. "The whole managed services piece turned out to be a lot of aggravation and expense," he says bluntly. "I would never do it again."
The day of the outsourcer
Still, analysts like Jeff Kaplan of THINKstrategies in Wellesley, Mass., believe managed services -- which he calls "out-tasking" -- are the wave of the future for midmarket companies. The catalyst can come from multiple angles: regulatory compliance, internal growth, mergers or acquisitions, spiraling Web site traffic, changing disaster recovery needs, even the loss of key personnel. "Out-tasking includes a variety of managed services, such as the storage function, the security function or the networking function," he explains. "Rather than putting yourself at risk, you parcel out bits of your data center, measure the outsourcer's success and incrementally give them more responsibility over time."
New technologies in virtualization, remote monitoring and data center management are also multiplying the options available to midsized companies today. "Midmarket companies can have better visibility into what the service provider is doing," says Kaplan. "I'm seeing a lot of growth in managed services. The deals are getting smaller, and the time frames shorter. That means less risk."
Here are some of the midmarket players that analysts and users identify for managed hosting services:
- Everdream Corp., Fremont, Calif.
Desktop management services
- mindSHIFT Technologies Inc., Fairfax, Va.
Managed technology services with a vertical focus in law firms, associations and professional services
- CBE Technologies, Boston
Managed IT services for small and midsized companies, schools, and government agencies
- AimNet Solutions, Norwalk, Conn.
A range of managed services
- SAVVIS Inc., St. Louis, Mo.
A range of managed services
- Perimeter Internetworking, Milford, Conn.
Managed network and security services
- Rackspace Managed Hosting Ltd., San Antonio
Managed Web and infrastructure hosting
There are unquestionably more outsourcing deals involving midmarket companies these days. And many have never outsourced before, says Richard Matlus, a vice president of research at Gartner Inc. "We're seeing more selective outsourcing, and along with that is multi-vendor outsourcing."
On the vendor front, there are the Big Six in outsourcing, which Summit Strategies in Boston identifies as IBM, EDS, Hewlett-Packard, Capgemini, Accenture and CSC, followed by a plethora of smaller players (see "Midmarket Players," at right). But midmarket companies are historically leery of striking deals with these providers. In a September 2005 survey of 179 customers at small to midsized companies, only 6% to 7% identified IBM or HP as "strategic vendors" in their outsourcing plans. The percentages shrank further for the remaining four. Even more telling, 21% of those surveyed by Summit Strategies said they would "prefer to avoid" EDS. Yet roughly one-third were willing to consider these major vendors "if the price was right."
Analysts don't track the size or number of deals with midsized companies yet, but everyone agrees that the market is growing. "Ten years ago, vendors weren't giving midmarket customers a second look," says John Madden, a practice director at Summit Strategies. "The vendors' goals were to rake in mega-deals, but now those kinds of deals are fewer at the enterprise level. The vendors all need to look at the midmarket as having greater potential growth." Just last month, IBM Business Consulting Services announced that it will sell services to small and midsized businesses through regional resellers.
Still, there are plenty of midsized providers ready to serve users with big-vendor aversion. To win business, they emphasize their solitary focus on the hosting business. "This is all we do: hosting Web sites, infrastructure and IT systems," says CTO John Engates of Rackspace Managed Hosting Ltd. in San Antonio. "We're not a phone company. We don't do development or consulting." The managed hosting provider -- itself a midsized company at $140 million in annual revenues -- competes for business against giants such as IBM, EDS and AT&T.
Rules of the outsourcing road
When it comes to what should or shouldn't be outsourced, Gartner analysts recommend against handing off key technology architecture such as operating systems. "Instead of the entire data center, [companies are looking] at outsourcing a portion of it," Matlus explains. "I see a lot of clients looking to outsource their mainframe because the people supporting mainframe systems are nearing retirement age and the younger members of the IT staff ... want to work on Linux and Windows machines."
For midmarket enterprises operating under the classic constraints of fewer resources and smaller staffs, offloading labor-intensive processing tasks can provide "a chance to look at the business," says Summit Strategies' Madden. "If the CIO is busy plugging holes in the data center, he won't have time to deal with the business side of IT."
Madden identifies four main factors involved in evaluating data center outsourcing:
- Control. "You need to lay down the rules about who is responsible for what and who is responsible if something goes wrong."
- Security. "Customers of all sizes, particularly in the midmarket, need to know that the assets are going to be protected."
- Trust. "Some customers are more inclined to look to a third party for non-core competencies such as infrastructure and management."
- Cost. "It's not just about 'How much can I save?' It's about measuring the value and the business benefits that outsourcing will produce."
Show me the savings
But the cost factor does loom large for midsized firms on either side of the equation, whether the data center is an in-house operating expense or an outsourced capital expense.
The investment required to build your own data center ranges from $250 per square foot to $1,000 per square foot or more for a high-end, fully redundant tier-one data system, according to various experts. "I don't think [midmarket companies] can afford not to outsource. The stakes are too high and the costs too great" to build your own, says THINKstrategies' Kaplan.
So you're in the market to outsource your data center but leery of the enterprise-focused Big Six (IBM, EDS, Hewlett-Packard, Capgemini, Accenture and CSC). Where do you start?
"When deciding to outsource, look at your own company culture," recommends John Madden of Summit Strategies. "If you have a bunch of employees who are support-dependent and they need a lot of help from your team, think about how are they going to react if that goes to a third party."
Madden says he is often surprised to find CIOs who don't have a clear idea of the assets, applications and servers in their environment. "Take an entire inventory. And then look at the skill set internally: Does it match what you have in that inventory?"
Gartner's Richard Matlus agrees that the first step is taking a comprehensive inventory of hardware, software, people costs and facilities. "When you start to do that, you see what's important," he says. Other experienced IT executives and experts offer these additional tips:
- Check in with your own network of IT contacts and colleagues. They'll tell you the truth about service levels and support.
- Use analysts' services to "map out the network of players," recommends CIO Greg Schueman of Employers Direct. "You hear a bit about the players, and you can plan your next steps from there."
- Google the companies you're most interested in to see when they've been in the news. That also helps identify customer companies that aren't on your candidates' reference lists. Call those customers and find out why.
- Ask customer references these questions: "If you were to do this over, what would you change?" and "What would you have taken out of the contract?"
- Plan to do real-life responsiveness testing once you have the vendor under evaluation. "When I call in with a question, I want an answer quickly, or tell me what the process is to get me that answer," says Lawrence Tam, VP of engineering at Constellation HomeBuilder Systems.
- Offer to be a reference customer in exchange for special consideration. "Pretty up before you go out on the main floor," says Schueman. "What's your key value statement on being their customer? Will you be willing to talk about your experiences and help them out?"
- Consider hiring legal help for contract negotiations, particularly if you're a first-timer or it's a larger company than your own.
"For us, it's about the reliability and the facility," says CIO Justin Yaros of Kelley Blue Book in Irvine, Calif., a publishing company that rents data center space from Missouri-based SAVVIS Inc. but keeps all applications in-house. A joint advertising campaign with General Motors -- and the additional server workload on the publisher's Web site (www.kbb.com) -- underscored the importance of turning to outside help. "We're servicing 130 million automotive consumers a year through our Web site. That means that availability is critically important," Yaros notes. "But for a company like ours to bear the cost of maintaining the data center would be unbearable."
Monthly hosting costs are far from trivial. At Rackspace, for example, the hosting costs among its midmarket enterprise customers range from $5,000 to $10,000 a month on the low end to $300,000 to $400,000 in the upper echelons. But there are ways CIOs can drive down hosting costs through better negotiating or a willingness to make compromises to get a certain price.
At Employers Direct, Schueman's vendor proposals ranged from $20,000 to $60,000 a month. He ended up in the "high 20s" with his final contract, which doesn't include disaster recovery protection. "I was looking at it based on how much it would cost me to do all this stuff and then trying to drive it as low as possible. I'm a tenacious negotiator," he explains. "We did some things others might not choose, like using AMD CPU equipment from a white-box vendor and Sun Microsystems' low-end Opteron-based line."
Schueman was also willing to try a vendor relatively new to the managed hosting space, Phoenix-based Avnet, an IBM partner that is better known as a product distributor than a data center hoster. "We're a good catch for Avnet in the midmarket space where they're aiming," he notes. His actual contract is signed with IBM. Before the deal was signed, Employers Direct made sure to have its Los Angeles-based legal counsel hammer out the actual terms. "Our L.A. firm has experience dealing with IBM contracts, and I've used them very successfully in the past," says Schueman. "IBM will start out with a contract not favorable to you."
The CIO watchdog
Along with contracts come even bigger challenges: control issues that range from security to support. For any CIO, loss of control over key parts of an environment has a nightmare-come-true quality to it. Failure to keep core competencies inside the IT organization can allow the outsourcer to hold your company hostage.
Meet the Terminator
Barry Brunetto will tell you straight up: He's not the easiest customer for vendors to handle. IBM learned that the hard way.
"I'm a Sicilian from New Jersey. My job is to basically watch out for the company assets," says Brunetto, vice president of information systems for Blount Inc., a manufacturer of outdoor products and industrial cutting tools in Portland, Ore.
Brunetto recently terminated his contract with IBM and its subsidiary Corio for service and support failures in an outsourcing deal involving the management of Blount's SAP system.
"We live in a touchy-feely world where you get a lot of people singing 'Kumbaya' about being partners with your vendors. So a lot of CIOs don't say anything when there's a problem," Brunetto says. "Well, I have a simple rule: I pay you for a service, and I expect one to be delivered."
"IBM doesn't comment on the specifics of customer engagements," IBM spokesman Jim Larkin responds. "But we can say that we're working closely with this client to address the concerns."
Brunetto's resolute philosophy sits well at a company like Blount, founded 60 years ago by a World War II pilot and his brother, who got their start building fish ponds and bridges in Alabama. Today, this $750-million concern employs more than 3,000 people at operations worldwide.
Three years ago, Blount executives decided to outsource management of a new global enterprise SAP system (along with its data center hosting) to a small applications service provider (Nexus Technology), which was later acquired by Corio. The production environment is hosted at an AT&T facility in Mesa, Ariz. "In our minds, hosting and managed services went together -- that's how the vendors want to sell it," Brunetto explains. "We wanted to pick a place that we felt would be a safe place, protecting it from a disaster. If we kept it in Portland, we could still be hit by earthquakes."
Early last year, IBM acquired the financially troubled Corio and folded its 300 employees into IBM's on-demand business unit. Services had already started to falter for Blount by that point, Brunetto recalls. When IBM took over Blount's contract, Brunetto was characteristically blunt with the new managers. "I warned IBM: 'We're going to open the gates of hell on you if you mess this up.'"
But mess it up they did. The real trouble hit last summer with the apparent mismanagement of the SAP application on a storage area network (SAN) at the AT&T facility. The worst of it hit during an 18-hour period one weekend last July during a planned shutdown to switch Blount's SAP application over to a new SAN. "They didn't have the SAN laid out correctly, and they didn't follow SAP's recommendations," Brunetto says.
"They changed our disk to low-speed disk to save money." Once the system move was completed on Sunday, Blount's IT group started letting people sign on from four manufacturing and distribution locations.
"We did not see the problem until Monday morning, when the typical load was placed on the environment," Brunetto recalls. Users were asked to limit use of the system to customer-related processes and critical business needs. In the meantime, his in-house technical team -- which fortunately included experts in SAP Basis and server technologies -- evaluated what was happening. "It seems that we were not getting all the facts from IBM, and it was evident that they did not spread the files properly across [the] spindles based on SAP specifications," Brunetto says. "I authorized them to bring down the system and install more disk spindles. Later that evening, they released the system back to us."
Now it was Tuesday morning, and system performance was worse than the day before. "I was getting nasty e-mail from around the globe," the VP recalls. "It was horrendous."
By now he was on the phone with account executives from SAP as well as his own contacts within IBM, who were trying to help behind the scenes. "Already we had started hearing excuses from IBM that sounded like the problem was not related to the SAN and that we needed to increase the number of servers, which of course meant more money," he says.
With yet another conference call pending later that Tuesday morning -- involving a widening circle of irate company officials -- Brunetto's team figured out that a particular switch in the SAN was probably set incorrectly. But they didn't have access to the production environment in Arizona to verify their suspicions.
"Before that last call, my team contacted the technical team at IBM to inquire about this setting. We never got a verification on what this setting was. But by the time we had the call, the system performance had improved drastically," he says. "So we really believed they had this set wrong. ... And from that point in time, we had a workable system. It wasn't optimal, but at least acceptable."
In the aftermath, Blount decided to stay with the AT&T facility but invested $850,000 for its own Dell and EMC hardware to install at the hosting site and replace the boxes owned by IBM. The company also hired one more SAP expert to join the IT staff in Oregon.
The data center hosting charge will be $88,000 a year now, versus the $505,000 annual cost of the hosting plus managed services piece. "If we need a new application server, it won't affect our monthly hosting fee," Brunetto says. "We can add one for about $3,000 instead of paying IBM $2,700 per month to add a new server to the contract. That's how they charge their management fee, so as you grow, that becomes a big ticket."
What was destroyed in the whole experience was the company's trust in IBM as a managed services provider. "The one key thing we all realized was that if we hadn't raised all holy hell, it just would have gone on," Brunetto says grimly. "And that could have really hurt our business."
So how much expertise do you need in-house to keep a proper eye on your vendor partners?
"If we had outsourced all the SAP environments like some of the vendors wanted us to do, there would have been no reason to have SAP Basis and technical support people in-house," says Brunetto. "If we did not have these people with the knowledge base looking out for the company, keeping IBM honest and focusing on the real problem ... this whole episode would have gone on for a much longer period of time and had a real impact on our business." Brunetto was able to prevent major business disruption when he grabbed the reins of his production application back.
"I don't think a lot of people, when they are looking to outsource all of their technical expertise, realize this," he adds. "I don't believe the outsourcing companies purposely try to harm their customers. But The Wall Street Journal and Fortune magazine are full of stories of stupid things that companies do today. My story shows that having technical expertise on staff can prevent major problems for the operations."
Keeping in-house expertise has been critical at Kelley Blue Book as well. The publisher (which is privately held and doesn't disclose revenues) chose to outsource only the "ping, power and pipe," says Tim Abbott, director of infrastructure. (Ping for sending a packet to the server to see if it responds, power for the electricity and pipe for the high-reliability Internet connection.) The hosting provider takes care of Kelley Blue Book's 90 servers at three data centers in Santa Clara and Irvine, Calif., and St. Paul, Minn. All applications remain within the company's control.
"We're in a period of rapid growth, and this is a time when we'd rather have our own hands on managing our applications and hardware for availability reasons," CIO Yaros explains. "I've gone through the process of implementing managed services. It's not something to take lightly."
When the first GM/Kelley Blue Book commercial aired, "Tim and his people were actively monitoring to see if the commercial had bogged down our servers," the CIO recalls. "We could see firsthand if it had an effect on the server loads. Tim was there to make sure there were no surprises. This makes us more comfortable."
Analysts and other experts also stress the importance of technical oversight of the outsourcing contract -- no matter how extensive the third-party arrangement may be.
"There are some other functions that you must retain. When you outsource, you also have to have a good contract manager," notes Gartner's Matlus. "Your data center manager knows what went right in the data center and what was happening, so someone should be [managing] the overall contract ... , looking at your [service-level agreement] monthly and validating that [the outsourcer] is delivering what is in the contract."
One Gartner client who had outsourced his wide-area network ended up paying much more than he did before the deal due to the lack of a technical person involved in oversight. "Accounts payable people were looking at the bill and just paying for it," Matlus recalls. "What the finance people didn't know is there was a charge on the bill for a $100,000 service that they were not getting."
A matter of trust
The "feel of the deal" may sound like an intangible factor, but it's one that experienced IT executives heed closely, especially in choosing between a large provider like IBM or EDS and a smaller services provider.
"Size doesn't matter. It's how comfortable I feel with the service they offer," asserts Lawrence Tam, vice president of engineering for Constellation HomeBuilder Systems Inc., a software provider for the home-building market and a division of $150-million Toronto-based Constellation Software Inc.
Over the past few years, Tam has been an owner and a renter. Those data center experiences made him a confirmed fan of outsourcing the infrastructure. "I really want my team to focus on software development for customers," he explains. "We outsource everything but the application layer and support to Rackspace. We're renting 20 servers, and they take care of the security, patching, database backup, whatever comes up on the hardware side."
The engineering executive likes not having to worry about the myriad issues involved in running a data center. "We make money by selling software, not by hosting it. I don't have to hire my own security expert, my own configuration expert, my own network expert," he points out. "The outsourcer has that expertise in-house. Their work with other companies means they're learning from those other customers and solving problems that might apply in my own environment."
The joys of ownership
Outsourcing the data center was never a serious consideration for $425-million Silicon Laboratories in Austin, Texas. "We're a technology organization supporting a technology company. Everything we do ties right back into the business," says IT Director Kristi Browder.
About three years ago, this semiconductor manufacturer and designer of integrated circuits consolidated a variety of smaller data centers into one new facility at corporate headquarters. The $1-million build-out turned an employee fitness room into a state-of-the-art, 24/7 computing facility. The driving factors behind the data center project -- in order of importance -- were company growth, regulatory issues, lower costs and greater security, Browder says.
Showing a visitor around one day recently, Browder barely breaks stride as she sweeps her access card over the wall-mounted scanner and presses her finger onto the companion biometric device. The door to Silicon Labs' data center clicks open with a rush of cool air and a muted roar of sound. If you close your eyes inside a tier-one data center like this, you could swear you were standing under a waterfall. A trio of industrial-strength cooling units makes its presence known; these units keep nearly 500 systems humming along at optimum temperatures. "Oh, we love that sound," the IT director says with a satisfied smile, raising her voice a bit to be heard. "Quiet means something is wrong."
Every Wednesday morning, the company fires up its enormous 800-kilowatt generator stationed out back in a test run that belches black smoke into the air and startles everyone in the building with its rumbling. When the power fails, which has happened only once this past year, the generator takes over within seconds.
A staff of 18 tends to the data center, and another 18 are assigned to the software infrastructure and user needs among the 650 employees. "I've worked for outsourcers before, and I know it's really a question of what your core competency is," says Browder, a lifelong resident of Austin, which has become something of a magnet city for large data centers due to the lack of natural disasters in the city. "Are you building widgets or building infrastructure?"
Virtualization and open source are both key technologies for Silicon Labs in running its data center, and the 350 Windows servers require only 80 actual boxes. "We ask two questions about every application. Does it run on Linux? Does it run on VMware?" she says. "From a supportability standpoint, we love Linux. And virtualization lowers costs."
The escalating sophistication of such technologies also means her data center will be able to accommodate more growth without enlarging its footprint. Even so, Browder keeps an eye on outsourcing trends (which she carefully refers to around staffers as "complementary sourcing strategies") and doesn't rule out such partnerships in the future.
During his evaluation of Rackspace and other larger players, Tam subjected his finalists to an unusual test. "I wake up at 3 a.m. and call them to see how fast they pick up the phone. I'm looking for a two-second response time," he says. "That's important because in real-life situations, Murphy's Law shows us that bad things happen at night. If there's no one around with the knowledge of my environment, I'll end up playing phone tag."
And who wants that in the middle of the night?
Tam's 3 a.m. test call is only the tip of his requirements iceberg, however (see "Outsourcing Checklist"). "There are a number of things to evaluate," he says. "One is the financial stability of the vendor. You don't want them out of business in two or three years, because every data center migration is a big task. Two, the customer support has to be outstanding and responsive. And three, the customer references have to be there."
At another company a few years ago, Tam had the unpleasant experience of pulling an outsourced data center in-house after his CFO decided the hosting was too expensive. "He looked at the numbers and said, 'The incremental cost of supporting 20 servers isn't that great.'"
But setting up the data center took the IT group more than a year, and uptime was rarely better than 98%. "It was really bad. On a daily basis, we were down anywhere from 10 to 20 minutes," the engineer recalls. "When I look at a holistic picture of what it takes to really host an application, it's not just about the network. It's about providing information to the customer -- and about application availability at the right time -- without worrying about security or scalability."
Outsourcing's high ground
Providing business continuity in the aftermath of a disaster is another deciding factor for some companies when they reach their data center crossroads.
For $1.4-billion MPS Group in Jacksonville, Fla., it was fear of flooding in its downtown Jacksonville data center that sparked the initial move to outsource part of the data center infrastructure in December 2003. MPS provides environmental and pollution control services to the automotive, chemical, steel and utilities industries.
"We also had a changing of the guard with a new CIO [Richard White] who had experience in colocation," says Allen Rittscher, vice president of application development for MPS. The company's aging facility had 300 servers, with maintenance costs climbing and no dedicated staff to oversee the environment, he adds. That facility now serves as a test environment.
Yet rather than going with an outsourcing vendor or managed hosting provider, MPS chose CSX Corp., a Fortune 500 transportation company that makes its tier-one data center -- built to withstand a Category 4 hurricane -- available for colocation. The fact that CSX has powerful industry influence in the railroad business was also a big draw for MPS.
"Since CSX is in transportation, they have strict guidelines they need to follow," says Cary Jankowski, director of MPS' system administration group. "Their data center is on the highest point above sea level in Jacksonville. It has lightening suppression and enough diesel fuel to run over 30 days at full capacity. CSX's relationship with fuel vendors, being a transportation company, is also very strong."
That additional business imperative -- to keep the railroad running -- makes CSX a stronger disaster recovery ally for MPS, Rittscher adds.
MPS rents 680 square feet from CSX, but without managed services included. "CSX was pushing us early on to take advantage of their expertise, but after talking to us they realized we have that management expertise in-house," he explains. "We just needed their facility."
Stefanie McCann, former editor at large for CIO Decisions, contributed to this article.
Maryfran Johnson is the founding editor in chief of CIO Decisions. To comment on this story, email [email protected].