The call came at 3 a.m.
Francis Juliano was working as the CIO at a former employer he prefers not to name when he was woken from a sound sleep by his chief executive officer, who was traveling overseas. The CEO couldn't get or send email through his corporate account and wanted to know why -- instantly. "It was a very heated phone call," Juliano recalls.
As it turns out, the marketing department had changed its outbound email management vendor a few weeks earlier without telling the CIO, and it had made a few mistakes.
First mistake: While the marketers collected data on opt-out requests and new customers who responded to marketing emails, they didn't send this data to their new vendor. So people who opted out were still getting emails, and new customers weren't showing up in the company's database.
Second mistake: The marketers were sending email campaigns through the company's own domain instead of using the vendor, which vets email campaigns to ensure that they comply with antispam laws as well as antispam rules set up by major email carriers like America Online. Within three weeks, the company surpassed its traffic limits and was flagged as a spammer. And so the provider shut down all its email.
It took Juliano three days to restore order. "What really blew me away was the initial reaction [from marketing]," he says. "They said, 'How could you, IT, let this happen?' I said, 'Who? We didn't let it happen. You decided to go out on your own, and now you want to tell us it's our fault?'"
Welcome to the underworld of rogue IT projects.@pb
Damn Those Covert Operators
They happen in all the obvious ways: a power user just wants to experiment with that server underneath his desk; a temp worker installs a network-hogging application on his desktop to save time; sales guys expense a wireless access point so they can use it in their part of the office. Then there's the guy who knows just enough technology to be dangerous and thus builds his own database because he's tired of waiting for someone else to spec it out.
Although it can happen in any unit of a company, the CIOs we interviewed agree on the most likely culprits: engineering and research groups, who typically install their own computers and software for developing new tools or running experiments. "They're notorious," says Bart Stanco, a senior vice president at Gartner Inc. and its former CIO. Software developers, sales and marketing are offenders too, he adds.
These folks might be entrepreneurial, tired of waiting for IT or determined to do projects behind the curtain because they got cut from the official budget. So there they operate, their software code spinning in stealth mode until the hapless CIO discovers them during a vulnerability scan or production mysteriously grinds to a halt.
Midmarket companies are especially vulnerable because they have entrepreneurial histories. For years companies grew without an official head of IT. So finance executives created spreadsheets that grew into mission-critical applications. And everyone got used to running around the corner to CompUSA to buy what was needed to get through the day. Suddenly, a CIO arrives -- but few want to give up control.
Although business rebels who launch these projects do so to improve how they do their jobs, they often end up hurting the company. If these projects don't cause an immediate disaster like the one Juliano encountered, they have other downsides. They cost more money than they're worth because they don't integrate well with the company's existing platforms. They make the corporate network more vulnerable to viruses, or they don't scale. Among other problems, sometimes they just don't work.@pb
Rogue projects happen between IT groups, too. Stuart Williams, a former consultant to CIOs at Magenic Technologies Inc., a Minneapolis-based consulting firm, recalls one instance where IT at a large financial services company set up a Web-based quote service without telling corporate IT. Unfortunately, the department forgot to secure the site against hackers.
As Williams tells it, a "script kiddie" discovered the unsecured Web service and wrote a script that queued thousands of transactions just to see if he could wreck it (which he did). The company's back-end servers were offline for six hours during prime time. The cost: hundreds of thousands of dollars, says Williams. When it was discovered that the department misled corporate IT about the project, heads rolled, he says.
In theory, good IT governance should help. If a procurement office won't pay for IT equipment unless the CIO has signed off on the purchase, that should nip rogue activity in the bud. But analysts say anecdotal evidence paints a different picture: As the price of technology has decreased, rogue IT has proliferated. Ten years ago, it was fairly difficult to buy a PC without anyone noticing. Now, even a low-level manager can use his company credit card to set up a stealth mini-network.
Too much governance also encourages rogue IT, says Marc Cecere, a Forrester Research Inc. principal analyst and vice president. Take project initiation processes. Ideally, they turn an idea into a working IT project efficiently. But Cecere has seen cases where it takes months to work out the process, let alone start the project. "The whole process becomes so onerous that essentially no one even starts the process because they know it will take forever. So they go to the person sitting next to them," Cecere says.
Other times, CIOs who are flat-out oppressive drive users to rebellious behavior. Cecere has seen CIOs forbid employees from hooking up to a hotel network while traveling. "They'll say, 'We want you to use only our [secure Internet access] software,' which slows systems down to a crawl, or 'We want you to use a dial-up facility,'" Cecere says. "People say, 'Well, that's nuts. We're not going to do that.' And they'll figure out ways around corporate dictates."
Magenic's Williams puts it more bluntly: "Business people need to get things done. Their salaries are tied to their performance. When you stand between them and [their] dollars, they will lash out and operate how they please."@pb
Corralling Wayward Tech
Williams says that technically network "sniffers" could put the kibosh on most rogue IT. After all, they can alert IT whenever new assets are plugged into the network. Fancier versions can even disable new assets until they've been cleared by IT. But in reality, a lot of CIOs who install network intrusion detection software don't use it properly, Williams says, much like a homeowner who puts in a fancy security system but rarely turns it on.
CIOs who have had to deal with rogues say they first try to see things from the employee's point of view. "If you get confrontational, all you're going to get is a lot of static," says Dennis Roell, a CIO who discovered three years ago that his engineering department was trying to automate his company's warehouse without telling him. "It's not personal. They had a need. They tried to fill it. They're trying to do their job."
Roell works for Betts USA Inc., a privately held manufacturer of toothpaste tubes in Florence, Ky. Roell is Betts' one-man IT department, serving about 150 employees. Roell found out about the warehouse plan when the shipping department called to say a controller had arrived, followed shortly by a call from the head of engineering, who couldn't make the system work.
The stealth project started when engineers tried to solve a problem on their own: Tiny parts, from toothpaste caps worth pennies to $10,000-a-pop controllers, were getting lost in the warehouse's giant room. The engineers' solution was to place the parts in trays, stack the trays in three 20-foot towers, and use a robotic arm to retrieve and replace the trays.@pb
The head of engineering figured he could go ahead without consulting Roell because the system seemed to work well at a sister company. He didn't expect Roell to help much anyway. After all, Roell's predecessor had been more of a systems babysitter, the kind of guy who buries his nose in manuals but rarely offers solutions. So when a sweet-talking salesman pitching robotic parts sealed the deal, the head of engineering was off and running.
Roell makes rounds of the company but had somehow missed hearing about the warehouse problem. "Anytime a CIO encounters rogue IT, it should always come as a wake-up call," he says. "Did I miss something that they really needed and they couldn't get from me? Or was this something where they had a request and I told them they had to wait? In either case, I'm sort of disappointed that I wasn't involved. But now I want to understand the need."
Indeed, the robotic warehouse kept better track of inventory. But the system couldn't plug into the network, and it lacked security. Roell ended up ditching the software that came with the robot and writing his own. He secured the inventory by placing the three towers in a small room that requires a security badge for entry.
Then he set up the computer interface so that employees had to identify themselves with a passcode and a biometric hand scan before they could tell the robotic arm what to retrieve. If they requested parts they weren't allowed to have, the system would generate a permission slip for their supervisor to sign. For additional security, Roell installed a video camera in each tower and one for the entire room. By the time he was done, breaking into the system had become next to impossible.
Around the same time, Roell discovered a piece of machinery that didn't have an asset tag, an object he now calls "the $50,000 ruler." Again, engineering had installed a piece of equipment without permission. The head of engineering told Roell that he had to buy the device because a major customer had insisted they have it. Roell agreed on the necessity but wished he'd been informed. "I guess I could have told you," Roell says the head of engineering told him.@pb
The Business Trump Card
Despite the department's chronic independence, Roell has not been in a position to scold the head of engineering. "He's above me in another department. What can I tell you?" Roell says. "Since IT has no direct reports, when operations wants to go in their own direction, they go in their own direction." If you want to control IT, "you have to do it by persuasion."
But after calmly salvaging the engineering chief's two rogue projects, Roell gained his respect. Roell also made a point of prowling around the plant more often for hints of rogue activity. "Call it a good lesson learned," Roell says. As for the head of engineering, "Since then, he doesn't even install anything on his computer without consulting me first," Roell adds.
Stanco agrees with Roell's noncombative approach. "If you're going in with guns blazing, you might alienate your business partners, even if you're right. You win the battle but lose the war," Stanco says. For a CIO, the most important thing is having "a good relationship with the business."
When he was Gartner's CIO, Stanco also encountered rogue projects. He didn't simply toss them out -- even if they weren't the best way to solve a problem. He preferred to cultivate trust. "I'd just say, 'I don't want to take it away. Let me figure out how to work with it and make it better.' There have got to be benefits in it for them. Otherwise, it's going to be, 'I don't need you. I'm running fine. Stay out of my business.'"
Months or even a year later, at the start of a new upgrade cycle, Stanco would cash in on his goodwill and persuade business owners to turn the rogue project over to him. Then he could replace the parts that didn't work well or rebuild them to work better with the corporate network. He found patience and humility to be virtues.
Then again, some rogue projects require a CIO to take a hard line. Rich De Brino realized he had to play both good cop and bad cop in a past job. (De Brino is now CIO at Compass Health, a behavioral health nonprofit in Everett, Wash.)@pb
De Brino was managing IT for a software company when an email virus struck. He and his staff worked through the night checking and fixing computers. Then De Brino walked into an office and spotted a server under a desk. "What's this box?" he thought. By the end of the night, his crew had discovered a few more. None had asset tags.
The mystery computers were in the engineering department, so De Brino confronted the manager of engineering, who confessed. De Brino told him that the servers put the corporate network at risk for viruses. The manager replied there was no cause for worry. "Technically they're not on your network; we have our own switch," De Brino recalls the manager saying. Engineering had built a whole rogue network.
The engineers thought the previous CIO had a propensity for shooting down projects, so they learned the art of stealth. Even once De Brino came on as the new CIO, they continued their covert ways by secretly expensing motherboards and hiding servers under their desks. But ironically, the rogue private network was a good idea. It allowed software developers to test code before launching it on the corporate network.
De Brino held out the stick before offering the carrot to make the carrot more appetizing. He remembers telling them, "Your choices are, we're going to take the servers and confiscate them -- they're corporate assets that aren't being managed -- or let us adopt them to our standards. We'll give you guys full access." The engineering manager agreed to the latter plan. As part of his effort to build a rapport with the rogues, De Brino didn't tell his boss: "My boss would have gone ballistic, [saying], 'There's assets we don't know about?'"
Juliano, of course, wasn't so lucky. After his CEO berated him at 3 a.m. for downed email and the marketing department blamed him, Juliano faced an uphill battle to build rapport. So he turned to booze. The marketing and IT teams went out to "quite a few after-work beer bashes," Juliano says. Did it help? "Absolutely." (Now Juliano is CIO and vice president of marketing for Wine Enthusiast Cos., a midmarket company that sells wine accessories, publishes wine magazines and arranges conferences. It is based in Elmsford, N.Y.)@pb
Respect for the Rebels
Perhaps the best way to build bridges and keep users out of trouble is regular communication. At Betts USA, Roell now regularly walks around the 60,000-square-foot facility talking to as many people as he can, from the chief executive on down. He uses hallway encounters to find out what people are up to and what they think he should know about.
Since he's outgoing, Roell says people feel comfortable going to him for their technology needs. Roell prioritizes requests based on how critical they are to "getting tubes out the door," taking on critical projects immediately and telling others to wait two weeks for less-pressing requests. "Then they can't go rogue on me. If I've already looked at the problem and it's in my notes and we put you on the schedule," Roell says, it's harder to justify moving ahead without IT.
Juliano has his own method for letting line managers know where they stand. He publishes a list of IT projects in order of priority on the company intranet, where all executive-level employees can see it. "I want to communicate in a clear, visual way so all business owners and stakeholders are aware of what the key projects are," he says.
Of course, rogue IT happens even in the best-run companies. Successful companies often encourage employees to think outside the box to achieve business goals. Some of the smartest moves in corporate history have involved business units making end-runs around their own bureaucracies.
It was a stealth engineering project, designed by the chairman behind the chief executive's back, that allowed a few midlevel engineers at Compaq Computer Corp. to secretly develop a new line of low-end PCs in just three days in 1991. The group was under orders not to alert anyone -- not even their bosses -- let alone IT. That rogue initiative ushered in half a decade of seemingly unstoppable growth.
To some degree, Juliano respects business units that start rogue IT projects. "If someone's gone through the effort [to work outside IT], they are so passionate that it's the right thing to do that they've decided that they're willing to roll the company's future on the dice," he says. Passion can be a good thing, especially when a CIO prevents it from doing any harm.
Joan Indiana Rigdon was a contributing writer for CIO Decisions. To comment on this story, email firstname.lastname@example.org.