Like the cobbler whose children have no shoes, CIOs toil away at making business processes more agile. But when it comes to IT work processes and metrics, the job isn't getting done.
Or keep your job, for that matter. "In the setting I work in -- retail -- there is a great deal of dissatisfaction with the IT organization's ability to respond quickly to changing business requirements," Kilcourse said. "In today's business cycle, the notion of an 18-month backlog within IT is not acceptable."
Collect solid metrics at the right time, said Kilcourse and other experts, manage the application development lifecycle, and the speed of software development can improve its chances of keeping up with the business.
Postmortem metrics = risk dead on arrival
Application development organizations lag behind their business peers, as well as IT peers, in using metrics to improve their work processes, according to Forrester Research Inc. Metrics collected tend to be the wrong ones. In a study on application development metrics commissioned by application lifecycle management (ALM) provider Borland Software Corp., Forrester found that application development teams rarely collect "in-flight metrics," or metrics gathered while the project is under way.
Instead, application development teams are more apt to use postmortem metrics in judging a project's relative success. Unfortunately, on-time, on-budget and on-scope measures "say more about estimation accuracy than about whether a project took as long as it ought to have," the Forrester authors note. Metrics like "business value delivered" and customer satisfaction are far more valuable, but also far more difficult to collect.
RSR's Kilcourse, whose 30-year IT career includes a stint as CIO of Longs Drug Stores Corp., agreed. "We measure things like did it arrive on time, or on budget. We calculate the number of errors per lines of code. But the fundamental question that keeps getting missed is: Did the effort deliver what the business asked for?"
Marc Brown, vice president of product marketing at Austin, Texas-based Borland, said the Forrester study underscores the reality that app dev is the "weakest link in the chain" and software delivery a "black box." In December, Borland announced that it would introduce a number of new "open ALM" platform products in 2008 that will track the development process from idea to delivery. The products promise to help customers determine which metrics to choose and how to automate the collection of those metrics, thus optimizing development processes through real-time or "in-flight" metrics that provide insight into things that matter to the business, such as risk, cost and value.
"We wanted to demystify it and make it more of a science," Brown said. "The open platform allows us to help organizations make that transition to a managed business process by utilizing and leveraging their choice of technology process and best practice."
Direct competitors in the field include VMware Inc., Microsoft, Hewlett-Packard Co., IBM Rational and Compuware Corp., but not all offer an open platform.
That's an important consideration, said Kilcourse, who serves on the CIO advisory board for Borland Software. Most CIOs at large companies have invested in some kind of application lifecycle management tools. The Borland approach of selling an integration platform for disparate tools, as well as its own tools, makes "good sense," he said.
"It sounds like what Oracle is doing with its AIA [Application Integration Architecture], but applied to the IT function," Kilcourse said, adding that IT deserves the same kind of advantages and opportunities as the business.
Costly and complex
What prevents app dev organizations from benefiting from the metrics their peers use?
It costs a company to collect data, for one, the Forrester study found. And because the metrics that organizations use are often superficial, or just wrong, the benefits are questionable -- and questioned by the people being measured.
According to the Forrester interviews, the No. 1 obstacle cited was the manual effort required. Even development shops that obtained metrics from tools told Forrester the collection was dependent upon someone, usually a project manager, entering the raw data. Manual metrics collection is tedious, error-prone and expensive. Tools that automate collection are also expensive.
Reporting and analyzing app dev metrics is also technically complex, the study found. Nearly half of those surveyed by Forrester said they were unable to trend or aggregate the metrics they collected. Heterogeneous collection tools account for some of the opacity. Tracing the development lifecycle from start to finish often spans multiple development tools and tool vendors, the study showed.
Then, there is the fear factor. "It takes courage to stop working and measure all the potential sources of trouble when you're afraid that the very act of measurement might slow you down," the authors noted.
The reality is that software development remains a difficult process, Kilcourse said, despite advances made in automating the delivery of code. And the difficulty is something many "non-ITers" don't get. He offers the example of a COO who presents the CIO with an idea that could give the company a real competitive advantage: identify the company's top 10 customers and at the point of sale direct them to those offers of most interest to them. Don't bother them with the other stuff.
Sounds great, the COO is excited, he can see it, but then begins the long process of breaking it down into components: How do you define the top 10, what are relevant offers, how do you get those offers to customers when they need them, etc. And oh, by the way, the COO adds, can you get those messages to them on their cell phones in an SMS message?
"As an IT-er, you're rolling your eyes, thinking, 'My God, he just came up with a 36-month, $50 million project,'" Kilcourse said.
Let us know what you think about the story; email: Linda Tucci, Senior News Writer.