Managing the application development lifecycle requires solid metrics

By Linda Tucci, Senior News Writer 15 Jan 2008 | SearchCIO.com

IT news and analysis for CIOs Digg This!     StumbleUpon     Del.icio.us

More on app. development BPM takes root, but still an adjustment for many ALM and PPM: A marriage of convenience or a match made in heaven?

"CIOs have been pushing the concept of business process management and business process optimization to the business. But they themselves don't have the advantage of a systemized, measured and optimizable process for the single most expensive business activity they undertake -- systems development," said Brian Kilcourse, managing partner at research firm Retail Systems Research LLC (RSR) and self-described recovering CIO. "If you can't put metrics around it, how can you possibly improve?"

Or keep your job, for that matter. "In the setting I work in -- retail -- there is a great deal of dissatisfaction with the IT organization's ability to respond quickly to changing business requirements," Kilcourse said. "In today's business cycle, the notion of an 18-month backlog within IT is not acceptable."

Collect solid metrics at the right time, said Kilcourse and other experts, manage the application development lifecycle, and the speed of software development can improve its chances of keeping up with the business.

Postmortem metrics = risk dead on arrival

Application development organizations lag behind their business peers, as well as IT peers, in using metrics to improve their work processes, according to Forrester Research Inc. Metrics collected tend to be the wrong ones. In a study on application development metrics commissioned by application lifecycle management (ALM) provider Borland Software Corp., Forrester found that application development teams rarely collect "in-flight metrics," or metrics gathered while the project is under way.

A true story by a recovering CIO Brian Kilcourse, on the futility of pointing out that turning language into digital assets is not like building a store: "In one of my moments of extreme clarity on the job, I remember I was trying to explain to my CEO some of the compromises we had to make on a system in order to meet a hard deadline. "My boss looked at me, out of frustration, and said, 'I just don't understand why building a system is any different than building a store. We have a plan for building stores and we know exactly from the moment we put the shovel in the ground when that store is going to open up, and you can't do that. Is that what you're telling me?' "I said, 'That's what I'm telling you.' He said, "And you know why that is not acceptable to me, don't you?' 'Yeah, I do know why it's unacceptable to you,' I said. 'But I have to tell you something really important. It's not because our people aren't good, and aren't smart and don't have the best tools available to them, it's that it is really really hard.' And he said, 'Get out of my office!'"

"Lacking real in-flight project data, app dev teams rarely have objective data about the true status of their projects or about the root causes of their problems," the Cambridge, Mass.-based consulting firm concluded. The July 2007 study was based on 20 in-depth interviews with IT professionals in charge of application development and project management.

Instead, application development teams are more apt to use postmortem metrics in judging a project's relative success. Unfortunately, on-time, on-budget and on-scope measures "say more about estimation accuracy than about whether a project took as long as it ought to have," the Forrester authors note. Metrics like "business value delivered" and customer satisfaction are far more valuable, but also far more difficult to collect.

RSR's Kilcourse, whose 30-year IT career includes a stint as CIO of Longs Drug Stores Corp., agreed. "We measure things like did it arrive on time, or on budget. We calculate the number of errors per lines of code. But the fundamental question that keeps getting missed is: Did the effort deliver what the business asked for?"

Marc Brown, vice president of product marketing at Austin, Texas-based Borland, said the Forrester study underscores the reality that app dev is the "weakest link in the chain" and software delivery a "black box." In December, Borland announced that it would introduce a number of new "open ALM" platform products in 2008 that will track the development process from idea to delivery. The products promise to help customers determine which metrics to choose and how to automate the collection of those metrics, thus optimizing development processes through real-time or "in-flight" metrics that provide insight into things that matter to the business, such as risk, cost and value.

"We wanted to demystify it and make it more of a science," Brown said. "The open platform allows us to help organizations make that transition to a managed business process by utilizing and leveraging their choice of technology process and best practice."

Direct competitors in the field include VMware Inc., Microsoft, Hewlett-Packard Co., IBM Rational and Compuware Corp., but not all offer an open platform.

That's an important consideration, said Kilcourse, who serves on the CIO advisory board for Borland Software. Most CIOs at large companies have invested in some kind of application lifecycle management tools. The Borland approach of selling an integration platform for disparate tools, as well as its own tools, makes "good sense," he said.

"It sounds like what Oracle is doing with its AIA [Application Integration Architecture], but applied to the IT function," Kilcourse said, adding that IT deserves the same kind of advantages and opportunities as the business.

Costly and complex

What prevents app dev organizations from benefiting from the metrics their peers use?

It costs a company to collect data, for one, the Forrester study found. And because the metrics that organizations use are often superficial, or just wrong, the benefits are questionable -- and questioned by the people being measured.

According to the Forrester interviews, the No. 1 obstacle cited was the manual effort required. Even development shops that obtained metrics from tools told Forrester the collection was dependent upon someone, usually a project manager, entering the raw data. Manual metrics collection is tedious, error-prone and expensive. Tools that automate collection are also expensive.

Reporting and analyzing app dev metrics is also technically complex, the study found. Nearly half of those surveyed by Forrester said they were unable to trend or aggregate the metrics they collected. Heterogeneous collection tools account for some of the opacity. Tracing the development lifecycle from start to finish often spans multiple development tools and tool vendors, the study showed.

Then, there is the fear factor. "It takes courage to stop working and measure all the potential sources of trouble when you're afraid that the very act of measurement might slow you down," the authors noted.

The reality is that software development remains a difficult process, Kilcourse said, despite advances made in automating the delivery of code. And the difficulty is something many "non-ITers" don't get. He offers the example of a COO who presents the CIO with an idea that could give the company a real competitive advantage: identify the company's top 10 customers and at the point of sale direct them to those offers of most interest to them. Don't bother them with the other stuff.

Sounds great, the COO is excited, he can see it, but then begins the long process of breaking it down into components: How do you define the top 10, what are relevant offers, how do you get those offers to customers when they need them, etc. And oh, by the way, the COO adds, can you get those messages to them on their cell phones in an SMS message?

"As an IT-er, you're rolling your eyes, thinking, 'My God, he just came up with a 36-month, $50 million project,'" Kilcourse said.

Tags: Enterprise application development,  Business process management,  Tools and Technologies: IT metrics, SaaS at heart of revised BPM,  VIEW ALL TAGS

Digg This!     StumbleUpon     Del.icio.us

<< PREVIOUS | NEXT >>: Autonomic policy management cures complex business...

VIEW ALL IN THIS CATEGORY

RELATED CONTENT Enterprise application development SOA implementation evolves from open source to Oracle SOA suite Six ways to fail with your SOA implementation Agile development methodology not easy but worth the effort, users say FAQ: Lean thinking for IT Integrated business intelligence strategy spans app, BI developers CIO returns from two years as operations SVP ready to drive IT value Using SaaS CRM for application integration gives users single sign-on Implementing SharePoint enterprise-wide requires governance, partners On-demand data integration ties HD Supply to customers IT offshore outsourcing deals being re-evaluated in weak economy Business process management The Real Business of IT: Download a free chapter BPM tool selection: Strategies for success FAQ: IT and organizational change management ITSM and ITIL best practices for process improvement Looking for low-cost business processes? Check out GE WorkOut and FTD FAQ: Lean thinking for IT Proven IT ROI strategies in an economic downturn Get the most out of your lean BPM solution Tips to keep your IT job and move along CIO career path in a recession Business process management quiz for enterprise CIOs Tools and Technologies: IT metrics, SaaS at heart of revised BPM E-Book: The BPM Payoff BPM Platform as a Service turns IT shops into SaaS providers Business process management newest SaaS deliverable Autonomic policy management cures complex business systems Project portfolio management takes flight at Sabre HP takes another shot at network management, buys Opsware

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary