Get started Bring yourself up to speed with our introductory content.

Agile project management, from agile to waterfall

A seasoned agile project manager shares an insider's view of agile definitions, lingo and frameworks, as well as the mind-set needed to kick-start agile project management.

Joseph Flahiff has more than 15 years' experience in agile and traditional project management. For the past five years he has managed exclusively agile projects in enterprise organizations. Currently he works for a multistate health insurance company, providing agile project management and training for a three-year $20 million project that coordinates the work of more than 100 team members. He is certified as a Project Management Professional, Six Sigma Green Belt and Scrum Professional, and lectures internationally on agile and waterfall integration. In May 2011 he will speak on that topic at the PMI Global Congress in Dublin, Ireland. Video, posts and free resources about current trends in enterprise agile project management may be found at his Whitewater Projects blog.

You probably have heard of agile project management from your IT department. You may have heard things like, "It is a new approach to doing IT projects that is better, faster and cheaper, and will guarantee you results." Or you may have heard things like "Agile projects don't need documentation," or "Agile projects don't have to follow a plan."

Are these claims true? What is this agile thing? When is it right or not right to use?

Agile project management is individuals working together to create working software in close collaboration with their customers, and responding quickly and easily to those customers' changing needs.

Traditional projects follow a pattern: defining requirements, documenting designs, building and testing what was designed, and releasing it all at once. This is often, in agile circles at least, referred to as "waterfall" project management.(See "Agile project management frameworks"). Most of the time when people say they are using agile, they mean that they are biting off very small pieces of the project scope and delivering them very quickly (every two to four weeks) in such a way that these small pieces can be put into production and feedback can be gained for the next round. This is the most strikingly visible difference between agile and traditional projects: many small releases versus one release.

I was working on a health care insurance reporting project. We were planning the next two-week chunk of work, aka iteration (one of the terms explained in "Agile project management terminology", below), when my IM popped up:

SUSAN: Joseph.


SUSAN: The sales team won't promote the reporting package until we have PDF printing.

JOSEPH: We didn't plan on doing that feature until about three months from now.

SUSAN: I know, but we need it now.

JOSEPH: No problem, it is a big piece of work. We will need to stop work on the rest of our features to tackle it.

SUSAN: I understand.

JOSEPH: OK, we are on it.

And with that we jumped on the PDF printing. The project sponsor knew the impact of what she was asking, and the team was fine with the change. It was not a problem or a change in scope, because the work we had done in the previous two weeks was completely self-contained. It was done and ready to be shipped, so we could slip the PDF printing feature up in front of the other work easily, and not have any lost work. It is this kind of adaptability that makes agile a valuable approach, especially for customer-facing software development.

Where did agile get its start?

Agile is a grassroots movement started by software developers and forward thinkers after many years of project failure. At the time, IT projects, software projects in particular, had a horrific reputation of always being late and over budget. In reaction, numerous lightweight approaches arose in the early- to mid-1990s: test-driven design, feature-driven design, Dynamic Systems Development Method, Crystal Clear, Scrum, Extreme Programming (see "Agile project management terminology") and others.

In 2001, a group of 17 representatives from these disciplines gathered and discussed what they held in common. In the end, they wrote the Manifesto for Agile Project Development (what good rebellion doesn't have a manifesto?) as "an alternative to documentation-driven, heavyweight software development processes," according to Jim Highsmith, one of the manifesto authors, who also wrote the project's history.

The best description of the real heart of agile I have ever seen is buried in that history. According to Highsmith, at the end of the summit, Bob Martin, another manifesto author, talked about the process of creating it and about how he "felt privileged to work with a group of people who held a set of compatible values … based on trust and respect for each other, and promoting organizational models based on people, collaboration and building the types of organizational communities in which we would want to work."

This is what lies deep at the heart of agile: a set of values that put people first and foremost. Above all other things, in work of any kind, people are the most important.

The agile manifesto identifies four values and 12 principles that are common to these software development approaches. The manifesto states: We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools.

• Working software over comprehensive documentation.

• Customer collaboration over contract negotiation.

• Responding to change over following a plan.

Let's unpack these four statements to delve further into their meaning.

1.  Individuals and interactions over processes and tools

Agile is a mind-set that believes that software development is done by people -- period. People, not processes or tools, make good software. You can use all the Unified Modeling Language you want or set up status reports all day, but they won't make good software -- that is done only by your people. Those people are the most important ingredient in making quality software: individuals interacting as a team. Take away the unique individual as contributor and contributors' role in a team, and you will cripple the software development process. This value is not typically one that most people have a problem with in the manifesto. Many businesses claim, "People are our most important asset", even if they don't really act like it.

2. Working software over comprehensive documentation

In an agile environment, working software is the one and only measure of progress. Until a piece of code is working as you expected it to work, you have no empirical evidence that you are making progress. You have documents and plans, and you hope they may result in software that does what you want it to do; but you do not have empirical progress and you don't have proof.

3. Customer collaboration over contract negotiation

Over many years of software development during the dot-com boom, it became clear to the authors of the agile manifesto that the copious time spent developing contract documentation would be better spent developing working software, because many of the assumptions made in a contract are volatile, and by the time the software is delivered the assumptions have changed significantly. If they left the contract vague, and committed to partnering with the customer to elaborate their changing requirements along the way, they could get moving and deliver value to market much more quickly.

You may think this doesn't apply to you because you do only internal projects, not contracts. However, a project charter is a contract for the project. Test this yourself in your organization. Do you have a "change control" process for managing project changes? Do you need multiple signatures and a governance body's approval to make changes to a project? If you do, then you are using a project charter like a contract, and this value will help you.

4. Responding to change over following a plan

Responding to change does not mean throwing planning out, but it shifts the focus from a locked-down plan requiring a heavyweight, give-up-your-first-born change control process to a more fluid planning approach, one that not only accommodates change but also welcomes it, knowing it will come. I find it telling that this value is stated last even though it often is stated first by both software developers promoting agile and traditional managers trying to discredit it. Maybe that's the case because it is the most striking difference between promoters and detractors, or maybe that's the case because developers have been abused by too many bad managers who hold to their plan and make change onerous, even when obvious change is required.

So, while there is value in processes, documentation and planning, people are the heart of good software development. If you keep that in mind, you will be on your way to being agile.

Agile project management frameworks

Unique attributes
Project management Sequential process of development: requirements specification, design, development, testing, deployment Highly applicable to construction. Use problematic in projects with dynamic requirements. Requires BUFD (big up-front design). Often relied on for outsourced statement of work projects. Uses strict change control process to amend the original scope if changes occur.
Project management and development (minimal)
A framework for managing projects with minimal documentation and high interactivity. Easily the most popular agile framework.
Daily scrum meeting for team member coordination that's simple to implement and lasts no more than 15 minutes. Each member answers three questions: What did you get done yesterday? What do you commit to today? What are your impediments?
Extreme Programming (XP)
Programming method that fits well within many agile approaches.
Paired programming: Two developers sit together writing one piece of code. Has proved useful in multiple studies, particularly in novice-novice pairs.
Crystal Clear
Project management and programming
A family of frameworks designed to suit varying levels of rigor, based on team size and project risk.
Defines multiple levels of management rigor, based on team size and project risk.
Adaptive software development
Software should be constantly refined and improved as a product is developed, in a cycle of speculate, collaborate, learn.
An overarching philosophy that is found in almost all agile frameworks.
Feature-driven design
Project structure
Systems modeling based on features: develop overall model; build a feature list; plan by feature; design by feature; build by feature.
Break a project down by features and build them up incrementally.
Test-driven design
Development approach; project structure
Write the test first, see it fail; write code to meet only the test case. Can be used to structure projects around higher-level tests as well, or may be used at a low level for unit testing only.
Tests are written first as failing tests. Then minimal code is written to make the tests pass.
Dynamic Systems Development Method (DSDM)
Project management
Iterative waterfall mixture. Popular in Europe. Three phases: pre-project, project, post-project. Four steps: study (feasibility and business); functional model; design and build; implementation. Within the steps, the delivery method the team uses may vary.
Strong focus on holding time and budget constant and allowing requirements to be the variable in making project delivery tradeoffs.

Agile project management terminology

Iteration: A fixed, regular and repeating period of time in which a piece of work is created and delivered by a project team.

Agile: The ability to be adaptable.

Release: Working software that will provide value to the customer.

Product backlog: A list of items the product owner would like to be created.

Project backlog: A list of items the product owner or project leadership has defined as the scope of the project.

Iteration backlog: A list of tasks the team has committed to execute in order to create the increment of working software for that sprint (see definition below).

Story: A shorthand way of writing requirements for an agile project. Writing requirements in the form of a story forces a focus on who the customer is and what the customer wants from the feature.

Story card: A story card is a promise of a conversation. It is not intended to describe a requirement in any depth, but is merely a placeholder to remind the customer representative and the developers to discuss this item and add more detail, just in time to create it. It's often actually written on a 3x5 card; if what you have to say will not fit on a 3x5 card, you are adding too much detail.

Epic or epic story: A parent story card, which describes a large set of features that are then put separately onto story cards.

Points: A way of estimating work based on relative sizing, which is the key. Various models are used, but a popular one is to consider the size, complexity and uncertainty of a piece of work when estimating the "points value" for that work. Points do not equal hours. Hours are a function of points and velocity (see definition below). Two teams may estimate the same piece of work with radically different results; the key to points is that, for the team doing the estimate, they will be relative to the way the team estimates everything.

For instance, one team may estimate a piece of work at 8 points and be able to do that work in two weeks. That team's velocity is 8. Another team may estimate the same piece of work at 21 points and be able to do that work in two weeks as well. That team's velocity is 21. Thus, the same work estimated by two teams will have different values (8 vs. 21), but the team's velocity is relative to their points estimation.

Sprint: An iteration in Scrum projects.

Daily scrum: A daily meeting (lasting 15 minutes or less) in which the team answers three questions and only three: What did I complete yesterday? What do I commit to completing today? What are my impediments? The focus is on commitment and interdependencies.

Time box: A fixed period of time. Agile practitioners use time boxes to focus and limit the amount of detail that is covered. Time-boxing a discussion keeps the team focused and prevents off-topic or unnecessarily detailed discussions.

Velocity: The amount of work a specific team can complete in a given time box. Velocity is specific to a given team. If you subtract or add anyone from the team, the velocity must be recalculated. It is not always a given that adding more people to a project will increase velocity. As a matter of fact, it has been shown (Brooks' law) that adding more people late in a project will actually slow a team down.

Next Steps

Compare development methodologies

How to ensure an Agile team isn't bogging down in to Groupthink

Dig Deeper on Enterprise application development, DevOps and software agility