News Stay informed about the latest enterprise technology news and product updates.

Agile best practices can combine waterfall and Scrum

Agile projects can mingle Scrum and waterfall approaches, but proceed with caution. Program managers share agile best practices and pitfalls.

Agile best practices were created to rebel against traditional waterfall project approaches that follow a sequential...

and often inflexible chain of command and release schedule.

More often, however, program managers are finding value in mixing up projects to combine waterfall and Scrum.

Joseph Flahiff, program manager with a large health insurance carrier, is at the helm of what he estimates will be a $20 million HIPAA 5010 implementation project. He's in charge of about 100 people taking part in this electronic data interchange (EDI) project, which encompasses four Scrum teams and other teams handling Web platform and hardware platform development.

Despite the predictability of the overall project -- 5010 includes federally mandated requirements that are not going to change and therefore could be viewed as a fit for a predefined waterfall approach -- a big chunk of the EDI project is being performed in agile.

"The value we're finding in agile is incremental delivery of features so that we see early success and can build, and continue to build on that," Flahiff said. He compares agile projects to building a pyramid: The tip of the pyramid was built first because it was unknown when the pharaoh would die, and the rest of the pyramid was built in layers underneath the tip.

For those working on the platform and services side of the project, using agile would be like trying to fit a square peg in a round hole, Flahiff said. "Sure, you can do it if you pound hard enough … but putting in 20 servers makes sense for a more traditional [waterfall] approach."

Buying and implementing servers lends itself to waterfall because the process is predictable: The box is ordered, it's built and shipped, a space in the data center will be prepped for it, and an OS will be installed and tested.

Testing as you go

A benefit of Scrum vs. waterfall is the ability to solve problems as they happen. Waterfall projects follow a given set of requirements and often are tested at the end, a somewhat "Big Bang" approach to software development. Agile best practices allow for software bugs to be fixed within a three-week development phase, so the IT team doesn't end up with a mountain of code fixes at the end -- or a fatal flaw in the system.

"When you get down to testing in a more traditional, waterfall approach, you don't know how many defects you're going to have, or how bad they will be," Flahiff said.

Flahiff also is able to forecast better. With each iteration of a Scrum project, he can see if a target will be hit or if a phase of a project is not coming together on time. "You can take an issue to the steering committee as it happens so there isn't a firefight at the end of the project," he said.

I've seen [agile project] work thrown into a pile and they work on it. They don't worry about all the integration and interdependencies, and they make changes that cascade.

Alex Keenan, legacy systems analyst

Figuring out which aspect of the project is better suited for an agile approach, such as Scrum, or for a sequential, sometimes regimented approach, like waterfall, is half the battle.

A project can get derailed, for example, if one approach, say, waterfall, is used up front for the project-planning stage (defining the scope of the project, objectives, requirements and implementation schedule) but there's a shift to agile best practices for the project development stage. That's mainly because waterfall does not account for change or failures, while Scrum does.

"Waterfall assumes that everything will hold constant for the duration of the project," said Ross Pettit, client principal at Chicago-based ThoughtWorks Inc. "What happens when a defined requirement isn't needed anymore, or you find that the technology doesn't scale or perform well, or key people on the team quit?"

Friction will occur when a fixed, waterfall plan is chunked up and delivered in small phases, which is a sequential approach, not necessarily an agile one. "With an agile plan, you are revising the requirements and priorities with every release, and if you have a fixed scope -- even if it's chunked up -- you will not get the benefits of that learning process," Pettit said.

When good Scrum goes bad

For Alex Keenan, a legacy systems analyst who works for a large grocery chain, both waterfall and Scrum create their own set of challenges. He is working on a project to develop new Web interfaces for 30-year-old systems and business intelligence applications. A waterfall approach is being followed by team members building an SAP/BusinessObjects universe, for example; while a two-person Scrum team is developing the Web interface for the back-end reporting system.

Like many people on the team, Keenan plays a small role, but in the end all the pieces have to come together.

"I've seen [agile project] work thrown into a pile and they work on it. … [T]hey don't worry about all the integration and interdependencies, and they make changes that cascade," Keenan said. "Good Scrum looks for cascading issues or integration issues up front."

The ability to work together as a team, often and in unison, to deliver working software in small increments is at the heart of agile best practices as laid out by the Agile Manifesto. But that is in an ideal world.

Often, team members have other full-time jobs, as is the case with Keenan. "I've seen seven Scrum groups going at the same time, and it's like herding cats if you don't get the architecture right first and the ground rules in place … and yet [the program managers] don't do that," he said.

Keenan has also been involved in agile projects in which his manager asked him to deliver all his questions at the end of the day, or during weekly or biweekly meetings -- which is more of a waterfall approach. "Waterfall doesn't really allow you to increase competency anywhere near as fast as Scrum," he said.

Herding the iterative pieces together from several Scrum teams is a common problem that Flahiff is encountering also. "We have one team working on one component, and another team working on another component that may not be related, and that's a problem because they all need to be delivering the same functionality," he said.

Up-front architecture design and scope, which may be developed using waterfall, are key to determining the requirements each team member will have to meet -- and in turn, the agile best practices needed to reach the project's end goals.

"The right project scope and vision allow you to continue to develop features that work together with each iteration and you keep building on top of that, and on top of that; so what's being built isn't coming together at the end, it's together all along," Flahiff said.

Let us know what you think about the story; email Christina Torode, News Director.

Dig Deeper on Enterprise application development, DevOps and software agility