Data-driven decision making propels Yammer to edge of Agile and beyond

Data-driven decision making and the scientific method pushed Yammer Inc. to the boundaries of Agile development and beyond.

Adam PisoniAdam Pisoni

As the co-founder of a tech startup, Adam Pisoni knew surviving in the land of digital disruption boiled down to two things: One, figuring out what customers really want (rather than what they say they want); and two, building those products faster than the competition. Both objectives require nimbleness, making Agile development a natural fit. But it didn't take long for Pisoni and his team to see Agile development just wasn't agile enough.

What was holding Pisoni and his team back? The answer started to take shape when one of Pisoni's engineers introduced him to Conway's law, which essentially states organizations create products and services that are reflections of themselves. What needed to change was how the organization operated.

"It dawned on us that in order to iterate on a product quickly, we have to be prepared to iterate on the organization quickly," said Pisoni, chief technology officer and co-founder of the enterprise social software company Yammer Inc.

And so Yammer threw tradition out the window and opted instead for what Pisoni calls post-Agile: a data-driven, collaborative engineering environment that experiments on just about everything, including organizational structure.

In this first installment of a two-part article, Pisoni explains how data and the scientific method play important roles in a post-Agile world, but don't go far enough. The second installment looks at how Yammer rebuilt its organizational structure, inspiring even Microsoft, which acquired Yammer in 2012, to experiment with some of the same practices.

It dawned on us that in order to iterate on a product quickly, we have to be prepared to iterate on the organization quickly.

Adam Pisoni, CTO, Yammer

Driving with data

Since its founding in 2008, Yammer has prided itself in understanding that the customer always comes first. But tapping into what customers really want from a product is easier said than done, Pisoni explained. In studies independently conducted by Google, Intuit and Microsoft, research showed product managers have to do a lot of guesswork about customer wants, making correct decisions only about a third of the time. According to Pisoni, even customers themselves struggle to articulate what they want, asking for enhancements to products that they don't much like. 

"This is the difficult thing about software development: A lot of the right choices are counterintuitive," he said.

Hence the need for data-driven decision making. Data doesn't dictate what Yammer builds, Pisoni said, but it does pull back the curtain on how customers are using the product. So when customers clamored for new functionality within the search feature, for example, Yammer engineers turned to the data before building anything. That's how they discovered a troubling pattern: A lot of people used the search feature once, then never again.

Rather than build on top of a search feature customers didn't like in the first place, the data convinced engineers to think about why search wasn't driving interest and to work out the kinks. To that end, the company used A/B testing, a method enabled by cloud computing that compares the behavior of a randomly selected group of users given access to new features against those who weren't. Each modification to the product is tested.

"It's the change against the control; it's the scientific method," Pisoni said. "We know the only difference this set of users is experiencing is this change we've made, and so if they post more, if they return more, if they engage more, we can attribute the [feature's] change to the behavioral change."

Companies that use longer development cycles often don't get to cull user data from one edition of the product to influence the next. The same holds true for companies operating on shorter, four-month release cycles, according to Pisoni.

"You can't test fast enough to have the data be meaningful," he said. That's why breaking down bigger projects into smaller ones is key. Smaller projects mean more flexibility, enabling engineers to move from hypothesis to testing to iteration and back to hypothesis quickly, Pisoni said.

Optimization and the people bottleneck

Adopting short, quick, iterative development cycles, however, didn't automatically result in building products the customer wants at the speed Yammer was hoping for.  

More on where data and science meet

A crash course in data science

HMS CIOs: Big data scientists, not big data practitioners

Learn about data science and mining available info for hidden value

"Even if you could release every minute of every day, the thing that's going to slow you down most is a term we call agreement costs," Pisoni said.  

For a company to do just about anything -- but especially to do something that cuts across the organization -- employees from various departments have to brainstorm, plan and generally agree to work on that something together before they start working on anything. Absent that, according to Pisoni, agility and data-driven development is doomed.

"You’re forced to plan too much because nobody wants to go back and re-agree every week," he said. "And so you come up with big plans you all agree on, and go off in your separate directions."

Once Pisoni and his team realized the kink in the organizational system, it became evident that Agile development wasn't enough for Yammer. "Instead of optimizing for efficiency, which is what we believe Agile software development is optimized for, we needed to optimize for throughput," he said.

In part two, Pisoni describes how Yammer razed a traditional organization structure in favor of a post-Agile world.

Let us know what you think of the story; email Nicole Laskowski, senior news writer, or follow her on Twitter at @TT_Nicole.

Dig Deeper on Enterprise collaboration strategy