CIOs are, by now, familiar with the phrase “fail fast.” It’s the idea that experimentation and getting new products to market quickly can expose flaws and weaknesses (and sometimes all around duds) without all the heavy lifting — and heavy investing. Rather than having to anticipate and fix every possible flaw –a potentially costly process — you let the customer of the product or service tell you what’s wrong.
While failing fast sounds great in theory, how can CIOs build an IT culture that celebrates experimentation when they’re often tasked with figuring out how to spend less and be more efficient with what they have? Is it even worth it? So I’ve been asking experts like Mary “Missy” Cummings, director at the Humans and Autonomy Lab at Duke University, to provide examples of how experimentation works for them and what they get out of it.
Cummings, who was one of the Navy’s first female fighter pilots, is a researcher and scientist by training, and so the test-and-learn cycle is practically second nature for her. Still, as someone well versed in experimental testing, the results (and the lessons learned based on those results) sometimes surprise even her: expert assumptions are often proved wrong. I spoke with here at the recent MIT Sloan CIO Symposium. Here’s what she told me about the value of experimentation.
Mary “Missy” Cummings: There was a case once where we had single operator control of multiple air and ground unmanned vehicles [also known as drones]. We were watching [the operators] interact with the system and we thought it was a high workload study because we were giving people more and more tasks to do. But it turns out that people are far more capable than we gave them credit for in terms of being able to handle high workloads.
At the same time, we were testing low workloads. We thought that if we slowed things down, people would take a break and they would slow their pace of activity. What we found was that, because of this inherent need for brain stimulation, the lower workload ended up driving people to frenetic activity, and they had substantially worse performance. These were a younger generation of people. They felt they had to get in there and do more work and, thus, caused a lot more problems in the system.
These kinds of results are really important for people to realize that humans have an incredible predictability in unpredictability. It’s important to get that testing in early so that you understand what issues you’re going run up against with your new technology.