Lean Analytics: Use Data to Build a Better Startup Faster, by co-authors Alistair Croll and Ben Yoskovitz, builds on the Lean Startup methodology, a trendy approach to product and market development with the rallying cry of build, measure, learn. Croll and Yoskovitz zero in on the middle phase of this iterative cycle: the key metrics that can help businesses shorten the time between build and learn.
In the first installment of SearchCIO's New Books interview, Yoskovitz talked about the Lean part of Lean analytics and how it applies to both CIOs and established companies. Here he talks about why it's important for companies -- big and small -- to find the "One Metric That Matters" or OMTM for every phase of the business.
Lean Analytics provides key metrics for different industries. How did you go about compiling that information and why was it important to include?
Ben Yoskovitz: All of the numbers we've shared have to come with a big asterisk because it's difficult to come up with lines-in-the-sand benchmarks that everybody should automatically follow. But we did a fair amount of research, talked to hundreds of people about different numbers they'd seen or experienced. We wanted to provide numbers because we wanted to provide some amount of specifics without making it overly prescriptive or saying, for example, "If you are building a consumer mobile application, you must have 10% daily active [users] in order to succeed." That's not necessarily true; it's just a good benchmark to measure yourself against. I'll always say if you're at 2% daily active, you're orders of magnitude away from where you probably need to be. Do you have to be at 10% just because a few people say that's a good number? Not necessarily.
But it's not just key metrics for the sake of key metrics. You write about finding OMTM -- the "One Metric That Matters." Why is that so important for startups?
Yoskovitz: The "One Metric That Matters" is an important thesis of the book, and it basically says that at any given point in time, there's one number that an entire company, at least in a startup, should be focused on. That number can change and will change and it may change quickly. You may only be focused on one thing for a week and you may run multiple, quick experiments, move the needle and then move to the next thing.
Does the "One Metric" idea translate from the startup to the enterprise?
Ultimately what you're looking for is a number that changes your behavior.
Yoskovitz: I don't think it's any different for [enterprise] CIOs. It may be on a project-by-project basis where they apply that kind of rigor to something like the "One Metric that Matters." But if their work is supporting an internal IT operation, there may be a metric that eventually bubbles up from that. … There are examples of that like HubSpot, which has what they call the "customer happiness index." … It's the one thing they look at when, for example, they're doing sales to ask, 'Are we selling to customers we believe are going to be happy and are those customers happy at the end?' I could see CIOs having that "One Metric That Matters" around the usage of IT systems or the efficiency their projects are driving within an internal organization.
How can CIOs identify the "One Metric That Matters" and avoid, as you wrote about in Lean Analytics, "vanity metrics?"
Yoskovitz: When we're looking at what makes a good metric overall, whether it's our "one metric" or not, there are a few factors [to consider]: It has to be understandable; there's always that risk of overcomplicating things and looking for things that don't necessarily make sense, so I always look for the simplest numbers. If you tell somebody, 'Here's the problem I'm trying to solve, and here's the number I'm tracking,' they get it.
More on metrics
Why CFOs love cloud metrics -- and when CIOs need to push back
CIO: Meeting business needs -- not tech metrics -- is measure of success
Discovering the right metrics for scalability testing
The number itself, ideally, is a ratio or a rate, and that's because ratios and rates are very easy to compare against each other. So, for example, a vanity metric that we often see with startups is number of users, and it might be the same internally for CIOs looking at an internal deployment of the system. But [number of] users is a bit of a vanity metric, because it doesn't tell us how often they're using. What we're really looking for is percent of active users. "Active" could be defined in any number of ways -- once a month, twice a month, once a week, whatever it is. But now I can, irrespective of how many users I bring onto this system, compare the percent of active users month over month and see if usage is going up or down as a measure of engagement and, by extension, as a measure of retention.
Ultimately what you're looking for is a number that changes your behavior. This is the hardest thing for people to wrap their heads around. If we think about number of users, and we see number of users is growing, what does that mean? What do I do with it, how do I change what I'm doing with this project or task? It's very difficult to know.
What you're really looking for is a number that says if this number goes up, I know what I should be doing; if this number goes down, I know -- maybe not what I should be doing -- but I know something has to change. You're looking for numbers that are going to encourage or drive decision-making and behavior change.