Metrics are merely a reflection of the product strategy that you have in place
Data is powerful because it is concrete. For many entrepreneurs, particularly with technical backgrounds, empirical data can trump everything else – best practices, guys with fancy educations and job titles – and for good reason. It’s really the skeptic’s best weapon, and it’s been an important tool in helping startups solve problems in new and innovative ways.
It’s easy to go too far – and that’s the distinction made between “data-informed” versus “data-driven,” which I originally heard at a Facebook talk in 2010 (included underneath the post). Ultimately, metrics are merely a reflection of the product strategy that you already have in place and are limited because they’re based on what you’ve already built, which is based on your current audience and how your current product behaves. Being data-informed means that you acknowledge the fact that you only have a small subset of the information that you need to build a successful product. After all, your product could target other audiences, or have a completely different set of features. Data is generated based on a snapshot based on what you’ve already built, and generally you can change a few variables at a time, but it’s limited.
This means you often know how to iterate towards the local maximum, but you don’t have enough data to understand how to get to the best outcome in the biggest market.
This is a messy problem, don’t let data falsely simplify it
So the difference between data-informed versus data-driven, in my mind, is that you weigh the data as one piece of a messy problem you’re solving with thousands of constantly changing variables. While data is concrete, it is often systematically biased. It’s also not the right tool, because not everything is an optimization problem. And delegating your decision-making to only what you can measure right now often de-prioritizes more important macro aspects of the problem.
Let’s examine a couple ways in which a data-driven approach can lead to weak decision-making.
Data is often systematically biased in ways that are too expensive to fix
The first problem with being data-driven is that the data you can collect is often systematically biased in unfixable ways.
It’s easy to collect data when the following conditions are met:
- You have a lot of traffic/users to collect the data
- You can collect the data quickly
- There are clear metrics for what’s good versus bad
- You can collect data with the product you have (not the one you wish you had)
- It doesn’t cost anything
This type of data is good for stuff like, say, signup %s on homepages. They are often the most trafficked parts of the site, and there’s a clear metric, so you can run an experiment in a few days and get your data back quickly.
In contrast, if you are looking to measure long-retention rates, that’s much more difficult. Or long-term perceptions of your user experience, or trying to measure the impact of an important but niche feature (like account deletion). These are all super difficult because they take a long time, or are expensive, or are impossible datapoints to collect – people don’t want to wait around for a month to see what their +1 month retention looks like.
And yet, oftentimes these metrics are exactly the most important ones to solve.
Worse yet, consider the cases where you take a “data-driven” mindset and try to trade off the metrics between concrete datapoints like signup %s versus long-term retention rates. It’s difficult for retention to ever win out, unless you take a more macro and enlightened perspective on the role of data. Short- vs long-term tradeoffs require deep thinking, not shallow data!
Not everything is an optimization problem
At a more macro level, it’s also important to note that the most important strategic issues are not optimization problems. Let’s start at the beginning, when you’re picking out your product. You could, for example, build a great business targeting consumers or enterprises or SMBs. Similarly, you can build businesses that are web-first (Pinterest!) or mobile-first (Instagram!) and both be successful. These are things where it might be nice to have a feel for some of the general parameters, like market size or mobile growth, but ultimately they are such large markets that it’s important to make the decision where you feel good about it. In these cases, you’re forced to be data-informed but it’s hard to be data-driven.
These types are strategy questions are especially important when the industry is undergoing a disruptive innovation, as discussed in Innovator’s Dilemma. In the book, Clayton Christensen discusses the pattern of companies who are successful and build a big revenue base in one area. They find that it’s almost always easier to increase their core business by 10% than it is to create a new business to do the same, but this thinking eventually leads to their demise. This happened in the tech industry from mainframes vs PCs, hardware vs software, desktop vs web, and web vs mobile now. The incumbents are doing what they think is right- listening to their current customer base, improving revenues from a % basis, and in general trying to do the most data-driven thing. But without a vision for how the industry will evolve and improve, the big guys are eventually disrupted.
Leverage data in the right way
It’s important to leverage data the same way, whether it’s a strategic or tactical issue: Have a vision for what you are trying to do. Use data to validate and help you navigate that vision, and map it down into small enough pieces where you can begin to execute in a data-informed way. Don’t let shallow analysis of data that happens to be cheap/easy/fast to collect nudge you off-course in your entrepreneurial pursuits.
Facebook on data-informed versus data-driven
I leave you with the Facebook video that inspired this post in the first place – presented by Adam Mosseri. He uses the example of multiple photo uploads, and how they use metrics to optimize the workflow. Watch the video embed below or go to YouTube.
