5 steps towards building a metrics-driven business at andrewchen

5 steps towards building a metrics-driven business

Don't ask me about viral marketing, ask me about metrics
Given my history blogging about viral marketing, I'm occasionally approached by folks who ask me, "For product X, how would you promote it and make it viral?" I think there's an expectation that there's a playbook which you can directly apply to every situation.

Unfortunately, there's no real answer to this – ultimately, I think any advancements that can be made to your business function based on the fact you make very gradual improvements based on creating goals, measuring subcomponents, making hypotheses, and testing them. There's no better way to do this than to just do it.

But to get the metrics novice started, here's how I'd approach assessing each particular business:

Step One: Create clear, measurable goals

First off, it's important to identify the main goals of your business, based on current strategy. What are you focused on now? Is it total users acquired, is it number of photos uploaded, is it revenue generated? Whatever it is, you want to focus on something that's not too soft ("increase brand recognition!"), but also not too tactical ("increase pageviews per session!"). I usually prefer something that's a qualified overall metric. I don't care about total registered users, for example, but I do care about total registered users that come back at least 2 times.

Step Two: Make an uber-model that breaks down key variables

Now that you have an overall goal in mind, you want to focus on breaking that down into key variables that you have control over. It might take several layers of modeling before you can get to those controllable variables, but it's important to get there.

For example, if you worry about total registered users, then you want to track incoming users to the site. But the first page doesn't create a registration, they have to go through multiple pages for that. So measure incoming users, then the % that try to register, and then the % that complete the registration form.

The reason why I write a bunch of cohort analysis is that in my perspective, a lot of what you're aiming for is a "flow-based" model for users. You want to focus on separating out variables so that if you get 1000 new users one day, you know how many are coming in via being invited from active friends versus how many are coming in from ads. The more you break this down, the sooner you'll get into variables that you can control.

Step Three: Collect both quantitative and qualitative data

One of the biggest headaches when you're generating quantitative models on your business is that after the key variables are broken down, it's difficult to figure out how to improve a particular metric. Oftentimes, the surest ways to improve end up as local maxima, whereas the highest yield increases are only offered as hazy global maxima.

Let's take an example where you're a photo-sharing site, and you need more people to upload their pictures. Local maxima could be reached by doing things like:

A/B testing your upload page to make people more likely to upload
Delivering a ton of email notifications prompting users to upload
Using switch-and-bait tactics like information-hiding, creating false incentives, etc.
Creating a gimmicky points system to upload photos

In many cases, I feel like many Facebook apps are trying to solve their problems by enacting the solutions as above. I think the quantitative side lends itself well to the above approaches, yet you rapidly hit diminishing returns.

Compare this to much harder (but higher payout) approaches like:

Repositioning the product for a higher resonating value proposition
Going after a different kind of audience to target their needs
Recalibrating the "core mechanic" of the product to make uploading photos a natural part of using the product (like HotOrNot, for example)

These qualitative approaches are much higher risk, because you can't collect significant amounts of data to validate your responses. You end up doing lots of user interviews, conducting ethnographic studies, and other methodologies that generate lots of data, but it's still up to you as the entrepreneur to figure it out. Not easy!

Ultimately, I think you have to combine the above approaches, to make sure you have views of the local maxima as well as potential paths into global maxima. Without both pieces of data, it's like navigating a mountain range with a map that's been torn into lots of different pieces.

Step Four: Generate hypotheses around key variables and variable combinations

Another key effort is to be able to follow the scientific method: Observe the data, generate many different hypotheses, and figure out what metrics are influenced. Build out an experiment, and conduct it! And remember that you can focus on an idea that hits just one variable, or even better, come up with higher-risk/higher-yield concepts that hit multiple variables.

In general, the more hypotheses you brainstorm the better – not all of them can be directly measurable, but sometimes you can figure out things that are related or proportional to what you're trying to accomplish.

Step Five: Execute test and control methods, and don't confuse correlation with causality!

Finally, it's important to execute your scientific approach with proper test and control methods:

A/B testing is a method of advertising testing by which a baseline control sample is compared to a variety of single-variable test samples. A classic direct mail tactic, this method has been recently adopted within the interactive space to test tactics such as banner ads, emails and landing pages.

Employers of this A/B testing method will distribute multiple samples of a test, including the control, to see which single variable is most effective in increasing a response rate or other desired outcome. The test, in order to be effective, must reach an audience of statistical significance.

This method is different than multivariate testing which applies statistical modeling which allows a tester to try multiple variables within the samples distributed. (from Wikipedia)

The entire point is, you have to separate out the variables that CAUSE the positive effects you're looking for, versus merely related things. The only way to separate these variables out is via A/B testing.

What are the tools you'll need to do this?

Now, all the steps above might sound like a lot, and in a way, it is. But your first focus is to have the inclination to even want to get started :)

In general, the first two steps (creating goals and breaking down variables) can just be done using spreadsheet models and talking. It's just figuring out how metrics really plays into your business – and even if you can't measure anything right away, it'll start to solidify how everything fits together.

Similarly, the hypothesis generation stage is all about getting in a conference room and doing brainstorms. The entire point of those discussions is just to generate ideas, with the constraint that the assertions have to be falsifiable.

For quantitative data collection, I typically do NOT recommend Google Analytics. Perhaps it's possible, through their events API, to collect some section of data you're looking for. But ultimately, the reporting has be built custom, by actual engineering staff. (Sorry, I know you didn't want to hear this) For the folks that are serious, you can spend as much as 50% of your energy building analysis and optimization tools, but of course, that can be the difference between a viral site that retains users well versus a crappy site that bleeds users. I generally prioritize this as a peer to the product experience, perhaps even higher, since I often overrule product functionality based on real data.

Same with A/B testing – in-house – unless you are just optimizing a page or two. In that case, you can use Google Website Optimizer.

Summary

To summarize:

Create clear, measurable goals
Make an uber-model that breaks down key variables
Collect both quantitative and qualitative data
Generate hypotheses around key variables and variable combinations
Execute test and control methods, and don't confuse correlation with causality!

For those who are dipping their toes in the water, I hope this helps!

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Views expressed in “content” (including posts, podcasts, videos) linked on this website or posted in social media and other platforms (collectively, “content distribution outlets”) are my own and are not the views of AH Capital Management, L.L.C. (“a16z”) or its respective affiliates. AH Capital Management is an investment adviser registered with the Securities and Exchange Commission. Registration as an investment adviser does not imply any special skill or training. The posts are not directed to any investors or potential investors, and do not constitute an offer to sell -- or a solicitation of an offer to buy -- any securities, and may not be used or relied upon in evaluating the merits of any investment.

The content should not be construed as or relied upon in any manner as investment, legal, tax, or other advice. You should consult your own advisers as to legal, business, tax, and other related matters concerning any investment. Any projections, estimates, forecasts, targets, prospects and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Any charts provided here are for informational purposes only, and should not be relied upon when making any investment decision. Certain information contained in here has been obtained from third-party sources. While taken from sources believed to be reliable, I have not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. The content speaks only as of the date indicated.

Under no circumstances should any posts or other information provided on this website -- or on associated content distribution outlets -- be construed as an offer soliciting the purchase or sale of any security or interest in any pooled investment vehicle sponsored, discussed, or mentioned by a16z personnel. Nor should it be construed as an offer to provide investment advisory services; an offer to invest in an a16z-managed pooled investment vehicle will be made separately and only by means of the confidential offering documents of the specific pooled investment vehicles -- which should be read in their entirety, and only to those who, among other requirements, meet certain qualifications under federal securities laws. Such investors, defined as accredited investors and qualified purchasers, are generally deemed capable of evaluating the merits and risks of prospective investments and financial matters. There can be no assurances that a16z’s investment objectives will be achieved or investment strategies will be successful. Any investment in a vehicle managed by a16z involves a high degree of risk including the risk that the entire amount invested is lost. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by a16z is available at https://a16z.com/investments/. Excluded from this list are investments for which the issuer has not provided permission for a16z to disclose publicly as well as unannounced investments in publicly traded digital assets. Past results of Andreessen Horowitz’s investments, pooled investment vehicles, or investment strategies are not necessarily indicative of future results. Please see https://a16z.com/disclosures for additional important information.

Andrew Chen Archives

5 steps towards building a metrics-driven business