Reader comments on user retention post
In my previous post on user retention strategies from the catalog marketing world, I received a number of comments commenting on the difficulty of implementing analytics systems when you don’t have the resources of a F500 company.
Another Andrew Chen writes:
I agree with the general sentiment of the post, but how would you suggest a startup deal with the problem? Without the marketing budgets of larger companies and without the time to do detailed analysis of its customers, how can a startup find the best messaging and segmentation?
Similarly, Rachanap writes:
RFM is a good idea in theory, however, even in traditional marketing world, like in CPG, companies don’t have the bandwidth to adopt it, let alone successfully implement it. Implementing such a truly effective and scalable marketing system needs a tool in place for segmenting the data, an then group the users (often needing expensive software) as well as additional infrastructure to capture that data, neither of which is an easy task. For startups this would be difficult, given their low bandwidth.
In essence, the question is, how do you take advantage of all of these different analytics systems given the constrained resources of a startup. Good question. Let’s tackle this below.
The role of analytics in startup decision-making
In general, a philosophy on the role of analytics within a startup is:
If you’re not going to do something about it, it may not be worth measuring.
(Similarly, if you want to act to improve something, you’ll want to measure it)
Don’t build metrics that aren’t going to be part of your day-to-day operations or don’t have potential to be incorporated as such. Building reports that no one looks at is just activity without accomplishment, and is a waste of time.
So instead, I recommend a “layers of onion” approach on figuring out what analytics you require. Early on, your goal may be to focus on creating a solid product that people like and stay engaged on. Obviously you could slap in Google analytics, but at the very least I’d recommend cohort-based analysis to get an actual feel for how well your products are retaining people. Similarly, you might reach a point where you want to be focused on traffic, in which case you’ll want to make sure you properly instrument to capture your viral loop. Later on, you’ll want to do the same for your ad inventory, to figure out what segments of audience and what sections of your site monetize the best.
The point is, develop the necessary metrics alongside whatever feature development that makes sense, don’t do any more than you need. Now when you are far enough along that segmenting your users based on behavior matters – likely this only because relevant once you hit a certain user acquisition threshold – then it may be important to implement RFM-based segmentation. Or not. It just depends on your product goals.
Metrics as a “product tax”
In fact, one way to view analytics is that they are a double-digit “tax” on your product development process because of a couple things:
- It takes engineers lots of time and development effort
- It produces numbers that people argue about
- It requires machines, serious infrastructure, its own software, etc
- Fundamentally, it slows down your feature development
As a rough estimate, I’ve found that it takes between 25-40% of your resources to do analytics REALLY well. So for every 3 engineers working on product features, you’d want to put 1 just on analytics. This may seem like a ton (and it is), but it throws off indispensible knowledge that you can’t get elsewhere, like:
- Validating your assumptions
- Pinpointing bottlenecks and key problems
- Creating the ability to predict/model your business to make future decisions
- It tells you which features actually are good and what features don’t matter
Question: Is it better to build 10 features where you don’t know what worked and what didn’t, or is it better to build LESS features but have a clear sense for what and why something worked? In my opinion, you want to learn as much as you can so you can “run up the score” on the features that work.
Prioritizing metrics development
The key to this philosophy is figuring out how to prioritze the metrics that you build relative to your features. Again, you want to only develop the analytics that you need to build out your product correctly – no more, no less.
In fact, it’s often wise to make a distinction between Operational reports versus Investigative reports,
each of which have their own goals and structure. The operational reports should reflect all the issues you care about in your business, and the investigative stuff generally satisfies curiosity (which is always a good thing).
So for folks with startups, I’d encourage you to ask: What are the outcomes that you care about? What are the assumptions you are making? If you’re a startup that’s spending copious amounts of time building features X, Y, and Z, what are you making the assumption of? That these features will make people want to extend their engagement times on the site? That these features will build word-of-mouth? What are the metrics that would prove you right or wrong in your assumptions?
Let’s discuss an example of a sample roadmap for a online photo-site.
Example product roadmap
I’ll close out this post with an example roadmap for a product that roughly had the same featureset as Imageshack.us. You may end up making these product decision out-of-order of what I have them, but the philosophy remains the same: Build analytics only when you need them, and align them to the key efforts behind your product development process.
Here’s an example sequencing of product development:
1) Core product and features
- Obviously, you want people to be able to upload a photo and give a URL for it
- These photos then can be displayed on other sites
1a) Analytics for core product
- First add Google analytics since it’s free :-)
- Build out a cohort matrix that tells you how many users join each week, how many come back, and what they do during their visits
- Where are these images being uploaded to? A breakdown of image displays by the domain they were displayed on would be nice
2) Working on growing traffic
- Once you have the core product done, perhaps you want to make some whizbang features to grow traffic
- You might make a photo slideshow, or some awesome photo effects, etc.
- Also, you might want to make one-click integration with Typepad/MySpace/etc via something like Gigya
2a) Analytics for traffic growth features
- One key issue to measure is how much additional traffic is gained by these different methods. You may want to start annotating visits that come to your site based on what kind of widget or photo created it. Do sparkly photo galleries generate the most visits? Why? Make a report
- Second feature might be an A/B testing framework. If you divide two groups, A and B, and show them different featuresets to see how the traffic grows for each
etc.
UPDATE: Thanks to Jeremy for the link and additional thoughts on spec’ing out metrics in the product process. Check them out.