Andrew Chen Archives

Flash Games Summit
Yesterday I attended the first ever Flash Games Summit which has a bunch of informative sessions and interesting people involved. Thanks to Mochi Media for inviting me!

Monetization panel
I moderated a panel called “Monetization and Business Models for Flash Games” with:

• Adam Caplan from Super Rewards
• Kate Connally, AddictingGames
• Jameson Hsu, Mochi Media
• Kenny Rosenblatt, Arkadium Games

This was a nice mix of people because it represented a virtual goods-focused payments platform (Super Rewards), an ad network (Mochi Media), a content portal (AddictingGames), and a developer (Arkadium), so there was a variety of interesting viewpoints.

Panel topics
We covered a variety of topics, including:

• How do the different players (ad network, portal, etc.) make money?
• What are the biggest factors in driving monetization?
• What are the differences in monetizing demographics and geographical areas?
• What kinds of games are more successful at monetization?
• How are social gaming folks different than flash gaming creators?
• What are the key metrics they look at?
• How has the recession changes their business, and how will it affect the industry overall?
• .. and many more!

Hope you enjoy watching the discussion.

Here’s the video – enjoy!
(click here if you don’t see the embed)

Online TV Shows by Ustream

BFF means “best friends forever” for those of you who are wondering why there’s a monkey and banana at the top of this blog post

Examining the power of one-way friending AKA “follow”
When I first joined Twitter, I found the one-way following mechanic pretty weird – but now, it’s clear that it’s very powerful and provides a richness that you can’t get from two-way friend requests. Initially though, I was confused. After all, hadn’t all social networks standardized around two-way friend requests that both parties have to accept? Why try to fix it? It seemed like it’d just be confusing, and potentially freak some people out that they were being followed by random people they didn’t know.

This post examines the strengths of the one-way “follow” design, in particular, the ability for this paradigm to support 4-tiers of relationships rather than the simple 2-tiered model in the classic friends case.

First, let’s discuss the social groupings issue.

Social groupings and friend segmentation
At the same time as Twitter was just getting started, the rapid explosion of users on Facebook, MySpace, and other social networks raised a bunch of really core and important questions about these social applications. Among these issues:

• Will one social network rule them all?
• Or alternatively, will you use one social network for work, one for your personal life, and possibly others for other vertical interests?
• If it’ll be one, how will you group your work friends in one, and your personal friends in another?
• How will this work at a design level? How about at a technical level? (aka Data Portability?)

These are all great questions, and point out a number of potentially fundamental weaknesses to the all-in-one social networking model. If you look at many other communication channels, like phone, email, etc., you’ll often see people segment their identities. Their work voicemail will be boring, and their personal voicemail will be funny, and they’ll use different phone numbers for each.

Of course, the initial petri dish that social networks grow – high school and college students – don’t really have to deal with this. Their social groupings are more or less homogenous, because they only have personal friends. But after you’ve worked a couple places, moved around, and have your friends’ careers diverge into lawyers and slackers, then your social network becomes more complex and segmented.

The approach that many social networks have taken to solve this is to group people into networks and friend lists. Either through self-assignment or you assigning them, people go into different lists. Of course this hurdle is basically a type of boring security configuration that consumers have historically had trouble with.

Twitter’s “follow” model
The amazing thing about Twitter’s model of allowing one-way following is that it adds depth and a couple simple segmentations to your friend list, without needing to do any configuration beyond hitting a button.

With the one-way follow design, you have:

1. People who follow you, but you don’t follow back
2. People who don’t follow you, but you follow them
3. You both follow each other (Friends!)
4. Neither of you follow each other

Having these 4-tiers of relationships on Twitter is nice – combined with Protected Updates, it creates a nuanced set of definitions, executed with just one button: Follow.

The advantages are numerous: First, it’s easier to get started by opting into a number of feeds that pre-exist, and you can populate your timeline without anyone accepting your friend requests. Second, it makes it possible to have interactions with lots of people (@replies), but your timeline only has information you care about, as you don’t have to follow folks you’re not interested in. Third, some profiles are inherently appealing to a cross-section of users – these include celebrities, companies, media content, etc. – and it the one-way follow design supports all of these nicely whereas two-way friending makes things complex. 

Two-way friending with public profiles?
Compare the above to the traditional two-way friending case, supported by social networks

1. You’re friends
2. You’re not friends

So how do you deal with Sean Combs aka P. Diddy (aka @iamdiddy)? If you were to friend him (and he friends you back), all of a sudden, you are exposed to the random people (like you) who are interacting with him, which creates a lot of low-value information on your newsfeed.

As a result, it only makes sense to separate Diddy’s profile into two separate ones, a public and private profile, where the private is the “real” friends and the public one is everyone else. For MySpace, they opted to differentiate these public profiles as “Artist Profiles” whereas Facebook decided to call them “Pages.” I imagine that they treat information flowing in and out of these pages specially, so that they know not to public crazy amounts of information from random people, and they can segment those interactions out.

Note that MySpace was very early in having these celebrity profiles, which has led to the right of so-called MySpace celebs like Tila Tequila, Forbidden, etc. whereas I’m not aware of any Facebook celebs emerging ;-)

Maybe this two-way friends with public/private profiles works, but it’s much less elegant than a single “follow” button. In the dual profile version, you end up needing either lots of configuration (what photos to publish, which friends belong in which), or you end up with two distinct pieces of content. This would mean multiple photos, multiple profile content, and two places to do everything. Not attractive, in my opinion.

Conclusion
Ultimately, both approaches have their advantages – the two-way friending model is better at supporting strictly real-life relationships. That ability has obviously led MySpace and Facebook to conquer a lot of real estate and build eyeballs. At the same time, this model requires them to design around the complexity introduced by celebrities, brands, and companies, which are all important folks to have in your ecosystem for long-term monetization as well as mass appeal.

As always, leave a comment with your thoughts! See any other friending models that have advantages?

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.

Here are some links I’ve posted to my twitter account over the last week or two. You can follow me on Twitter if you like these! Many are work unrelated.

• Spot Runner Is Running On Fumes: Another 60 To Lose Their Jobs http://tinyurl.com/dj89n2
• rt @mkapor If you read anything about what’s happening to newspapers, read this essay by @cshirky (Clay Shirky) http://bit.ly/18tDhy
• NYT article, In Italy, a Vending Machine Even Makes the Pizza: http://tinyurl.com/cwuy5q
• Candy-Maker Tries To Ignore Kids Pretending To Smoke On YouTube http://tinyurl.com/casw5j
• Awesome Peter Funch Photo Series of Manhattan http://cli.gs/z67bZu
• Don’t launch http://tinyurl.com/bz5vop
• Struggling in-game ad firm IGA Worldwide seeks investments or possible sale http://tinyurl.com/c8ujol
• Yahoo Search Streak Snapped, Loses Share In Feb (Google Gains) http://tinyurl.com/b3v6dz
• demographic changes in myspace->FB movement: http://tinyurl.com/cb53or
• Introduction to the ad auction http://tinyurl.com/bmsjnb
• Pivots for change http://tinyurl.com/bu4tdr
• Sam Spenser’s Umbrella Bloom http://tinyurl.com/d95zbq
• i thought google thought behavioral targeting was evil? Now they just unveiled their new internet-wide cookie tracking: http://bit.ly/X5xWM
• slate article explaining icelandic elf detection: http://www.slate.com/id/2213353
• How Google Will Invade Your Privacy While ‘Protecting’ It http://tinyurl.com/d5nqwf
• graph of microlending vs venture capital: http://tinyurl.com/dza5r9
• Not Enough Work http://xkcd.com/554/
• can’t believe msft surface is a real product you can buy – i always thought it was a demo: http://tinyurl.com/brlalo
• Graham on Boston vs Silicon Valley, why YC left Boston (his opinion: Silicon Valley wins hands down) http://bit.ly/gAh6
• Some zone design lessons http://tinyurl.com/cjbuuc
• Cellphones vs Women (or Men) [Comic] http://tinyurl.com/bb5z32
• massive tetris flash game. Fun? http://sovietrussia.org/f/src/tetoris.swf
• 27 Huge Publishers Join To Replace The Bannerhttp://tinyurl.com/bpdv7r
• More Details & Stats on User Communication Patterns from Facebook Data Team http://tinyurl.com/bbvhnl
• 8 Ways the Changing Facebook Home Page Will Affect Application Virality http://tinyurl.com/dzzrkz
• Dilbert and The Smart Talk Trap: The Dangers of Skilled Bullshitting http://tinyurl.com/bt742t
• Branded Virtual Goods Boost Purchase Intent By 20% Or More On Social Networks http://tinyurl.com/boq7mo
• Unauthroized viewing of TV and movie content expected to grow http://tinyurl.com/aeet87
• The Last Days of the Oligarchs? http://tinyurl.com/cdluh5
• What is your game design style? http://tinyurl.com/br8sh7
• Facebook Extends Lead Over MySpace in US Traffic in February http://snipurl.com/d9qld
• Lessons Learned from imeem http://snipurl.com/d9w4s
• Opinion: Designing For Free Takes More Than ‘Just’ Game Design http://tinyurl.com/bqu6j6
• # of iphone apps in the appstore is accelerating!http://apple20.blogs.fortune.cnn.com/2009/03/05/apples-app-store-25000-apps-and-counting
• yahoo mail still 5X the size of Gmail: http://tinyurl.com/cezzco
• Don’t Throw Out the Subscription Model http://tinyurl.com/d92fah
• wikipedia article on the large % of icelandic people who believe in elves: http://en.wikipedia.org/wiki/Huldufólk
• nyt article on icelandic elves:http://www.nytimes.com/2005/07/13/international/europe/13elves.html
• Watch this! Great speaker, and he’s 10 years old: http://tinyurl.com/a9rdhm
• rt @adachen Wow, YouTube hits 100 million *US* viewers http://bit.ly/8CWQH
• Icelandic logic behind the meltdown http://tinyurl.com/aloche

New monetization option for app developers launches
Today, my friend Noah Kagan launched a new payments service called Gambit which you can check out here. The focus is on virtual currency-based Facebook/OpenSocial applications, and supports credit cards, mobile payments, and offers-based monetization. He also has a blog and twitter account.

Given the proliferation of these services, I wanted to spend a couple minutes talking through the new monetization funnel for apps and some of the metrics that are being thrown around.

Spreadsheet model
First off, I wanted to quickly share a very simple spreadsheet model which you can download here.

Funnel steps
At a high level, the key issues to track for an offers-driven monetization looks something like this:

• total installs / registered users
• monthly active uniques
• daily active uniques
• daily paypage uniques – how many users go page where you can pay or fill out offers?
• daily lead clickthroughs
• daily net lead completions (lead completions – chargebacks)
• daily revenue
• monthly revenue

From top to bottom, you can see that the focus is around uniques, and how many of them translate to completed offers and ultimately revenue. Of course, many of these transactions will actually end up as direct payment via credit card or mobile, and that is trivial to add to this model – I won’t cover those just for simplicity.

One quick note on leadgen though, for those who are unfamiliar – essentially, leads are opt-in forms that users can fill out in order to generate virtual currency. This might be subscribing to a Netflix offer, for example, or giving out a real address to get a direct mailing from a university. More about the leadgen industry here. As a result of this construct, users may not have to use credit cards in get money to the publisher, which can be great.

As a result of this offer-based monetization, it becomes important to track not only how many people click through to begin filling out a lead, but also how many folks complete leads, and ultimately how much revenue each lead is worth. Different demographics, geographical areas, and lead types generate different kinds of revenue. There’s also chargebacks that happen when the leads are rejected for being complete or incorrect.

Example numbers
If you plug this into a monetization table, then you can see the flow.

Here are some example numbers derived from games that Noah’s company Kickflip had previously developed and operated – he was comfortable sharing the data that came out of his own apps. The numbers below would represent a large and successful app with millions of actives per month:

and of course these numbers are driven by all the percentages of how much dropoff there is at each step. For quick reference, the percentages are listed below and drive the numbers in the table above.

Now that we have these metrics, we can start to calculate to other metrics.

Let’s define a new term, “ACPM” which stands for “App CPM”
As someone from the online ad industry, I was saddened to hear that the term “CPM” had been co-opted by these app monetization companies to mean something entirely different and weird.

In the app monetization world, this is the definition:

App CPM = (daily revenue / daily uniques to the paypage) * 1000

Recall that this is very different than the standard definition:

Online ad CPM = (daily revenue / daily ad impressions) * 1000

They are certainly not apples-to-apples, even though they are represented in some literature as such. And unfortunately, now that some of the market leaders are using these misguided terms, everyone is following suit. Yuck!

So as you can see, the “app CPM” (which I’ll refer as ACPM) is defined by uniques to the payments/offer page, rather than by pageviews or impressions. Using the above numbers, we’d get:

ACPM = ($5,670 / 45,000) * 1000 = $126

I’ve been told by multiple people that numbers from $50-$200 are all possible here.

Measuring ARPU
You’ll notice that the ACPM has no relation to the overall usage of the product – in fact, you might have $300-$400 app ACPMs but still have low revenue, since the ACPM is only defined once the users hit the payments page. Maybe you have a small number of users who do so, or maybe you only have a small % of users who are active at any given time.

To measure how the numbers fit together from top to bottom, instead we’ll have to calculate the ARPU:

ARPU = revenue / total actives

This means that this would include any and all actives, regardless of whether or not they visited the payments page. For the numbers above, they’d translate to:

Monthly ARPU = $172,935 / 3,000,000 = $0.058

On Facebook, I’ve been told from multiple sources that numbers from $0.01 to $0.25 are all reasonable, and that off of the social platforms you’ll find more niche destinations that generate closer to $1 ARPU.

Conclusion
Ultimately, it’s very exciting that multiple monetization platforms are getting created in the industry, and that competition will be great for everyone. Gambit is certainly one of many new creative companies that  will come out with great stuff.

At the same time, it’ll be up to publishers to figure out how to squeeze as much monetization they can from their properties, but without compromising the user experience. By looking at the numbers above, it’s clear how you can increase both ARPU and ACPM in very systematic ways, but it’s potentially at the cost of retention or engagement within the product.

Ideas or questions? Leave me a comment.

Want more?
If you liked this post, please subscribe or follow me on Twitter.

UPDATE: thanks to Jared Fliesler for correcting a silly mistake in my arithmetic ;-)

Today we have a fantastic guest blog from Ranjith Kumaran, on his adventure going from an ad-supported free service to a subscription-based freemium model. Ranjith is the Co-Founder & CTO of YouSendIt.com, a Silicon Valley company that allows businesses and individuals send, receive and track digital content securely and easily. Enjoy! -Andrew

Free to Freemium: 5 Lessons Learned
by Ranjith Kumaran

Introduction
A tech reporter recently asked me if YouSendIt.com had made the switch from a free ad-based business model to a subscription-based freemium model “just in the nick of time”. After all, with death chasing every ad-revenue-fueled startup these days, surely we must have been scrambling over the last few months!

The truth is that we got our first paid subscriber at YouSendIt on the night of February 28th, 2006, over three years ago. The company recently passed the 100,000 paid subscriber mark but that first customer was where it all started: the transition from free to freemium. As startup pundits we expect business models to iterate but this particular switch was a thrill-a-minute ride.

So if you’re ready to take the plunge or are still on the fence between free vs. freemium then read on. I’ll highlight five key lessons learned over the last three years as we went from a 100% free model to freemium:

Lesson 1: It’s all about DNA
Lesson 2: Funnels come in all shapes and sizes
Lesson 3: Compound growth is a double-edged sword
Lesson 4: Don’t let pricing psyche you out
Lesson 5: “Boring” things can give you lots of conversion lift

It’s all about DNA
It doesn’t matter how smart your team is or how hard you work, everyone has to want to make the switch from free to freemium. The thesis of our first venture round of investment was to test both models to see how they scaled. But the reality was we already had a significant free business (advertising revenue helped my co-founders and I keep the lights on, sound familiar?) so the lion’s share of the first six months were spent building a team that could keep the viral, ad-impression-generating parts of the business growing. When the subscription service launched and showed great promise (we collected our first payment within 4 minutes of pushing the code live), the business model was changed but many of the team’s mindsets were not. Reconciling these differences was exhausting but we got there.

Do yourself a favor and pull the band-aid off quickly. If you re-channel all effort into improving conversion and building your brand your subscription business will get out of the blocks much faster. A change in DNA is the hardest thing a company can endure and some don’t; get through it early.

Funnels come in all shapes and sizes
Once you’ve made the switch a number of things will happen:

• People who don’t believe in paying for web-based services will call you a sell-out. Unsurprisingly, these folks aren’t in your target market. If you provide a valuable service the majority of your users will stay with you (most for free and some percentage will subscribe right away). YouSendIt.com’s traffic took a 30% haircut in traffic during this process. If we didn’t have anything further down the funnel this would have been devastating.
• Expect to see a drastic change in the mix of users you serve going forward. YouSendIt’s business is international (everyone sends files), including geographies that any startup will struggle to effectively monetize showing ads; in general the same geographies also yield weaker subscriber numbers. This pruning of who you serve and how much (by, say, asking for payment) is very, very, common and often more deliberate; it’s a cost-to-serve discussion every web business that thinks beyond customer acquisition will eventually have. Over time we found that the users who were willing to pay for our services attracted similar users to the site.
• Plan to change the metrics by which you measure your business. Our dashboard went from plotting CPMs, impressions and make-goods to conversion rates, churn, and ARPU. Acquisition cost, cost-to-serve, and lifetime value start to rear their ugly heads. If you want to fully understand your freemium business, learn to love them.

Compound growth is a double-edged sword
Once the freemium engine has run for a while you’ll see that, unlike fluctuations in ad-impressions and CPMs, subscription revenue is very predictable; your shareholders will appreciate this. Step functions in revenue are seen when new products are launched (including up-selling to the current base and convincing more users to subscribe) and new channels into the top of the funnel are created (making our way onto the desktop was a big one). Compounded subscriber growth is very powerful: convert more users in January and you’ll have a chunk of the year’s revenue in the bag, provided you’ve got churn under control; fall behind and revenue shortfall amplifies just as quickly over time.

Don’t let pricing psyche you out
Balancing market penetration and the fear of leaving money on the table is no fun and more than one startup has failed to even launch a paid product because of the pricing hurdle. Here’s a quick and dirty way to put a stake in the ground:

1. Make a list of your competitors or find adjacent markets / potential substitutes with similar users and use cases. You should already have this list.
2. Plot the spectrum of all the price-points of their offerings.
3. Plan to release at least two paid tiers: one at the bottom end of the spectrum that is driven by volume and one at the top that is clearly differentiated by value.

By doing this you can accomplish the following: fill in any market share vs. revenue maximization discussion rat-holes (now you can test both); give customers a way to compare between three offerings (free, a little more, and lot more; being able to compare is an important part of any purchase decision); feel good that you’ve done some diligence on pricing without prematurely shelling out a lot of cash on market research.

If the above exercise seems unscientific that’s because it is. Your pricing work has just begun: constantly observe the rates at which users move through the funnel at different price-points, use promotions to get buyers off the fence, and re-price as you get more price elasticity data. At YouSendIt we raised prices (yes, it can be done) successfully several times in the early days as we learned more and more about buying behavior.

“Boring” things can give you lots of conversion lift
Conversion lift doesn’t always come from groundbreaking changes in product, offers, or funnel analysis. These days I will look for ten 1% lifts in conversion before one 10% magic bullet; in reality there probably aren’t a lot of 10% lifts left after the first handful. Get into the groove of turning knobs a little at a time, learning, and iterating; you never did this further down the funnel when you were selling ads and you are likely out of practice. Other mundane things that you haven’t invested in start to get a lot of play: customer service SLAs, quality of service, and even the right terms of service are all areas which can drive conversion. Look for a 1% lift in conversion right now, it’s in there somewhere; then do it again a million times.

Conclusion
With any luck there are enough examples above to convince you that switching from free to a freemium business model can be done with a little perseverance and a lot of belief. I’ve experienced the rush of going from 0 loyal users, to thousands, hundreds of thousands, and millions a few times in my career. But there is a different kind of satisfaction you and your team will get when your business starts to amass paid subscribers: users who believe the things your company has worked so hard to create are good enough to pay for. This is the ultimate validation of your efforts.

These topics will be covered in more detail at a couple of panels (one on Understanding Freemium Business Models, another on Customer Acquisition in a Down Economy) I’m helping to organize in the coming weeks. If you’d like to participate as an attendee, panelist, or moderator or if you’re simply interested in hearing about more lessons learned (the hard way), please follow the Twitter feed I’ve set up.

Here are some links I’ve posted to my twitter account over the last week or two. You can follow me on Twitter if you like these! Many are work unrelated.

• TiVo Loses 125,000 Subscribers In January ‘09 Quarter http://tinyurl.com/c9gxvu
• Chinese Students Want To Know: How Do I Get Rich? – Forbes.com http://cli.gs/LXUBSp
• Crazy Numbers: $300 Billion in Annual Revenue from Pachinko & Pachislots in Jap.. http://tinyurl.com/c52zvu
• Why Amazon Is Bucking the Trend http://tinyurl.com/dzcf7y
• Facebook’s Thiel Explains Failed Twitter Takeover http://tinyurl.com/cp6d64
• YouTube TV News Upload And Viewing Numbers, Week ending February 27 http://tinyurl.com/asug59
• Today’s Threat To Broadcast TV Networks http://tinyurl.com/aqlck9
• Berkshire: Worst Year Ever, plus Credit Default Swaps http://tinyurl.com/ddf8c5
• Max Levchin Is Bored With Silicon Valley Startups http://tinyurl.com/d735ez
• The ugly truth about your favorite social networks http://tinyurl.com/bgtdda
• As first-big-domino Rocky Mountain News folds, Google begins to place ads on Google News. http://bit.ly/S37ew
• THIS IS SO COOL — Oregon Trail ONLINE: http://www.virtualapple.org/oregontraildisk.html
• Golden Parachute http://tinyurl.com/b48ry9
• Surprise! Yahoo Outperforming Google http://tinyurl.com/b9onk5
• The Shakeout Begins! Video Startups in Play http://tinyurl.com/d78zyd
• termsheets still getting done, or at least VCs still talking about them on twitter ;-) http://tinyurl.com/dk97bw
• challenges/opportunities in brand online ad market (via @joshk) http://tinyurl.com/aa2z2z
• Apartment Buyers Abandoning 6-Figure Deposits http://tinyurl.com/dyyamd
• what happened to avg time per visit on MySpace in July? This graph looks like it’s bad data to me: http://bit.ly/4My6o
• Launch: Pitching Hacks, The Book http://tinyurl.com/by9bm6
• Building Quake Live: Carmack Speaks http://tinyurl.com/br6h4r
• Moral Hazard http://tinyurl.com/aoesd3
• What do fourth generation ad networks look like? http://tinyurl.com/ahh9of
• Bailout Hearings http://tinyurl.com/bmp4wu
• State of the Computer Book Market 2008, part 4 — The Languages http://tinyurl.com/dauyzk
• Eliminating Channel Conflict Between Publishers, Ad Networks http://bit.ly/D3Ra7
• Reboot: How to Reinvent a Technology Startup http://tinyurl.com/aodcbm
• How IMVU learned its way to $10M a year http://tinyurl.com/aqjnb3
• 5 Reasons Chernin’s Exit Puts Hulu In Danger http://tinyurl.com/alwbnr
• Leonardo di Vinci was a big time procrastinator: http://tinyurl.com/dmj74r
• Pic of Vegas, 1954: http://tinyurl.com/cj8579
• Online Video Viewing Up, Impact on TV “Negligible” http://tinyurl.com/c3naft
• Exploring a ‘Deep Web’ That Google Can’t Grasp http://tinyurl.com/dgd2fy
• Why the Click Is the Wrong Metric for Online Ads http://bit.ly/Gm0SZ
• Foxtrot comic, Jason figures how to make money in a down economy: http://bit.ly/11iwtm
• 4Chan.org: 300m page views/mo; about $6k in revenue http://tinyurl.com/czeppu
• Education Fail http://tinyurl.com/cb32u2

Lots of Bay Area startup chatter on Twitter
As I’ve written about many times, I’m of the often-repeated notion that the SF Bay area is a one-of-a-kind place for entrepreneurs. There’s lots of advantages, but one great one is the constant casual chatter of other entrepreneurs and CEOs here.

I quickly combed my tweets for today and made a list of active users of Twitter who also happen to be founder/CEOs in the companies around here. Almost all of them are running VC-backed startups, and they are among the best folks to learn from. I’m purposely leaving out obvious choices like @ev and conference types that get enough visibility as it is ;-)

The quick list of folks you might follow on Twitter
Here we go:

If you’re not on the list, then I didn’t see you use Twitter in the last day ;-)

And last but not least, you can follow me on Twitter here.

Am I missing anyone? Just comment if you have recommendations.

Above: Hollywood sequels follow from risk-averse design decisions, like the widely panned Godfather Part 3

The dangers of the metrics-driven design process
Many readers of this blog are expert practitioners of metrics-driven product development, and with this audience in mind, my post today is on the dangers of going overboard with analytics.

I think that this is an important topic because the metrics-driven philosophy has come to dominate the Facebook/OpenSocial ecosystem, with negative consequences. App developers have pursued short-term goals and easy money – leading to many copycat and uninspired products.

At the same time, it’s clear that A/B testing and metrics culture serves only to generate more data-points, and what you do with that data is up to you. Smart decisions made by entrepreneurs must still be employed to reach successful outcomes. (Thus, my answer to the title question is that no, A/B testing does NOT lead to crappy products, but poor decision-making around data can absolutely lead to it)

So let’s talk about the dangers of being overly metrics-driven – here are a couple of the key issues that can come up:

1. Risk-averse design
2. Lack of cohesion
3. Quitting too early
4. Customer hitchhiking
5. Metrics doesn’t replace strategy 

Let’s dive in deeper…

#1 Risk-averse design
The first big issue is that when you design for metrics, it’s easy to become risk-averse. Why try to create an innovative interaction when something proven like status/blogging/friends/profiles/forums/mafia/etc already exists? By copying something, you’re more likely to quickly converge to a mediocre outcome quickly, rather than spending a ton of effort potentially creating something bad – but of course this also eliminates amazing, ecstatic design outcomes as well. 

This risk-averse product design can lead to watered down experiences that combine a mish-mash of features that your audience has already seen elsewhere, and done better too. So while it’s an efficient use of effort, it’s unlikely that your experience will ever be a great one. It’s a recipe for mediocrity. 

Risk aversion is responsible for a whole bunch of bad product decisions outside of the Internet industry as well: Why do Hollywood sequels get made, even though they are usually much worse than the original? Why do companies continually do “brand extensions” that dilute the value of their brand position? The reason is that it’s an efficient thing to do, and it’s pretty easy to make some money even if the end product is not that great. But it’ll hurt in the long run, since these products will inherently be mediocre rather than great.

In my opinion, the only way to avoid this is to never get lazy about design, and to always take the time to create innovative product experiences. Of course you’ll always have parts of your product which will borrow from the tried-and-true, yet I think it’s always important that the core of the experience is differentiated and compelling.

#2 Lack of cohesion
As hinted above, the next issue is that A/B tested designs often create severe inconsistency within an experience. The bottoms-up design process that results from lots of split testing is likely to come up with many local effects, which may rule global design principles.

Here’s a thought experiment to demonstrate this: Let’s say you tested every form input on your website, with different labels, fonts, sizes, buttons, etc. You’re likely, if you picked the best-performing candidate, to have wildly different looking forms across the site. While it may perform better, it also makes the experience inconsistent and confusing.

Ultimately, I think resolving this has to do with striking a balance between global design principles and local effects. One great way to do this is to split out the extremely critical parts of your product funnel to be locally optimized, and keep the rest of the experience the same. For a social gaming site, the viral loop and the transaction funnel should be optimized separately, whereas the core of the game experience should be very internally consistent.

#3 Quitting too early
Another way to get to uninspired products is to quit too early while iterating an experience because of early test data. When metrics are easy to collect on a new product feature, it’s often very tempting to launch a very rough feature and use the initial metrics to judge the success of the overall project. And unfortunately, when the numbers are negative, there can be a huge urge to quit early – this is a very human reaction to wanting to not waste a bunch of time on something that’s perceived to fail.

Sometimes a product requires features A, B, and C to work right, and if you’ve only done A, it’s hard to figure out how the entire experience will work out. Maybe the overall data is negative? Or maybe it inspire dynamics that go away once all the features are bundled? Interim data is often just that – interim. But people are great at extrapolating data, but sometimes the right approach is just to play out your hand, see where things go, and evaluate once the entire design process has completed.

#4 Customer hitchhiking
A colleague of mine once used the term “customer hitchhiking” to describe how it’s easy to follow the customer on whatever they want to do, rather than having an internal vision of where YOU want to go. This can happen whenever the data overrules internal discussion and resists interpretation, because it’s so uncompromising as hard evidence. The important thing to remember, of course, is that the analysis is only as good as the analyst, and it’s up to the entrepreneur to put the data into the context of strategy, design, team, and all the other perspectives that occur within a business.

Today, I think you see a lot of this customer hitchhiking whenever companies string together a bunch of unrelated features just to please a target audience. This reminds me of what is often called the “portal strategy” of the late-90s. Just combine a bunch of stuff in one place, and off you go. The danger of that, of course, is that that it leads to incoherent user experience, company direction, and numerous other sources of confusion.

In the Facebook/OpenSocial ecosystem, of course, this manifests itself as companies that have many unrelated apps. You can dress this up as a “portfolio” or a “platform” but at the same time, it can be a recipe for crappy product experiences.

#5 Metrics doesn’t replace strategy
What do you think it would be like to write a novel, one sentence at a time, without thinking about the broader plot? I’m sure it’d be a terrible novel, and similarly, I bet that testing one feature at a time is likely to lead to a crappy product.

Ultimately, every startup needs to decide what they want to do when they grow up – this is a combination of entrepreneurial judgement, instinct, and strategy.

Every startup has to figure out how big the market is, they have to deliver a compelling product, and they need a powerful marketing strategy to get their services in front of millions of people. Without a long-term vision of how these things will happen, an excessive amount of A/B testing will surely lead to a tiny business.

To use a mountaineering analogy: Metrics can be very helpful in helping you scale the mountain once you’re on top of the right one – but how do you figure out whether you’re scaling the right peak? Analytics are unlikely to help you there.

Conclusions
My point on this – nothing is ever a silver-bullet, and as much as I am an evangelist for metrics-driven approaches to startup building, I’m also very aware of the shortcomings. In general, these tools are great for optimizing specific, local outcomes, but they need to be combined with a larger framework to reach big successes.

Ultimately, quantitative metrics are just another piece of data that can be used to guide decision-making for product design – you have to combine this with all the other bits of information to get it right.

Agree or disagree? Have more examples? Leave me a comment! 

Want more?
If you liked this post, please subscribe or follow me on Twitter.