Andrew Chen Archives

Metrics are merely a reflection of the product strategy that you have in place
Data is powerful because it is concrete. For many entrepreneurs, particularly with technical backgrounds, empirical data can trump everything else – best practices, guys with fancy educations and job titles – and for good reason. It’s really the skeptic’s best weapon, and it’s been an important tool in helping startups solve problems in new and innovative ways.

It’s easy to go too far – and that’s the distinction made between “data-informed” versus “data-driven,” which I originally heard at a Facebook talk in 2010 (included underneath the post). Ultimately, metrics are merely a reflection of the product strategy that you already have in place and are limited because they’re based on what you’ve already built, which is based on your current audience and how your current product behaves. Being data-informed means that you acknowledge the fact that you only have a small subset of the information that you need to build a successful product. After all, your product could target other audiences, or have a completely different set of features. Data is generated based on a snapshot based on what you’ve already built, and generally you can change a few variables at a time, but it’s limited.

This means you often know how to iterate towards the local maximum, but you don’t have enough data to understand how to get to the best outcome in the biggest market.

This is a messy problem, don’t let data falsely simplify it
So the difference between data-informed versus data-driven, in my mind, is that you weigh the data as one piece of a messy problem you’re solving with thousands of constantly changing variables. While data is concrete, it is often systematically biased. It’s also not the right tool, because not everything is an optimization problem. And delegating your decision-making to only what you can measure right now often de-prioritizes more important macro aspects of the problem.

Let’s examine a couple ways in which a data-driven approach can lead to weak decision-making.

Data is often systematically biased in ways that are too expensive to fix
The first problem with being data-driven is that the data you can collect is often systematically biased in unfixable ways.

It’s easy to collect data when the following conditions are met:

• You have a lot of traffic/users to collect the data
• You can collect the data quickly
• There are clear metrics for what’s good versus bad
• You can collect data with the product you have (not the one you wish you had)
• It doesn’t cost anything

This type of data is good for stuff like, say, signup %s on homepages. They are often the most trafficked parts of the site, and there’s a clear metric, so you can run an experiment in a few days and get your data back quickly.

In contrast, if you are looking to measure long-retention rates, that’s much more difficult. Or long-term perceptions of your user experience, or trying to measure the impact of an important but niche feature (like account deletion). These are all super difficult because they take a long time, or are expensive, or are impossible datapoints to collect – people don’t want to wait around for a month to see what their +1 month retention looks like.

And yet, oftentimes these metrics are exactly the most important ones to solve.

Worse yet, consider the cases where you take a “data-driven” mindset and try to trade off the metrics between concrete datapoints like signup %s versus long-term retention rates. It’s difficult for retention to ever win out, unless you take a more macro and enlightened perspective on the role of data. Short- vs long-term tradeoffs require deep thinking, not shallow data!

Not everything is an optimization problem
At a more macro level, it’s also important to note that the most important strategic issues are not optimization problems. Let’s start at the beginning, when you’re picking out your product. You could, for example, build a great business targeting consumers or enterprises or SMBs. Similarly, you can build businesses that are web-first (Pinterest!) or mobile-first (Instagram!) and both be successful. These are things where it might be nice to have a feel for some of the general parameters, like market size or mobile growth, but ultimately they are such large markets that it’s important to make the decision where you feel good about it. In these cases, you’re forced to be data-informed but it’s hard to be data-driven.

These types are strategy questions are especially important when the industry is undergoing a disruptive innovation, as discussed in Innovator’s Dilemma. In the book, Clayton Christensen discusses the pattern of companies who are successful and build a big revenue base in one area. They find that it’s almost always easier to increase their core business by 10% than it is to create a new business to do the same, but this thinking eventually leads to their demise. This happened in the tech industry from mainframes vs PCs, hardware vs software, desktop vs web, and web vs mobile now. The incumbents are doing what they think is right- listening to their current customer base, improving revenues from a % basis, and in general trying to do the most data-driven thing. But without a vision for how the industry will evolve and improve, the big guys are eventually disrupted.

Leverage data in the right way
It’s important to leverage data the same way, whether it’s a strategic or tactical issue: Have a vision for what you are trying to do. Use data to validate and help you navigate that vision, and map it down into small enough pieces where you can begin to execute in a data-informed way. Don’t let shallow analysis of data that happens to be cheap/easy/fast to collect nudge you off-course in your entrepreneurial pursuits.

Facebook on data-informed versus data-driven
I leave you with the Facebook video that inspired this post in the first place – presented by Adam Mosseri. He uses the example of multiple photo uploads, and how they use metrics to optimize the workflow. Watch the video embed below or go to YouTube.

Don Valentine, who founded Sequoia Capital, talks about what makes Sequoia Capital effective. It’s one of my favorite talks, and I find myself watching and re-watching it from time to time, and I’d encourage everyone to hear the wisdom themselves.

Markets, not team
In the beginning of the video, Don Valentine asks, why is Sequoia successful? He says that most VCs talk about how they finance the best and the brightest, but Sequoia focuses instead on the size of the market, the dynamics of the market, and the nature of the competition.

This is, of course, super interesting because in many ways it’s contrarian to the typical response that investing is all about “team.”

Creating markets versus exploiting markets
Another choice quote: “We’re never interested in creating markets – it’s too expensive. We’re interested in exploiting markets early.”

In consumer internet, when the divisions that separate product categories are so fuzzy, it can be hard to understand when you’re creating a market versus when you’re attacking an existing one. My rule of thumb is that:

If people know how to search for products in your category then you are in an existing market.

I’ve written more about this in posts here and here

Watch the video of Don Valentine of Sequoia capital on “Target Big Markets” on YouTube or in the embed below:

Using retweets to assess content virality
Recently I’ve been running an experiment:

1. Tweet an insight, idea, or quote
2. See how many people retweet it
3. If it catches, then write a blog post elaborating on the topic

My recent Growth Hacker post was the result of one such tweet, which you can see above in my Crowdbooster dashboard. I wrote it on a whim, but after the retweets, I developed it into a longer and more comprehensive blog post. (Note that sometimes a tweet is not suitable to developed into a blog post, but most of the time this technique works)

Why this works
This works because the headline is key. It spreads the content behind it.

This is especially true on Twitter, but it’s also true for news sites that will pick up and syndicate your content. If that headline is viral and the content behind it is high quality, there’s a multiplier effect – sometimes a difference of 100X or more. Naturally, you want to optimize the flow of how people interact with your content, starting with what they see first: The title.

After all, what’s a better test for whether the following will be viral:

New blog post: Growth Hacker is the new VP Marketing [link]

than the tweet:

Growth Hacker is the new VP marketing

It’s a natural test.

I’ll also argue that if you can express the core of your idea in a short, pithy tweet, then that’s a good test for whether the underlying blog post will be interesting as well. Great tweets are often provocative insights or mesmerizing quotes, and there’s a lot to say by examining the issues more deeply. Contrast this to writing a long, unfocused, laundry-list essay examining a topic from all angles, taking no interesting positions or risks along the way – now that’s a recipe for boredom.

Combining virality with a high-quality product, of course, is the key to a lot of things – not just blogging :)

Don’t waste your time writing what people don’t want to read
Testing your ideas like this allows you to invest more time and effort into the content – a clear win.

Personally, I love writing long-form content that dives deep into an area, and also enjoy reading it as well. Unfortunately, writing a blog post often takes a long time – an hour or more. Use this technique to make it safer to spend more time, think more deeply, and research more broadly on you write. In my experience, writing a high-quality, highly retweetable blog post once per month is better than writing a daily stream of short, low-quality posts that no one will read. Plus, it takes less time.

As a smart guy once said: “Do less, but better.”

I recently asked, and then answered my own question on Quora and wanted to share here as well.

Has Facebook’s DAU/MAU always been ~50%?

According to public info, Facebook’s DAU/MAU is 58% these days. Here’s a link.

It states:

• 901 million monthly active users at the end of March 2012
• 526 million daily active users on average in March 2012

Has Facebook’s DAU/MAU always been this good, as a consequence of its product category (communication/photo-sharing/etc.)? Or was it once a lot worse and was improved over time?

(UPDATE: Here’s a followup question I have about the same topic- Was Facebook’s DAU/MAU ~50% prior to launching the Newsfeed in 2009?)

Answer: Yes, Facebook’s DAU/MAU has been close to 50%, at least since 2004.

Based on their media kit from 2004, their DAU/MAU was already 75%.

Since this media kit, their DAU/MAU data has been included in their financials since 2009. However, I theorize that Facebook’s DAU/MAU has always been high as a natural outcome of the communication-oriented usage of the product. Contrast this to a product category like ecommerce which you are unlikely to use and purchase with every day.

In their recent financial filings, the following chart is shown for Facebook’s DAU and MAU since 2009:

If you do a graph of the DAU/MAU on this data, since 2009, you’ll see that it starts around 45-47% and goes up to a very impressive 58% recently.

(As an aside, another interesting aspect is that Facebook’s MAU growth looks pretty much like a straight line, and so the % growth has been slowing down as of late. The MAU growth was around 23% starting in 2009, but is now down to 6-7% in recent months. See below for a graph on MAU vs % MAU growth)

After writing my recent article on Growth Hackers, I’ve been asked by quite a few folks on how to learn the discipline. The best answer is, learn from someone who’s already good at it – if you’re technical and creative, it’s well worth the time.

I would encourage everyone to also read Andy Johns’s Quora answers on What is Facebook’s User Growth team responsible for and what have they launched? and
What are some decisions taken by the “Growth team” at Facebook that helped Facebook reach 500 million users?– it lays out a lot of the key activities used in a well-run growth team.

The list below includes some of these folks I know personally, some just by reputation- but collectively they’ve grown products up to millions, 10s of millions, and in some cases, 100M+ users. Typically they use quantitatively-oriented techniques centered on virality across different channels such as iOS, Facebook, email, etc. There’s lots of iteration, A/B testing, and experimentation involved. There’s also really great growth hackers centered around SEO, SEM/ad arb, and other techniques, but for the most part I’m just listing out the folks around quant-based virality. The important thing about virality is, it’s free :) So it’s an important skill for startups.

Missing from this list are many unsung heroes over at Zynga, Dropbox, Branchout, Viddy/Socialcam, lots of ex-Paypal/Slide people, etc., etc. Also, all of these guys typically have co-founders or entire growth teams around them that are experts, even if I don’t know them by name.

If others in the community would like to make suggestions, tweet me at @andrewchen or just reply in the comments.

UPDATE: My friend Dan Martell’s new company, Clarity, provides a way to access experts like this via phone and email. Here’s the directory of folks with expertise on growth.

The rise of the Growth Hacker
The new job title of “Growth Hacker” is integrating itself into Silicon Valley’s culture, emphasizing that coding and technical chops are now an essential part of being a great marketer. Growth hackers are a hybrid of marketer and coder, one who looks at the traditional question of “How do I get customers for my product?” and answers with A/B tests, landing pages, viral factor, email deliverability, and Open Graph. On top of this, they layer the discipline of direct marketing, with its emphasis on quantitative measurement, scenario modeling via spreadsheets, and a lot of database queries. If a startup is pre-product/market fit, growth hackers can make sure virality is embedded at the core of a product. After product/market fit, they can help run up the score on what’s already working.

This isn’t just a single role – the entire marketing team is being disrupted. Rather than a VP of Marketing with a bunch of non-technical marketers reporting to them, instead growth hackers are engineers leading teams of engineers. The process of integrating and optimizing your product to a big platform requires a blurring of lines between marketing, product, and engineering, so that they work together to make the product market itself. Projects like email deliverability, page-load times, and Facebook sign-in are no longer technical or design decisions – instead they are offensive weapons to win in the market.

The stakes are huge because of “superplatforms” giving access to 100M+ consumers
These skills are invaluable and can change the trajectory of a new product. For the first time ever, it’s possible for new products to go from zero to 10s of millions users in just a few years. Great examples include Pinterest, Zynga, Groupon, Instagram, Dropbox. New products with incredible traction emerge every week. These products, with millions of users, are built on top of new, open platforms that in turn have hundreds of millions of users – Facebook and Apple in particular. Whereas the web in 1995 consisted of a mere 16 million users on dialup, today over 2 billion people access the internet. On top of these unprecedented numbers, consumers use super-viral communication platforms that rapidly speed up the proliferation of new products – not only is the market bigger, but it moves faster too.

Before this era, the discipline of marketing relied on the only communication channels that could reach 10s of millions of people – newspaper, TV, conferences, and channels like retail stores. To talk to these communication channels, you used people – advertising agencies, PR, keynote speeches, and business development. Today, the traditional communication channels are fragmented and passe. The fastest way to spread your product is by distributing it on a platform using APIs, not MBAs. Business development is now API-centric, not people-centric.

Whereas PR and press used to be the drivers of customer acquisition, instead it’s now a lagging indicator that your Facebook integration is working. The role of the VP of Marketing, long thought to be a non-technical role, is rapidly fading and in its place, a new breed of marketer/coder hybrids have emerged.

Airbnb, a case study
Let’s use case of Airbnb to illustrate this mindset. First, recall The Law of Shitty Clickthroughs:

Over time, all marketing strategies result in shitty clickthrough rates.

The converse of this law is that if you are first-to-market, or just as well, first-to-marketing-channel, you can get strong clickthrough and conversion rates because of novelty and lack of competition. This presents a compelling opportunity for a growth team that knows what they are doing – they can do a reasonably difficult integration into a big platform and expect to achieve an advantage early on.

Airbnb does just this, with a remarkable Craigslist integration. They’ve picked a platform with 10s of millions of users where relatively few automated tools exist, and have created a great experience to share your Airbnb listing. It’s integrated simply and deeply into the product, and is one of the most impressive ad-hoc integrations I’ve seen in years. Certainly a traditional marketer would not have come up with this, or known it was even possible – instead it’d take a marketing-minded engineer to dissect the product and build an integration this smooth.

Here’s how it works at a UI level, and then we’ll dissect the technology bits:

(This screenshots are courtesy of Luke Bornheimer and his wonderful answer on Quora)

Looks simple, right? The impressive part is that this is done with no public Craigslist API! It turns out, you have to look closely and carefully at Craigslist in order to accomplish an integration like this. Note that it’s 100X easier for me to reverse engineer something that’s already working versus coming up with the reference implementation – and for this reason, I’m super impressed with this integration.

Reverse-engineering “Post to Craigslist”
The first thing you have to do is to look at how Craigslist allows users to post to the site. Without an API, you have to write a script that can scrape Craigslist and interact with its forms, to pre-fill all the information you want.

The first thing you can notice from playing around with Craigslist is that when you go to post something, you get a unique URL where all your information is saved. So if you go to https://post.craigslist.org you’ll get redirected to a different URL that looks like https://post.craigslist.org/k/HLjRsQyQ4RGu6gFwMi3iXg/StmM3?s=type. It turns out that this URL is unique, and all information that goes into this listing is associated to this URL and not to your Craigslist cookie. This is different than the way that most sites do it, where a bunch of information is saved in a cookie and/or server-side and then pulled out. This unique way of associating your Craigslist data and the URL means that you can build a bot that visits Craigslist, gets a unique URL, fills in the listing info, and then passes the URL to the user to take the final step of publishing. That becomes the foundation for the integration.

At the same time, the bot needs to know information to deal with all the forms – beyond filling out the Craigslist category, which is simple, you also need to know which geographical region to select. For that, you’d have to visit every Craigslist in every market they serve, and scrape the names and codes for every region. Luckily, you can start with the links in the Craiglist sidepanel – there’s 100s of different versions of Craigslist, it turns out.

If you dig around a little bit you find that certain geographical markets are more detailed than others. In some, like the SF Bay Area, there’s subareas (south bay, peninsula, etc.) and neighborhoods (bernal, pacific heights) whereas in other markets there’s only subareas, or there’s just the market. So you’d have to incorporate all of that into your interface.

Then there’s the problem of the listing itself – by default, Craigslist works by giving you an anonymous email address which you use to communicate to potential customers. If you want to drive them to your site, you’d have to notice that you can turn off showing an email, and just provide the “Contact me here” link instead. Or, you could potentially fill a special email address like listing-29372@domain.com that automatically directs inquiries to the right person, which can be done using services like Mailgun or Sendgrid.

Finally, you’ll want the listing to look good – it turns out Craigslist only supports a limited amount of HTML, so you’ll need to work to make your listings work well within those constraints.

Completing the integration is only the beginning – once it’s up, you’d have to optimize it. What’s the completion % once sometime starts sharing their listing out to Craigslist? How can you change the flow, the call to action, the steps in the form, to increase this %? And similarly, when people land from Craigslist, how do you make sure they are likely to complete a transaction? Do they need special messaging?

Tracking all of this requires additional work with click-tracking with unique URLs, 1×1 GIFs on the Craigslist listing, and many more details.

Long story short, this kind of integration is not trivial. There’s many little details to notice, and I wouldn’t be surprised if the initial integration took some very smart people a lot of time to perfect.

No traditional marketer would have figured this out
Let’s be honest, a traditional marketer would not even be close to imagining the integration above – there’s too many technical details needed for it to happen. As a result, it could only have come out of the mind of an engineer tasked with the problem of acquiring more users from Craigslist. Who knows how much value Airbnb is getting from this integration, but in my book, it’s damn impressive. It taps into a low-competition, huge-volume marketing channel, and builds a marketing function deeply into the product. Best of all, it’s a win-win for everyone involved – both the people renting out their places by tapping into pre-built demand, and for renters, who see much nicer listings with better photos and descriptions.

This is just a case study, but with this type of integration, a new product is able to compete not just on features, but on distribution strategy as well. In this way, two identical products can have 100X different outcomes, just based on how well they integrate into Craigslist/Twitter/Facebook. It’s an amazing time, and a new breed of creative, technical marketers are emerging. Watch this trend.

So to summarize:

• For the first time ever, superplatforms like Facebook and Apple uniquely provide access to 10s of millions of customers
• The discipline of marketing is shifting from people-centric to API-centric activities
• Growth hackers embody the hybrid between marketer and coder needed to thrive in the age of platforms
• Airbnb has an amazing Craigslist integration

Good luck, growth hackers!

I was reading today’s NYT article on Google+’s new redesign and found myself continually puzzled by the key metric Google continues to report as the success of their new social product: Registered Users.

In the very first sentence, Vic Gundotra writes:

More than 170 million people have upgraded to Google+, enjoying new ways to share in Search, Gmail, YouTube and lots of other places.

The use of registered users is a vanity metric, and reflects how easily Google can cross-sell any new product to their core base of 1 billion uniques per month. What it doesn’t reflect, however, is the actual health of the product.

Ultimately, this misalignment of metrics is due to the curse of instant distribution. Because Google can cross-sell whatever products they want against their billion unique users, it’s easy to grade on that effort. Plus it’s a big number, who doesn’t love a big number?

Google+ should be measured on per user metrics
Here’s what metrics are more important instead: Given the Google+ emphasis on Circles and Hangouts, you’d think that the best metrics to use would evaluate the extent to which these more personal and more authentic features are being used. These would include metrics like:

• Shares per user per day (especially utilizing the Circles feature)
• Friends manually added to circles per user per day (not automatically!)
• Minutes of engagement per user per day

Point is, the density and frequency of relationships within small circles ought to matter more than the aggregate counts on the network. As I’ve blogged about before, you use metrics to reflect the strategy you already have in place, and based on the Google+’s focus on authentic circles of friends, you’d think the metrics would focus on the density of friendships and activities, and not the aggregate numbers.

The curse of instant distribution
Every new product for a startup goes through a gauntlet to reach product/market fit, and then traction. In the real world, product quality and the ability to solve a real problem for people ends up correlating with your ability to distribute the product. Google+ is blessed, and cursed, with the ability to sidestep this completely. They are able to onboard hundreds of millions of users without having great product/market fit, and can claim positive metrics without going through the gauntlet of really making their product work.

Adam D’Angelo of Quora (and previously CTO of Facebook) wrote this insightful commentary regarding Google Buzz a while back:

Why have social networks tied to webmail clients failed to gain traction?
Personally I think this is mostly because the social networking products built by webmail teams haven’t been very good. Even Google Buzz, which is way ahead of the attempts built into Yahoo Mail and Hotmail, has serious problems: the connections inside it aren’t meaningful, profiles and photos are second class, comments bump items to the top of the feed meaning there’s old stuff endlessly getting recycled, and the whole product itself is a secondary feature accessible only through a click below the inbox, which hasn’t gotten it enough distribution to kick off and sustain conversations.

I’m pretty sure that if Google, Microsoft, or Yahoo had cloned Facebook almost exactly (friends, profiles, news feed, photos) and integrated it well into their webmail product, that it could have taken off (before Facebook got to its current scale; at this point it will be hard for any competitor, even with a massive distribution channel pushing it).

So I think this question is really, why are social networks that webmail teams build always bad? Here’s my guess:

• The team building the social network knows that they’re going to get a huge amount of distribution via the integration and so they aren’t focused on growth and making a product that people will visit on their own.
• Integrating any two big products is really hard.
• Any big webmail provider is going to have a big organization behind it, and lots of politics and compromises probably make it difficult to execute well.
• Teams that work on webmail products have gotten good at building a webmail product, and haven’t selected for the skills and culture that a team that grows around building a social network will have.

(The bolding is from me). I couldn’t agree more with this answer. I think a key lesson behind the recent success of products like Instagram and Pinterest is that there’s still a lot of room in the market for great social products to take off- but the emphasis has to be on the product rather than the superficial act of onboarding a lot of new users into Google+.

Ultimately, it comes down to how realistic the Google+ folks are in looking at their metrics. If they drink their own kool-aid and think they have product/market fit when it’s in fact the traction is solely dependent on the power of their distribution channels, they may never get their product working.

On the other hand, if they have a balanced view on their metrics and know they don’t have product/market fit yet, then they have a fighting chance. Unfortunately, I think the changes they’ve made to the product recently are more efforts to optimize, rather than fundamental improvements to the product. I think Google+ needs much bigger changes to make it as engaging as the best social products.

The first banner ad ever, on HotWired in 1994, debuted with a clickthrough rate of 78% (thanks @ottotimmons)

First it works, and then it doesn’t
After months of iterating on different marketing strategies, you finally find something that works. However, the moment you start to scale it, the effectiveness of your marketing grinds to a halt. Sound familiar?

Welcome to the Law of Shitty Clickthroughs:

Over time, all marketing strategies result in shitty clickthrough rates.

Here’s a real example – let’s compare the average clickthrough rates of banner ads when debuted on HotWired in 1994 versus Facebook in 2011:

• HotWired CTR, 1994: 78%
• Facebook CTR, 2011: 0.05%

That’s a 1500X difference. While there are many factors that influence this difference, the basic premise is sound – the clickthrough rates of banner ads, email invites, and many other marketing channels on the web have decayed every year since they were invented.

Here’s another channel, which is email open rates over time, according to eMarketer:

While this graph shows a decline, the other graph (which I don’t have handy) is that the number of emails sent out has increased up to 30+ billion per day.

All these channels are decaying over time, and what’s saving us is the new marketing channels are constantly getting unveiled, too. These new channels offer high performance, because of a lack of competition, big opportunities for novel marketing techniques, and these days, the cutting edge is about optimizing your mobile notifications, not your banner placements.

There are a few drivers for the Law of Shitty Clickthroughs, and here’s a summary of the top ones:

• Customers respond to novelty, which inevitably fades
• First-to-market never lasts
• More scale means less qualified customers

Novelty
Without a doubt, one of the key drivers of engagement for marketing is that customers respond to novelty. When HotWired showed banner ads for the first time in history, people clicked just to check out the experience. Same for being the first web product to email people invites to a website – it works for a while, until your customers get used to the effect, and start ignoring it.

One of the most important tools you have at your disposal is the creative and calls to action that you use in your marketing – this might be like “X has invited you to Y” or it might be the headline you use in your banner ads. Recently, Retargeter posted an interesting analysis on the Importance of Rotating Creatives, which showed how keeping the same ad creative led to declining CTRs over time:

Publishers often have a similar problem in consumers ignoring the advertising on their site, which drives down clickthrough rates for both of them (bad for CPMs). This problem is often described as banner blindness, and you can see it clearly here in an eye-tracking study by Jakob Nielsen:

You can see here how users, almost comically, avoid looking at any banners.

The point is, humans seek novelty yet are pattern-recognition machines. Your initial marketing strategy will work quite well as your users try it for the first time, but afterwards, they learn to filter your marketing efforts out unless they are genuinely useful (more on that later).

First-to-market never lasts
It’s bad enough that your own marketing efforts drive down channel performance, but usually once your marketing efforts are working, your competitors quickly follow. There’s a whole cottage industry of companies that provide competitive research in the area of how their competitors are advertising and give you the information needed to fast-follow their marketing efforts.

For example, with a quick query, I know how much Airbnb is spending on search marketing (turns out, millions per year) what keywords they are buying ads on, and who their competitors are. And this is just a free service! There are much more sophisticated products for every established marketing channel:

Airbnb Search Engine Marketing

• Daily ad budget: $10,638
• Keywords: 62,729
• Example ad: Find Affordable Rooms Starting From $20/Day. Browse & Book Online Now!
• Main competitors: Expedia.com, booking.com, hotels.com, Marriott.com

Any clone of their business can quickly fast-follow their marketing efforts and use the same ads in the same marketing channels. This quickly degrades the performance of the marketing channel as the novelty wears off and clickthroughs plummet.

Any product that is first to market has a limited window where they will enjoy unnaturally high marketing performance, until the competition enters, in which case everyone’s marketing efforts will degrade.

More scale means less qualified customers
Another important way to think about the available market for your product is in terms of the popular Technology Adoption Lifecycle, in which early adopters actively seek out your product, while the rest of the mainstream market needs a lot of convincing. The quant marketing way to look at this is that early adopters respond better to marketing efforts across any given metric (signup %, CTR, CPA) than the later customer segments. In the TAL framework, the early market seeks out novelty, whereas the mainstream market just cares if you solve a problem for them.

As a result, a marketing strategy focused on early adopters is bound to look better than what you get later. You can get some limited traffic from PR and targeted advertising from niche communities and media properties. However once you get past this group, the CTRs can drop substantially.

If you’re a SaaS or ecommerce company that’s road-tested your marketing strategy by acquiring limited batches of customers, the problem is that whatever assumptions and projections you make off of this base end up fundamentally skewed positive. If your model indicates that you can acquire customers at $10 and break even within 6 months, it’s not hard for a 30% increase in CAC and 30% decrease in LTV to double the time it takes to get to profitability. This could be the difference between life and death for a company.

Lesson to investors is: Beware marketing metrics done at a small scale, and beware marketing tech companies that facilitate momentary marketing opportunities without a bigger vision. These are arbitrage opportunities that will disappear over time.

How to fight the Law of Shitty Clickthroughs
I call it a Law, of course, because I really believe it’s a strong gravitational pull on all marketing on the web. You can’t avoid it, and in many ways, it’s counter productive to try.

You can always get incrementally better performance out of your marketing by taking a nomad strategy – always keep developing new creative, testing new publishers, and so on. That’s all easy, but is mostly about maintaining some base level of performance. This can push the Law of Shitty Clickthroughs to act over years rather than degrading your marketing efforts over months.

Similarly, this law provides a litmus test as to the difference between advertising and information. When you are marketing with useful information, then CTRs stay high. Advertising that’s just novelty and noise wrapped in a new marketing channel has a limited shelf life.

The real solution: Discover the next untapped marketing channel
The 10X solution to solving the Law of Shitty Clickthroughs, even momentarily, is to discover the next untapped marketing channel. In addition to doubling down on traditional forms of online advertising like banners, search, and email, it’s important to work hard to get to the next marketing channel while it’s uncontested.

Sometimes I get asked “have you ever seen someone do XYZ to acquire customers?” Turns out, the highest vote of confidence I can give is, “No I haven’t, and that’s good – that means there’s a higher chance of it working. You should try it.”

Today, these (relatively) uncontested marketing channels are Open Graph, mobile notifications, etc. If you can make these channels work with a strong product behind it, then great. Chances are, you’ll enjoy a few months if not a few years of strong marketing performance before they too, slowly succumb.

I had a nerdy conversation on what might be the next mainstream framework for building web products, and in particular whether the node.js community would ultimately create this framework, or if node.js will just be a fad. This blog post is a bit of a deviation from my usual focus around marketing, so just ignore if you have no interest in the area.

Here’s the summary:

• Programming languages/frameworks are like marketplaces – they have network effects
• Rails, PHP, and Visual Basic were all successful because they made it easy to build form-based applications
• Form-based apps are a popular/dominant design pattern
• The web is moving to products with real-time updates, but building real-time apps hard
• Node.js could become a popular framework by making it dead simple to create modern, real-time form-based apps
• Node.js will be niche if it continues to emphasize Javascript purity or high-scalability

The longer argument below:

Large communities of novice/intermediate programmers are important
One of the biggest technology decisions for building a new product is the choice of development language and framework. Right now for web products, the most popular choice is Ruby on Rails – it’s used to build some of the most popular websites in the world, including Github, Scribd, Groupon, and Basecamp.

Programming languages are like marketplaces – you need a large functional community of people both demanding and contributing code, documentation, libraries, consulting dollars, and more. It’s critical that these marketplaces have scale – it needs to appeal to the large ecosystem of novices, freelancers and consultants that constitute the vast majority of programmers in the world. It turns out, just because a small # of Stanford-trained Silicon Valley expert engineers use something doesn’t guarantee success.

Before Rails, the most popular language for the web was PHP, which had a similar value proposition – it was easy to build websites really fast, and it was used by a large group of novice/intermediate programmers as well. This includes a 19-yo Mark Zuckerberg to build the initial version of Facebook. Although PHP gained the reputation of churning out spaghetti code, the ability for people to start by writing HTML and then start adding application logic all in one file made it extremely convenient for development.

And even before Rails and PHP, it was Visual Basic that engaged this same development community. It appealed to novice programmers who could quickly set up an application by dragging-and-dropping controls, write application logic with BASIC, etc.

I think there’s a unifying pattern that explains much of the success of these three frameworks.

The power of form-based applications
The biggest “killer app” for all of these languages is how easy it is to build the most common application that mainstream novice-to-intermediate programmers are paid to build: Basic form-based applications.

These kinds of apps let you do a some basic variation of:

• Give the user a form for data-entry
• Store this content in a database
• Edit, view, and delete entries from this database

It turns out that this describes a very high % of useful applications, particularly in business contexts including addressbooks, medical records, event-management, but also consumer applications like blogs, photo-sharing, Q&A, etc. Because of the importance of products in this format, it’s no surprise one of Visual Basic’s strongest value props was a visual form building tool.

Similarly, what drove a lot of the buzz behind Rails’s initial was a screencast below:

How to build a blog engine in 15 min with Rails (presented in 2005)

Even if you haven’t done any programming, it’s worthwhile to watch the above video to get a sense for how magical it is to get a basic form-based application up and running in Rails. You can get the basics up super quickly. The biggest advantages in using Rails are the built-in data validation and how easy it is to create usable forms that create/update/delete entries in a database.

Different languages/frameworks have different advantages – but easy form-based apps are key
The point is, every new language/framework that gets buzz has some kind of advantage over others- but sometimes these advantages are esoteric and sometimes they tap into a huge market of developers who are all trying to solve the same problem. In my opinion, if a new language primarily helps solve scalability problems, but is inferior in most other respects, then it will fail to attract a mainstream audience. This is because most products don’t have to deal with scalability issues, though there’s no end to programmers who pick technologies dedicated to scale just in case! But much more often than not, it’s all just aspirational.

Contrast this to a language lets you develop on iOS and reach its huge audience – no matter how horrible it is, people will flock to it.

Thus, my big prediction is:

The next dominant web framework will be the one that allows you to build form-based apps that are better and easier than Rails

Let’s compare this idea with one of the most recent frameworks/languages that has gotten a ton of buzz is node.js. I’ve been reading a bit about it but haven’t used it much – so let me caveat everything in the second half with my post with that. Anyway, based on what I’ve seen there’s a bunch of different value props ascribed to its use:

• Build server-side applications with Javascript, so you don’t need two languages in the backend and frontend
• High-performance/scalability
• Allows for easier event-driven applications

A lot of the demo applications that are built seem to revolve around chat, which is easy to build in node but harder to build in Rails. Ultimately though, in its current form, there’s a lot missing from what would be required for node.js to hit the same level of popularity as Rails, PHP, or Visual Basic for that. I’d argue that the first thing that the node.js community has to do is to drive towards a framework that makes modern form-based applications dead simple to build.

What would make a framework based on node.js more mainstream?
Right now, modern webapps like Quora, Asana, Google Docs, Facebook, Twitter, and others are setting the bar high for sites that can reflect changes in data across multiple users in real-time. However, building a site like this in Rails is extremely cumbersome in many ways that the node.js community may be able to solve more fundamentally.

That’s why I’d love to see a “Build a blog engine in 15 minutes with node.js” that proves that node could become the best way to build modern form-based applications in the future. In order to do this, I think you’d have to show:

• Baseline functionality around scaffolding that makes it as easy as Rails
• Real-time updates for comment counts, title changes, etc that automatically show across any viewers of the blog
• Collaborative editing of a single blog post
• Dead simple implementation of a real-time feed driving the site’s homepage

All of the above features are super annoying to implement in Rails, yet could be easy to do in node. It would be a huge improvement.

Until then, I think people will still continue to mostly build in Rails with a large contingent going to iOS – the latter not due to the superiority of the development platform, but rather because that’s what is needed to access iOS users.

UPDATE: I just saw Meteor on Hacker News which looks promising. Very cool.

Will CPE (Cost Per Engagement) advertising ever take off?
I doubt it – the reason is that it’s targeting metrics at the kind of marketers that don’t care too much about metrics.

Broadly speaking, there’s two kind of marketers in the world – a ton could be written about this, so I’ll just provide some sweeping generalizations:

Direct response marketers are companies that are typically very focused on ROI when they buy advertising – often these include companies you’ve never heard of in ecommerce, online dating, financial services, etc., where it’s easy to calculate the value of a customer and they are primarily getting their traffic through paid marketing channels. They like to back everything out to ROI by comparing lifetime value to cost per customer, and if not that, then at least cost-per-action or some similarly concrete metric.

In many cases, these kinds of marketers prefer search marketing, email marketing, telesales, and other things where it’s easy to quantify what’s going on – they stay away from Super Bowl ads though. They prefer CPA and CPC versus CPM or sponsorships.

Brand marketers are companies you’ve heard of and have seen a lot of advertising for – they are typically targeting a large consumer base, they want to position their products differently relative to their competition and don’t have great ways to quantify the value of a customer. For example, Coca-Cola doesn’t know the LTV of a customer nor what the cost-per-customer looks like for a billboard ad they’ve bought.

For these guys, they are used to hiring big ad agencies to help them advertise on billboards, television, the front page of Yahoo, etc. They may buy search marketing, but have different goals than ROI. (For example, they may just want the top ad, and don’t care too much about ROI)

Why CPE is a weird metric for both DR and brands
The reason why cost-per-engagement is a weird metric is that ROI-focused marketers (that is, direct response marketers), don’t care about “engagement.” They want to know if people are going to buy, and if their media spend is going to be profitable.

As a result, the “E” part of CPE is really only a part that brands care about. And yet, they don’t care that much about CPE because they aren’t focused on the cost of the campaign as the #1 priority. Instead, it’s more important where the ads are being placed, how strong the ad creative is being used, etc.

One scenario to demonstrate this: If they could buy the front page of YouTube, even if that had a higher CPE, a brand advertiser would be happier with that than being shown in random footers of YouTube (the “remnant”) even at a lower CPE. They are looking to establish their brand, not optimize their spend.

What will be prevalent instead?
I think even with the advent of lots of ad opportunities on social sites, the dominate business model will still be CPM/sponsorships for brand advertisers, and CPC/CPA for direct response. Basically, nothing much will change.

If it turns out that CPE correlates to CPA/CPC, then DR marketers will end up liking it.

Also, CPE might turn into a secondary metric that you use alongside really strong placement of ads- maybe as a way to establish a bonus or upside on the campaign, but I don’t think it’ll ever happen that the dominant form of advertising on the web will be that ad agencies will put in a CPE “bid” into self-serve systems :)

I answered this question on Quora – more great answers over there.

Facebook, early 2006
Sometimes, you need to be horribly, embarrassingly wrong to remind yourself to keep an open mind. This is my story of my failure to understand Facebook’s potential.

In 2006, I was working on a new ad network business that experimented a lot with targeting ads with social network data, broadly known as “retargeting” now. The idea was that we’d be able to take your interests and target advertising towards them, which would lead to higher CPMs. As part of this project, we did a meeting with Facebook when they were ~12 people. I had read bits and pieces about the company in the news, but since I was a few years out of college, I hadn’t used their product much. We got a meeting and since I was based in Seattle at the time, I flew down with some coworkers and chatted with them at their new office in Palo Alto.

We met the Facebook team at their office right next to the Sushitomo on University Ave. The place looked like a frat house – a TV and video game console on the ground, clothes and trash everywhere – the result of a handful of young people working very hard. After waiting a few minutes, we were escorted into a meeting room where we met with Sean Parker, Matt Cohler, and Mark Zuckerberg. Sean led the meeting, and told us a lot about Facebook, the amazing job he did raising their recent VC round from Accel, and all the good things that were happening at the company. Mark and the other folks there didn’t say a thing.

Ultimately, we didn’t get to work with them though we did eventually sign 1000s of publishers including MySpace, AOL, Wall St. Journal, NY Times, and others. But that meeting opened my eyes and convinced me of a horribly wrong thing: Facebook would never be a billion dollar company.

The metrics for Facebook – high growth, very low CPMs
As part of our meeting, we talked a bit about the metrics around Facebook, and I was immediately struck by a few things:

• Facebook was growing fast- very fast, and impressively handled by a super young team (like me!) sitting on a site with millions of uniques/month
• Their CPMs were terrible, lower than $0.25 (the revenue earned per thousand ad impressions) and the site was covered (at the time) with crappy remnant ads like online poker, dating, mortgages, etc. (ironically, which we now associate with MySpace)
• They didn’t know much about advertising, and that their CPMs were really bad and unlikely to improve- their monetization strategy seemed superficial at best

From these numbers, I did a quick calculation:

$0.25 CPM * 5 billion ad impressions per month max?
= $1.25M/month = $15M/year = $150-300M value business?

I figured that Facebook hitting 5B ads/month would be incredible – after all, it was just a college social network, right? Hitting 5B impressions/month would make Facebook bigger than our largest client at the time, ESPN.com, a top 10 internet property. The only thing larger were big portals like Yahoo, MSN, and AOL. The idea that Facebook would one day be bigger than all the portals never crossed my mind.

I was confident especially in the CPM number staying low because I had multiple proprietary datapoints from across the industry – from MySpace, Friendster, Hi5, Dogster, and many other social networks. I was convinced that I had a unique understanding about Facebook’s true potential – that convinced me even more that it could never be big.

And of course, I was totally, horribly wrong :)

The case at Yahoo for buying Facebook
While I was doing these calculations after my meeting, Yahoo was also doing a similar analysis on the value of Facebook for their ill-fated attempt at buying the company. I would first read about it in the WSJ, but later saw this fascinating slide on Techcrunch.

The slide below starts out with a projection of how many registered users Facebook had at the time and projected very logically what it would mean for them to saturate more of the core userbase of “high school and young adult” – I’m sure at the time, these felt like aggressive projections to ultimately be able to justify a big purchase price:

If you look at these numbers and compare them to what really happened, it’s pretty hilarious. Comparing their projected 2010E and what actually happened, they were only off by a few hundred million users!

Furthermore, I would say that even the Yahoo numbers were very optimistic about the increase from CPM going from $0.25 to >$5 over time. There were a lot of problems with brand advertisers putting themselves next to user-generated content that had not been worked out, and these numbers would have also ultimately involved Facebook doing homepage takeovers and such. And in fact, it’s true that no large user-generated content or social networking site has been able to generate CPMs close to the $5 level, at scale.

So what was wrong with my reasoning?
Ultimately, all my conclusions were wrong by several orders of magnitude – Facebook would go on to become the #1 site on the internet and would break all attempts at reasoning based on historical datapoints, interpolation, expert opinions, etc.

To contrast how silly my reasoning turned out to be:

My 2006 prediction: Facebook would max out at 3-5B pageviews/month
Reality: Facebook is at 1 trillion pageviews/month, and growing

I was ultimately right on the CPMs not improving by much, but it didn’t matter because I was off by 200-300x on pageviews/month! Total fail. The big insight, of course, was that Facebook wouldn’t just stay a social network for college students – ultimately the product targeted the market of everyone in the world. Confined within this the college niche, the idea that Facebook would one day reach a trillion pageviews per month seemed ludicrous. But because of the vision of the founding team, Facebook broke through this niche to build a new product that the world had never seen, and got to the numbers I had never predicted.

The most exceptional cases defy simple pattern-matching
As I mentioned in my previous post on group think vs innovation in Silicon Valley, there’s a strange contradiction between the mental tools we use to analyze and categorize businesses versus what it looks like when there’s an exceptional company that takes off. Pattern matching, deductive reasoning, and expert opinion tell you how things work in the “typical” case, but of course, we’re not interested in the typical case – we’re trying to find the exceptional ones, the rocketship companies that define the startup landscape.

That’s exactly when our logical reasoning and historically-based reasoning fails us the most.

For example, after years of failures from the entire category of social shopping sites like ThisNext, Kaboodle, and others, Pinterest has become the hottest company of the year. After years of Google impressing upon all of us that every startup needed to have an algorithm called X-rank and a 10X technology advantage, a simplistic webapp known as Twitter would emerge. And after 10 VC-funded search companies were started, and people at Yahoo thought search was a loss-leading feature that would best be outsourced, Google emerged. The list goes on and on.

Legendary VC Mike Moritz, who invested in Google/Yahoo/PayPal/Apple/etc has a relevant quote here:

I rarely think about big themes. The business is like bird spotting. I don’t try to pick out the flock. Each one is different and I try to find an interestingly complected bird in a flock rather than try to make an observation about an entire flock. For that reason, while other firms may avoid companies because they perceive a certain investment sector as being overplayed or already mature, Sequoia is “careful not to redline neighborhoods.

There’s a lot to be said for investing in the ugly duckling. When Don Valentine led Sequoia Capital’s investment in Cisco, many others had passed on the husband and wife founding team of Len Bosack and Sandy Lerner.

Never has a more profound thing been said about birdspotting :)

The biggest lesson I took away
The concrete lesson to be learned from this is: In the modern era, business models are a commodity. I never want to hear about people asking, “But what’s their business model?” because in a world where you can grow a userbase of 1 billion in a few years, displaying remnant ads and getting a $0.25 CPM will do. Or just throw some freemium model on it, and monetize 1% of them. If you can build the audience, you can build a big business.

The more abstract lesson to learn is: Be humble, and keep an open mind towards weird new companies. After a few years in Silicon Valley, you can gather a lot of useful heuristics about what’s worked and what doesn’t work. That will help you most of the time, but when it comes to the exceptional cases, all bets are off. So keep your mind open to weird, young companies that you meet that don’t fit the established pattern: Maybe the founders will all be recent MBAs, or be a spinout from a stodgy old corporation. Or maybe it’ll be in a slow-moving market, or it’ll be a married couple, or there’s 10 founders, or some other stereotypically bad thing. Remember that you’re helping/investing/working for the company right in front you, not a mutual fund of all companies with that characteristic!

If you had looked at social networking companies as a group, as I did, you would have found a flock of companies with questionable business models. However, if you had been prescient enough to pick out Facebook specifically, then you would have seen a company break through all historic precedents and become a huge success. Hats off to all 12 employees I met that day in 2006.

Once you’ve been working in Silicon Valley for a bit, you’re often offered advice such as:

• Are you launching at X conference?  … where X is whatever hot conference is coming up, like SXSW or Launch
• Do you have an X app? … where X is whatever new platform just emerged, be it Open Social, iPhone, or whatever
• Have you pitched X venture capitalist? … where X is a prolific headline-grabbing investor with a recent hot deal
• You should do feature X that company Y does! … where X is some sexy (but possibly superficial feature) that a hot startup has done
• Do you know what your X metric is? … where X is some metric a recent blog post was written about
• Have you met X? … where X is some highly connected expert in the field
• Maybe you should pivot into X space! … where X is a space with a hot company that just raised a ton of funding
• Did you think about applying framework X to this? … where X is a new framework, be it gamification or viral loops or Lean

Sound familiar? I confess that I’ve both received and given much advice along the lines of the above. I call it “advice autopilot.”

The perils of “advice autopilot”
Advice autopilot is when you’re too lazy to think originally about a problem, instead regurgitate whatever smart thing you read on Quora or Hacker News. If you’re a bit more connected, instead you might parrot back what’s being spoken at during Silicon Valley events and boardrooms, yet the activity is still the same – everyone gets the same advice, regardless of situation. The problem is, the best advice rarely comes in this kind of format – instead, the advice will start out with “it depends…” and takes into account an infinite array of contextual and situational things that aren’t obvious. However, we are all lazy and so instead we go on autopilot, and do, read, say, and build, all the same things.

That’s not to say that sometimes generic advice isn’t good advice – sometimes it is, especially for noob teams who are working off an incomplete set of knowledge. Often you may not have the answers, but the questions can lead to interesting conversations. You may not be able to say “you should do an iPhone app” but it’s definitely useful to ask, “how does mobile fit into this?” This can help a lot.

The other manifestation of this advice autopilot is the dreaded use of “pattern matching” to recommend solutions and actions.

Pattern-matching in a world of low probability, exceptional outcomes
One of Silicon Valley’s biggest contradictions is the love of two diametrically opposed things:

• The use of pattern-recognition to predict the future…
• … and the obsession with a small number of exceptional successes.

Exceptional outcomes for startups are limited – let’s say it’s really only 5-10 companies per year. In this group, you’d include companies like Facebook and Google that have “made it” and hit $100B valuations. On the emerging side, this would include startups who might ultimately have a shot at this, like Dropbox, Square, Airbnb, Twitter, etc. This is an extraordinarily small set of companies, and it isn’t much data.

The problem is, we’re hairless apes that like to recognize patterns, even in random noise. So as a result, we make little rules for ourselves – Entrepreneurs who are Harvard dropouts are good, but dropping out of Stanford grad school is even better. It’s good if they start a company in their 20s unless they’re Jeff Bezos. Being an alum of Google is good, but being an alum of Paypal is even better. Hardcore engineers as founders is good, but the list of exceptions is long: Airbnb, Pinterest, Zynga, Fab, and many others. And whatever you do, don’t fund husband-wife teams, unless they start VMWare or Cisco, in which case forget that piece of advice.

As anyone who’s taken a little statistics knows, when you have a small dataset and lots of variables, you can’t predict shit. And yet we try!

The intense focus on a small set of companies also introduces a well-known logical fallacy called Survivorship Bias. Here’s the Wikipedia page, it’s interesting reading. Basically, the idea is that we draw our pattern-recognition from well-publicized successful companies while ignoring the negative data from companies that might have done many of the same things, but end up with unpublicized failures. We’re all so intimately familiar with stories like “two PhDs from Stanford start Google” that we ignore all the cases where two PhDs from Stanford try to start a company and fail. Or similarly, YCombinator has built a great rep on companies like Airbnb and Dropbox, and yet you’d think that if you invest in 600+ startups that you’d get a few hits. Because of factors like this, it might seem as though A predicts B when in fact, it does nothing of the sort – we’re just not taking the entire dataset into account.

Conformity leads to average outcomes when we seek exceptional outcomes
The problem with giving and taking so much of the same advice is that ultimately it breeds conformity, which is another way if saying it reduces the variance in the outcome. And if you conform enough, you end up creating the average outcome:

The average outcome for entrepreneurs is, your startup fails.

Lets not forget that. And so one part of Naval Ravikant’s talk on fundable startups that resonated with me is the idea of playing to your extremes. He says in the talk:

“Investors are trying to find the exceptional outcomes, so they are looking for something exceptional about the company. Instead of trying to do everything well (traction, team, product, social proof, pitch, etc), do one thing exceptional. As a startup you have to be exceptional in at least one regard.” –Naval Ravikant @naval

Be extremely good at something, and invest in it disproportionately relative to your competition – this gives you the opportunity to actually create an extreme outcome. Otherwise, the average outcome doesn’t seem so good.

The flipside of innovation
The funny thing with all of this, of course, is that this is what innovation looks like. The remarkable ability for practical knowledge to disseminate amongst the Bay Area tech community is what makes it so strong. Before something becomes autopilot advice for a wide variety of people, often a small number of hard-working teams who know what they’re doing leverage it to great success. Follow those people, and you might find yourself successful – just like them.

So the billion dollar question is – how do you separate out trendy/junk advice from what really matters?

… well, it depends!

I recently dug through Favstar.fm and found a bunch of the tweets over the last few months that were saved/retweeted the most. Wanted to save them here for posterity:

Teardowns

• Facebook, Google, Twitter, eBay, YouTube, Wikipedia, Amazon, Hotmail, Blogger, Apple: How they used to look http://t.co/fL2zDHu0
• The Secret To Pinterest’s Astounding Success: A Brilliant Sign-Up Process You Should Copy http://t.co/AsGi9pBx
• Airbnb’s first pitch deck http://t.co/3BTSY6dO
• Android Gripes, Why do apps from the same company look worse on Android than on iPhone? http://bit.ly/h19EKL
• Why Angry Birds is so successful and popular: a cognitive teardown of the user experience http://bit.ly/dN3W3d

Compilations

• 11 Books Every Leader Should Read – Bob Sutton http://t.co/gwViu9DQ
• 5 Articles on Rapid Prototyping you Should Have Read – LaunchBit http://t.co/cF10LzKI
• Platforms and Networks: Managing Startups: Best Posts of 2011 http://t.co/mdeyv3wK
• 5 Former Design Trends That Aren’t Cool Anymore (So Stop Using Them) | Design Shack http://t.co/59prlt1N

Quotes

• “Success is like being pregnant: everybody congratulates you but nobody knows how many times you were fucked…” via @NatalieSEO
• “No matter how beautiful, no matter how cool your interface, it would be better if there were less of it.” –Alan Cooper (via @destraynor)
• Very appropriate for entrepreneurs: “A casual stroll through the lunatic asylum shows that faith does not prove anything.” -Nietzsche
• grid is the new feed, custom cover is the new custom background, real name is the new username, and repin/reblog is the new embed code.
• “The easiest way to get 1 million people paying is to get 1 billion people using” -Phil Libin, CEO Evernote http://bit.ly/f1SY7U

Media

• Steve Jobs – 25 years old, rare footage of him presenting about early Apple http://t.co/AiNbyW0S
• INFOGRAPHIC: How rich are the superrich? http://bit.ly/hxavXB
• INFOGRAPHIC: Carbs Are Killing You http://t.co/FMdZ5pEG
• Amazing PDF with lots of conversion %s to compare your product against http://bit.ly/e2t6Ip

Articles

• Bye Bye, Long Tail http://tcrn.ch/gfTOYP
• interesting lesson on listening to customer self-reporting: Walmart’s $1.85 billon dollar mistake – Daily Artifacts http://t.co/8GLLajdx
• Design is Horseshit! http://t.co/12vw2MZN
• If You’re Competing On Features You’ve Already Lost http://bit.ly/jbfk8p
• My answer on Quora to: What are the best metrics for measuring user engagement? http://qr.ae/OyAD
• i find myself explaining scalable startups vs smallbiz / lifestyle all the time. This blog breaks it down. Via @sgblank http://t.co/DJNSkTx
• Why Do Some People Learn Faster? | Wired Science | Wired.com http://t.co/Zd53Z3Bl
• The Top Ten Signs the Valley is on Tilt Again http://bit.ly/fjZUhk

Here’s the form as a link if you can’t see it – questions, thoughts, etc appreciated.

Lobby of the Pixar offices in Emeryville, CA

“My product isn’t quite there yet.”
You’ve said this before. We all have.

Anyone working on getting their first product out to market will often have the feeling that their product isn’t quite ready. Or even once it’s out and being used, nothing will seem as perfect as they could be, and if you only did X, Y, and Z, then it woould be a little better. In a functional case, this leads to a great roadmap of potential improvements, and in a dysfunctional case, it leads to unlaunched products that are endlessly iterated upon without a conclusion.

About a year ago I visited Pixar’s offices and learned a little about this product, and I wanted to share this small story below:

Over at Pixar…
Matt Silas (@matty8r), a long-time Pixar employee offered to take me on a tour of their offices and I accepted his gracious offer. After an hour-long drive from Palo Alto to Emeryville, Matt showed up while I was admiring a glass case full of Oscars, and started full tour. I didn’t take great photos, so here’s some better ones so you can see what it’s like: Venturebeat, Urbanpeak.

I’ve always been a huge fan of Pixar – not just their products, but also their process and culture. There’s a lot to say about Pixar and their utterly fascinating process for creating movies, and I’d hugely recommend this book: To Infinity and Beyond. It gave me a kick to know that Pixar uses some very collaborative and iterative methods for making their movies – after all, a lot of what they do is software. Here’s some quick examples:

• Pixar’s teams are ultimately a collaboration of creative people and software engineers. This is reflected at the very top by John Lasseter and Ed Catmull
• The process of coming up with a Pixar movie starts with the story – then the storyboard – then many other low-fidelity methods to prototype what they are ultimately make
• They have a daily “build” of their movies in progress so they know where they stand, with sketches and crappy CGI filling holes where needed – compare this to traditional moviemaking where it’s only at the end
• Sometimes, as with the original version of Toy Story, they have to stop doing what they’re doing and restart the entire moviemaking process since the whole thing isn’t clicking – sound familiar, right?

The other connection to the tech world is that Steve Jobs personally oversaw the design of their office space. Here’s a great little excerpt on this, from director Brad Bird (who directed The Incredibles):

“Then there’s our building. In the center, he created this big atrium area, which seems initially like a waste of space. The reason he did it was that everybody goes off and works in their individual areas. People who work on software code are here, people who animate are there, and people who do designs are over there. Steve put the mailboxes, the meetings rooms, the cafeteria, and, most insidiously and brilliantly, the bathrooms in the center—which initially drove us crazy—so that you run into everybody during the course of a day. [Jobs] realized that when people run into each other, when they make eye contact, things happen. So he made it impossible for you not to run into the rest of the company.”

Anyway, I heard a bunch of stories like this and more – and as expected, the tour was incredible, and near the end, we stopped at the Pixar gift shop.

There, I asked Matt a casual question that had an answer I remember well, a year later:

Me: “What’s your favorite Pixar movie?”
Matt: *SIGH*
Me: “Haha! Why the sigh?”
Matt: “This is such a tough question, because they are all good. And yet at the same time, it can be hard to watch one that you’ve worked on, because you spend so many hours on it. You know all the little choices you made, and all the shortcuts that were taken. And you remember the riskier things you could have tried but ended up not, because you couldn’t risk the schedule. And so when you are watching the movie, you can see all the flaws, and it isn’t until you see the faces of your friends and family that you start to forget them.”

Wow! So profound.

A company like Pixar, who undoubtedly produces some of the most beloved and polished experiences in the world, ultimately still cannot produce an outcome where everyone on the team thinks it is the best. And after thinking about why, the reason is obvious and simple – to have the foresight and the skill to refine something to the point of making it great also requires the ability to be hugely critical. More critical, I think, than your ability to even improve or resolve the design problems fast enough. And because design all comes to making a whole series of tradeoffs, ultimately you don’t end up having what you want.

The lesson: You’ll always be unhappy
What I took away from this conversation is that many of us working to make our products great will never be satisfied. A great man once said, your product is shit – and maybe you will always think it is. Yet at the same time, it is our creative struggle with what we do that ultimately makes our creations better and better. And one day, even if you still think your product stinks, you’ll watch a customer use it and become delighted.

And for a brief moment, you’ll forget what it is that you were unhappy about.

Special thanks to Matt Silas (@matty8r, follow him!) for giving me a unique experience at Pixar. (Finally, I leave you with a photo of me posing next to Luxo Jr.)

Quick blog post to congratulate my sister Ada Chen and her husband Sachin Rekhi, who have just announced the acquisition of their startup Connected to LinkedIn. The company was backed by a seed investment from Ignition Partners and Trinity Ventures.

Here’s an excerpt from AllThingsD:

LinkedIn has acquired Connected, a small contact management startups that unifies and dynamically updates users’ connections on email, social networks, calendars and phones, according to sources close to the company.

Connected is similar to Xobni/Smartr, but it’s more of a dashboard than a plug-in, and it costs $9.99 per month. The company had raised a seed round of $500,000 led by Trinity Ventures in June. The service has been called “bloody awesome” by Tim O’Reilly.

Ada and Sachin posted some additional info via blog post with the annoucement. They’ll soon be moving down to Mountain View as part of the purchase so I’ll get to see them more often!

Yesterday I attended the 500Startups demo day – it was a fun event and will be interesting to compare to the YC demo day coming up later this month as well.

For everyone who didn’t make it, I wanted to share all the slides:

Read the rest of this entry »

The “Ultimate Driving Machine” is a classic slogan that makes BMW compete based on position, not features.

It’s hard to keep things simple, especially when adding so many new features
In my recent post on the virtues of marketing simple products, a couple readers wrote in to write a really interesting questions – here’s a particularly interesting one by Mark Hull:

How do you ensure that by simplifying your product too much, you are not losing a competitive edge by a lack of additional features/functions?

Every product team struggles with this question- it seems like naturally adding more featureset adds more power to the product, yet at the same time adds complexity that makes it hard for new users to even get started. This is a common problem in the initial version of a product, because most of the time the first version doesn’t work, and the most obvious way to solve the problem is to just keep adding features until it starts to click. Yet does this ever work?

Don’t compete on features. If your core concept isn’t working, rework the description of the product rather than adding new stuff.

Make sure you’re creating a product that competes because it’s taking a fundamentally different position in the market. If the market is full of complex, enterprise tools, then make a simpler product aimed at individuals. If the market is made up of fancy, high-end wines, then create one that’s cheaper, younger, and more casual. If the market is full of long-form text blogging tools, then make one that makes it easy to communicate in 140 character bursts. If computers are techy and cheap, then make one that’s human and more premium. These ideas are not about features, these are fundamentally different positions in the market.

BMW is the Ultimate Driving Machine
My favorite example of differentiated market positioning in a very crowded market is BMW’s “Ultimate Driving Machine” slogan. It’s not just a marketing message, you know it’s true when you sit inside a BMW and turn on the engine. Among other things, you’ll notice that:

• The center console is aimed towards you, the driver
• The window controls are next to your stick so it’s easier for your right hand*
• … and obviously the remarkable driving experience

Furthermore, when you go to the dealership, the entire experience keeps reinforcing the “Ultimate Driving Machine” message. The point is, the positioning is about the driving experience and the engineering to back that up.

In a price and features comparison, it’s unlikely that BMW would ever come on top- it’s expensive, and very little of the money goes into the interior and niceties that you’d expect out of a Mercedes. Yet people end up buying BMWs not for the features, but because it’s a fundamentally different car than a Mercedes (or at least it feels that way).

I’ve always felt that Apple goes this way too, where their products are more expensive and often do a lot less than competitive devices, yet win because they have a more cohesive design intention across their whole UX. Again, the idea here is more about competing via a differentiated positioning rather than based on a feature checklist.

You’ll never win on features against a market leader
The other important part to remember is that for the most part, if there’s a winning product X on the market, you’re unlikely to win by creating the entire featureset of X+1 by adding more features. Here’s why:

• First off, that’s crazy because you have to build a fully featured product right away, and that might already take years to match a market leader
• Secondly, as described in the Innovator’s Dilemma, if you’re mostly copying the market leader and then adding features, those features are likely to be sustaining innovations that is likely on the incumbents roadmap already- by the time you’re done, they’ll either have it or just copy you

Instead, the idea is to have a simpler product that attacks the low-end of the market leader’s product by taking a completely different market positioning. That way, you don’t have to build a fully featured product and you can take a completely different design intention, which leads to a disruptive innovation.

Ramifications for startups building initial versions of a product
I think there are three key ramifications for teams building the first version of a product.

The first is: Don’t compete on features. Find an interesting way to position yourself differently – not better, just differently – than your competitors and build a small featureset that addresses that use case well. Then once you get a toehold in the market, you can figure out what to do there. This doesn’t mean that new features are inherently bad, of course- they are fine, as long as they support the differentiation that you’re promising.

The second thing is: If your product initially doesn’t find a fit in the market (as is common), don’t react by adding additional new features to “fix” the problem. That rarely works. Instead, rethink how you’re describing the product and how you deliver differentiated value in the first 30 seconds. Rework the core of the experience and build a roadmap of new features that reflects the differentiated positioning. Avoid add-ons.

The third is: Make sure your product reflects the market positioning- this isn’t just marketing you know! If your product is called the Ultimate Driving Machine, don’t just slap that onto your ads and call it a day. Instead, bring that positioning into the core of your product so that it’s immediately obvious to anyone using it- it’s only in that way your product will be fundamentally differentiated from the start.

* UPDATE: An astute reader, Greg Eoyang, pointed out that the modern generation BMWs (E90s) are different now- I have an E46 that’s a few years old, so I was basing my observation on that. He writes:

First of all, a most modern BMWs do not have the window controls near the stick, that’s like 2 generations old, they are on the windows just like Honda’s these days.  BMW doesn’t even tell you about a lot of the features that have been standard for a long time – such as speed variable volume on the radios – Wide Open Throttle switch (back in the non-CPU days, it cut off the air conditioner when you floored it) – They have improved the concept of a car which is more than the features.

Thanks for the additions Greg!

This is reposted from my answer on Quora here.

Question: What UX considerations were built into Google+?
The most interesting design choice I’ve seen for G+ has been deploying it across all the Google properties within a navbar, and via the notifications – I’m talking about this thing right here:

(btw, looking at it now, I notice it’s the same coloring scheme as Quora, hilarious)

Building G+ on top of pre-existing, high-retention products
Obviously this is a smart decision because it lets them build on top of their own high-retention, pre-existing products: Google Search and Gmail, in particular. Contrast this to an approach where they would have started up G+ as its own independent property, which Google users could choose to adopt or not- but then that looks like Orkut.

Anyway, as a result of adding this new global navbar across all the Google properties, they have to deal with a very small amount of real estate to create some pretty rich interactions. Thus, it was very interesting to then see them building a mobile-like interface for interacting with comments, follows, etc., inline, without leaving whatever experience you’re already in:

And if you click on any of these, you see a quick sliding motion that lets you interact with the different notifications inline, without going anywhere:

Contrasting with Facebook
In comparison, the Facebook notifications dropdown is almost more like a inbox of “pointers” to the actual content. As a result, while you can see what’s new, you can’t actually do anything about it without leaving where you are. I found this a nice interaction on G+’s part given that they are building on top of things like email or search where you may not want to leave yet.

Someone should obviously do a much longer design discussion of the G+ main site, but I personally found the new navbar and notifications system pretty interesting so I thought I’d write a bit about it.

Simple products aren’t only better designed, they’re easier to market too.
Marketing and product UX are seen as conflicting with one another, but there are, in fact, many opportunities for the two to work together. Some of the best tools for increasing metrics are the same ones that are used to create effective interaction designs. These techniques include things like adding “soft” onboarding experiences, stripping out unnecessary features, having clear visual hierarchy and calls to action, and many more tactics. Ultimately, these tactics serve to create simple product experiences that are both desirable and well-optimized.

Let’s explore the different reasons why simplicity is a virtue for both designers and marketing quants.

Highly optimized flows make it easy to understand “what do I do next?”
Every product lives and dies based on how well new users are able to sign up and get oriented with the product’s core value. High signup and onboarding rates depend on a large % of users completing each step.

As a result, it’s important for each page to be as simple and directed as possible, so it’s constantly obvious what to do next. If each page gives the user too many options, thus distracting from the primary goal of the funnel, then the %s will decrease. As a result, some of the best landing pages and funnels fundamentally depend on extremely simple, stripped down designs. Here, removing things like navigation chrome, extraneous links, etc is not only simpler, but also better performing from a metrics standpoint.

More data and faster learning cycles
A metrics-informed team depends on deploying A/B tests and evaluating the results as the core of their product iteration process. Early on however, you often don’t have enough users to quickly evaluate tests at a statistically significant level. This data is then further diluted when you have a complex featureset, since only a small % of users interact with each option. However, if you have a simple product, where almost 100% of the users go through the same signup, invite, and sharing flows, then you’ll be able to collect data sooner and thus make decisions faster too.

This is a huge advantage because when you can run A/B tests in 3 days instead of 9 days, for instance, you can learn 3x faster and find product breakthroughs sooner. Think about this like compounding interest in the bank- finding 10% improvements faster leads to exponentially better performance.

Simple products are easier to optimize and pivot
Ultimately, it’s the optimized flow through your product that wins – you don’t get any credit for complexity. One optimized funnel beats any number of unoptimized funnels, because you only get credit for average conversion rate across all the funnels. Thus, more funnels means that on a practical level, it’s harder to keep them all optimized. It’s easier and better to push users through a small number of signup flows that you can keep well-designed and well-optimized, so that the overall quality stays high.

This is especially true if you decide to make some product changes in a classic “pivot,” or otherwise test significant new additions in a signup funnel like adding Facebook sign-on. If you have a simple product with a small number of onboarding flows, then it’s easy to experiment to see if it’ll work, collect data quickly, and then add it to 100% of new users’ experiences. Contrast this to a complex product where shifting the design takes a lot of time because you have to update so many different places in the product.

Keeps the focus on top of funnel rather than low-impact add-ons

When a product isn’t working, often the knee-jerk response is to “fully bake” the product by adding more features. However, I’ve found that when examining the data of new startups, the problem most often lies on the first couple pages of a product- often an unattractive value proposition, or clunky signup flow that kills the new user experience. Adding metrics to simple products often makes it clear exactly what’s going on, and most of the time, it’s a fundamental issue that needs to be fixed on the first page.

In this way, simple products with the “right” value prop will end up with better signup rates- this lets you put your attention on top-of-funnel issues rather than low-impact feature add-ons that won’t 10x the destiny of your product.

Short funnels result in more conversions
One of the most powerful things you can do to a key product flow is to shorten it*. Generally, because you lose a % of users at each step, reducing the amount of work to get started is a highly effective tool- rather than presenting a complicated homepage and asking for tons of information upfront from a user, perhaps you just let them signup with Facebook- that might reduce the number of steps, leading to a simpler product and better metrics too.

Ultimately, this all aligns with the highly opinionated design ethos that prioritizes what users most often want to do, rather than presenting many options equally. As is discussed in the Palm story in the book “Designing Interactions” the features of a product are used in a Power Law distribution- a small number of features are used constantly and the rest are long tail. As a result, you want to make the most commonly used features convenient while putting the unused features available but hidden.

(*in some outliers, lengthening signup flows with the right steps can help too)

Increasing the prominence of high-value actions by removing low-value actions
One of the most common (bad) design patterns I see among metrics-oriented products is continually layering more and more prominent calls to action for sharing or other viral mechanics. This got especially bad in early Facebook apps. The problem is that the user’s attention is easily diluted, and each new feature competes with the last- as a result, after a few iterations of this, it’s pretty easy to end up with a frankenstein of a product that’s cluttered and messy.

Instead, a compelling tool is to remove features in order to make what remains more prominent. Instead of making the high-value actions bolded and highlighted in yellow, simply remove the actions that are no longer necessary. This leads to both a simpler product experience as well as raised prominence for whatever actions you want to emphasize.

Conclusion- let’s make design and metrics work together
Ultimately, the key to the tools above are that they increase the effectiveness of the UI while simultaneously increasing the metrics. This can happen because highly optimized products are dead simple to use- they have landing pages that communicate a compelling value, soft onboarding flows, clear calls to action, and simple mechanics that drive a lot of value. The same things that make it a highly marketable product are the same things that make it well-designed, and a great thing for which every product should strive.

To use these tools effectively, those who are metrics-informed must also become design-informed. While it’s obvious that you can increase the prominence of something by making it blink and highlighted in red, there are many more tasteful tools that lead to less visual clutter and provide an even greater metrics benefit. Even Dave McClure!

There’s more than one way to use this tool
A/B testing is a very useful tool that can be used to develop better product designs, rather than just evaluating landing pages.

In a classic A/B test, you’re metrics-driven and want to pick whatever test variant ends up with the higher numbers. This is a useful tool, but is only applicable to scenarios like signup flows where the conversion is obvious. This post will describe some different tactics that are metrics-informed and end up as an aid to your product design process, rather than driving it.

The tactics I’ll describe are for:

• Updating your product without negatively impacting numbers
• Streamlining your product by measuring and removing unused features
• Designing for the right level of prominence

Updating your product without negatively impacting numbers
Product teams are constantly pushing small updates to their products in response to customers and what’s happening to the market. When an update affects a key part of the product, particularly to the main signup flow or core viral loop, it’s often important to ensure that it doesn’t hurt the numbers.

For example, let’s say you’re building a new social site and you have a Facebook-integrated “friend finder” option that you want to add. If you build this and test it, you’ll likely find that since it’s unoptimized, it’ll have worse initial numbers. A classic A/B test will often eliminate the new design because it performs worse. But instead of killing it prematurely, you can use an A/B test to iteratively “bake” the new design with a small % of users until it’s ready to replace the old one.

If you know that it’s important to have this type of Facebook integration in your product design, what you do is leave it in, but only expose 10% of your users to it. Then keep making small updates to the design, working on the copy, call to action, and other aspects, until the new design performs as well as the original.

In this way, you can update your product without impacting the numbers negatively. And unlike a classic A/B test where you aim to just pick a winner, instead you are using it to incrementally benchmark a new design until it’s ready to replace the existing one. For this, you are design-led because you know you want to execute this product in a particular way, but you use the A/B test as a safety net to make sure you don’t push out something that’s not ready.

Streamlining your product by measuring feature usage
There’s an important design principle that says, “Do less, but better.” I’ll elaborate on my POV of this philosophy more in a future post, nevertheless many product teams struggle to remove features, or even to quantify unused features.

For example, you might have a legacy feature that suggests people to follow on your social site, which you’d like to replace with a Facebook-based “friend finder” screen instead. Sometimes it can be difficult to get rid of navigation on something like this because it’s not clear how many people are really using it and how that affects their behavior overall, especially new users

A nifty way of using A/B tests to handle this is to run an A/B test to remove the feature, and get the following information back:

• How many people actually get exposed to this feature? (Based on what % of people get added into the experiment versus your active users during the test’s time period)
• What metrics are affected by people who have this feature removed? (As long as the metrics are neutral to positive, then you can remove it safely)
• If some metrics are bad, can you counteract it by adding something else to the new design?

Similar to the process of updating your product, the important notion here is that you have a particular action you want to take on a design level (simplify the UX) and you use the A/B test as a tool to aid that design goal. In this case, rather than going with whatever has better metrics, instead the goal is to go with the better design as long as it’s neutral or better on the numbers.

Designing for the right level of prominence
As you model out the key metrics for your product, there’s often important assumptions that need to be made on things like what % of your users invite their friends, or how many friends they invite, etc. Oftentimes, entire product strategies hinge on making sure that certain kinds of metrics get hit- it could mean the difference between being a viral eyeballs business versus one based on lifetime value and ad spend.

From a product standpoint, this manifests itself as trying to figure out how prominent to make things like “Invite friends” or “Import your addressbook” or “Subscribe to the Pro version.” To build a great UX, you often want to make something as low-prominence as possible while still making sure it’s easy and accessible for users.

A/B testing can help a lot here since you can test multiple versions of prominence and see where it takes you. If you want to prove that a model is even possible (for example, in the very best case could we get 20% of our users to invite their friends?) then you can make a popup that asks for friend invites constantly and see if you are even close. The point here isn’t that you would ever actually close the experiment with the obnoxious popup, but rather, it helps you do a sensitivity analysis of what might even be possible, to see are realistic values within your model.

You can use this technique hand-in-hand with the other ones listed above so that you eventually take a high-prominence version of it and iterate until it’s acceptable to show to 100% of the users.

Final thoughts
The thing that all of these ideas share is that you are using A/B testing as a tool to aid in a broader and stronger design POV rather than slavishly following whatever has the better metrics outcome. As others have discussed before, it’s the difference between data-informed versus data-driven. Many features you’ll want to do in your product have lots of qualitative value, even if the short-term quantitative benefits are difficult to measure or not there at all- using these advanced tactics lets you continue to push out dramatic new designs but without hurting the metrics your business depends on.

Do you live outside of Silicon Valley? Watch this video
For all the startups and entrepreneurs outside of Silicon Valley, I want to direct you to this incredible summary by Naval Ravikant of AngelList and VentureHacks on the anatomy of a fundable startup.

This video is a much more comprehensive and detailed version of what I often talk to non-Valley entrepreneurs and startups about. It’s part of the “grooming” process that startups out here get to become fundable and to focus on the right things to get there. People spend a surprising amount of time on things that will contribute little or no value to getting them to a seed round, and this talk is the best I’ve seen in terms of presenting the issues in its entirety.

Naval broke down the 5 main qualities of an “exceptional startup,” in the following order:

1. Traction
2. Team
3. Product
4. Social Proof
5. Pitch/Presentation

And while all these qualities are important, Naval explained, the most important thing is to understand that:

“Investors are trying to find the exceptional outcomes, so they are looking for something exceptional about the company. Instead of trying to do everything well (traction, team, product, social proof, pitch, etc), do one thing exceptional. As a startup you have to be exceptional in at least one regard.”

(via Founder Institute)

Anyway, please watch it all the way through and enjoy!

7th Founder Showcase – Naval Ravikant Keynote from Founder Showcase on Vimeo.

Does anyone care about your new product?
A question every entrepreneur asks is, if I build it will they come?

You can have a cool idea for a new product, but how do you know if anyone cares about it? And for a consumer product, how do you know that tens of millions of people will care about it?

One of the key points that I argue in my 2011 blogging roadmap is that tapping into a large market with pre-existing demand makes things easier. More discussion here. This means going after markets where users are already familiar with with your product category, the different options, and you build a product that is better against some competitive axis.

This post is about one way to figure out if anyone cares about your product.

Innovating on product execution vs. Creating a new product category
Some of the Valley’s best companies, like Google, Facebook, and Apple’s mobile products, entered their respective markets late in the game and effectively competed with differentiated products to win. They competed by executing great products in pre-existing categories rather than creating brand new categories. At first I resisted this point of view, since it’s more fun to paint on a blank canvas and do something that is completely new and innovative. Yet over time, I’ve come to believe that “blank canvas” ideas may be superficially innovative, they are much riskier and “first mover advantage” is wildly overrated, especially when there’s ample room to innovate in existing product categories. And for ideas in spaces with lots of competition, if you are able to get to scale and differentiate along some key dimensions, ultimately your traction lets you do all sorts of fun innovative stuff later on.

It’s easier to reinvent something than to invent it.

The reference example of this is Apple, which has created amazing and differentiated products in huge pre-existing categories like computers, laptops, MP3 players, phones, etc., and only occasionally go for new product categories (like the Newton and iPad). When Apple picks an existing category, they can take something that’s OK but fragmented, and take it to an entirely new level on design- and they can do this without the risk that the market is zero.

So these days, as I meet new startups, I like to think about the new/risky stuff in their product as part of a careful and coherent strategy to tap into pre-existing markets, rather than trying to create new categories. One of the key tools that you can use here is the Google Keyword Tool.

Introducing the Google Keyword Tool
The GKT was created for advertisers looking to buy Adwords- in fact, some of the best market research tools on the internet are designed for ad buying, so that advertisers can actually understand how much inventory is available to buy. I’ll do similar writeups for Google’s Ad Planner tool, Quantcast, the Facebook ad buying UI, and others when I have time.

The GKT lets you do something very simple – Plug in some keywords, and it’ll tell you:

• How many searches are happening on those keywords
• Related keywords to what you put in

Here’s a screenshot:

You can use this for a lot of different scenarios, but my favorite use cases are to validate how mainstream a product category is, make sure you are using customer-centric wording to describe your product, and to identify nearby product positioning options.

Here’s an example of how I might use the tool to research a movies site:

1. Go to Google Keyword Tool
2. Plug in “movies” and sort by searches
3. Notice that some words, like “film” or “theater” are related, and add them to the search
4. Repeat, to collect a large collection of related keywords
5. Start scrolling through the results (again, sorted by searches)

Based on a GKT search like this, you find all sorts of interesting things, which let you both validate pre-existing demand and make sure you’re speaking the same language as your customers.

Validating pre-existing demand
First off, the # of searches is a pretty interesting. If you plug in keywords related to your business and find very low numbers, it might mean you’re using wording that mainstream users don’t understand or don’t care about. This happens commonly when a phrase is used to describe the product to other entrepreneurs and to investors, and includes abstract/strategic notions of what the product encompasses, but not what end-users are actually doing.

One practice I might suggest would be:

Steer your product towards a category with millions of pre-existing consumer searches – this shows mainstream understanding and demand for your product category

In fact, this is an easy way to define a new versus existing market that only applies to consumer internet products:

If people are searching for products in your category then you are in an existing market.

I personally find this a very nice and clear-cut way to figure out where you are is in the spectrum of new versus existing markets, and how much consumer behavior risk a product takes.

Are you using the words your customers use?
One of the best uses of the GKT is for finding the right words to describe your product. Oftentimes people like to use “X for Y” descriptions, which are convenient, and the Google Keyword Tool can help you refine that thinking.

If you plug in your high-concept pitch and you get millions of searches back, then you’re in good shape. For example, it turns out tens of millions of people are looking for “free movies,” so if you can do that legally, you’re all set. But sometimes you plug in a term and it falls flat. For example, imagine a startup that self-describes as “a marketplace for whatever” – in this case, “marketplace” is the X and the “whatever” is the Y. If you look up “marketplace” in GKT, what you’ll find that is that there are very few searches for that keyword, and there may be better options for the product positioning. Why does “marketplace” get so few searches?

My theory is that a “marketplace” is an abstract, businessy description for a startup’s strategy, whereas consumers likely only care about 1) selling stuff, or 2) buying stuff, and they are only in one mode at a time. As a result, I’d argue that as a marketplace startup might want to consider one of the following strategies for positioning their product:

• Targeting primarily one audience (buyers usually?) and have some secondary UI/flows to bring in sellers
• Picking a clearer attribute for what’s being listed, like “free” “collectible” or “upscale”
• Using colloquial terms “buy and sell” vs more abstract terms like “marketplace” or “exchange”

This product position then informs how you describe the product through all of your marketing channels. You’re probably better off buying ads or having site invites that say, “sell your useless junk online” rather than describing it as a marketplace.

Don’t build what your customers are asking for, says Henry Ford
An important reminder for this type of exercise, which is so dependent on what customers are searching for, is that what you build and how you describe it are two very different things.

There’s a famous Henry Ford quote:

If I asked my customers what they want, they simply would have said a faster horse.

This is absolutely true, and of course we’re all fortunate that Henry Ford ended up building a car. Yet at the very beginning, remember that they ended up calling cars “horseless carriages” so that the product could be anchored against something consumers already understood. Horseless carriage hints at many things – how it’s used and why it’s valuable, most importantly, and also the primary axis of differentiation. It’s horseless but still gets you from X to Y!

So maybe an addendum to the quote would be:

Build a car, not a faster horse, yet start by describing as a faster/better horse until people understand what cars are. Afterwards, build on that term.

Research potential segments of a market
Once you have a big juicy market to go after, the other interesting question is how you’re going to segment it. Basically, what is your product going to do that’s better/different than what’s already out there? Steve Blank has written a bunch of interesting content on this, including this post and this slide deck.

The Google Keyword Tool can also help with this – when you do a GKT search for a big “X” like “movies” you find all sorts of interesting modifiers to that phrase. Just glancing at the results, you see potential subcategories focused on:

• free movies
• imdb
• movie times
• movie quotes
• movie trailers

Beyond these, you’re starting to get to very few searches per month. Looking at this, if a startup had a really compelling product for one of the above scenarios, I’d certainly be really excited about them.

(One of the reasons I’m optimistic about the new YC startup Hellofax.com is knowing how many searches are online for easy and free email-to-fax services. Just search for “email” and you’ll see what I mean).

When you search for these keywords on Google.com, sometimes there’s already great products that cater specifically towards this group. For example, there are a ton of cheap airfare ticket sites. But sometimes, you find millions of users searching for something that doesn’t exist- that’s pretty awesome, and a great opportunity if you can execute a really compelling product there.

Google Keyword Tool product research checklist
If you already have a product in mind, here’s a quick list of things to think about:

• What’s your “high-concept pitch” for your product? (Often an “X for Y”)
• Are you using terminology that millions of consumers actually understand and know how to search for? Or do only other smart hackers or social media douchebags know what you’re talking about?
• Along what dimensions does your product compete with substitutes? (hopefully only 1 or 2 axes)
• Are there millions of consumers who care about that competitive axis?

And again, to repeat the observation re: the Henry Ford quote, have a vision for your product, and execute against that. Invent and build cars, but market it as a horseless carriage so that people know what they’re buying and why.

Is this too conservative?
Short answer: Yes, if followed too strictly.

Remember that the Google Keyword Tool test can validate an idea, but I’d be hard pressed to use it to invalidate an idea. Sometimes people will use things, and be passionate about it, but not search for it. Or maybe the product is mostly on mobile or Facebook and the searches are happening there, not Google. Either way, it’s a conservative tool, that’s to be sure.

I think there’s a spectrum of risk on introducing new products- the safest thing to do is to execute the hell out of a product design for a huge pre-existing product category and a competitive axis that people care about. Ideally you could validate that this market was already pre-existing and the competitive axis was an important one.

Yet most startups aren’t started this way, and instead take some risk in either picking a new category, or a market segmentation that’s driven by intuition. Ultimately I think it’s still important to try this out, when your entrepreneurial intuition says it’s the right way to go- but remember that you might have to go through a step of cleaning up the marketing and messaging around your product to slot it into something consumers understand.

Credit for inspiring this post
I have to give credit to Sean Ellis, who inspired me with a coffee conversation a few years ago to think about using search data to identify new versus existing markets. He was describing his consulting stints at Xobni versus LogMeIn as “demand creation” versus “demand harvesting” and using Google SEM as a test of which mode you’re in. Hopefully he will blog about that comparison with more nudging sometime :)

There’s been some excellent Groupon analysis
Since the S-1 has come out, there’s been some incredible analysis done – two of my favorites are Rakesh Agrawal’s Quora answer on “What are some notable aspects of the Groupon S-1?” and also Yipit’s analysis on the deterioration of fundamentals in Groupon’s oldest markets. I would highly encourage everyone doing anything in daily deals or the (misnamed) “social commerce” space to check those out. Additionally, please comment if there are some other great blog posts that I’m missing.

“There’s no tech! Anyone can do it!”
One of funny things that you used to hear about Groupon is how easy it is to start a clone, and how any startup could do it. A lot of people, especially developers, also say the same thing about product like Twitter, which are easy to code v1.0s for. That’s often a critique of consumer internet companies because what they do seems deceptively simple- there’s often no tech and no “barriers to entry” that a lot of the more B2B/enterprise investors like to see.

After all, let’s look at something like Groupon:

• Technology: Trivial, it’s just a mailing list and a landing page
• Market: Trivial to enter, because it’s huge and fragmented
• Sales: Trivial, you need 1 sales guy/gal initially who can sell some local deals

Seems like there ought to be tons of successful local daily deal sites right? And yet Groupon and Livingsocial control the vast majority of the market, and I have no idea who the #3 is? In fact, the most interesting competition ends up being other huge companies with big established userbases, like Yelp, Google, Facebook, Amazon, etc.

Email subscriber costs
The real reason is that there was a temporary arbitrage in buying tons of demographically targeted ad inventory that no longer exists. The Yipit blog post referenced earlier has this handy diagram:

That’s a huge increase from 2010 to 2011.

So if you think about it, this is one of the key bottlenecks to getting a Groupon clone actually started- if you want to build a list of 100,000 users, that’s actually going to cost you $3M right off the bat.

The backend is scary too
Furthermore, to even be able to monetize to break even, you start to need to contort the backend of your business to get there. This means that you have to be comfortable with things like:

• Extended time periods before your LTV catches up to your CAC (for example, 12 month breakeven on your LTV)
• Large # of deals per week at high margin
• To support the # of quality deals, a high-quality sales team

If you need 5 deals per week, every week, for 12 months to break even, then you’ll need a great sales team for that. Then you’ll need someone to optimize your ad spend, a bunch of customer support people, and all of a sudden it doesn’t look so easy.

Getting to scale, let’s say to 1M or 10M email subs, costs gobs of money that very few people in the world would be able to raise in venture capital. That’s why I imagine the most successful Groupon “clones” start in other geography where the arbitrage still looks like <$5 and not $30, or where it’s a high-end niche with some built-in distribution to get the first 10k-100k on board.

The same is true for viral products too
I wrote this post originally about Groupon, but it’s important to note that the same is true for viral marketing channels as well. As with other marketing vehicles, users get “inoculated” over time to the same approaches. Getting an “invite” was a big deal in 2003, so addressbook importers were super effective. Banner ads used to get 10% clickthrough rates, and now they’re 0.1%. Over time, marketing channels naturally become saturated and that creates a built-in defense against new entrants in the market.

Thus, even if something looks easy to build, you better do it quick otherwise you may never be able to catch up. A corollary to this is that if you discover a new marketing channel or some new viral mechanics, you’ll have a huge advantage early on since your response rates will be great.

Send me any other interesting analyses of their S-1 or others!
As all of these S-1s are coming out, I’ll try to stay on top of any interesting analyses, but feel free to email any that I might be missing. Just shoot me a note or comment on this post.

UPDATE: A post drilling down into how Groupon defines “customer” and ratios in their oldest markets (via Dru Wynings). Also, Yipit did a great followup post called “Reports of Groupon’s Death are Greatly Exaggerated“.

Today we have a guest post from my sister Ada Chen Rekhi about user acquisition based on experiences at her new startup Connected. Connected is a new contact management product they’re working on for professionals to easily manage their relationships across their email, calendar and social networks. Enjoy! -Andrew

When Does Paid Acquisition Work for SaaS Startups?
by Ada Chen Rekhi

Introduction
After recently moving on from adventures building a consumer gaming portal at Mochi Media (acquired last year for $80 MM), I’m now working on a new startup called Connected, which provides contact management without the work. I decided to blog some of my thoughts based on my experience thus far with deciding on the right user acquisition channels to focus on.

When does ad buying work for SaaS businesses?
It’s a convenient belief that after you decide to build your software as a service (SaaS), Google AdWords and other networks will enable you to outsource all of your marketing efforts and focus less about user acquisition. This is not always true. Here’s a “napkin math” model to quantitatively decide whether or not ad buying is right for your startup based on reality, not guesswork.

A model for user acquisition

Paid user acquisition works for you when the following proves true

• LTV > CAC

The lifetime value (LTV) of your users should exceed the cost of acquisition (CAC) to get them in the door. As a reminder

• LTV = Expected Life x Average Revenue Per User (ARPU) x Gross Margin

In addition, for SaaS, you care quite a bit about costs and conversion rate for your funnel to trial, and from trial to paid. In specific, these look like

• CPC – cost per click to get traffic
• % trial conversion rate – users who convert to a trial of your product
• % paid conversion rate – users who convert to paid account

To estimate your cost of acquisition, you can base it off of estimates for your trial and paid conversion rates.

• CAC = CPC / (% trial x % paid)

An example of cost of acquisition
Let’s pick an example and work backwards. Let’s say you have a

• $20/monthly subscription
• 5% paid conversion rate – from trial to paid
• 10% trial conversion rate – from visits to trial

Then let’s pick a two different points for cost per click

• $0.50 CPC
• $2.50 CPC

In order to get a user at these CPC points

• CAC = CPC / (% trial x % paid)
• CAC = $0.50 / (10% x 5%) = $100
• CAC = $2.50 / (10% x 5%) = $500

In this example, it costs anywhere from $100 to $500 to get a single paying user at $20 per month. If you were trying to acquire 100 users ($2000/month), at $0.50 CPC that’s $10k ad spend, and at $2.50 it’s $50k. Drew Houston from Dropbox brought up very similar issues from his Dropbox Startup Lessons Learned presentation, where their initial search marketing test had a whopping $233-388 cost per acquisition for a $99 product!

Compare this against lifetime value
Compare this against the lifetime value of your user, or the total amount of profit you expect to receive over the user’s use of your product. This value should factor in the churn that you’re seeing from users canceling their subscription over time as well as what the payback period and working capital which you expect. Even though you might expect a user to be retained over a period of years, most startups don’t have the capital necessary to tie up their money for that long.

Let’s go back to the example above. We have the two users who cost

• $100
• $500

Assuming zero churn and zero operating costs on their $20/month subscription, you would recoup your cost on these user over a fixed period of time

• $100 / $20 = 5 months
• $500 / $20 = 25 months

In the case of second user, it would take over two years to recoup the initial $500 you spent to acquire them. You can offset this issue of working capital by setting the value at the amount of revenue you receive over a fixed period of time, or by being more aggressive with pushing them to prepay for longer periods of subscription cost upfront.

For example, what if you could get these users to pre-purchase their $20/mon subscription for $149/year? You’d be able to recoup the first user’s cost instantaneously, and get back a significant percentage of the second user’s acquisition cost.

Making the model work
The path to achieving profitability looks like making the model of having your cost of acquisition beneath your lifetime value work. You can quickly get a back of the envelope idea of whether paid acquisition is for you based on the examples and model above.

Doing this will help you determine whether or not you can profitably use ad buying as a source for getting users. You can also fine-tune your model to incorporate even more granularity such as

• virality
• traffic source
• retention
• working capital
• churn
• etc.

Trying paid acquisition on for size

Now that we have the framework down, the question is whether or not paid acquisition works for you.

If this works for you, then congratulations- you are on the path to scalable riches! ;-) If it doesn’t work, then you should think about how far off it is. Getting ad arbitrage to work out profitably is extremely sensitive to changes in the steps of your conversion funnel, as well as the source of the traffic. So if you’re not many factors off, it may make sense to spend a few months refining your funnel and trying to optimize the channel the traffic is coming from. Here’s a few things to consider-

Does the math work?
Once you launch your product and get a sense of what the conversion rates are in each step along the funnel and the churn rate, it may be that the math doesn’t work out. If you’re not too far off, then it may be worth spending time trying to make the metrics work out through landing page optimization, increasing conversion along the steps of your funnel and trying to optimize your traffic sources. However, if you’re several factors off (this is common in highly competitive markets) paid acquisition may not make sense as a strategy for you.

Is your product in an existing market or a new market?
Intent-based paid acquisition channels like search advertising work best in an environment where users are aware of the problem and actively searching for solutions which your product meets. You can look up potential search terms and volumes through Google AdWords Traffic Estimator, including estimated average cost-per-click and monthly search volumes. If not, you can also experiment with targeting sites that reach the demographics of your users.

How much working capital do you have?
While theoretically you might be willing to pay up to the full LTV of the user, you may want to limit the amount you’re willing to pay based on a fixed time period, for example the expected value from the user over 6 months. This may be because at some point you run into working capital issues paying for users who may take years to break even.

I’ve had two recent conversations in which people have mentioned the “grow first, monetize later” philosophy as one of the signs of the coming bubble apocalypse, and this post is to argue why it’s very smart and rational to focus on getting millions of users first. (This post is part of my 2011 blogging roadmap)

Regarding monetization, I’ll note that…

• in general, consumer products mostly suck at monetizing
• any business model built on 1% subscriptions of 0.1% ads need millions of users
• costs are ridiculously low for new startups, and N millions of users is not expensive
• ignore this for products like marketplaces where monetization is part of the value prop

An ad-based example
Here’s some quick math- let’s say that you are a typical seed-stage team of 4 trying to get your startup off the ground, and your burn is approximately $40k/month. If you monetize at $0.25 CPM, which is a pretty typical ad rate for every thousand ad impressions, then that means you need a whopping 160M ad impressions per month to break even[1]. Even if you get 2X or 10X that ad rate, you’re still in the millions of users to get there. Scary right?

A subscription-based example
Similarly, if your site has a freemium business model, you’ll find that something like 1% of users subscribe for a pretty nicely tuned freemium configuration. So if you have 1% of registered users paying you $5/month, that means your average user is worth $0.05. Given this, you’d need 800k registered users, and if only 10% of your users register, you’ll need millions of users to get there.

Ultimately, the key is new user growth
Given the difficulty of monetization for consumer products, ultimately the best way to get to breakeven isn’t to try to optimize the 1% subscription rate to 2%, but rather to pick a huge market, create a killer product, and try to acquire millions of users. Because this is the biggest risk, you want to focus on growth first and foremost.

Here’s a different analogy that Steve Blank uses to get at this- let’s say that you wanted to create a cancer-curing drug. You don’t need to crunch the business model for that- if you had it, it’s valuable. You don’t need to price test or do customer development. All the risk is in the science, so you just focus on the science.

Similarly, I’d argue that in consumer internet, the real risk is that you can’t get millions of users actively engaged in your product, and that risk is ultimately driven by growth and long-term user retention. Thus focus on that first, then figure out the monetization once you’re at scale.

Stuff is so cheap these days
Note also that running a site with millions of users is cheap. The cost of hiring developers/designers will vastly overshadow the cost of maintaining the infrastructure- all you need are a few dedicated servers or just use Amazon Web Services- unlike the 90s, you don’t need a huge datacenter to get started. Because these costs are pretty low, you can just focus on making sure your designers and developers are productive and you’re getting to product/market fit.

Ignore this advice for products where revenue is part of the value prop
Of course for products where you are helping people make money as the central value, you need to do this sooner rather than later, so that you can make the entire network happen. So if you’re building a marketplace, collect money early, even if you don’t take very much profit. Same with Groupon-esque startups.

[1] CPM to revenue calculation
$0.25 CPM = $0.25 for every 1000 impressions
$0.25 / 1000 = $0.00025 per ad
$40k burn / $0.00025 per ad = 160M ad impressions per month

This post is part of my recent 2011 blogging roadmap post, where I created an outline of going from zero to product/market fit. Getting to this endpoint is obviously a good goal in theory, but question is, what does it even mean to hit this goal?

The original definition
In Marc Andreessen’s original post on the topic, he writes:

Product/market fit means being in a good market with a product that can satisfy that market.

You can always feel when product/market fit isn’t happening. The customers aren’t quite getting value out of the product, word of mouth isn’t spreading, usage isn’t growing that fast, press reviews are kind of “blah”, the sales cycle takes too long, and lots of deals never close.

And you can always feel product/market fit when it’s happening. The customers are buying the product just as fast as you can make it — or usage is growing just as fast as you can add more servers. Money from customers is piling up in your company checking account. You’re hiring sales and customer support staff as fast as you can. Reporters are calling because they’ve heard about your hot new thing and they want to talk to you about it. You start getting entrepreneur of the year awards from Harvard Business School. Investment bankers are staking out your house. You could eat free for a year at Buck’s.

His partner, Ben Horowitz, follows it up with a bunch of other observations about the fact the event isn’t a “big bang” kind of event – instead, there’s lots of gray area as your product starts working for the market: I’d encourage everyone to read his subsequent post here.

So the short answer is, there’s no easy test.

Now given that caveat, I’m going look at this through the lens of consumer internet to add some additional thoughts.

What is a market anyway? And how do you validate it’s real?
How do you even define a market for consumer internet? Ultimately, I concluded that the most useful definition of “market” is 100% consumer-centric. Here’s an attempt at a simple definition, focused on consumer internet:

A market consists of all the consumers who can search for and compare products for a use case they already have in mind.

This definition is very focused on the notion of pre-existing demand for products in your market, and is scoped narrowly to avoid confusion.

The most concrete test of pre-existing demand is using the Google Keyword Tool, which tells you how many people are searching on Google for a particular keyword. To try this out, you’d execute the following steps:

1. What keyword do people search to get to your site?
2. Put those keywords into Google Keyword Tool
3. How many people are searching for this keyword?

If the answer to #3 is large (millions or more), then you have a large market. This test is very concrete, and also very finicky. By design, terms like “vacation package” score high on this test, whereas “travel experiences” do not, even though an educated entrepreneur or investor might abstractly group them together. Similarly, by design, a person who’s building a “social network for musicians” might be inclined to list the # of musicians in the US as part of their market sizing, but under this test, you’d quickly see that there’s not too many people are specifically looking for that. Also interestingly enough, you’d never say there was a “Photoshop market” but a quick search will show that in fact almost 40 million searches per month on “photoshop,” and it might be a great strategy to position yourself relative to that keyword.

Validating that you are part of a pre-existing market comes with all sorts of benefits, which I’ll address in later posts. But for now, the most important benefit is that you know the # of potential customers is large.

(In general, I’ve been constantly confused about how to even define a market in consumer internet, given that there’s so much similar featureset between otherwise very different products. For example, early on, people talked about “social” as if it were a type of site, whereas now it’s seen as an aspect for all new products coming to the web. Similarly, people sometimes talk about “Facebook apps” as if it’s a market when, again, it’ll probably just end up an aspect of every new online service.)

What’s a great market?
What are other attributes that make a market attractive? For consumer internet, a great market is commonly defined by:

• a large number of potential users
• high growth in # of potential users
• ease of user acquisition

Not competition, in my opinion, because for consumer internet there is often literally billions of potential users, and you’re mostly competing against obscurity. So even if there’s a ton of competition, if it’s easy to acquire consumers to your product, that’s great! Then get a good enough product, and you’re ready to go.

Not monetization, in my opinion,  because making money is pretty straightforward. You can throw on some ads and get $0.1-$1 CPMs, or you can charge subscription rates and get 1% to convert, or you can do the virtual goods thing. The biggest risk in all of these monetization models is really about whether or not you can get millions of users or not.

Picking a great market leads to better products
Leading with a great market helps you execute your product design in a simpler and cleaner way. The reason is that once you’ve picked a big market, you can take the time to figure out some user-centric attributes upon which to compete. This leads to a strong intention for your product design, which drives a clean and cohesive UX. In a market of all black Model Ts, you can sell otherwise identical cars of different color and that’ll work. Picking the right attribute is it’s own topic though!

The important part here is that you can usually pick some key things in which your product is different, but then default the rest of the product decisions. This means that your product’s design can be more cohesive because you’re trying to do less, but better.

Once you’ve executed your product, then there are various ways to validate that it’s “good enough” and your product fits the market:

• When user testing, do people group your product in with the “right” competitive products?
• Do they understand the differentiation of your product versus your competitors?
• Will some segment of users in the overall market switch to your product?
• Are some users who’ve “rejected” the products in the market willing to try your product?
• How do your underlying metrics (DAU/MAU, +1 week retention, etc.) compare to your competitors?

All of the above are signals towards product/market fit. Thee above tests are interesting in that they fundamentally anchored on pre-existing competitive products in the category. In a new market, you don’t have the luxury of comparing yourself to other things.

In future posts, I’ll try to give some more concrete metrics based on my research for what are good numbers in each of these cases, but for now, the important idea is just that in a large existing market you have more datapoints to at least say, “my product is at least as good as the other guy’s.”

New markets are a danger to good product design
In fact, one of the scariest things to me about new markets is that doing great product design for them is extremely hard. It’s so unconstrained that it’s hard to do anything other than add features, see what sticks, and iterate. This is fun except that keeping a cohesive product experience is quite hard, and removing features is usually harder than adding them. So at the end, you incur tons of product design debt that never gets paid off. (It’s not a surprise to me that Apple has a history of simplifying already successful product categories, rather than inventing brand new ones from scratch)

Conclusion
To summarize my main points in this essay, I’ve come to some simplifying definitions on how to validate product/market fit in consumer internet. For market, if you constrain the definition to people who know how to search for products in your category, you can develop a pretty concrete test evaluating pre-existing demand. And by leading with a market, you can develop a central design intention that leads to better product design. This in turn can then be validated by comparing your product metrics to competitor numbers, as well as user tests that focus on grouping and differentiation.

This leaves lots of unanswered questions, but hopefully is a start to my new blogging roadmap! More to come soon.

One important question that comes up all the time is, what makes a product easy to market?  I had a fun chat about the topic with Eric Florenzano and Eric Maguire who worked together at Mochi Media with my sister Ada. They also recently did YCombinator.

After the chat, eflo wrote up a helpful summary of some of the ideas we covered. I wanted to quickly share them with some comments:

1. Come up with one resounding use case–one thesis for how people should use the product.  Preferably this fits in with something that users already do and already understand.

I’m going to try to start blogging again!
It’s been a long time since I was in a good blogging rhythm, and I’m going to try to start doing it again :-) In preparation for this, I put together an outline of an output-driven set of milestones around product, that takes you from zero to a P/M fit product thats ready to scale on marketing/tech/etc.

As far as I can tell, this is all standard fare for companies in Silicon Valley. My desire to write these posts is ultimately about documenting what’s working for people and spreading the knowledge beyond Palo Alto, CA :-) All of these topics are ultimately derived by both my own projects as well as my advisory roles at venture-backed startups. (Some of these are listed here)

If you like the outline and want to stay up to date, just subscribe and follow me on Twitter.

Without further ado, here’s the outline- I hope to write at least a post or two per week:

Blogging roadmap goals

• “output-driven” roadmap for going from zero to product/market fit
• for small hackerish teams building consumer internet products
• the intention is to create a scalable startup that is going after a huge market, and generate huge returns for venture capital investors
• goal is to get to P/M fit in shortest time possible, defer everything else
  • defers monetization
  • defers marketing
  • defers scaling
  • (this is all by design)
• P/M fit takes a non-deterministic amount of time to get there, insanely hard, you’ll probably fail anyway
• the problem is 90% contextual, make up your own rules as you go

Concept prototype

Picking a product and market

• build for yourself (start with intuition)
• have a long-term vision
• base it off something that’s already big and already working
  • big makes it easy to test and collect feedback
  • already working means you have a good sense for minimum product
  • also, there’s pre-existing distribution channels as well
• figure out the options for competitive differentiation – this is the core design intention
  • talk to a lot of users, do a lot of research, compare a lot of products in the space
• dimensions for competitive differentiation
  • competitive dimensions
  • vertical audience
  • design intention
  • cheaper/niche
  • targeting rejectors
• validating that there’s LOTS of pre-existing “pull” for the market
  • search keywords
  • app leaderboards
• ideal goal: simple product with fundamentally different core design intention for large pre-existing market
  • bonus points for baked-in distribution, monetization, etc. but don’t let this lead the idea!!!
  • usually one killer feature (not a bunch of features)
• prototype: Landing page
  • what’s a good landing page experiment?
  • headlines, copywriting, hero shot, etc.
  • unique URLs
• anti-patterns:
  • “someone’s already done this” (desire for originality)
  • monetization/strategy-driven product ideas
  • technology in search of a market
  • “Wall Street” markets
  • lumping yourself into an aspirational market
  • comprehensive featureset done poorly

Paper/Wireframe prototype

Designing the initial product

• go for the minimum desirable product
  • might work :-)
  • the central design intention drives the product design
  • supports only the core use case, as minimum as possible
  • core UX should be 2-3 pages
  • limited functionality, done well. “Less but better”
  • Should build bare bone prototype in less than 2 weeks (really!)
  • flow-based product design
  • user quotes, then fill in with UI
• low-fidelity prototyping tools
  • easier and cheaper to make changes
  • fix defects earlier (Toyota lean manufacturing model)
  • engineers always want to prototype in code, but then sunk-cost fallacy
  • get feedback from people and iterate
• prototype: Core user flows, mocked up and ready to build
• anti-patterns:
  • “database-up” design
  • feature creep and low product self-esteem (v1 should look like a feature!)
  • comprehensive featureset all of it done poorly
  • lots of pet features that don’t fit into the core design intention

Code prototype

Coding the initial product

• build the prototype as fast as possible
• fill in any blanks left out of the prototype
• use the product yourself, iterate on it while keeping with the core design intention
• focus on key flows and prioritize over ancillary ones
• don’t worry about corner cases
• get it ready to be used by other people
• prototype: Live product, usable by other people
• anti-patterns:
  • taking too long
  • losing focus of the central design intention
  • not adjusting based on intuition and usage
  • overarchitecting, trying to make it scalable or modular or future-proofing in general

Friends and family alpha testing

• private beta goals
  • clean up core experience
  • make product usable over multiple visits
  • validate the core design intention
  • not scalable
• recruiting friends and family
  • focus on retention
  • are users coming back?
• recruiting random people
  • Find people from the existing market, rejectors, and outside the market
  • Learn from extreme users
  • Craigslist
  • Usertesting
• user testing
  • do they get it?
  • how would you describe this to a friend?
  • usability – remove the friction
  • would they switch? (for existing market users)
  • Net promotor score
• interpreting user feedback and learning to say “no”
  • which users fall into the target market? Hear them out
  • which users don’t? It’s OK (and maybe even good!) to have them reject
  • try not to add new features unless absolutely necessary
  • what features can you remove that aren’t part of the core?
• prototype: Simple product, polished by real use
• anti-patterns
  • Delusion- it’s not working but you think it is
  • Melancholy from user testing
  • Adding features without interpreting
  • Adding features that violate core design intention
  • Listening to out-of-market users
• is it working?
  • people understand the product
  • some subset of your users like it and use it
  • you like it :-)

Random people beta testing

• traffic testing goals
  • start polishing your onboarding flow
  • develop options for distribution
  • build some basic stats infrastructure
  • not meant to be scalable
• User acquisition tactics
  • ads
  • PR + launch page + slow stream
  • partnerships
  • power through it
• Collecting feedback
  • surveys
  • help and problems
  • recruit users to talk to
• prototype: Spreadsheet for signup flow, more polished signup flow
• is it working?
  • signups are happening
  • people are going through the core flow
  • retention/recurring usage from target users
  • product still works for you, and your friends/family

User flow optimization

• model your usage and figure out your core drivers
  • this is completely product specific
  • two examples- daily deal versus a chat site
  • whats your “metric of love?”
• prototype your funnel – explore!
  • flow chart
  • excel
  • SQL
  • formalize/finalize with dashboards
• identify major bottlenecks for why the product’s not working
  • start at the beginning of the flow
  • fix bottlenecks with A/B tests
• is it working?
  • how do the metrics compare to the usage model?
  • 10% signup
  • +1 day retention and +1 week retention
  • DAU/MAU
• anti-patterns:
  • trying to fix problems in core UX when signup is the problem
  • over-architecting stats infrastructure
  • trying to use a generic analytics product to answer situational questions

Ready to scale?

• Hopefully the major checkboxes are checked – at this point you’d have:
  • Huge market
  • Differentiated product
  • Product makes sense to normal people
  • Product is working for IRL people
  • Product is working for non-IRL people
  • Well-understood and optimized user flows
  • Ready to scale up
• Non-scaleable marketing, tech, and otherwise- that’s fine
• Now scale everything else :-)

Crisis, terror, and melancholy

• Is it good enough?
• Nobody likes my product!
• My product is a mess!
• It’s taking too long!
• Investors hate my product!
• I’m iterating in circles!
• When to work on a completely new idea?
• Iterations are getting diminishing returns and people still don’t love the product

Final note: Thanks to my friends who helped review and add to this: Vinnie at Yipit, Alex at Penzu, Rob Fitzpatrick, Kevin at Hyperink, Jamie/Justin at Mocospace, Ada/Sachin at Connected, Noah at Appsumo, Jason at Kima, and the other folks who helped

Joshua’s slides on Metrics-Driven Design got tweeted out during SXSW and I wanted to share them.

In general, I think all of the various MVP/customer-development oriented startups out there are struggling with how to incorporate design into their product process. And at the same time, more designery teams are trying to figure out how to get more agile. It’s hard. As someone smarter than me has observed, the vast majority of the MVP-oriented companies end up with pretty uninspired, incoherent products- and they don’t seem to get any better over time. So I think it’s a great challenge for the whole community to get more informed about design and figure out how to really make it work.

Great Google color-testing followup
In particular, in the first few slides there’s a really funny followup to Doug Bowman’s complaints about Google testing shades of blue. These slides claim that in fact, the color choice really did matter, and quite a bit so, and quotes Bing search guy saying that the decision was actually worth $80M. I suppose in retrospect it’s not surprising, because the bluer something is, the more it looks like a link- so given the visual signal, it is meaningful for users over billions of searches.

Metrics-informed versus metrics-driven
All that said, I do have to say that I much prefer the term “metrics-INFORMED design” rather than “metrics-driven.” You should really be driven based on your vision of the product and where you want it to go, not the metrics that you use to validate or learn about your vision. (I first read the distinction of being data-informed over data-driven in a talk by some Facebook product folks, and have much preferred it ever since – this topic probably deserves an entire post by itself).

Finally, the slides
Anyway, the Joshua’s slides are excellent and I’d encourage you to flip through them. The official place to read the details around this presentation is here, on his site. His Twitter is here.

I’ve been struggling with trying to blog more often yet being in the mood to write the kinds of posts that I do. Quora helps a little bit, but I find myself mostly just tweeting instead :-)

Anyway, I have a question for the readers of this blog- please vote and/or leave any comments on what you’d prefer to see on here:

I wrote this on Quora a while back, but forgot to cross-pollinate it on my blog, so apologies if you’re seeing this twice. As those who have been following this blog know, I had a great deal of respect for MySpace back in the early days and worked with the initial team back when the site was just a few million members- I’ve written about it here and here.

Anyway, here’s the question…

How did MySpace, with a smart team of people, do such a bad UI/UX job with the new design?

The answer’s simple:
In the new redesign, MySpace prioritized short-term monetization ahead of user experience due to its failing business fundamentals.

First off- let me state that I think the new MySpace is actually better than the old one. However the new MySpace is still not good enough, obviously, to turn around the product.

I recently spoke to an interaction designer who worked on the new MySpace, who told me an anecdote that blew my mind:

When the team was working on the new feed at the heart of MySpace, the interaction designers wanted to make bigger images so that it’d be easy to see what users’ friends were doing. Similarly, they wanted to make the feed more easily scannable and have more content per page on the feed. Basically, to turn the feed into a modern implementation the way Facebook, Twitter, Quora, and many others have set up.

However, they were aware that if they did this, then users would be less likely to click through to the images and thus would decrease pageviews. Given MySpace’s declining revenues, the interaction designers there were asked to actively design with the goal of more pageviews. So they added smaller images than they thought optimal, and fewer images per page than they thought optimal, just so that they could generate more pageviews. Basically they were now designing a worse newsfeed to generate short-term revenue.

As I understand, this happened systematically within the product which led to many compromises in the user experience, and the business needs won every time.

When the folks who ought to be the strongest user advocates at the company design for the business goals as a priority, you do not end up with an inspired product experience.

You have to prioritize having a great product experience to end up with a great product experience- it doesn’t happen by accident.

Anyway, the site is still huge and influential in many ways, so let’s hope the team there figures it out and there’s a resurgence in the future.

[ed: I also wanted to add the following answer from Sizhao Zao Yang, co-founder of myminilife, which created Farmville and then was acquired by Zynga]

Sizhao’s additional commentary on this question:

In addition to Andrew Chen‘s comment, I want to emphasize MySpace’s short term perspective seeped into product/engineering such that management actually believed that MySpace was special because people liked to generate pageviews.

MySpace invited a number of the application developers to MySpace last year including Zynga, and I was the Zynga representative. During the sessions, they asked for specific suggestions on product. I told them to make it feed based: increase the size of the pictures, have more descriptions on the activity, etc. Have different ways to surface social content with counters/toasters, and make social feedback very easy with one click functionality. Multiple general managers and product people at MySpace told me that MySpace people just like to click more. I told them that they were on “the wrong side of history.” Little did I know that this session was broadcast to all of MySpace. So, overall, my comments to the management/product ppl/everyone didn’t resonate at all, and most of it was never incorporated in the MySpace 2.0 launch. They said my comments were “interesting, but we’ll see,” with an underlying mood/attitude that I was wrong about the pageviews generating MySpace crowd.

I remember also that when FB was on the rise, MySpace execs would publicly say they were: cooler, more about self expression, celebrities, and that the newsfeed/app platform didn’t matter (in the early days) because it was too geeky. They didn’t know what was going on and positioned MySpace as brand and used lifestyle marketing to promote MySpace. Ultimately, you can probably blame the non- product focused culture, or you can blame the completely wrong judgment/perspective. They just didn’t get it. In b-school/MBA talk, these were strategic/product mistakes and a focus on the wrong metrics.

Unfortunately, when a company is in a downward spiral and think they are differentiating by encouraging pageviews, there’s not a lot you can do to help them. At the end of the day, you sometimes either get it or you don’t, unfortunately, MySpace went viral but didn’t understand social, which is about retention, not customer acquisition, and FB completely out executed/maneuvered them.

I posted this answer to Quora and figured I would share it here as well. You can find me on Quora here.

What are the best metrics for measuring user engagement?
Metrics are merely a reflection of the product strategy that you have in place.

What you are trying to do should lead what you want to measure, not the other way around. It’s for this reason that the blanket questions and answers around “best” metrics are meaningless- the question is, what are you trying to do.

For example, if you are an ecard that is driven based on holiday traffic, your strategy might be:

• people should come to my site at every major holiday
• people should send as many ecards as possible

In this case, your week-to-week retention isn’t important. The only question is whether or not you are sending out ecards, and whether or not you’re a new user or if you came back last holiday.

On the other hand, if you are trying to be more of a communications product, then you might want something like:

• people should come back for short durations every day to check their messages
• people should write messages occasionally, but mostly read a bunch of messages

In this case, you care a lot about DAUs and +1 day and +1 week retention. You might also put in a qualifier to that to make sure that people are actually reading/writing messages and not just showing up to an empty messages area.

So ultimately, the important part is to figure out what you are trying to do and what the expected behavior is around it. Only once you have that should you then ask yourself how you’d validate and test it using metrics.

Silicon Valley is mean to MBAs
This tumblr, Whartonite Seeks Code Monkey, made me laugh.

It’s full of emails from clueless Wharton MBAs which read like this:

LOL right?

This also reminds me of the famous quote on valuing startups:

Add $1,000,000 in value for every engineer.
Subtract $500,000 in value for every MBA.

Here’s why it’s hard: The nerd perspective is, they don’t need you
Much of the reason why it’s insanely hard to find a really good technical cofounder is that the best ones really don’t need you. Or at least they don’t think they need you.

Because there’s an illustrious track record of engineering-founded companies succeeding, spanning from HP to Facebook, there’s a lot of datapoints that say that a 20-yo Stanford computer science major can do it himself, or at least with his other CS roommates. Similarly, the very best alums out of places like Facebook and Google have lots of access to capital, advice, and people- these are all recipes for making you (the biz founder) completely irrelevant.

So I think the right point of view is just to accept that the amount of leverage strong technical folks in the Valley have is just the facts, and you’ll have to work around that.

Remember this:

They are not the code monkey. You are the biz monkey.

That’s just how it is.

Picking the right idea
One key way to mitigate this is to pick the right idea that doesn’t require ridiculous amounts of technical expertise upfront. You can build a great company that’s extremely sales driven rather than product driven in categories like:

• Enterprise sales
• Groupon for X
• Blog/media sites aka Content farms
• Marketplaces
• Ad network

I’m sure I’m leaving many other categories out.

For anything above, a lot of the work is in sales, and the actual technical infrastructure doesn’t require a strong engineer to pull together, at least initially. You’ll need them to scale it, but at that point hopefully you’ll have more money and more momentum.

For the kinds of ideas above, they might be easy enough to build in the short-run that you can get a different kind of coder at first. You can get someone who can code up a site and potentially have some visual design background, rather than an “engineer” who has theoretical understanding of computer science, understands performance tradeoffs, etc. There are more of the former than that latter in the world.

At the same time, note that many of the ideas above may not be particularly exciting to an engineer that wants to play with technologies. So perhaps something that combines the two can help – for example, MySQL is a great example of a cool technology (at the time) but clearly couldn’t have been turned into a company without a lot of business types running around.

Understanding and communicating what you really bring to the table
If you read through the Wharonite Seeks Code Monkey blog,  you can see that obviously they are mostly noobs and don’t know what exactly is the valuable part of what a biz cofounder can do versus not. This is true of many startups, both biz and geek-led, but there is huge overvaluation of the initial idea.

What do geeks really need help with? It’s very simple- there’s a class of purely business-related stuff that adds value:

• selling stuff and making money
• getting partnerships and marketing/distribution of the product
• funding the company
• scalable marketing/monetization strategy (ad arb / viral / freemium / etc.)
• team recruiting, particularly of other engineers and disciplines (not other MBAs please)

If you are an expert at any of the above and can show it, then there’s a lot more value. Very few business folks, particularly newly-minted MBAs (with the exception of Stanford folks) or industry-switchers can really deliver on these though, which is why they’re not bringing much to the table.

Then there’s a class of things that are much more product-oriented, and while it overlaps with the skillset of some engineers, if you have great skills in any of the following, they are clearly valuable too:

• design, especially visual design
• UI/frontend skills – HTML/CSS/JS – even if mediocre!
• copywriting within the product for help text, marketing, etc
• user research and customer development
• usability testing

Again, it all depends on what you’re really good at and what the particular product needs – enterprise might require less of the above, but a more solid initial product might help.

Worst comes to worst, write it yourself
And finally, there’s a nice track record of technical-enough people writing the first version of something and then having great engineers build it up later. Foursquare was like that, for example. More recently, David Binetti of Votizen wrote the first version of his product. I have immense respect for folks who do this, because it means they’re making “good-enough” progress without waiting for exactly the right technical partner to show up.

Any other thoughts or tips to share?
If you guys have other thoughts on ideas or thoughts on this topic, especially from those who are on the technical side, of how to attract and partner with engineers, write me a note in the comments! I’ll update this post as we go.

Per my previous post on Quora’s stats via the Facebook interface, I wanted to also share another hot startup at the moment, Instagram. Obviously you can use Instagram without Facebook- for example, only connecting it to Twitter, but again it shows the relative growth. In this case, as a mobile app, there’s very little data about how many folks are using it. Facebook’s data gives us unprecedented detail.

Here are the graphs, from AllFacebook – you can see they’re on a nice growth curve and doing quite well:

I ♥ Quora. Oh yes, I’m a fanboy.
As my many of my friends and family know, I love Quora and get a ton of value out of it. It’s incredible to see what some of the most intelligent and influential folks in Silicon Valley and beyond have to say, and it’s some of the most valuable content I read every morning.

My prediction for Quora is that it’s going to turn into a huge, important Internet property- it’ll break out of the Valley network easily and inevitably. The experience of Facebook going from college-to-college will inform a strategy of going topic-to-topic, profession-to-profession, and network-to-network. I can easily see how the Q&A mechanics would apply to many other things, especially the political blogosphere and Beltway insiders, the entertainment industry in LA, the media and advertising industry, as well as random everyday stuff. And of course, it’s one of the best executed products I’ve seen in a long time- the interaction design in the product is amazing.

Writing down all the things that product designers and entrepreneurs can learn from studying Quora would be many blog posts in itself. Like I said, I’m a fanboy :-)

Facebook app data shows stats for connected sites and products, including Quora
All the fanboys want to know: Quora’s been growing, but how fast? I recently realized that because Facebook sign-in is used so aggressively by Quora, they will end up getting listed as an app just like everyone else. As a result, Facebook (for better or worse) ends up publishing their DAU and MAU stats, which are then stored and graphed on services like AppData, AllFacebook’s stats service, and others. This establishes a lower-bound for all the core users who have authed to Facebook, but obviously doesn’t count users who bounce or who don’t sign up, etc.

I included the public listings for Quora excerpted from both AllFacebook and AppData. (Both are great services, I’d encourage you to try them). As you can see from the graphs, Quora is growing on a very nice clip, over 2X larger in MAU over the last month. Very nice!

Caveats: These numbers ought to be a lower bound since not everyone is going to either 1) register on Quora, nor 2) connect their Facebook accounts. As a result, I’m sure the real uniques number is much larger, but this is probably a good estimate of the active Quora community. I’ve also seen a lot of Quora answers in my search engine result pages, so I’m sure there’s easily a multiple that come purely to look at the answers and then bounce, that ought to be added to the totals. So again, think of it as a lower bound, but I imagine that the relative growth trajectory is right.

Again, this is really impressive growth and I look forward to seeing Quora’s progress continue.

PS. For the data geeks out there: If someone else does a more thorough analysis on their historic growth rate, sticky ratio, benchmarks/comparisons, etc., please write it in a comment and I’ll link you. And please point out if I’m misinterpreting these stats!

Just returning briefly from blog vacation to share a couple links and slides I had collected on retention metrics. There’s so little public information out there that I wanted to call out the various articles and presentations that actually do contain real data. Given the difficulty of getting exponentially viral on Facebook these days, most companies are focused on great lifetime value and making it work with big ad buys. Obviously good long-term retention is important for that.

After you calculate out some basic cohort retention analysis, where do you go from there? One key thing is comparing it to existing benchmarks to see if things are going well or badly. Below are some of the few public articles and slides with real data in them.

Here’s a collection of public retention data and discussion

First, from Daniel James of Three Rings:

Twitter retention analysis
RJMetrics, another analytics vendor, did this analysis of Twitter a while back: Twitter Data Analysis. Has some nice graphs like so:

Social gaming data for Facebook apps
And finally, Mixpanel based in San Francisco has some aggregated social gaming data:

• Social Game Developers Use Tutorials to Get Crucial Early Retention
• How to analyze Traffic funnels and retention in Facebook apps

Here’s a nice graph from them:

So what to make of all of this?

For now, you’ll have to look through this data yourself. I have a few rough notes I’ve written up about all of this retention data, and time allowing, I’ll publish some of them later. In the meantime, enjoy the presentations and links.

OK, back to blogging vacation :-)

Just a quick FYI on an upcoming conference – here’s the details if you’re interested:

Startup Lessons Learned is the first event designed to unite those interested in what it takes to succeed in building a lean startup. The goal for this event is to give practitioners and students of the lean startup methodology the opportunity to hear insights from leaders in embracing and deploying the core principles of the lean startup methodology. The day-long event will feature a mix of panels and talks focused on the key challenges and issues that technical and market-facing people at startups need to understand in order to succeed in building successful lean startups.

• Conference agenda
• Speaker list
• Where it is

I’ll be on a panel on Minimum Desirable Product with Dave McClure and others. We’ll be talking about the dynamics of incorporating design into a lean startup methodology, with all the difficulties and tradeoffs that entails.

25% discount if you use the link below:

Register for the conference.