Andrew Chen Archives

Will CPE (Cost Per Engagement) advertising ever take off?
I doubt it – the reason is that it’s targeting metrics at the kind of marketers that don’t care too much about metrics.

Broadly speaking, there’s two kind of marketers in the world – a ton could be written about this, so I’ll just provide some sweeping generalizations:

Direct response marketers are companies that are typically very focused on ROI when they buy advertising – often these include companies you’ve never heard of in ecommerce, online dating, financial services, etc., where it’s easy to calculate the value of a customer and they are primarily getting their traffic through paid marketing channels. They like to back everything out to ROI by comparing lifetime value to cost per customer, and if not that, then at least cost-per-action or some similarly concrete metric.

In many cases, these kinds of marketers prefer search marketing, email marketing, telesales, and other things where it’s easy to quantify what’s going on – they stay away from Super Bowl ads though. They prefer CPA and CPC versus CPM or sponsorships.

Brand marketers are companies you’ve heard of and have seen a lot of advertising for – they are typically targeting a large consumer base, they want to position their products differently relative to their competition and don’t have great ways to quantify the value of a customer. For example, Coca-Cola doesn’t know the LTV of a customer nor what the cost-per-customer looks like for a billboard ad they’ve bought.

For these guys, they are used to hiring big ad agencies to help them advertise on billboards, television, the front page of Yahoo, etc. They may buy search marketing, but have different goals than ROI. (For example, they may just want the top ad, and don’t care too much about ROI)

Why CPE is a weird metric for both DR and brands
The reason why cost-per-engagement is a weird metric is that ROI-focused marketers (that is, direct response marketers), don’t care about “engagement.” They want to know if people are going to buy, and if their media spend is going to be profitable.

As a result, the “E” part of CPE is really only a part that brands care about. And yet, they don’t care that much about CPE because they aren’t focused on the cost of the campaign as the #1 priority. Instead, it’s more important where the ads are being placed, how strong the ad creative is being used, etc.

One scenario to demonstrate this: If they could buy the front page of YouTube, even if that had a higher CPE, a brand advertiser would be happier with that than being shown in random footers of YouTube (the “remnant”) even at a lower CPE. They are looking to establish their brand, not optimize their spend.

What will be prevalent instead?
I think even with the advent of lots of ad opportunities on social sites, the dominate business model will still be CPM/sponsorships for brand advertisers, and CPC/CPA for direct response. Basically, nothing much will change.

If it turns out that CPE correlates to CPA/CPC, then DR marketers will end up liking it.

Also, CPE might turn into a secondary metric that you use alongside really strong placement of ads- maybe as a way to establish a bonus or upside on the campaign, but I don’t think it’ll ever happen that the dominant form of advertising on the web will be that ad agencies will put in a CPE “bid” into self-serve systems :)

I answered this question on Quora – more great answers over there.

Facebook, early 2006
Sometimes, you need to be horribly, embarrassingly wrong to remind yourself to keep an open mind. This is my story of my failure to understand Facebook’s potential.

In 2006, I was working on a new ad network business that experimented a lot with targeting ads with social network data, broadly known as “retargeting” now. The idea was that we’d be able to take your interests and target advertising towards them, which would lead to higher CPMs. As part of this project, we did a meeting with Facebook when they were ~12 people. I had read bits and pieces about the company in the news, but since I was a few years out of college, I hadn’t used their product much. We got a meeting and since I was based in Seattle at the time, I flew down with some coworkers and chatted with them at their new office in Palo Alto.

We met the Facebook team at their office right next to the Sushitomo on University Ave. The place looked like a frat house – a TV and video game console on the ground, clothes and trash everywhere – the result of a handful of young people working very hard. After waiting a few minutes, we were escorted into a meeting room where we met with Sean Parker, Matt Cohler, and Mark Zuckerberg. Sean led the meeting, and told us a lot about Facebook, the amazing job he did raising their recent VC round from Accel, and all the good things that were happening at the company. Mark and the other folks there didn’t say a thing.

Ultimately, we didn’t get to work with them though we did eventually sign 1000s of publishers including MySpace, AOL, Wall St. Journal, NY Times, and others. But that meeting opened my eyes and convinced me of a horribly wrong thing: Facebook would never be a billion dollar company.

The metrics for Facebook – high growth, very low CPMs
As part of our meeting, we talked a bit about the metrics around Facebook, and I was immediately struck by a few things:

• Facebook was growing fast- very fast, and impressively handled by a super young team (like me!) sitting on a site with millions of uniques/month
• Their CPMs were terrible, lower than $0.25 (the revenue earned per thousand ad impressions) and the site was covered (at the time) with crappy remnant ads like online poker, dating, mortgages, etc. (ironically, which we now associate with MySpace)
• They didn’t know much about advertising, and that their CPMs were really bad and unlikely to improve- their monetization strategy seemed superficial at best

From these numbers, I did a quick calculation:

$0.25 CPM * 5 billion ad impressions per month max?
= $1.25M/month = $15M/year = $150-300M value business?

I figured that Facebook hitting 5B ads/month would be incredible – after all, it was just a college social network, right? Hitting 5B impressions/month would make Facebook bigger than our largest client at the time, ESPN.com, a top 10 internet property. The only thing larger were big portals like Yahoo, MSN, and AOL. The idea that Facebook would one day be bigger than all the portals never crossed my mind.

I was confident especially in the CPM number staying low because I had multiple proprietary datapoints from across the industry – from MySpace, Friendster, Hi5, Dogster, and many other social networks. I was convinced that I had a unique understanding about Facebook’s true potential – that convinced me even more that it could never be big.

And of course, I was totally, horribly wrong :)

The case at Yahoo for buying Facebook
While I was doing these calculations after my meeting, Yahoo was also doing a similar analysis on the value of Facebook for their ill-fated attempt at buying the company. I would first read about it in the WSJ, but later saw this fascinating slide on Techcrunch.

The slide below starts out with a projection of how many registered users Facebook had at the time and projected very logically what it would mean for them to saturate more of the core userbase of “high school and young adult” – I’m sure at the time, these felt like aggressive projections to ultimately be able to justify a big purchase price:

If you look at these numbers and compare them to what really happened, it’s pretty hilarious. Comparing their projected 2010E and what actually happened, they were only off by a few hundred million users!

Furthermore, I would say that even the Yahoo numbers were very optimistic about the increase from CPM going from $0.25 to >$5 over time. There were a lot of problems with brand advertisers putting themselves next to user-generated content that had not been worked out, and these numbers would have also ultimately involved Facebook doing homepage takeovers and such. And in fact, it’s true that no large user-generated content or social networking site has been able to generate CPMs close to the $5 level, at scale.

So what was wrong with my reasoning?
Ultimately, all my conclusions were wrong by several orders of magnitude – Facebook would go on to become the #1 site on the internet and would break all attempts at reasoning based on historical datapoints, interpolation, expert opinions, etc.

To contrast how silly my reasoning turned out to be:

My 2006 prediction: Facebook would max out at 3-5B pageviews/month
Reality: Facebook is at 1 trillion pageviews/month, and growing

I was ultimately right on the CPMs not improving by much, but it didn’t matter because I was off by 200-300x on pageviews/month! Total fail. The big insight, of course, was that Facebook wouldn’t just stay a social network for college students – ultimately the product targeted the market of everyone in the world. Confined within this the college niche, the idea that Facebook would one day reach a trillion pageviews per month seemed ludicrous. But because of the vision of the founding team, Facebook broke through this niche to build a new product that the world had never seen, and got to the numbers I had never predicted.

The most exceptional cases defy simple pattern-matching
As I mentioned in my previous post on group think vs innovation in Silicon Valley, there’s a strange contradiction between the mental tools we use to analyze and categorize businesses versus what it looks like when there’s an exceptional company that takes off. Pattern matching, deductive reasoning, and expert opinion tell you how things work in the “typical” case, but of course, we’re not interested in the typical case – we’re trying to find the exceptional ones, the rocketship companies that define the startup landscape.

That’s exactly when our logical reasoning and historically-based reasoning fails us the most.

For example, after years of failures from the entire category of social shopping sites like ThisNext, Kaboodle, and others, Pinterest has become the hottest company of the year. After years of Google impressing upon all of us that every startup needed to have an algorithm called X-rank and a 10X technology advantage, a simplistic webapp known as Twitter would emerge. And after 10 VC-funded search companies were started, and people at Yahoo thought search was a loss-leading feature that would best be outsourced, Google emerged. The list goes on and on.

Legendary VC Mike Moritz, who invested in Google/Yahoo/PayPal/Apple/etc has a relevant quote here:

I rarely think about big themes. The business is like bird spotting. I don’t try to pick out the flock. Each one is different and I try to find an interestingly complected bird in a flock rather than try to make an observation about an entire flock. For that reason, while other firms may avoid companies because they perceive a certain investment sector as being overplayed or already mature, Sequoia is “careful not to redline neighborhoods.

There’s a lot to be said for investing in the ugly duckling. When Don Valentine led Sequoia Capital’s investment in Cisco, many others had passed on the husband and wife founding team of Len Bosack and Sandy Lerner.

Never has a more profound thing been said about birdspotting :)

The biggest lesson I took away
The concrete lesson to be learned from this is: In the modern era, business models are a commodity. I never want to hear about people asking, “But what’s their business model?” because in a world where you can grow a userbase of 1 billion in a few years, displaying remnant ads and getting a $0.25 CPM will do. Or just throw some freemium model on it, and monetize 1% of them. If you can build the audience, you can build a big business.

The more abstract lesson to learn is: Be humble, and keep an open mind towards weird new companies. After a few years in Silicon Valley, you can gather a lot of useful heuristics about what’s worked and what doesn’t work. That will help you most of the time, but when it comes to the exceptional cases, all bets are off. So keep your mind open to weird, young companies that you meet that don’t fit the established pattern: Maybe the founders will all be recent MBAs, or be a spinout from a stodgy old corporation. Or maybe it’ll be in a slow-moving market, or it’ll be a married couple, or there’s 10 founders, or some other stereotypically bad thing. Remember that you’re helping/investing/working for the company right in front you, not a mutual fund of all companies with that characteristic!

If you had looked at social networking companies as a group, as I did, you would have found a flock of companies with questionable business models. However, if you had been prescient enough to pick out Facebook specifically, then you would have seen a company break through all historic precedents and become a huge success. Hats off to all 12 employees I met that day in 2006.

Once you’ve been working in Silicon Valley for a bit, you’re often offered advice such as:

• Are you launching at X conference?  … where X is whatever hot conference is coming up, like SXSW or Launch
• Do you have an X app? … where X is whatever new platform just emerged, be it Open Social, iPhone, or whatever
• Have you pitched X venture capitalist? … where X is a prolific headline-grabbing investor with a recent hot deal
• You should do feature X that company Y does! … where X is some sexy (but possibly superficial feature) that a hot startup has done
• Do you know what your X metric is? … where X is some metric a recent blog post was written about
• Have you met X? … where X is some highly connected expert in the field
• Maybe you should pivot into X space! … where X is a space with a hot company that just raised a ton of funding
• Did you think about applying framework X to this? … where X is a new framework, be it gamification or viral loops or Lean

Sound familiar? I confess that I’ve both received and given much advice along the lines of the above. I call it “advice autopilot.”

The perils of “advice autopilot”
Advice autopilot is when you’re too lazy to think originally about a problem, instead regurgitate whatever smart thing you read on Quora or Hacker News. If you’re a bit more connected, instead you might parrot back what’s being spoken at during Silicon Valley events and boardrooms, yet the activity is still the same – everyone gets the same advice, regardless of situation. The problem is, the best advice rarely comes in this kind of format – instead, the advice will start out with “it depends…” and takes into account an infinite array of contextual and situational things that aren’t obvious. However, we are all lazy and so instead we go on autopilot, and do, read, say, and build, all the same things.

That’s not to say that sometimes generic advice isn’t good advice – sometimes it is, especially for noob teams who are working off an incomplete set of knowledge. Often you may not have the answers, but the questions can lead to interesting conversations. You may not be able to say “you should do an iPhone app” but it’s definitely useful to ask, “how does mobile fit into this?” This can help a lot.

The other manifestation of this advice autopilot is the dreaded use of “pattern matching” to recommend solutions and actions.

Pattern-matching in a world of low probability, exceptional outcomes
One of Silicon Valley’s biggest contradictions is the love of two diametrically opposed things:

• The use of pattern-recognition to predict the future…
• … and the obsession with a small number of exceptional successes.

Exceptional outcomes for startups are limited – let’s say it’s really only 5-10 companies per year. In this group, you’d include companies like Facebook and Google that have “made it” and hit $100B valuations. On the emerging side, this would include startups who might ultimately have a shot at this, like Dropbox, Square, Airbnb, Twitter, etc. This is an extraordinarily small set of companies, and it isn’t much data.

The problem is, we’re hairless apes that like to recognize patterns, even in random noise. So as a result, we make little rules for ourselves – Entrepreneurs who are Harvard dropouts are good, but dropping out of Stanford grad school is even better. It’s good if they start a company in their 20s unless they’re Jeff Bezos. Being an alum of Google is good, but being an alum of Paypal is even better. Hardcore engineers as founders is good, but the list of exceptions is long: Airbnb, Pinterest, Zynga, Fab, and many others. And whatever you do, don’t fund husband-wife teams, unless they start VMWare or Cisco, in which case forget that piece of advice.

As anyone who’s taken a little statistics knows, when you have a small dataset and lots of variables, you can’t predict shit. And yet we try!

The intense focus on a small set of companies also introduces a well-known logical fallacy called Survivorship Bias. Here’s the Wikipedia page, it’s interesting reading. Basically, the idea is that we draw our pattern-recognition from well-publicized successful companies while ignoring the negative data from companies that might have done many of the same things, but end up with unpublicized failures. We’re all so intimately familiar with stories like “two PhDs from Stanford start Google” that we ignore all the cases where two PhDs from Stanford try to start a company and fail. Or similarly, YCombinator has built a great rep on companies like Airbnb and Dropbox, and yet you’d think that if you invest in 600+ startups that you’d get a few hits. Because of factors like this, it might seem as though A predicts B when in fact, it does nothing of the sort – we’re just not taking the entire dataset into account.

Conformity leads to average outcomes when we seek exceptional outcomes
The problem with giving and taking so much of the same advice is that ultimately it breeds conformity, which is another way if saying it reduces the variance in the outcome. And if you conform enough, you end up creating the average outcome:

The average outcome for entrepreneurs is, your startup fails.

Lets not forget that. And so one part of Naval Ravikant’s talk on fundable startups that resonated with me is the idea of playing to your extremes. He says in the talk:

“Investors are trying to find the exceptional outcomes, so they are looking for something exceptional about the company. Instead of trying to do everything well (traction, team, product, social proof, pitch, etc), do one thing exceptional. As a startup you have to be exceptional in at least one regard.” –Naval Ravikant @naval

Be extremely good at something, and invest in it disproportionately relative to your competition – this gives you the opportunity to actually create an extreme outcome. Otherwise, the average outcome doesn’t seem so good.

The flipside of innovation
The funny thing with all of this, of course, is that this is what innovation looks like. The remarkable ability for practical knowledge to disseminate amongst the Bay Area tech community is what makes it so strong. Before something becomes autopilot advice for a wide variety of people, often a small number of hard-working teams who know what they’re doing leverage it to great success. Follow those people, and you might find yourself successful – just like them.

So the billion dollar question is – how do you separate out trendy/junk advice from what really matters?

… well, it depends!

I recently dug through Favstar.fm and found a bunch of the tweets over the last few months that were saved/retweeted the most. Wanted to save them here for posterity:

Teardowns

• Facebook, Google, Twitter, eBay, YouTube, Wikipedia, Amazon, Hotmail, Blogger, Apple: How they used to look http://t.co/fL2zDHu0
• The Secret To Pinterest’s Astounding Success: A Brilliant Sign-Up Process You Should Copy http://t.co/AsGi9pBx
• Airbnb’s first pitch deck http://t.co/3BTSY6dO
• Android Gripes, Why do apps from the same company look worse on Android than on iPhone? http://bit.ly/h19EKL
• Why Angry Birds is so successful and popular: a cognitive teardown of the user experience http://bit.ly/dN3W3d

Compilations

• 11 Books Every Leader Should Read – Bob Sutton http://t.co/gwViu9DQ
• 5 Articles on Rapid Prototyping you Should Have Read – LaunchBit http://t.co/cF10LzKI
• Platforms and Networks: Managing Startups: Best Posts of 2011 http://t.co/mdeyv3wK
• 5 Former Design Trends That Aren’t Cool Anymore (So Stop Using Them) | Design Shack http://t.co/59prlt1N

Quotes

• “Success is like being pregnant: everybody congratulates you but nobody knows how many times you were fucked…” via @NatalieSEO
• “No matter how beautiful, no matter how cool your interface, it would be better if there were less of it.” –Alan Cooper (via @destraynor)
• Very appropriate for entrepreneurs: “A casual stroll through the lunatic asylum shows that faith does not prove anything.” -Nietzsche
• grid is the new feed, custom cover is the new custom background, real name is the new username, and repin/reblog is the new embed code.
• “The easiest way to get 1 million people paying is to get 1 billion people using” -Phil Libin, CEO Evernote http://bit.ly/f1SY7U

Media

• Steve Jobs – 25 years old, rare footage of him presenting about early Apple http://t.co/AiNbyW0S
• INFOGRAPHIC: How rich are the superrich? http://bit.ly/hxavXB
• INFOGRAPHIC: Carbs Are Killing You http://t.co/FMdZ5pEG
• Amazing PDF with lots of conversion %s to compare your product against http://bit.ly/e2t6Ip

Articles

• Bye Bye, Long Tail http://tcrn.ch/gfTOYP
• interesting lesson on listening to customer self-reporting: Walmart’s $1.85 billon dollar mistake – Daily Artifacts http://t.co/8GLLajdx
• Design is Horseshit! http://t.co/12vw2MZN
• If You’re Competing On Features You’ve Already Lost http://bit.ly/jbfk8p
• My answer on Quora to: What are the best metrics for measuring user engagement? http://qr.ae/OyAD
• i find myself explaining scalable startups vs smallbiz / lifestyle all the time. This blog breaks it down. Via @sgblank http://t.co/DJNSkTx
• Why Do Some People Learn Faster? | Wired Science | Wired.com http://t.co/Zd53Z3Bl
• The Top Ten Signs the Valley is on Tilt Again http://bit.ly/fjZUhk

Here’s the form as a link if you can’t see it – questions, thoughts, etc appreciated.

Lobby of the Pixar offices in Emeryville, CA

“My product isn’t quite there yet.”
You’ve said this before. We all have.

Anyone working on getting their first product out to market will often have the feeling that their product isn’t quite ready. Or even once it’s out and being used, nothing will seem as perfect as they could be, and if you only did X, Y, and Z, then it woould be a little better. In a functional case, this leads to a great roadmap of potential improvements, and in a dysfunctional case, it leads to unlaunched products that are endlessly iterated upon without a conclusion.

About a year ago I visited Pixar’s offices and learned a little about this product, and I wanted to share this small story below:

Over at Pixar…
Matt Silas (@matty8r), a long-time Pixar employee offered to take me on a tour of their offices and I accepted his gracious offer. After an hour-long drive from Palo Alto to Emeryville, Matt showed up while I was admiring a glass case full of Oscars, and started full tour. I didn’t take great photos, so here’s some better ones so you can see what it’s like: Venturebeat, Urbanpeak.

I’ve always been a huge fan of Pixar – not just their products, but also their process and culture. There’s a lot to say about Pixar and their utterly fascinating process for creating movies, and I’d hugely recommend this book: To Infinity and Beyond. It gave me a kick to know that Pixar uses some very collaborative and iterative methods for making their movies – after all, a lot of what they do is software. Here’s some quick examples:

• Pixar’s teams are ultimately a collaboration of creative people and software engineers. This is reflected at the very top by John Lasseter and Ed Catmull
• The process of coming up with a Pixar movie starts with the story – then the storyboard – then many other low-fidelity methods to prototype what they are ultimately make
• They have a daily “build” of their movies in progress so they know where they stand, with sketches and crappy CGI filling holes where needed – compare this to traditional moviemaking where it’s only at the end
• Sometimes, as with the original version of Toy Story, they have to stop doing what they’re doing and restart the entire moviemaking process since the whole thing isn’t clicking – sound familiar, right?

The other connection to the tech world is that Steve Jobs personally oversaw the design of their office space. Here’s a great little excerpt on this, from director Brad Bird (who directed The Incredibles):

“Then there’s our building. In the center, he created this big atrium area, which seems initially like a waste of space. The reason he did it was that everybody goes off and works in their individual areas. People who work on software code are here, people who animate are there, and people who do designs are over there. Steve put the mailboxes, the meetings rooms, the cafeteria, and, most insidiously and brilliantly, the bathrooms in the center—which initially drove us crazy—so that you run into everybody during the course of a day. [Jobs] realized that when people run into each other, when they make eye contact, things happen. So he made it impossible for you not to run into the rest of the company.”

Anyway, I heard a bunch of stories like this and more – and as expected, the tour was incredible, and near the end, we stopped at the Pixar gift shop.

There, I asked Matt a casual question that had an answer I remember well, a year later:

Me: “What’s your favorite Pixar movie?”
Matt: *SIGH*
Me: “Haha! Why the sigh?”
Matt: “This is such a tough question, because they are all good. And yet at the same time, it can be hard to watch one that you’ve worked on, because you spend so many hours on it. You know all the little choices you made, and all the shortcuts that were taken. And you remember the riskier things you could have tried but ended up not, because you couldn’t risk the schedule. And so when you are watching the movie, you can see all the flaws, and it isn’t until you see the faces of your friends and family that you start to forget them.”

Wow! So profound.

A company like Pixar, who undoubtedly produces some of the most beloved and polished experiences in the world, ultimately still cannot produce an outcome where everyone on the team thinks it is the best. And after thinking about why, the reason is obvious and simple – to have the foresight and the skill to refine something to the point of making it great also requires the ability to be hugely critical. More critical, I think, than your ability to even improve or resolve the design problems fast enough. And because design all comes to making a whole series of tradeoffs, ultimately you don’t end up having what you want.

The lesson: You’ll always be unhappy
What I took away from this conversation is that many of us working to make our products great will never be satisfied. A great man once said, your product is shit – and maybe you will always think it is. Yet at the same time, it is our creative struggle with what we do that ultimately makes our creations better and better. And one day, even if you still think your product stinks, you’ll watch a customer use it and become delighted.

And for a brief moment, you’ll forget what it is that you were unhappy about.

Special thanks to Matt Silas (@matty8r, follow him!) for giving me a unique experience at Pixar. (Finally, I leave you with a photo of me posing next to Luxo Jr.)