Andrew Chen Archives

A great new blog from IMVU cofounder
I recently have been reading a new blog by Eric Ries, one of the founders of IMVU. The URL is http://startuplessonslearned.blogspot.com and the RSS feed is here. I’ve found it to be a great source of thinking around both games and metrics, two of my favorite topics. It’s quickly become one of my favorite blogs.

For those of you guys who haven’t spent much time looking at IMVU, here’s a good summary from GigaOm, including the statistics below:

• About 1.7 million user-generated items have been uploaded to the IMVU catalog
• Users create most content offline with industry-standard modeling tools such as Maya and Blender
• Individual content creators can take their earnings and sell them on third party web sites
• As with Second Life, IMVU users retain intellectual property rights to content submitted to the IMVU catalog
• About 100,000 users are registered to be IMVU content creators, but the number of consistent creators is in the tens of thousands
• IMVU users often sell their credits to other users via third-party web sites — also fine with Cary. “We observe it sort of casually,” he said.
• Demographic: Largest cohort is teens; 60 percent of users are female, 60 percent are American
• 50-70K concurrency is typical, with peaks in the high 70s
• IMVU areas are more like virtual rooms restricted to 10 avatars or fewer; Cary estimated 90 percent of those room interactions are person-to-person, anyway.

Eric has already written some great articles from his experience at IMVU, including articles on:

• common distribution models for online businesses
• gaining distribution advantages on the iPhone
• doing market research for product feedback
• a/b testing products

I’d encourage you to read more games+web+metrics goodness.

KP beefing up on games
Another interesting note is that it looks like KP has recently been beefing up their games focus. These days, Eric is a venture advisor for Kleiner Perkins, after 4 years at IMVU. He also joins the recently made partner Bing Gordon, a cofounder of EA, who quickly went on a tear making investments in games companies like:

• Facebook social gaming powerhouse Zynga
• mobile games company Ngmoco (from EA veterans)
• GoGii (from the Jamdat founders)

They also have Chi-Hua Chien, who sourced the Facebook for Accel Partners, and who is obviously a great resource for the intersection of social web and games.

Anyway, I hope you enjoy the blog!

My friend Dan Cook over at Lost Garden put together some slides on productivity research which I thought I’d share below. Much of it references the typical “crunch” periods when delivering large traditional games:

Last week’s Twitter links

If you want to grab these in real time, you can follow me on Twitter at @andrew_chen.

Wow, there has been a ton of investment into games-related startups.

I hadn’t seen this table until today – I thought I’d share it. It shows the tremendous amount of investment that’s been thrown into the “new” games space, although I’ll note that some of the companies on this list are relatively mature. But lots of them were funded in the last year or so. Given that there are still tons of stealth companies out there, I bet that a comprehensive list would be >2x as large.

Read more about this on Jussi’s excellent games blog:

UPDATE: There was a set of new updates on Jussi’s blog, which I’ve added here.

I’m speaking at the Virtual Good Summit (plus a discount code)

I’ll be speaking at this year’s Virtual Goods Summit on October 10th, 2008 and I just wanted to share some details about the event. The Virtual Goods Summit 2008 is a one day conference focused on the emerging market opportunity for virtual goods and economies. This year’s conference will build on the success of last year’s event and dive even deeper into some of the key themes facing this emerging industry. The conference will feature a blend of panel discussions and expert-led breakout sessions covering everything from getting started with virtual goods to maximizing the revenue opportunity around virtual goods and virtual economies.

I’ll be giving a talk during one of the breakout sessions – Daniel James and I will be talking about how to apply a metrics-driven approach to analyzing and understanding a virtual goods business, using some of Daniel’s experiences at Whirled as an example. There will be a ton of great speakers and presenters at the conference – I’ll be joined by Amy Jo Kim (Shufflebrain), David King (Lil Green Patch), David Perry (Acclaim), Brian Balfour (Viximo), Sean Ryan (Meez), Nabeel Hyatt (Conduit Labs), and many other leading thinkers in this space.

I hope can you can make it out to the Virtual Goods Summit 2008! You can save 10% on the cost of registration by using the code “ANDREWCHEN” at checkout.

Conference Website: http://www.vgsummit2008.com 
Registration: http://vgsummit2008-andrewchen.eventbrite.com

Last week’s Twitter links

If you want to grab these in real time, you can follow me on Twitter at @andrew_chen.

As always, I can’t promise they are all work-related 

Quick intro to getting set up on analytics
I’ve been asked a few times recently, “Wow, these analytics you write about are great, but how does a startup begin to bite off the relevant parts?” This blog is to address these questions.

First, let me recommend reading a previous blog, called omg I’m just a startup, I can’t do those fancy metrics. In it, I cover some more general philosophical ideas about how to approach what to measure and what not to measure. Might be worth taking a look if it’s not too important.

Now let’s move on to the first couple topics:

Step 0: Pre-product
Initally, the product development process should likely be focused on big-picture qualitative information, like whether or not your business is addressing the right audience as well as the preferences for that audience. So don’t measure anything yet :)

Instead, spend your time gathering qualitative data, interviewing users, understanding the problem-behind-the-problem you’re trying to solve, and prototyping concepts.

Do this for a couple weeks!

Step 1: Prototypes
As you create prototypes of your product, you should throw up some free, simple analytics to get you some rough ideas of what’s happening inside the functionality. This likely means something like Google Analytics, although there is a very large universe of equivalent tools out there as well.

Google analytics can’t really tell you much – it’s not very actionable. The main things I like to look at are new versus return visitors, top content pages, what pages are causing bounces, etc. Again, at this stage you are still primarily driven by qualitative research and ideas, and it’s hard for analytics to drive much of your thinking.

This prototype phase might last a month or a couple months

Step 2: Traffic comes in, so data must be collected
As your product begins to mature, and you get a better sense for what you are trying to do with it, the next thing I might do is to figure out what the important pieces of data are, and confirm that it’s being measured. Nothing is worse than throwing data away that you might want to use later.

Generally, I prefer a single table or log that can be queried later that stores events. The right granularity of events is at the “business” event level, like “someone updated their profile” or “someone downloaded a video” rather than at the URL level. This ensures that you are getting a good amount of information from the logs but it’s not so overwhelming that you’re blowing up your database.

You might, for example, hold events in the rough key/value form:

user_id, event_name, value, datetime

Where it might look something like:

1000, profile.photo.update, 1, 9:30AM 3/14/2008

Make sense?

I prefer to start out via SQL so that the manipulations of the data are easy, although many large-scale systems eventually move to flat-files of some format.

Design-wise, here are some things to consider:

• What’s your “event” hierarchy and what level of granularity do you want?
• Do you want your analytics DB to be the same as your webapp DB?
• How should you join data between your webapp stats and your analytics stats?
• Where does it make sense to throw data away versus trying to store it forever?
• How do you pass data into the analytics DB? Via a JS interface called by the client (like Google Analytics) or server-side within your methods?

There’s really no wrong answers to the above – I’ve seen it done in many ways.

Step 3: Identifying your user flows
Every web product ultimately has a bunch of user flows contained within it. For example, there might be a series of flows in how users come into the site, starting with ads, SEO, or otherwise. Similarly, once they get on the site, you might be trying to optimize their usage of their site.Identifying these flows is key since you are trying to find the”critical path” that is then optimized. Figure these flows out, and make sure you’re collecting the right data to optimize.
A good place to learn about these user flows is to read about ecommerce “funnels” and how folks go about breaking those down and optimizing them.

Step 4: Trying ad hoc queries
As users are coming into the system, it can then become a good idea to start gathering data into a standard format. This means creating a small set of queries that you might try to run to learn more about the critical paths that users are taking, and where you can adjust their flow. At this point, it’s important to have the vision of the product become fairly stable so that you are starting to optimize the edges rather than reinventing the core constantly.

The kinds of ad hoc queries worth doing revolve around whatever are the tactical goals of your business. If you are trying to come up with a monetization strategy, you should try to figure out your average order size and what percentage of users that start a buying process finish it. Once you create a small list of these queries, then you can start to formalize the ideas into specific metrics that you track daily.

If any ad hoc queries return data that is similar to what you could get out of Google Analytics (for example, aggregate numbers like pageviews and uniques), it’s probably a dumb idea to try to do those in-house. Don’t do more work than you have to! Instead, the only homegrown stuff should be so specific to your business that it’s easier to do in-house than to shoehorn it into a 3rd party analytics stuff. Don’t waste your effort on numbers a off-the-shelf analytics pacakge would get you.

Assuming that your product is stable, most startups will want to tackle this within the first few weeks (but obviously not until you have data)

Step 5: Formal in-house reporting
Once the product features (and thus the user flows) are sufficiently mature to invest in this area, then it makes sense to formalize out the reports. Typically I would start out with a series of pretty plain HTML pages using tables that just print out SQL queries. You can add finishing touches like percentage %s, key ratios, etc. as you go. I generally invest zero time into cute visualizations and graphs, and prefer to read the key numbers.

How many reports should you generate? I find that it’s pretty addictive to build reports and get a clear understanding of what’s actually happening in your product. So create enough that you can make key decisions, but don’t go too far either – you’ll hit diminishing returns quickly. Generally, 2-3 reports are good enough to start, but ultimately you’ll probably track dozens of dashboards each focusing on specific aspects of your business like.

• System performance and uptime
• User acquisition via each method you use
• Aggregate metrics
• Retention
• Engagement
• Content creation?
• Ads and monetization?
• Pricing and revenue?
• etc.

Anyway, get enough data but not too much – it’s a fine balance. For timing, it probably only makes sense to do this once the product is quite stable and the key user flows are stable as well. This is likely at least a month or two out from the prototype stage.

Step 6: Too much data! Reports are too slow!
If you’re lucky, eventually your reports will be too slow. At Revenue Science, we were gathering somewhere like 1 billion pixel hits per day, and that had to be translated into reporting. Ouch. So you likely will go through a couple specific steps:

• Reports will initially query the production server – eventually this doesn’t work and slows down the site
• Reports and data are then moved off to a slave machine, where the queries still happen in real-time – but eventually this doesn’t work either because it’s too slow and there’s too much data
• Reports and data are then pre-processed every hour, and then served up – which is fine, until your queries take too long, and you have go keep moving
• Data is then replicated across a number of slave machines, where the pre-processing happens
• etc.

There are many many layers of incremental improvements you can make here – but the toughest nut to crack, in the case where your web product is HUGE is that you will be inserting more data into the system than the system can process within a reasonable time.

Then the more exotic technologies like Hadoop, HBase, Hypertable, etc start to make a difference. Most sites don’t have to deal with this so I’ll stop here!

Conclusion
Eventually, most serious analytics-driven businesses have to build their own internal analytics. It’s not pretty, but it has to be done. Hopefully the above article gives some background on the key issues you might want to look at as you scale up your product.

If you liked this blog post, please recommend it to a colleague and/or click here to get updates via email or RSS.

New mobile blog from KP
This morning, my friends at KP launched the iFundVC blog to specifically cover the iPhone and the open mobile ecosystem that’s emerging. Their first blog post details a number of investments that they’ve made in the space, in particular Pelago, iControl, ng:moco, and GOGII. Anyway, I’d encourage you to read more about this legendary firm’s perspectives on the impending transformation of the mobile landscape.

One of the important comparisons for me is between the Facebook platform versus the iPhone platform. They are two of the biggest ideas of the past few years, one to open up the monster traffic enabled by social networks, and the other to crack open the walled gardens of mobile carriers. 

Is the mindshare of hobbyist developers going to the iPhone platform?
A key issue is where the developers are going to go – most of my technical friends have spent the last year and a half hacking away building apps on the Facebook platform. However, as time as passed, the platform has grown less attractive for a number of obvious reasons – to resummarize:

• Lack of stability: Facebook may slap down your app and replace it with their own
• Lack of monetization: Beyond remnant ad networks, there isn’t much you can do with the <$0.25 CPM inventory
• Lack of investment: Many angel investors are no longer investing much into the space, as mature companies like Slide, Zynga, and others have established themselves
• Lack of market excitement: Everyone wants to find the next new thing!

I want to also note that for the hobbyists, it’s not even the money that really concerns them. They just want to tinker around with stuff and build cool products, and there’s a set of sexy features like geo-location and SMS that allow them to experiment with the interactions.

As a result, I’d argue that a class of social network “refugees” are forming who are looking to build the next new thing, and the mobile platform (iPhone and otherwise) will start looking pretty attractive for them. I wonder how many of the lessons learned on the FB platform, like social gaming, viral distribution, etc. will port over to iPhone as well.

Comparing the Facebook platform and the iPhone platform
I also asked the audience to fill in a couple of the datapoints around comparing the two platforms, and what axes they would use. Here are some of the thoughts I got back:

I’m sure there is a much more comprehensive table out there. Noah also had a fantastic table here that was very funny. Anyway, in the comments there was a fun discussion of whether or not Facebook’s huge relative audience was more important, or if having more direct monetization was more important – quite a good exchange.

Ultimately, some of the >1M adoption rates of free iPhone apps like Tap Tap Revenge show that there may be enough audience there to generate some sizeable returns. So regardless of whether or not it’s a favorable or not platform, it’s attractive enough (especially with the stronger monetization channels) for developers to get excited about it. The after-effects of the open mobile platform, across all the carriers, is likely a strong enough trend to get VCs and entrepreneurs to decide that “it’s finally here.”

Are you doing cool stuff on the iPhone platform?
If you are, drop me a note at voodoo [at] gmail, I’d love to hear more about it.

If you want to grab these in real time, you can follow me on Twitter at @andrew_chen.

As always, I can’t promise they are all work-related 

• Blog essay list is finally up to date! Added the most recent stuff… http://tinyurl.com/5m2qdq
• New site: http://AmINormalOrNot.com? My fave section is the Most Weird, with “I eat poopy” and “I have never seen star wars”
• scientific american article on how sleeping and dreaming helps you get better at tetris: http://snurl.com/3oddd 
• lots of neat aggregated stats on angel investing (via @fredwilson):http://www.angelsoft.net/industry/index.seam
• top 10 free to play publishers, representing $2B in revenue per year: http://tinyurl.com/5knx4d 
• david shen on competing via design versus features, depending on product category lifecycle: http://tinyurl.com/5jzspv 
• interview with will wright on spore “I think of the theme first, the genre last.” http://tinyurl.com/66rtk6 

Here’s the transcript:

Intro
Hi everyone. My name is Andrew Chen and I’m a blogger and entrepreneur, focused on consumer internet products here in San Francisco.
This afternoon, I gave a quick talk at O’Reilly Alphatech Ventures called “Growing Renewable Audiences” which I thought I would share.

The primary focus of this talk is about the fact that every internet startup needs to grow their audience to be successful. And for venture scale returns, you need 10s of millions of users, not 10s of thousands. So ultimately, this divides the growth efforts that you can use into two types: renewable and not renewable. I’ll get to defining what this means in the next few moments.

Techcrunch and press traffic
Let’s start with this Alexa graph. There’s really nothing special about it – they’re just another startup trying to grow their traffic. I will say, however, that this startup is one of the many that presented last year at Techcrunch 40, which leads to the question, how many of the companies that launched this week at Techcrunch 50 will look like this in a year?

My point is not to pick on Techcrunch, which I love and read on a daily basis. Techcrunch is great for getting introduced to potential partners and investors, but for a consumer internet product that’s trying to drive users to a site, it doesn’t do much. The reason is that press and blog traffic are ultimately nonrepeatable, nonsustainable audiences that doesn’t stick. You get the spike in traffic, and it melts away as quickly as it show up.

In fact, I’ll describe press and blog traffic as “fool’s gold” because of the associated emotions that it brings. It’s easy to overestimate the impact of this kind of traffic because it just feels good to have your name and company featured. It strokes your ego. You might get a bunch of inbound emails from other press and partners, and all of these things can contribute to a feeling that you’re on your way to getting tons of traffic. Problem is, you inevitably become yesterday’s old news.

So again, this is the kind of one-time traffic that I definitely discount and stop focusing on. Instead, let’s talk about what it means to build sustainable, renewable audiences.

Renewable audiences versus not
What’s the definition of renewable? I’m interested in the last part of this definition, which is to define it as “inexhaustable or replaceble by new growth.”

For a startup, this can only mean one thing: the hard-earned audiences you generate via buzz, beta testers, and other sources must beget more audiences. This means that you should focus on building repeatable, sticky traffic that will stay for the long term rather than getting the quick hits. This is the only way for startups to get big and create venture-scale returns.

So let’s talk about a bunch of methods that are renewable versus not renewable. I’ll start with the non-renewable stuff first. As I said, pr and blogs. Same thing for talking at conferences. PR, blogs, and conferences are great to attract investors, partners, potential employees, but terrible for trying to scale to 10s of millions of users.

Same for community building events like meetups, contacting influencer communities, and so on. This can help you build out intuition for your product, but it won’t help you grow your userbase to 10s of millions.

And finally, here in San Francisco we have the “cult of feature worship.” Every product must have better features than the next, and it’s easy to respond to issues of traction with thinking. But I’ll argue that features might increase your engagement, but have a tough time driving more users.

Compare this to the renewable strategies, like viral marketing, SEO, widgets, and ads, which can scale into 10s of millions of users but are primarily centered around tough, non-user centric work. These are things that if you get right, you can optimize your way into a big, sustainable audience.

So I’ll stop here and ask you: What strategies is YOUR company using? Are they more from the left hand column? Or the right hand column?

Taking a systematic view to growing audiences
Finally, let’s talk about the approach for how you execute these growth efforts and build up sustainable audiences. First, you need a mental model for how users enter your site, and the process in which they bring in other users. This is your growth funnel, which you should then measure in extreme detail, and then use A/B testing tools to optimize. If you approach this process scientifically, then you’ll end up generating a vast array of hypotheses which you then identify, measure, and optimize.

I’ll give an example of this in advertising, and why ads can be a renewable resource.

An example: Growing audiences using ad arbitrage
Let’s say that you have a product that has a great LTV backend, like a free-to-play MMO that uses virtual goods for monetization. Then ultimately the entire focus of your business should be around figuring out how to buy ads profitably. get them to your site, upsell the experience so that a % of users buy. This then enables you to focus on buying more advertising, which lets you reach more users. If successful, you’ll be able to grow your audience up to its maximum size, until you run out of ads you can arbitrage effectively.

I updated the Community Next sldies from last time with an audio track, so I thought I would publish these slides again. Pardon the intrusion if you feel like you’ve already grokked all the details!

Intro
Hi everyone. My name is Andrew Chen and I’m a blogger and entrepreneur, focused on consumer internet products here in San Francisco.

I recently gave a talk called People Not Pages at the Community Next, which I’ll share today.

This year, the online advertising industry has reached roughly 20 billion dollars, that’s a big number. And a big chunk of those dollars are here because of the fact that advertisers can target consumers in ways that are simply not possible in traditional media.

Talking about ad targeting
So today we’ll talk about the business of ad targeting, and in particular 3 things:

First off, how can publishers generate more revenue from ad targeting? How does this actually happen?

Secondly, how do advertisers evaluate the effectiveness of their targeted campaigns? By understanding how they think about the question “does it work?” publishers can serve them better as customers.

And finally, we’ll discuss a topic that’s near and dear to my heart, which is: Why do social networks suck so much at monetization?

Note that this talk will mostly focus on the publisher-centric view, since the startups here in San Francisco tend to be creators of ad inventory rather than buyers of it.

About me
So quickly, before we begin, a quick bio about me. Again, my name is Andrew Chen and I’ve spent the bulk of my career in online advertising. I was most recently an Entrepreneur-in-residence at Mohr Davidow Ventures, a time I greatly enjoyed, evaluating deals in ad infrastructure as well as companies that are ad-supported. Before that I spent 4 years running product at Revenue Science, a behavioral targeting ad company in Seattle, and before that in the automotive vertical at Cobalt Group.

About Revenue Science
Most my experience will be drawn from my time at Revenue Science, so I’ll talk briefly about the company. Basically, as a behavioral targeting ad provider, they work with some of the largest brands online – Reuters, NY Times, ESPN, AOL, and dozens of others – to help them increase their revenues via targeting. While I was there I worked on their brand publisher products as well as getting their ad network off the ground.

Generating more revenue as a publisher
On to our first topic: How does targeting generate more revenues for publishers?

To understand this, I think you have to go all the way back to the newspaper days. This is an article from the early 1900s, costing one cent, documenting when the Titanic struck an iceberg. You have to think about how advertising was bought and sold at that time to understand why things work the way they do online today.

Now imagine this: You’re a newspaper company, which means you actually have to print physical copies of your text. That means that if you’re going to sell advertising, you must also print your ads on that physical copy, next to the editorial that’s relevant. That means that as an ad sales professional, your job is to take an audience that an agency specifies, let’s take “small investors” as an example, and translate that into a particular section in your paper. And so you sell them the Finance section. This means the agency asks for people, but you sell them pages.

The modern day online publishers are not much different. For a publisher like NY Times, you’ll have a bunch of sections on your site. You have the homepage, which makes a good CPM because it gets the most traffic and is the most prestigious placement on the site. Then you have high-value sections like Technology, and Finance. Then you have low value sections like Opinion and World News, which are harder to sell because what advertiser wants to be next to a story about the War in Iraq? So all of the low value stuff gets classified as “remnant ad inventory” and is lumped together.

So, this means that you have a bunch of different sections, and you can sell each for different prices. Again, the agency asks for a specific kind of person, let’s say IT decisionmakers, and you give them back the Tech section.

But wait a minute here, the IT decisionmaker reads the tech section, but also browses around the site. Is it right to value them highly on one pageview, and then drop them on the next? This is really a vestige of the old newspaper mindset.

If the agency is fundamentally targeting an audience, and you can deliver this audience regardless of where they are on the NY Times. or potentially even across the internet, you can deliver them more value. And lucky for the publisher, they are then converting their low-CPM remnant inventory into high-value targeted inventory.

This process of finding the qualified audiences in the remnant inventory is really what a publisher strives for.

Targeting techniques
So all the different targeting techniques that are out there, be it Geo, SIC, Demographic, Content, Daypart, or Behavioral, is all about finding these valuable audiences.

Once you have your targeting toolset in place, then the value of your ad inventory is increased. You find the IT decisionmakers in the opinion section, and convert low-value ads into high-value ones.

Evaluating targeted ad campaigns
Great, now let’s move to the second topic. How do advertisers evaluate whether or not targeting campaigns are working?

To understand this topic, you have to understand that there are really two major groups of advertisers out there: Brand advertisers, and direct response. Let’s talk about each in succession.

Brand advertising metrics
Brand advertisers are the guys you see all over the place, like Coke, Budweiser, Honda, United Airlines, and others. For many of them, their biggest goal is to drive awareness of their brand messages regardless of strict ROI. Obviously the ads are meant to eventually sell product, but it’s often not accountable from a strict ROI basis.

This brand spend means big dollars. Take a Budweiser NASCAR vehicle, for example. These cars can cost up to $20MM to sponsor. This year, Facebook is rumored to gross $300MM in revenue, which only translates to 15 NASCAR sponsorships.

So what kind of metrics are they looking for?

Well, here’s a case study that I worked on many years back. American Airlines and WSJ worked together on a campaign which was evaluated using Dynamic Logic, an ad effectiveness company. They went and surveyed users who had seen the campaign and who hadn’t, and asked them questions around message awareness and purchase intent. And it led to some positive results, which I’ll discuss.

First, there was a metric called Audience Composition. The campaign was targeted at frequent business travelers, so the survey qualified who fit the bill. The targeted campaign had a higher % of business travelers than the untargeted one, in fact by more than 2X.  Same for the definition of heavy travelers, people who took more than 5 trips a year, where it was 145% better.

The other set of metrics were brand and persuasion metrics, like Aided Brand Awareness and Message association. For example, those questions might revolve around things like, “Of the following brands, which do you associate with higher comfort” and would name a series of airline brands. The ones in the campaign were correctly associating the brand message.

Direct response advertising metrics
Now let’s go to direct response advertising. I used a picture of a cheesy infomercial because that’s the kind of advertiser that really cares about their results. Every dollar they spend must generate more dollars. They care a lot about ROI, as a result.

The example case study here, which I did not work on, is from Yahoo. In their example, they created an audience segment of mobile users who searched for ringtones, specific mobile providers, and browsed relevant pages. For the advertiser targeting this audience, the campaign generated higher CTRs and conversion rates, which are good ROI metrics. The better the increase in conversions, the more you as the advertiser can spend on this audience to get them to purchase your products.

Social network monetization (and why it’s hard)
OK, and the final topic for today, a discussion of why social networks suck so much at generating revenue. This is a topic I’ve written a lot about on my blog.

First off, this slide shows what you get when you search for “myspace addict” in Google images. The fact is, people find MySpace insanely addicting, which is great for them on a pageviews and engagement perspective. Problem is, this same engagement is what makes MySpace users pretty uninterested in what you’re trying to sell them. They are simply not in the mood to buy stuff, and when social network audiences are in their “flow state” clicking around, the CTRs can be quite low. This is exactly what you see, with CTRs often 0.1% and below, whereas search CTRs are often in the 10s of percentage points.

Social network data = sucks for targeting
The next issue is the type of data social networks are amassing. I included a real profile from a friend of mine, Andrea, which states “I have one interest: Dane Cook. I’m kidding I have more: Movies, music, soccer” etc. This tells you that Andrea’s interested in these things, but not that she’s in the transactional window to buy something. This is completely different than the kind of data you get from users when they are searching for dane cook movies on Google, or searching for dane cook on Amazon.

This issue is really the distinction of Interest versus Intent, and it’s a really hard area to break through. There’s a lot of social advertising companies working on this problem, and good luck to them, because it’s very hard.

Don’t assume that all the data you’re collecting is useful, but it’s likely that it’s not, unless the user is very close to making a transaction.

Wrap up
OK great, so we’ve covered the main topics. We talked about how publishers make money, and the process of turning remnant into premium. We also talked about brand versus direct response metrics in evaluating targeting campaigns. And finally, we talked about why social networks are so bad at monetization.

If anyone has questions, please feel free to e-mail me or check out my blog. Thanks!

Comparing the iPhone platform versus Facebook platform
I’ve recently been pondering the key similarities and differences between the way the Facebook platform ended up getting executed versus the iPhone platform. It’s an interesting trend now that other mobile players are thinking about opening a platform and providing an app store as well.

How would you guys add to the following table?

What else should be in this table?

Do users really love your product?

If they did, how would you be able to tell?

I would argue that the single most telling metric for a great product is how many of them become dedicated, repeat users. This angle of thinking naturally leads to a number of metrics around user retention, which we’ll examine in this blog post.

User retention is especially important for social web products. Failure to consider the backend retention of a userbase can lead to catastrophic results – in particular, without the proper mechanics in place, it’s easy to hit the “shark fin” user curve, as well as the death spiral caused by reverse Metcalfe’s Law. In both cases, once the core audience of a site starts to erode, then the erosion can cause a negative feedback loop that causes the entire audience to fall away.

This raises a series of questions:

• What are the right metrics to track for user retention?
• (And as a corollary, what are the wrong ones?)
• What is a “good” retention number? What are bad retention numbers?
• How do you optimize and improve retention rates?

Let’s tackle these below.

Retention versus Engagement
First off, there’s an important distinction between engagement versus retention, which some folks often track in one bucket. I generally define retention is simply the act of getting users BACK to revisit, regardless of their actual activity on the site. Contrast this with engagement, which measures how much time they spend with the product, how many features they interact with, etc.

An implication of this is that the right metric to follow is visits rather than something like pageviews or time-on-site.

Here are a couple examples of the separation of engagement versus retention:

• Google is a high retention, low engagement site
• MySpace is a high retention, high engagement site
• News sites are often medium/high retention, low engagement sites (like checking a headline)
• etc.

Note the important point that engagement doesn’t necessarily correlate with monetization. Because many retail sites and reference properties are transactional in nature, oftentimes this implies that the closer you are to the money, the lower the engagement is.

Keep this in mind for people who espouse “addictiveness” and “engagement” as virtues for social media sites.

Retention versus Acquisition
Secondly, there’s the important issue of how to disambiguate newly acquired users from retained users. The problem with a traffic graph that’s going up-and-to-the-right is that it’s not clear what’s really happening – is the site bringing in lots of new users? Or is there a bunch of dedicated users that are extended their engagement? You need to figure out which of 4 scenarios are actually happening, which I’ve blogged previously about:

1. Pageviews are coming ONLY from new users
2. Pageviews are coming ONLY from one generation of users (like early adopters)
3. Pageviews are coming ONLY from retained users
4. Pageviews are coming from new users and retained users

The proper way to disambiguate retention from acquisition is to precisely track the following stats:

• How many new users are joining the site?
• Of these new users, what are the different funnels they are joining from? (be it SEO, direct navigation, etc.)

Then you separate out these users completely from the aggregate numbers, and the remaining folks you have left are ones who are coming back to the site. You can then further segment this group by cohort, which we’ll discuss below.

Building your first retention table: User cohorts vs Revisit rates
Using the points from above, you can now build a retention table that compares how many users are coming back. This table starts with three columns:

• Time period the user joined
• Number of users that joined that period
• Revisit percentage rate

The reason why you separate it out into cohorts is that it gives the ability to compare performance of the site over time. As new product features are added, ideally the revisit rates would also continue to rise.

Let’s put this together in a table, imagining that we’re at Week 5:

A couple points on the above table:

• Looking back as Week 5, you can see that Week 1 is now the “oldest” cohort, and those users have had many weeks to revisit the site
• The overall userbase is growing 10% per week, starting with an initial userbase of 1000
• The revisit rate is naturally <100% since whatever initial cohort you start out with, it can only decrease but not increase
• Note that the retention rate of the site seems to be around 30%, although you’d want to let the Week 1 cohort run for a while and see if it eventually stabilizes
• Week 5 is currently at 0% since in this example the week just started and no users have revisited yet
• The actual number of visits on any given day is weird to calculate using this table, since the view is not based on aggregate numbers

The key metric is really the number that the revisit rate converges to. You can use this number in your traffic models to understand whether you should be focused on acquiring new users, or if you can simply focus on extending the engagement levels of your site.

What’s your revisit rate? (Using Google Analytics to approximate it)
Google Analytics gives you an overall number for free, with some caveats. You can access this feature on the lefthand nav through “Visitors”, then “New vs. Returning.” Basically this is an OK approximation of the revisit rate, as long as you:

• Maximize the window in which you are doing the analysis (ideally starting the analytics window when the site was first made public), otherwise the numbers will skew high since you’ll be counting too many dedicated users
• Ideally, the site would isn’t adding exponentially more users every day, since it would skew lower because newer users are less likely to have returned

Essentially there’s some skew that comes into play since Google Analytics doesn’t let you segment your users based on when they first joined the site.

Willing to share?
For readers who are willing to share the numbers on their site, please comment below and if I get enough responses I’ll do a followup blog post on the subject.

Like this blog?
If you did, please recommend it to a colleague and/or click here to get updates via email or RSS.

If you want to grab these in real time, you can follow me on Twitter at @andrew_chen.

As always, I can’t promise they are all work-related

• new microsoft ad with seinfeld, i like it. http://snurl.com/3n4x6
• 11-year old girl saved her family by knowing an upturned car can explode because it happens in Grand Theft Auto. http://snurl.com/3n1vw
• Funny Flash game – Pigeon Projectiles: http://is.gd/2blg
• great discussion on offerpal on venturebeat by @eldon: http://snurl.com/3mn0i
• pretty sweet t-shirt: http://store.glennz.com/grounddefenses.html
• myspace continues to grow, both in uniques and overall engagement: http://snurl.com/3mfpw
• top advertisers online – note that these skew towards large, remnant buyers: http://snurl.com/3mfpq
• another example of america != SF, mapquest still has 46% marketshare in maps: http://snurl.com/3mfp6
• lol. “Pizza grabbed 4 of 10 spots in a recent list of mobile-directory restaurant searches in US metropolitan areas” http://snurl.com/3mfop
• there needs to be more advertising related open-source projects i wanted to give this one a plug: http://snurl.com/3md6l
• vanity fair article on rupert murdoch: http://tinyurl.com/55x6yd
• compete.com blog, ctr and conversion rate comparisons: http://tinyurl.com/663gq5
• pretty cool futuristic tennis video: http://www.lacoste-future.com/
• interview with mike moritz and john doerr: http://snurl.com/3m3ak
• behavioral targeting on the ISP-side continues to drift out of favor: http://snurl.com/3m0sz
• star trek the experience (vegas) closing down: http://www.cnn.com/2008/TRAVEL/09/01/star.trek.voyage.ending.ap/index.html

Online poker revenue datapoint
This article on the online poker industry in Italy was interesting enough that I had to post it:

Online poker players in Italy spend an average of $881 per person a year on the pastime producing a yearly haul of $514 million according to a new survey released as the nation’s Government gets set to legalise, license and control Internet gambling.

Normalized down to month, this nets out to $73 ARPU/month!

I have no idea how this compares to the broader set of online gambling, or if it mostly has to do with the specific demographics in Italy.

Comparison to virtual good games
Compare this some numbers that Jeremy Liew posted in his excellent analysis on free-to-play and Facebook app monetization.

• Second Life: $9.30 ARPU/month
• Club Penguin: $1.62 ARPU/month
• Habbo Hotel: $1.30 ARPU/month
• Runescape: $0.84 ARPU/month
• Facebook apps: $1.20 ARPU/month

For social networks, your customers are often remnant ad networks
In previous posts, I’ve discussed the reasons why social networks often resort to ad networks to monetize their sites. As a result, it could be said that remnant ad networks are often the largest customers for these sites.

Let’s spend some time in this post discussing these customers of social network ads, and how things are typically structured.

An example remnant ad:

What’s remnant advertising?
First off, let’s define some terms – feel free to skip this if you’re already familiar with online ads. For any site with a bunch of ad impressions, the entire “inventory” of all your impressions is broken up into different pieces. Quoting from one of my earlier essays:

• The first US impression in a session has the most value ($10)
• Then impressions 2-5 have some level of brand value or high CTR value ($3-5)
• Then after that, you’re hitting ad networks selling on category ($1)
• Then eventually, you hit remnant ad networks ($0.50)
• Finally, you hit pure CPA remnant networks ($0.10)

Usually at the high end, the advertiser gets to be pretty picky about what kinds of ads they get. They might be able to specify a bunch of targeting, how many impressions they want from each user, and most importantly, where the user will see the ad. So if you have a music section on the site, then a big record label is likely to spend brand dollars only on the first couple impressions, and only in the music section.

So after your ad sales team has sold all the premium impressions they can, what happens to the rest? Well, as my friend Jay Weintraub eloquently describes, the rest of the “remnant” impressions are typically offloaded to ad networks, CPA offer advertisers, leadgen companies, and other folks who don’t mind the fact that you as a publisher aren’t guaranteeing placement on their site.

What it means to be a “blind” ad network
Often times, one interesting characteristic of remnant ad networks is that they are generally described as “blind” networks – this means that the advertiser, in many cases, doesn’t know where their ads are going to appear. Unlike doing an ad deal with ESPN, where you know exactly what page your ads will show up on (the Golf section, let’s say), instead the ad network could put you across any of the publishers they work with. There isn’t much guaranteed, other than the fact it won’t be next to porn or other completely inappropriate content.

It’s not always true that ad networks will sell their ads blind, but it’s an important class of inventory since it often makes up a majority % of the ads that a network will touch. In the case of AdSense, I’d guess that number if >95%, for example. When it’s not blind, often publisher sites are classified into categories like “pets” or “cars” or other broad channels, but let’s focus on the blind inventory for now.

Advertisers who prefer blind ad inventory
Most of the advertisers that are willing to buy this blind inventory tend to be advertisers with very clear monetization strategies in the backend. This is often why you often see ads from companies like:

• Ringtones/wallpapers/etc (monthly mobile subscription)
• Mobile jokes/crushes/etc (monthly mobile subscription)
• Toolbars, like IAC’s Smiley Central (search toolbars = predictable ongoing revenues)
• Free ipods after you fill out an offer (leadgen where each lead as a concrete value)
• Dating sites (with a recurring subscription fee)
• etc.

And inversely, this is also why you don’t often see large brand advertisers or folks with very specific advertising goals (for example, if you did lawsuits for cancer caused by asbestos) advertising on blind inventory. Without control of the context, the dollars put into these campaigns can be wasted.

In fact, there are certain characteristics that advertisers who routinely buy remnant ads often share. Here they are below:

• Broad, horizontal offering
• Clear LTV in the backend
• More transactional than not

First off, you need a broad offering because you often don’t know where the ads are going to be placed. The broader the offering, in fact, the better, since you’ll also be buying primarily on inventory that other advertisers find low-value. A good example of this is ad inventory focused around communication (email/social networks/etc) or media/entertainment. So if your product or service is very broad, then you can maximize the # of likely people you’ll hit, and also be able to purchase on the cheapest ad inventory.

As discussed above, a clear LTV (that’s good!) is important since it lets you focus on the simple equation that your spending costs must be lower than the profit from each user you buy. In many cases, a higher LTV might even drive the advertiser to purchase more broadly, just because the value on the user is so good – I’d consider many “in-market” audiences like home buyers and such as a good example of this.

And finally, a transactional mindset is often important since it correlates with LTV. You want to drive the user down a fairly narrow funnel that gets them to act on a high-value action. A good example of this is compelling the user to download a toolbar, or fill out a lead form, or similar. That way the advertiser is within a very small steps of making money from the user they paid for, and can balance it all out in a “portfolio” basis of buying large number of clicks to their landing pages.

The future of remnant ads on social networks
Right now, most social networks and social network apps have a great degree of reliance on these remnant ad networks. Because there isn’t clear context in writing on someone’s wall or playing a game, these low-value impressions are disposed of fairly cheaply. It’s interesting to see some of these guys “going native” on social networks, like Super Rewards and Offerpal, particularly in regards to virtual currency. I have to imagine that this will only continue.

If you liked this blog post, you can get updates by email or RSS here.

Peer-to-peer lending
Prosper is one of my favorite startups, on a personal level, since what they are doing with peer-to-peer loans makes the market more efficient and generates gobs of graphs to play around with. For those of you who are not familiar, here’s a description of Prosper from Wikipedia:

Prosper Marketplace, Inc. is a San Francisco, California-based company that operates Prosper.com, an online auction website where individuals can buy loans and request to borrow money. Borrowers set the maximum interest rate they wish to pay^[1], and loan buyers, called “lenders,” bid on specific loans by committing a portion of the principal and setting the minimum interest rate they wish to receive on a particular loan^[2]. Prosper manages the reverse dutch auction, assembling bids with the lowest interest rates in order to fund the loan.

Prosper verifies selected borrowers’ identity and personal data before funding loans^[3] and manages loan repayment. These unsecured loans are fully amortized over three years, with no pre-payment penalty. Prosper generates revenue by collecting a one-time fee on funded loans from borrowers, and assessing an annual loan servicing fee to loan buyers. The idea for the service is derived from group banking concepts, such as rotating savings and credit associations. Other motivating ideas derive from the concept of microlending.

And because of their open API where you can query for pretty much every piece of loan data in their system (examples here), there’s been a large ecosystem of Prosper-related websites that have sprung up, including one of my favorites, LendingStats. They have a number of charts on their site which I’ll run through below.

The main thing I’m interested in is how the recession has affected the Prosper marketplace, and how it might affect other companies in this area as well. Obviously a bad recession is bad for almost every consumer startup, but it’s interesting to dive into this particular example.

Prosper Total Member Count
First off, the growth around Prosper has been good albeit mostly linear. That’s OK since the site actually involves money, versus non-revenue generating web 2.0 sites ;-)

Interestingly enough, you can see that new Borrowers outpace Lenders, and that right near the end of the chat (in the upper right), there is a bit of a plateau as membership growth stalls. Given the state of the economy, I’d assume that the reason is that Borrowers now have cheaper sources of credit and/or Lenders have less money, so fewer of them are on Prosper.

Sharp drop in membership growth and active lenders/borrowers
You can see the same drop here, where since April/May 2008 there’s been a substantial drop in membership.

Another view of this is just the Active members, rather than new users coming in:

One interesting note for the above graph – why do Lenders rapidly outpace Borrowers? My guess for why there’s more Active Lenders than Active Borrowers has to do with the fact that the Borrowers are just trying to get a big lump sum out of the system, whereas the Lenders are likely to re-invest their money and keep the returns coming in.

How does the economy affect YOUR startup?
This example of Prosper is an interesting datapoint of one particular company is being affected by the economic downturn. Being in the loans business obviously makes them affected by the fact the economic downturn has its underpinnings in the credit markets – but it’s still a non-advertising business affected by the recession.

If you liked this blog post, you can get updates by email or RSS here.