Andrew Chen Archives

Above: Many small business figured out the hard way why coupon sites generate worse users

Incentive programs often don’t perform
The people you attract with referral programs, free trials, coupons, and gamification — folks who are “incentivized” as a broad umbrella category — are usually MUCH WORSE than organic ones. Worse LTVs, worse conversion, less engaged, and so on.

In a previous life, I headed up Uber’s $300m+/year referral program (“give $5 and get $5”) and learned a ton. Much of the learnings apply to the next wave of gamified consumer apps, web3 games, etc.

So why are these users worse? Let’s discuss.

When CAC/LTV spreadsheets fail
When a new product comes to market, usually the team will measure a baseline set of metrics around lifetime value, etc. if the numbers look good, they might say OK let’s roll out some incentives and get more users like this. Spreadsheets are built, budgets are planned, growth is forecasted, and the new growth project kicks off.

The problem is, all of these forms of incentives usually end up attracting a different type of marginal user that wouldn’t have signed up earlier. They are less qualified, more discount seeking, and behave differently. There is negative selection.

This is especially true when the product has been out there for a while and the core market has mostly been saturated. You also see significant amounts of fraud as users scheme to profit from the incentives. This could be a simple as creating a new account to grab an incentive or it could be something much more organized and nefarious.

This is why core metrics like LTV and engagement can often be half as good or lower, which is often enough to defeat the mathematics that justified the program in the first place. An additional user at upside down mechanics feels good from a top line basis, but in fact, fewer users would be better for the business model. And all the attention towards a complex referral program might take away attention from innovation elsewhere in the product.

One final issue that’s quite subtle, but very important: Cannibalization. You have an target market and sometimes it takes time for a product to spread through its ideal users — this is magical because word of mouth is free. And when it happens in an organic way, the intent is even higher. But if these ideal users encounter the product via an incentive program, you often “pull forward” these users, thus costing you money, when you would have gotten them anyway.

If this all sounds like I might have suffered some trauma from Uber, it’s because I did! Not only did the rider-side referral programs perform worse over time, and perform worse than other channels, in fact, the users were much worse than even users bought from paid ads. It was millions of dollars of spend that didn’t need to happen.

Why this matters — in the world of web3, gamified apps, etc
The ramifications of this are wide, especially on the world of web3, consumer apps that are gamified, etc.

First, it tells you that if you take a game or an app that does not have inherent engagement and retention, it is not enough to add gaming mechanics. If anything, the new mechanics might make things worse, not better, as they attract a group of users who respond to the mechanics, but wouldn’t otherwise use the underlying product. I think we saw a lot of this in web3, where incentivized attracted speculators early on, but struggled to find fun gameplay to attain actual users. Similarly gamified consumer apps (the trad kind) might attract and sustain a certain type of user who is happy to engage in any gamified app, and who will quickly move on because the underlying app doesn’t engage either.

Second, all of these dynamics create sort of a related dynamic to the Law of Shitty Clickthroughs. Not only do individual marketing channels degrade, but many of the new channels you add over time — because they are incentivized — perform worse than the initial channels. Thus the entire machine gets slower and harder as you go.

Final story on this from Uber, funny enough the referral program on the driver side attracted very positively selected users. Whereas the rider referral program got discount seekers, the drivers were highly money motivated. Because they were so motivated and signed up for larger referral bounties, they actually performed better after sign up. Even though referrals was 15% of sign-up they were well over 30% of first trips.

Incentives are a form of selection and you need to make sure you know what you’re selecting for.