Andrew Chen Archives

Subscribe · Featured · Recent · The Cold Start Problem 📘
Dear readers, I have moved to Substack and I will be writing here from now on:
In the meantime, I will leave up for posterity. Enjoy!

How to keep visual design consistent while A/B testing like crazy

If you don’t watch out, after a couple months of A/B testing, your product will end up looking like Las Vegas!

Why A/B testing and visual design come into conflict
It’s great to implement consistent A/B testing in their product process, but then it becomes even harder to keep a consistent visual design while doing test after test. This tension comes from the fact that A/B tests push you towards local maxima, making the particular section of page you’re testing high-performance, but at the expense of the overall experience. As a result, there’s a lot of temptation to “hack in” a new design, the way that software engineers have to “hack in” a feature – but this is short-term at best. This often means adding a bold, colored link to the top of page with “NEW!” or adding yet another tab – these are all band-aid solutions because once you get to the next set of features, it’s not a scalable design to have 100 tabs.

Each of these competing features, taken by itself, moves the needle positively. However, there isn’t a great way to measure the gradual “tragedy of the commons” effect to the overall user experience. Each new loud page element competes with all previous page elements, and must be louder as a result – this leads to the Vegas effect that many Facebook apps end up in.

To really solve this problem, you need a central design vision – there’s no way around that. It also helps a lot to have a flexible design that embraces A/B testing – you can work with your designers to make this happen through modular, open elements.

Closed designs make it hard to add or remove content
Let’s take a particular example and look at it – this might be a standard example on a page like a video or otherwise:


It looks nice, but also has tremendous sensitivity to the content and an inflexible design that makes it hard to test new content. To be more specific, ask yourself the following quesitons:

  • If you wanted to add a comments count in addition to views and votes, how would you do that?
  • What happens when the views number gets beyond 10,000?
  • What if you wanted to add favorites, or flagging for inappropriate content?
  • If we decided to hide the thumbs down, how would this visually balance?
  • If we wanted to fit more thumbnails onto a browse page, how easy it is to shrink the main thumbnail?
  • etc.

The above design is an example of a “closed” design where everything fits just right, but makes it very difficult to add or remove elements. There’s an exact balancing of all the parts of the element, which makes it very sensitive.

Many of the solutions to the questions involve either require building out new pieces next to the element, which throws it off balance. Thus, if the above were used in an A/B test, the visual look would be immediately ruined.

Open designs that are A/B test-friendly
Let’s compare this to the elements below, which have a more modular design that can scale vertically:


The above elements don’t have the same “just right” visual appeal, but make it much easier to add and remove content. The key design decision is to add multiple bands of content which can be grouped together and extended vertically. Ideally, you would never end up with a repeating tile of 4-buttons and 3-stats, but you could certainly test it much more easily than with the closed design.

Here are some of the variations that can easily be tested:

  • Switch the title section and the stats/buttons sections
  • Add and remove buttons (or no buttons!)
  • Add and remove stats (or no stats!)
  • Combine price tags with other stats
  • Try different buttons
  • etc.

Following an open design on page elements enables substantial A/B testing within some flexible constraints. Now you may still be tempted to do something crazy like big hover overlays, <BLINK> tags, and other stuff, but at least you can make it easy to test a wide variety of low-hanging fruit. It also makes the owner of the overall visual design able to maintain a central “style guide” while still offering enough flexibility to keep people creative.

This same idea of open designs with horizontal bands of content can be applied to whole pages too – let’s examine a page from the king of A/B testing,

Open page layouts
From the snapshot below, you can see that Amazon groups the center column of content – each element has a title explaining how it is, a list of items, and a navigation link to see more. This is also true with the item detail pages, which use a similar grouping to show everything from similar books to reviews to other elements. These pages can get very long, but because most of it is below-the-fold, it’s easy to get away with.

I’ve been told that this modular design enables Amazon to take a “King of the Hill” approach to testing each horizontal band of content against each other. Different software teams will create different kinds of navigation and recommendation, and if it causes people to click through to buy, then it floats up higher in the page. This systematic A/B testing is much more easily enabled when there’s the design flexibility for that sort of thing.

Here’s a snapshot for a reminder of what this looks like:

While you may argue that Amazon’s design is cluttered and actuallysucks, on the other hand, this approach lets them take a very experimental approach to pushing out features. It makes it very low-cost to implement a new recommendations approach and try it out without needing to figure out how to design it into the UX.

What’s next? Modular user flows?
Of course, if you can take a modular approach to scaling individual page elements or entire pages, the next question is whether you can take this approach to user flows.

I’ve never seen anyone do this, but this is how it might work:

  • Any linear user flow is identified in a product (like new registration, payment flow, etc)
  • This flow might be 1 page, or broken into N pages
  • Similarly, every individual page might have a bunch of fields (like photo, about me, etc.)
  • As part of the A/B testing process, you might want to drop a new page (or new fields) into the flow
  • Then an optimization process shuffles pages throughout the flow to identify the best page sequence and page-by-page configuration

You might imagine something like this could be a very powerful process as it would allow you to identify whether you should offer a coupon pre-transaction or post-transaction, or on any given page, where an input field should be placed.

For those who want to know more, I have written a bunch more about A/B testing here.

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Views expressed in “content” (including posts, podcasts, videos) linked on this website or posted in social media and other platforms (collectively, “content distribution outlets”) are my own and are not the views of AH Capital Management, L.L.C. (“a16z”) or its respective affiliates. AH Capital Management is an investment adviser registered with the Securities and Exchange Commission. Registration as an investment adviser does not imply any special skill or training. The posts are not directed to any investors or potential investors, and do not constitute an offer to sell -- or a solicitation of an offer to buy -- any securities, and may not be used or relied upon in evaluating the merits of any investment.

The content should not be construed as or relied upon in any manner as investment, legal, tax, or other advice. You should consult your own advisers as to legal, business, tax, and other related matters concerning any investment. Any projections, estimates, forecasts, targets, prospects and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Any charts provided here are for informational purposes only, and should not be relied upon when making any investment decision. Certain information contained in here has been obtained from third-party sources. While taken from sources believed to be reliable, I have not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. The content speaks only as of the date indicated.

Under no circumstances should any posts or other information provided on this website -- or on associated content distribution outlets -- be construed as an offer soliciting the purchase or sale of any security or interest in any pooled investment vehicle sponsored, discussed, or mentioned by a16z personnel. Nor should it be construed as an offer to provide investment advisory services; an offer to invest in an a16z-managed pooled investment vehicle will be made separately and only by means of the confidential offering documents of the specific pooled investment vehicles -- which should be read in their entirety, and only to those who, among other requirements, meet certain qualifications under federal securities laws. Such investors, defined as accredited investors and qualified purchasers, are generally deemed capable of evaluating the merits and risks of prospective investments and financial matters. There can be no assurances that a16z’s investment objectives will be achieved or investment strategies will be successful. Any investment in a vehicle managed by a16z involves a high degree of risk including the risk that the entire amount invested is lost. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by a16z is available at Excluded from this list are investments for which the issuer has not provided permission for a16z to disclose publicly as well as unannounced investments in publicly traded digital assets. Past results of Andreessen Horowitz’s investments, pooled investment vehicles, or investment strategies are not necessarily indicative of future results. Please see for additional important information.