Experiments: A/B tests, holdouts and store pilots
An experiment is the most reliable way to know that a change caused a result. This lesson shows you how to plan an A/B test (hypothesis, metric, sample size and duration), read a result with a confidence interval in plain words, avoid the traps that fool smart teams, and run holdouts and store-level pilots when you can't randomise users.
By the end you can
- Write an experiment plan with a hypothesis, one primary metric, guardrails, a minimum detectable effect, a sample size and a decision rule
- Read a result's lift, confidence interval and p-value, and say what "not significant" does and doesn't mean
- Avoid peeking, metric fishing, sample ratio mismatch, novelty effects and interference
- Design a holdout or a store-level pilot with matched control stores, and estimate its effect with a difference in differences
- Summarise an A/B test in SQL without losing the people who didn't buy
The everyday picture
A restaurant changes its noodle soup recipe on Tuesday, and Tuesday's soup sales are up 15%. Was it the recipe? Tuesday was also payday. It rained, so more people ate near the office. And a food blogger posted about the place on Monday. Any of those could explain the 15%.
So the chef tries something else. For two weeks, each time a table orders the soup, the waiter flips a coin: heads, the new recipe; tails, the old one. Payday, rain and bloggers now hit both groups equally, because the coin knows nothing about them. If the new-recipe tables order the soup again more often, the recipe is the reason.
Two more rules finish the method. The chef decides before she starts how many bowls she needs to be sure, and doesn't stop on the day the new recipe happens to be ahead. And if she can't flip a coin per table, because the new recipe needs a different stove, she tries it in one branch and compares it with a similar branch that didn't change, over the same weeks.
Why randomise
Before-and-after comparisons fail because something else always changes too: a festival, payday, the weather, a marketing push, a new store opening. In an A/B test, users are split at random into a control group (A, the current version) and a variant group (B, the change), at the same time. Randomness spreads every other cause evenly across both groups, including causes you haven't thought of. A difference that is bigger than chance can then be credited to the change.
Three details make it work:
- The unit. What gets randomised: a customer, an order, a store, a city, or a time slot. Pick the unit at which the change acts and at which units don't affect each other (more on that below).
- Sticky assignment. The same customer sees the same version every time, or the groups blur.
- Exposure logging. Record when each user actually met the change (opened the cart page, say). People assigned but never exposed add noise and hide the effect.
Why bother, when the team is confident? Because confidence is a poor guide. In their 2017 Harvard Business Review article, Ron Kohavi and Stefan Thomke reported that at Microsoft about a third of experiments improved the metric they targeted, a third made no difference and a third made things worse; at Google and Bing only about 10% to 20% gave positive results. The same article describes a Bing idea to change how ad headlines were shown that sat in the backlog for more than six months as low priority. When an engineer finally tested it, it raised revenue by 12%, worth more than $100 million a year in the US alone.
Quick check
Imran ships a new receiving screen to every store on 1 March. In March, supplier deliveries received in the app are 6 points higher than in February. Why can't he credit the screen with 6 points?
With no comparison group, the screen and everything else that changed in March are tangled together. Six points may be large or small depending on cost; the problem is that we don't know how much of it the screen caused. A store-level pilot with control stores, later in this lesson, would have told him.