Designing and reading an A/B test
Most A/B tests go wrong before the first customer sees the variant, or in the week someone looks at the result too early. This lesson is the statistics under a trustworthy test: the unit you randomise, where the sample size formula comes from, the checks you run before reading, why peeking breaks the 5% promise, and how to read the result against a rule you wrote first.
By the end you can
- Choose the unit of randomisation, and say how clusters and interference shrink your real sample
- Work out the sample size per arm for a conversion test from the baseline, the MDE, alpha and power, and turn it into whole weeks
- Check a test before reading it: a sample ratio mismatch with a chi-square test, and novelty by week
- Explain why peeking inflates false positives, and plan one honest early look
- Read the result with a confidence interval, guardrails with a margin, and a decision rule written before the test
The everyday picture
Two football teams want to know which one is better. If one team is far stronger, a single match will probably show it. If the difference is small, one match is mostly luck: a deflected shot, a missed penalty. To separate a slightly better team from a lucky one you need a whole season of matches. The smaller the difference you care about, the longer the season.
The league also fixes its rules before the season starts: how many matches, what counts as a win, how points are added up. Nobody may declare the champion in October because their team is top of the table that week. Over a season, every team is top at some point by luck.
An A/B test is a short league with two teams. The difference you care about sets the length of the season, and rules written first stop anyone calling the result on a good day.
Pick the unit, then count it honestly
Mangrove's app team has built a one-page checkout. Today a customer goes through three screens after the cart: delivery slot and address, payment, then review. The new design puts all of it on one page. You and Tomás, Mangrove's data analyst, will design the test that decides whether it ships. The product view of experiments (hypothesis, holdouts, store pilots) is in pm-25; here you go into the statistics that make the numbers trustworthy. The test logic itself, null hypothesis, p-value, z-test and Welch's t-test, is in st-03.
The first decision is the unit of randomisation. Three choices are on the table.
- Session or cart. Each visit gets a coin flip. Over two weeks many customers open the cart several times, so one person sees both checkouts, and their visits are not independent of each other.
- Customer. Each customer gets one coin flip and keeps their version (sticky assignment). This matches how the change acts: a person learns one checkout. It is the right choice here.
- Store. Turn on the new checkout for every customer of 20 stores, and keep 20 stores as control. Easy to switch on, and very costly in statistics.
Why costly? Customers of the same store are alike: same neighbourhood, same stock, same riders. Statisticians measure this with the intraclass correlation (ICC). The sample you really have is the number of units divided by the design effect:
design effect = 1 + (m - 1) x ICC m = customers per store
40 stores, 20 per arm, about 2,000 customers each, ICC = 0.01
design effect = 1 + 1,999 x 0.01 = 21.0
40,000 customers per arm behave like 40,000 / 21.0 = about 1,900 independent onesAn ICC of 0.01 sounds tiny, yet the 40,000 customers per arm are worth about 1,900. A store-level test needs far more stores, which is why pm-25 uses store pilots for changes that can only work store by store. When you can randomise customers, do it.
Two rules follow. First, analyse at the unit you randomised. If customers were randomised, build one row per customer (converted yes or no) before you compute anything. If you compute conversion per session instead, the formula from st-03 treats every session as independent, the standard error comes out too small, and false winners appear more often than 5%. Second, watch for interference. Both arms share the same "Now" delivery slots and riders. If the variant fills slots faster, control customers find fewer free slots and convert less, and the variant looks better than it is. Tomás keeps the test to 20% of customers so the extra orders are too few to change slot availability. For changes that move a lot of rider time, use a switchback (pm-25).
Quick check
The checkout test randomises customers, but an analyst computes conversion per cart session, with 3.1 sessions per customer on average. What happens to the result?
Sessions from the same customer are correlated, so 3.1 sessions carry much less than 3.1 customers' worth of information. Treating them as independent shrinks the standard error and inflates false positives. More rows only help when the rows are independent; that is the trap in the first option. Compute one row per customer, the unit you randomised.