Free data template

A/B test readout

Conversion per variant, the lift, a confidence interval and a plain-language verdict.

Open in the Data LabSave and export

It opens with this template loaded and runs in your browser. You don't need an account to use it; a free account saves your work and exports it.

The job it does

When an experiment ends and someone asks "did B win?": a readout you can paste into the decision doc. It comes with its own checks, so after you change it you can see at a glance that it still does the job.

How to use it with your own data

Remove the sample experiment CSV and load your results as experiment.csv (columns variant, visitors, conversions; A is the control).

What's inside

This is the template exactly as the Data Lab opens it, read-only. Open it to run it, change it and export it.

  • 2 notebook cells
  • Your own CSV files
  • CSV file: experiment (2 rows)
The brief: A/B test readout

An experiment ended and someone asks "did B win?" This readout gives the conversion rate per variant, the lift, a 95% confidence interval for the difference, the p-value, and a verdict in plain words you can paste into the decision doc.

The sample: 10,000 visitors each saw the old checkout (A) and the new one (B).

Your own data: remove the experiment CSV file (Data panel), load yours as experiment.csv (columns variant, visitors, conversions; A is the control) and Run. The criteria pin the sample's numbers: change them to yours, or remove them.

Done when
  • The notebook runs top to bottom without errors
  • lift_pct equals 16 ± 0.01
  • p_value equals 0.0123 ± 0.001
  • readout has 2 rows
  • A bar chart plots rate_pct
Cell 1 · Python
import math
# Your own data: replace the "experiment" CSV (columns variant, visitors, conversions; A is the control).
exp = experiment.set_index("variant")
a, b = exp.loc["A"], exp.loc["B"]
rate_a = a["conversions"] / a["visitors"]
rate_b = b["conversions"] / b["visitors"]
lift_pct = float((rate_b - rate_a) / rate_a * 100)

# a two-sided z-test on the difference, and its 95% confidence interval
diff = rate_b - rate_a
se = math.sqrt(rate_a * (1 - rate_a) / a["visitors"] + rate_b * (1 - rate_b) / b["visitors"])
z = diff / se
p_value = float(2 * (1 - 0.5 * (1 + math.erf(abs(z) / math.sqrt(2)))))
ci_low_pts, ci_high_pts = round((diff - 1.96 * se) * 100, 2), round((diff + 1.96 * se) * 100, 2)

readout = experiment.assign(rate_pct=(experiment["conversions"] / experiment["visitors"] * 100).round(2))
readout
Cell 2 · Python
# The verdict in plain words, for the decision doc
if p_value < 0.05 and ci_low_pts > 0:
    verdict = f"B wins: conversion {lift_pct:.1f}% higher (B minus A is {ci_low_pts} to {ci_high_pts} points, p = {p_value:.3f})."
elif p_value < 0.05:
    verdict = f"B loses: conversion {lift_pct:.1f}% lower (p = {p_value:.3f})."
else:
    verdict = f"No clear winner: the difference ({lift_pct:.1f}%) could be chance (p = {p_value:.2f}). Keep A or run longer."
print(verdict)
chart(readout, x="variant", y="rate_pct", kind="bar", title="Conversion rate (%)")
Open in the Data Lab

The lesson behind it

Learn why it works, with a podcast and a quiz for each lesson.

  • Simple forecast

    A trend plus weekly pattern fitted to daily sales, the next 4 weeks forecast and a check against the last weeks.

  • KPI dashboard notebook

    Weekly revenue, orders, average basket and active customers, with a chart per KPI.

All data templates · Every template