Traps that fool smart people
Every number in a report can be correct and the story it tells can still be false. This last lesson of the course shows six traps that catch experienced analysts (Simpson's paradox, regression to the mean, many comparisons, survivorship, selection and ignored base rates) and the check that catches each one before it reaches a decision.
By the end you can
- Spot Simpson's paradox and report a rate at a fixed mix next to the overall rate
- Explain why the worst stores improve after almost any action, and design a test that separates a real effect from regression to the mean
- Work out the false-alarm risk of many comparisons and correct for it with Bonferroni or Benjamini-Hochberg
- Ask who is missing from a dataset (survivors, volunteers) and put the base rate next to any alert or test result
The everyday picture
Your knee has ached for weeks. On the worst day you finally try a cream a friend swears by. Three days later it feels better. Did the cream work? Maybe. You also reached for it on your worst day, and a worst day is usually followed by a less bad one, cream or no cream.
Your friend trusts the cream because everyone she knows who still uses it is happy. The people it failed stopped using it and stopped talking about it. The online reviews come mostly from people who felt strongly enough to write one. And she found this cream after trying twenty others, so one of the twenty was bound to coincide with a good week.
No number in that story is wrong. The trap is in which numbers reached you and what you compared them with. Data work has the same traps, with stores, riders and customers in place of knees.
Simpson's paradox: when the mix moves
Tomás's monthly delivery report covers "Now" orders in North city and South city. The share delivered within 30 minutes fell from 91.0% in September to 90.0% in October. The head of riders asks which city got worse. Tomás splits the number by city:
| City | Sep orders a day | Sep on time | Oct orders a day | Oct on time |
|---|---|---|---|---|
| North city | 12,000 | 94.0% | 12,000 | 95.0% |
| South city | 4,000 | 82.0% | 10,000 | 84.0% |
| Both cities | 16,000 | 91.0% | 22,000 | 90.0% |
Both cities got better. Together they got worse. This is Simpson's paradox, and there is no error in the data. The overall rate is a weighted average of the city rates, and the weights are each city's share of orders:
September: 0.75 x 94% + 0.25 x 82% = 91.0% (12,000 and 4,000 of 16,000)
October: 0.545 x 95% + 0.455 x 84% = 90.0% (12,000 and 10,000 of 22,000)
October's city rates at September's mix:
0.75 x 95% + 0.25 x 84% = 92.25% (+1.25 points against September)A marketing push in South city, where rides are longer and the on-time rate has always been lower, raised its orders from 4,000 to 10,000 a day. South city's share of orders went from 25% to 45%. More of the orders now come from the harder city, so the average falls even though each city improved.
Which number is right? Both, for different questions. "Did delivery get better?" is answered by the city rates, or by the mix-adjusted rate: October's city rates weighted by September's mix give 92.25%, up 1.25 points. "What share of customers got their order in 30 minutes?" is answered by the overall 90.0%: customers did wait longer on average, because more of them live in South city. The report should show both, and the paradox carries its own news: South city is growing fast, so plan riders there.
The best-known real case is from 1973. The University of California, Berkeley admitted about 44% of men and 35% of women who applied to its graduate programmes. Peter Bickel and colleagues looked department by department and found little sign of bias against women inside departments. Women had mostly applied to departments that admitted a small share of everyone who applied.
Suspect the paradox whenever the groups in a comparison change size: a city or a channel grows, new stores open, "Now" orders take share from "Later" orders, a test group gets more phones of one kind. The check takes two minutes: split the headline by the dimension whose shares moved, compare the shares, and recompute at a fixed mix.
On-time rate for one city and month (columns: city, month, orders, on_time):
=SUMIFS(D:D, A:A, "South", B:B, "2026-10") / SUMIFS(C:C, A:A, "South", B:B, "2026-10")
Mix-adjusted October rate: September shares in F2:F3, October rates in G2:G3
=SUMPRODUCT(F2:F3, G2:G3)import pandas as pd
df = pd.read_csv("delivery_week.csv") # city, month, orders, on_time
g = df.groupby(["city", "month"])[["orders", "on_time"]].sum()
g["pct"] = 100 * g["on_time"] / g["orders"] # a ratio of sums, never a mean of rates
print(g["pct"].unstack("month").round(1)) # one row per city
sep = g.xs("2026-09", level="month")
octo = g.xs("2026-10", level="month")
weights = sep["orders"] / sep["orders"].sum() # September's mix: 0.75 and 0.25
print((weights * octo["pct"]).sum()) # 92.25: October's rates at September's mixQuick check
The head of riders wants to know whether riders in each city got faster from September to October. Which number answers that?
Holding the mix fixed removes the effect of South city's growth, so what remains is the change in each city's performance. The overall rate is correct, but it answers a different question: what customers experienced. Nothing is broken; the paradox comes from the weights.