Hypothesis tests without the fog
A hypothesis test answers one narrow question: if nothing had changed, how surprising would data like ours be? This lesson builds that question from a simulation, runs the two tests analysts use most (the two-proportion z-test and Welch's t-test) in a spreadsheet and in Python, and shows how to report an effect with its interval so nobody mistakes "significant" for "important".
By the end you can
- Write a null and an alternative hypothesis, and choose alpha before you look at the data
- Explain a p-value in one plain sentence, and name the three most common ways people misread it
- Run a two-proportion z-test for conversion and a Welch t-test for averages, by formula and with scipy.stats or statsmodels
- Say what type I and type II errors and power are, with numbers from a real test
- Report the effect size with its 95% confidence interval and judge whether it is big enough to matter
The everyday picture
A court trial starts from a fixed position: the accused is innocent. The prosecution brings evidence, and the jury asks one question: is this evidence so hard to explain if the person were innocent that we can no longer believe it? Only then do they convict.
Three things about trials carry over to statistics. First, the starting position is "nothing happened", and the evidence has to push you off it. Second, the jury can make two kinds of mistake: convict an innocent person, or let a guilty one go. Making one mistake rarer tends to make the other more common. Third, a verdict of "not guilty" does not mean "proven innocent". It means the evidence was not strong enough. A test that finds "no significant difference" says exactly the same thing about your data.
The question a test asks
Mangrove's app team has a new checkout button. In the old cart screen the "Place order" button sits below the list of items, so customers with long baskets have to scroll to find it. The new design keeps the button fixed at the bottom of the screen. Tomás, Mangrove's data analyst, runs an A/B test for one week: 25,000 carts see the old screen (control) and 25,000 see the new one (variant). The control converts 3,000 carts into orders (12.0%, close to the app's usual cart-to-order rate). The variant converts 3,150 (12.6%).
A gap of 0.6 percentage points. Is that the button, or the normal wobble you get whenever you split customers into two groups? A test turns this into two precise statements.
- The null hypothesis (H0): the button makes no difference. Both groups have the same true conversion rate, and any gap is chance.
- The alternative hypothesis (H1): the true rates differ. We test in both directions (two-sided), because a new button could also hurt.
Now the simulation, before any formula. Suppose the null is true and every cart converts at the same 12.3% (the two groups combined). Tomás asks a computer to run 1,000 fake tests, called A/A tests: each one splits 50,000 carts into two groups of 25,000 that all convert at 12.3%, and records the gap between the groups. Nothing differs between the groups, so every gap is pure chance.
That picture is the whole idea. The p-value is the share of "nothing changed" worlds that produce a gap at least as extreme as yours. Here about 41 in 1,000, so p ≈ 0.041. In words: if the button made no difference, a gap of 0.6 points or more, in either direction, would turn up in about 4% of tests this size. That is fairly surprising, so the data argue against the null.
Notice what the p-value is about: the data, assuming the null. It is not the probability that the null is true, and it is not the probability that the button works.
Quick check
In the button test, what does the null hypothesis say?
The null is the "nothing changed" claim about the true rates. Option 3 describes what was observed in the sample, which is data, not a hypothesis. Option 1 is one specific alternative; the test does not assume it.