Statistics cheat sheet

8 minReference

Statistics cheat sheet

The statistics a data analyst, data scientist or PM uses every week: describing data, confidence intervals, which test for which question, A/B test sizing, regression, and the traps that turn noise into a headline. Formulas come with Excel, Python and plain-English versions. Keep it open when you read an experiment result.

Describe before you test

Look at the spread, not only the middle. Mangrove's basket sizes are skewed: most baskets are small and a few are huge, so the mean sits well above the median (Describing data).

Measure	Tells you	Excel / Sheets	Python
Mean	The balance point; pulled by outliers	`=AVERAGE(r)`	`s.mean()`
Median	The middle value; robust to outliers	`=MEDIAN(r)`	`s.median()`
Percentile (p90)	90% of values are below it	`=PERCENTILE.INC(r, 0.9)`	`s.quantile(0.9)`
Standard deviation	Typical distance from the mean	`=STDEV.S(r)` (sample)	`s.std()` (sample by default)
Interquartile range	Spread of the middle half	`=QUARTILE.INC(r,3)-QUARTILE.INC(r,1)`	`s.quantile(.75) - s.quantile(.25)`
Coefficient of variation	Spread relative to size, to compare stores of different sizes	`=STDEV.S(r)/AVERAGE(r)`	`s.std() / s.mean()`
Correlation	How tightly two numbers move together on a straight line, from -1 to 1	`=CORREL(a, b)`	`df["a"].corr(df["b"])`

Numbers worth remembering

Rules of thumb for roughly normal data and large samples (Describing data, Confidence intervals).

What	Value	Use it to
Within 1, 2, 3 standard deviations	about 68%, 95%, 99.7% of values	Spot unusual days
z for a 95% interval	1.96 (call it 2)	Build a quick interval
z for 90% and 99%	1.645 and 2.576	Change the confidence level
Standard error of a mean	s / √n	See how precise an average is
Standard error of a proportion	√(p(1 − p) / n)	Interval for a conversion rate
Halving the error	needs 4 times the sample	Price a more precise answer
Margin of error for a 50% rate, n = 1,000	about ±3 percentage points	Sanity-check survey results
Sample per group for an A/B test	about 16 × p(1 − p) / δ² (80% power, 5% two-sided)	Size a conversion test (δ = absolute lift)
"Rule of three"	0 events in n trials: 95% upper bound about 3 / n	"No failures in 300 deliveries" still allows up to 1%

Confidence intervals

A 95% interval comes from a method that captures the true value in 95% of repeated samples. Report it with every estimate: "average basket $23.40, 95% CI $22.90 to $23.90" (Confidence intervals).

Output

Mean:        x̄ ± 1.96 × s / √n
Proportion:  p ± 1.96 × √( p(1 − p) / n )
Difference:  (p1 − p2) ± 1.96 × √( p1(1 − p1)/n1 + p2(1 − p2)/n2 )

Example: 540 of 6,000 app orders had an item cut, p = 0.09
SE = √(0.09 × 0.91 / 6000) ≈ 0.0037, so 95% CI ≈ 8.3% to 9.7%

For small samples, or anything that is not a mean or proportion (a median, a ratio of sums), use the bootstrap: resample the rows with replacement a few thousand times, compute the statistic each time, and take the 2.5th and 97.5th percentiles (Confidence intervals).

Which test for which question

The test follows from the kind of outcome and how the groups were formed. Python names are from scipy.stats unless marked (Hypothesis tests).

Question	Test	Python	Lesson
Do two groups differ in a mean (basket size)?	Welch's t-test	`ttest_ind(a, b, equal_var=False)`	Hypothesis tests
Same stores, before and after a change?	Paired t-test	`ttest_rel(before, after)`	Hypothesis tests
Do two groups differ in a rate (conversion)?	Two-proportion z-test	`proportions_ztest` (statsmodels)	Hypothesis tests, A/B test design
Is a category spread different between groups?	Chi-square test of independence	`chi2_contingency(table)`	Hypothesis tests
Skewed values, small samples	Mann-Whitney U	`mannwhitneyu(a, b)`	Hypothesis tests
Three or more group means	One-way ANOVA, then pairwise tests with a correction	`f_oneway(a, b, c)`	Hypothesis tests
Any statistic, few assumptions	Permutation test or bootstrap	`permutation_test`, `bootstrap` (SciPy 1.8 and later)	Confidence intervals
How much does Y change per unit of X, holding others fixed?	Linear regression	`smf.ols("y ~ x + C(city)", df).fit()` (statsmodels)	Regression
Yes/no outcome with several drivers	Logistic regression	`smf.logit("churned ~ tenure + orders", df).fit()`	Regression

Reading a p-value

A p-value is the chance of seeing a difference at least this big if there were really no difference. It is not the chance the result is a fluke, and it says nothing about how big or useful the effect is (Hypothesis tests).

Term	Plain meaning	Typical value
Null hypothesis	"No difference": the claim you try to reject
Alpha (significance level)	False alarm rate you accept	0.05
Type I error	Calling a change a win when it did nothing	happens at rate alpha
Type II error	Missing a real effect	happens at rate beta
Power (1 − beta)	Chance of detecting the effect if it is real	0.8
Minimum detectable effect (MDE)	Smallest lift the test is sized to catch	chosen by the business
Effect size	How big the difference is, in units people care about	report it with its interval

A/B test checklist

Decide everything before the test starts, write it down, and do not change it after you see data (A/B test design, Experiments).

One primary metric, plus guardrail metrics that must not get worse (cancellations, delivery time, refunds).
The unit of randomisation matches the unit of the metric: randomise customers for a per-customer metric, stores for a store-level change.
Sample size and run length fixed in advance from the MDE; run whole weeks so weekdays and weekends both count.
Sample ratio check: a 50/50 split that comes out 50.8/49.2 on large traffic points to a bug. Test it with a chi-square.
No peeking: checking every day and stopping at the first p < 0.05 inflates false wins. Use a sequential method if you must look early.
Several metrics or segments: expect 1 in 20 to look significant by chance. Correct (Bonferroni: alpha divided by the number of tests) or call them exploratory.
Watch for novelty effects: a new button gets clicks for a week because it is new.

Python

from statsmodels.stats.proportion import proportions_ztest
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# Checkout conversion: 9.0% today, want to detect 9.5%
es = proportion_effectsize(0.095, 0.090)
n = NormalIndPower().solve_power(effect_size=es, alpha=0.05, power=0.8)
print(round(n))          # customers per group, about 53,000

stat, p = proportions_ztest(count=[2860, 2710], nobs=[30000, 30000])

Regression in one screen

Regression estimates how the outcome changes with each input while the others stay fixed. It shows association; only a randomised test or a careful design shows cause (Regression).

In the output	Means	Watch for
Coefficient	Change in Y for one unit more of X, others held fixed	Units: cents, minutes, orders
Standard error and CI	How precise that coefficient is	An interval that crosses zero
R²	Share of the variation the model explains	High R² does not mean the model is right or causal
Residuals	Actual minus predicted	Patterns in a residual plot mean a missing variable or curve
`C(city)`	Dummy variables for a category, compared with a base level	Know which level is the base
`np.log(y)` as the outcome	Coefficients become roughly percentage changes	Zeros cannot be logged
Two inputs that move together	Multicollinearity: coefficients swing and SEs grow	Drop or combine one
Logistic coefficient	Change in log-odds; `exp(coef)` is the odds ratio	Odds are not probabilities

If you see X, suspect Y

Most wrong conclusions in business data come from a short list of traps (Statistical traps).

You see	Suspect	Check
A trend reverses when you split by city	Simpson's paradox: the mix of groups changed	Compare within each group
The worst stores improved after a coaching visit	Regression to the mean	Compare with similar stores that got no visit
Customers who use the app more spend more	Confounding or reverse cause	A randomised test, or control for tenure and city
Results only from customers who answered a survey	Selection bias	Compare responders with everyone
Advice drawn only from stores that are still open	Survivorship bias	Include the stores that closed
A "win" in one of 20 segments	Multiple comparisons	Correct alpha or retest that segment
A test stopped the day it turned significant	Peeking	Run to the planned sample size
A 99% accurate fraud flag that is mostly wrong	Base rate neglect: fraud is rare	Look at precision, not accuracy
A big percentage change on a small base	Small numbers	Show the counts and an interval
Average delivery time fell, but complaints rose	The mean hides the tail	Look at p90 and the distribution

Glossary

Plain definitions, with the lesson that explains each one.

Bootstrap: Estimating uncertainty by resampling your own data with replacement many times. Confidence intervals
Confidence interval: A range, from a method that captures the true value at a stated rate such as 95%. Confidence intervals
Confounder: A third factor that drives both the cause you are studying and the outcome. Statistical traps
Control group: The group that does not get the change, so you can see what would have happened anyway. A/B test design
Distribution: How often each value occurs; its shape matters as much as its middle. Describing data
Effect size: How large a difference is, in real units or standardised. Hypothesis tests
Guardrail metric: A metric an experiment must not harm, watched beside the primary metric. A/B test design
Median: The middle value when sorted; half the values are below it. Describing data
Minimum detectable effect: The smallest true effect a test is sized to detect with the chosen power. A/B test design
Null hypothesis: The default claim of no effect, which a test tries to reject. Hypothesis tests
Outlier: A value far from the others; investigate before you drop it. Describing data
p-value: The chance of a result at least this extreme if the null hypothesis were true. Hypothesis tests
Peeking: Checking a running test repeatedly and stopping early on a good result, which inflates false wins. A/B test design
Power: The chance a test detects an effect that is really there. A/B test design
Randomisation unit: What gets assigned to A or B: a customer, a store, a session. A/B test design
Regression to the mean: Extreme results tend to be followed by more ordinary ones, with no intervention. Statistical traps
Residual: The gap between an actual value and the model's prediction. Regression
Sample ratio mismatch: An experiment split that differs from the planned ratio by more than chance allows, a sign of a bug. A/B test design
Simpson's paradox: A trend in every group that reverses when the groups are combined. Statistical traps
Standard deviation: The typical distance of values from their mean. Describing data
Standard error: The standard deviation of an estimate, such as a mean, across repeated samples. Confidence intervals
Statistical significance: A p-value below the chosen alpha; not the same as a result that matters. Hypothesis tests
Type I and Type II error: A false alarm, and a missed real effect. Hypothesis tests

Courses on Kolom

Statistics for Data Work: describing data, intervals, tests, A/B design, regression and traps.
Metrics & Product Analytics: experiments and metric reviews in a product team.
Data Analysis with Excel, Python & AI: the tools to compute all of the above.
Data Science and Machine Learning: from inference to prediction.