Population, sample and how you pick it

The population is everything you want to know about: all 320 Mangrove stores, every app order in October, every registered customer. The number you want, such as the average stock accuracy across all 320 stores, is fixed but unknown. A sample is the part you measure. You compute the same number on the sample and use it as your estimate.

Why sample at all? Because measuring everything costs time and money. Stock accuracy is the share of counted items where the system's stock figure matches what is on the shelf; at Mangrove it is about 82%. To check it, staff count items by hand. Counting every item in every store would stop the stores for days. Counting 200 random items in 40 random stores takes an afternoon per store.

How you pick the sample matters more than how big it is. In a random sample, chance decides which units get in, so every store has the same chance. In a convenience sample, people choose: the stores near head office, the managers who volunteer, the customers who answer a survey. Those units often differ from the rest. Managers who volunteer for a stock count tend to run tidy stores. That error is called bias, and a bigger sample does not fix it. Ask 400 volunteers instead of 40 and you get the same wrong answer, only more confidently.

Drawing a random sample takes a minute in any tool:

• Spreadsheet: next to the list of 320 stores, add a column with =RAND(), then sort by it and take the first 40 rows. RAND changes every time the sheet recalculates, so copy the column and paste it as values before you sort.
• SQL (Postgres): SELECT id FROM site WHERE kind = 'store' ORDER BY random() LIMIT 40;
• pandas: stores.sample(n=40, random_state=2026). The random_state number makes the draw repeatable, so a colleague gets the same 40 stores.

If cities differ a lot, you can also draw at random within each city in proportion to its store count. That is a stratified sample, and it makes sure no city is left out by bad luck.

How much does a sample wobble? Simulate it

Draw a different sample and you get a slightly different answer. You can see how much by trying it many times on a population you know. In Describing data honestly you met Mangrove's app baskets: average about $18.40, median about $14, with a long right tail from big monthly shops. Tomás built a simulated population of one million baskets with that shape. Its standard deviation is about $15.70.

Then he did what you can never do with real data: he drew a random sample of 100 baskets, wrote down its mean, and repeated that 1,000 times. The 1,000 sample means look like this:

• They pile up around the true $18.40. No single sample is far off very often.
• They spread much less than single baskets do. Their standard deviation is $1.55, against $15.70 for single baskets.
• The middle 95% of them fall between $15.71 and $21.55. The lowest was $13.51 and the highest $26.95.
• Their histogram is close to a bell shape, even though single baskets are heavily skewed.

That pile of possible answers is the sampling distribution. Its standard deviation has its own name, the standard error (SE), and you don't need 1,000 samples to know it. For a mean:

standard error = s / √n        s = standard deviation of the values, n = sample size

baskets: s = $15.70, n = 100  →  SE = 15.70 / √100 = 15.70 / 10 = $1.57
the simulation measured $1.55: the formula predicts the wobble from ONE sample

The s comes from your one sample (STDEV.S in a spreadsheet, stddev_samp in Postgres, .std() in pandas, which divides by n − 1 by default). So one sample tells you both an estimate and how much to trust it.

Why √n? Errors in single values partly cancel when you average them: a big basket here, a small one there. They cancel more slowly than the sample grows. Tomás's simulation, repeated for three sample sizes, shows the pattern:

sample size   SE by formula ($15.70 / √n)   SD of 1,000 simulated means
     25               $3.14                          $3.12
    100               $1.57                          $1.55
    400               $0.79                          $0.80

Four times the sample halves the error, because √4 = 2. To cut the error to a tenth you need 100 times the data. This is why precision gets expensive fast, and why a sample of 1,000 is common in surveys: going to 4,000 costs four times as much and only halves the error.

The bell shape at the bottom of the diagram is the central limit theorem at work. In plain words: averages of enough random values pile up in a bell shape, even when the values themselves are lopsided. How many is "enough" depends on the skew. For mildly skewed data 30 often does; for baskets with a long tail, means of 25 are still a little lopsided and 100 or more look like a bell. The theorem is about averages; it never makes the baskets themselves normal.

From standard error to a confidence interval

Because sample means form a bell, about 95% of them land within 1.96 standard errors of the true value. Turn that around and you get a recipe for a range around your one estimate, the confidence interval (CI):

95% CI ≈ estimate ± 1.96 × SE          the part after ± is the margin of error

With small samples you also estimate the spread s from few values, which adds uncertainty. So instead of 1.96 you use a slightly bigger multiplier from the t distribution with n − 1 degrees of freedom: 2.262 for 10 values, 2.201 for 12, 2.023 for 40, and 1.984 for 100. Above a few hundred values it is practically 1.96. A spreadsheet gives it with =T.INV.2T(0.05, n-1), Python with scipy.stats.t.ppf(0.975, n - 1).

For a proportion (a share of yes or no answers), the standard error has its own formula, because the spread of yes/no data depends only on the share itself:

SE = √( p × (1 − p) / n )

Store 17 counts 200 random items; 158 match the system:  p = 158 / 200 = 0.79
SE = √(0.79 × 0.21 / 200) = 0.0288
95% CI = 0.79 ± 1.96 × 0.0288 = 0.79 ± 0.056  →  73.4% to 84.6%

This shortcut works when there are at least about 10 yes and 10 no answers. With fewer, or with a share near 0% or 100%, it can even give a range below zero. Then use the Wilson interval, which statsmodels offers as proportion_confint(count, n, method="wilson"). For store 17 it gives 72.8% to 84.1%, close to the shortcut. For 1 cut item in 20 orders, the shortcut says −4.6% to 14.6% and Wilson says 0.9% to 23.6%.

A useful sense of scale: a poll of 1,000 people on a question near 50/50 has a margin of error of about 3 points, because 1.96 × √(0.25 / 1,000) = 0.031. You will hear "plus or minus 3 points" in news polls for that reason.

What a confidence interval does and does not mean

The "95%" describes the method. If you repeat the whole process (draw a fresh random sample, compute the interval) many times, about 95% of the intervals will contain the true value. The second diagram shows it: 19 of these 20 intervals catch $18.40. Your one interval either contains the true value or it doesn't, and you can't tell which. That is why careful people say "we are 95% confident" and describe the method behind it.

Four readings to avoid:

• "95% of stores lie in the interval." The interval is about the average. Single stores spread far wider: in the audit below, the stores ranged from 63.5% to 97%, while the interval for the average is about 5 points wide.
• "The interval covers every kind of error." It covers only the luck of the draw. A convenience sample gives an interval just as narrow around the wrong value. Miscounted shelves and a wrong system figure are not in it either.
• "Values near the edge are as likely as the middle." Values near the centre fit the data best; the edges are the least plausible values still in range.
• "The method is exactly 95% every time." It is close to 95% when its assumptions hold. With skewed data and small samples it misses more often. In Tomás's simulation the intervals caught the true basket mean about 91% of the time with samples of 25, about 93% with 100 and about 95% with 400.

The width is the useful part. A result of "81.9%, between 79.4% and 84.4%" tells a manager whether the answer is precise enough to act on. A bare "81.9%" hides that.

The bootstrap: a confidence interval by resampling

The formulas above work for means and proportions. For a median, a 90th percentile or a ratio, the formula is hard or missing. The bootstrap, introduced by Bradley Efron in 1979, gets an interval by simulation from the one sample you have:

1. Treat your sample as a small model of the population.
2. Draw a new sample of the same size from it with replacement (a store can be picked twice, another not at all).
3. Compute your statistic (mean, median, anything) on it.
4. Repeat 10,000 times. The middle 95% of the 10,000 results is a 95% interval (the percentile method).

It is the simulation from earlier, run on your sample instead of on a population you can't see. It needs no bell-shape formula, so it works for medians and ratios. It still needs a random sample, and with very small samples (say under 20) it tends to be too narrow, because a small sample shows too little of the population's spread.

In Python, a few lines do it by hand, and SciPy (version 1.7 or later) has it built in:

import numpy as np
from scipy import stats

values = np.array(accuracy_pct)              # the 40 measured stores
rng = np.random.default_rng(7)               # a seed makes the result repeatable
boot = [rng.choice(values, size=len(values), replace=True).mean()
        for _ in range(10_000)]              # 10,000 resampled means
print(np.percentile(boot, [2.5, 97.5]))      # middle 95% = the interval

# built in; its default method is "BCa", a corrected version of the percentile method
res = stats.bootstrap((values,), np.mean, confidence_level=0.95, n_resamples=10_000)
print(res.confidence_interval)

Spreadsheets have no bootstrap function. You can build one with RANDBETWEEN and INDEX across many columns, but it is slow and fragile; this is a good place to move to Python (the same analysis in Python).