Mean, median and skew

The mean is the sum divided by the count. Picture each value as a weight on a ruler: the mean is the point where the ruler balances. A weight far out on one side moves the balance point a lot. The median is the middle value after sorting. It only cares about order, so a far-out value moves it at most one place.

Here are ten Mangrove app baskets, in dollars: 7, 9, 10, 12, 13, 15, 17, 20, 23 and 58. The $58 order is someone's big monthly shop. The ten add up to $184, so the mean is $18.40. The median sits between the fifth and sixth values, 13 and 15, so it is $14. Seven of the ten baskets are below the mean. Remove the $58 order and the mean drops to $14.00, while the median only moves from $14 to $13.

When most values are small and a few are very large, the data is right-skewed: it has a long tail to the right. Money data is almost always like this: basket sizes, incomes, order values, time spent on a task. On right-skewed data the mean is above the median, and most values are below the mean. So "the average customer" describes a basket that most customers don't buy.

The same thing happens across all of Mangrove's app orders. On a normal day there are about 60,000. Their mean basket is $18.40 and their median is about $14. About two orders in three are below the mean.

Which one should you report? Ask what the decision needs. If you need a total, such as how much cash riders will collect tomorrow, the mean is right: mean × count = total, and the big orders really do add up. If you need the typical customer, such as what most people put in a basket, the median is right. When in doubt, report both, and treat a big gap between them as a signal about shape.

Percentiles and spread

A percentile answers "how big is the value that X% of orders stay under?" The median is p50. p90 is the value 90% of orders are below, and p99 the value 99% are below. For Mangrove's baskets, p90 is about $38 and p99 about $52. Percentiles describe the tail, which a mean hides.

Percentiles are how service promises work. When 91% of "Now" orders arrive within 30 minutes, that is the same fact as "the 91st percentile of delivery time is 30 minutes". A median delivery time of about 21 minutes says nothing about the 9% who waited longer. That is why people who run a service watch p90 and p99, the slow end, and not only the middle.

How is a percentile computed when it falls between two values? Most tools sort the values and use the position (n − 1) × p, counting from 0, and draw a straight line between the two neighbours. For the ten baskets, p90 sits at position 9 × 0.9 = 8.1, a tenth of the way from $23 to $58: $26.50. Excel's and Google Sheets' PERCENTILE.INC, numpy's np.percentile and pandas' quantile (with their default settings), and Postgres's percentile_cont all give $26.50. Other methods exist. PERCENTILE.EXC uses position (n + 1) × p and gives $54.50 here, and Postgres's percentile_disc returns a real value from the data, $23. With ten values the method changes the answer a lot; with 60,000 it barely matters. Pick one, and use the same one every week.

Small samples also make the extreme percentiles fragile. A p99 from 50 orders is decided by the two largest orders. With 60,000 orders a day, about 600 orders are above p99, so the number is stable. A rough rule: only report p99 when you have at least a few hundred values, and always state n beside it.

Spread is how far values sit from each other. Three measures are common:

• Range: maximum minus minimum. For the ten baskets, 58 − 7 = $51. It depends only on the two most extreme values, so one odd order decides it.
• IQR: p75 minus p25, the width of the middle half. For the ten baskets, Q1 = $10.50 and Q3 = $19.25, so the IQR is $8.75. For all 60,000 orders it is about $15 (from about $9.60 to $24.60). The extreme values don't touch it.
• Standard deviation (SD): take each value's distance from the mean, square it, add them up, divide by n − 1, and take the square root. For the ten baskets it is $14.77.

Squaring gives big distances a big voice. The $58 basket is 39.6 from the mean; squared, that is 1,568 of the total 1,964, about 80% of the sum. Without it the SD of the other nine is $5.27. One order nearly triples the SD. On skewed data, an SD describes the tail more than the typical spread.

Two versions of the SD exist. Dividing by n − 1 gives the sample SD, the right choice when your data is a sample from something bigger (the reason is part of Samples, uncertainty and confidence intervals, later in this course). Dividing by n gives the population SD. Spreadsheets have STDEV.S and STDEV.P. Watch the Python defaults: pandas' .std() divides by n − 1, while numpy's np.std divides by n unless you pass ddof=1. Postgres has stddev_samp and stddev_pop. With thousands of values the two barely differ; with ten values they differ by about 5% ($14.77 against $14.02).

A quick test for skew: compute mean − 2 × SD. For the ten baskets it is 18.40 − 29.55 = −$11.15, and for all app orders about −$6. No basket is negative. When "two SDs below the mean" is impossible, the data is skewed and the rules you may know for bell-shaped data, such as "95% of values within 2 SDs", do not apply.

Finding outliers: IQR fences and z-scores

An outlier is a value far from the rest. Two methods are common for finding them, and on skewed data one of them fails.

IQR fences, from John Tukey's box plot, set the fences at Q1 − 1.5 × IQR and Q3 + 1.5 × IQR. Anything outside is flagged. For the ten baskets: 10.50 − 13.13 = −$2.63 and 19.25 + 13.13 = $32.38. The $58 order is outside the upper fence and gets flagged. The fences are built from quartiles, so the outlier can't move them. Tukey also used 3 × IQR for values he called "far out".

A z-score is (value − mean) ÷ SD, and a common rule flags |z| > 3. For the $58 order: (58 − 18.40) ÷ 14.77 = 2.68. Not flagged. The outlier inflated both the mean and the SD it is measured against, so it hid itself. This is called masking. On skewed data the z-rule is also lopsided: the lower fence (mean − 3 SD) is far below zero, so it can only ever flag large values.

Fences have their own trap at scale. On all 60,000 app orders the upper fence is about $47, and about 2.5% of orders, roughly 1,500 a day, are above it. Nearly all of them are ordinary big scheduled shops. On right-skewed data, run the fences on the logarithm of the values (on a log scale the baskets look roughly symmetric); there they flag almost nothing, which is the truth.

A flag is a question, never a verdict. For each flagged value, find out which kind it is:

• An error: a test order, a duplicate, a price typed in dollars where the system expects cents. Fix it at the source if you can, remove it if you can't, and write down what you removed and why.
• Real but rare: a customer buying for a wedding. Keep it. If it distorts a summary, report a median or the result with and without it. Deleting real customers to make a chart prettier is how analyses start to lie.

Shape: bins, log scales and two groups in one column

A histogram splits the values into equal ranges, called bins, and draws how many fall in each. The bin width changes what you see. Too wide and real features disappear; too narrow and every bar is noise. Always try at least two widths. In most tools you can set the bin width directly (in Python, plt.hist(x, bins=...) takes a number of bins or a list of edges).

With $20 bins the app baskets look like one smooth slope: 69% of orders under $20, 23% from $20 to $40, 8% from $40 to $60. With $5 bins (the first diagram) the bars fall, then rise again around $35 to $40. A dip followed by a second rise is a sign of bimodal data: two groups mixed in one column.

Here the two groups are the two kinds of app order. About four orders in five are "Now" orders, 30-minute top-ups: a few items, median about $12, mean about $13.30. One in five is a "Later" order, a scheduled big shop: median about $38, mean about $38.80. Mixed together they give the overall mean of $18.40 (0.8 × 13.30 + 0.2 × 38.80), a number that describes neither group well. Most of the long tail in the first diagram is Later orders.

A log scale helps with right-skewed data. On a log axis, $5, $10, $20 and $40 sit at equal distances, because each is double the one before. The long tail gets squeezed and the bulk gets room, so you can see the shape of both. Basket sizes, incomes and response times often look roughly symmetric on a log scale. In a spreadsheet you can add a column with =LN(A2) and chart that; in matplotlib, ax.set_xscale("log") with bins spaced by np.geomspace.

The fix for mixed groups is to split before you summarise. Describe Now and Later orders separately, and only quote an overall number when the decision really is about all orders together. The same goes for any column that mixes stores and the app, weekdays and festival days, or new and returning customers.

Finally, always report n, the number of values a summary rests on. A ranking of 320 stores by average app basket once put a new store at the top with $31. It had 11 orders, two of them big monthly shops. An average of 11 orders and an average of 2,000 orders are different kinds of evidence, and the reader can only tell them apart if you write the n. How much an average from a small n can move is the subject of the next lesson.