Regression and correlation
Riders say rainy evenings are slower, and everyone knows far orders take longer. This lesson turns those hunches into numbers you can defend: how tightly two things move together, how many minutes each extra kilometre and each rain shower add, how sure you can be of those minutes, and when a fitted line says nothing about cause.
By the end you can
- Read a scatter plot and a Pearson correlation, and name what r cannot see
- Fit a least-squares line in a spreadsheet, Python or SQL, and explain its slope, intercept and R² in business units
- Read a multiple regression table with a 0/1 rain variable: coefficient, standard error, confidence interval and p-value
- Check a fit with a residual plot and refuse to predict outside your data
- Spot the confounder behind a tempting correlation and say what would show cause
The everyday picture
A taxi fare has two parts: a fixed charge when the meter starts, and a rate for every kilometre. Imagine you never saw the price card. All you have is a shoebox of 400 old receipts, each showing the distance and the fare. Plot them and they fall roughly along a straight line. Where the line meets zero kilometres is the starting charge. How steeply it climbs is the price per kilometre. The receipts don't sit exactly on the line, because of red lights, waiting and tips, and that scatter tells you how much the line leaves out.
Now suppose some receipts come from rainy nights, when the company adds a surcharge. Colour those receipts blue and they sit a little above the others at every distance. One more number, the rain surcharge, describes that gap.
Regression is reading the price card back from the receipts. It comes with one warning. If fares were higher on nights when more taxis were out, that would not mean taxis raise prices. Busy nights bring both.
Scatter plots and correlation
The course page promised this question: are rainy evenings really slower for riders? Tomás, Mangrove's data analyst, pulls 400 delivered "Now" orders from one city over two weeks in October. For each order he has three columns: distance_km from store to door, rain (1 if it was raining when the order was placed, 0 if not) and minutes from order to door.
Start with a scatter plot. Put the thing you think drives the other on the horizontal axis (distance) and the outcome on the vertical axis (minutes), one dot per order. Before you compute anything, look. Does the cloud slope up? Is it straight or bent? Are there dots far from the rest? Is the cloud the same width everywhere?
Then measure how tightly the dots follow a straight line with the Pearson correlation, written r. Turn each variable into z-scores (how many standard deviations each value is from its mean, as in st-01), multiply each order's two z-scores together, add them up and divide by n − 1. When long trips are also slow trips, most products are positive and r is close to +1. When there is no straight-line pattern, the products cancel and r is near 0. For Tomás's 400 orders, r = 0.72: a clear upward relationship with plenty of scatter.
- Spreadsheet:
=CORREL(C2:C401, A2:A401)in Excel or Google Sheets (the order of the two ranges doesn't matter for r). - Python:
df["minutes"].corr(df["distance_km"])in pandas, orscipy.stats.pearsonr(x, y), which also returns a p-value. - SQL:
corr(minutes, distance_km)in PostgreSQL.
r is useful and easy to misread. It misses four things:
- Curves. r only measures straight-line patterns. Delivery minutes by hour of the day rise at lunch, fall in the afternoon and rise again at dinner: a strong pattern with r close to 0.
- Outliers. One test order to an address 9 km away can push r up or down a lot, because its z-score is huge.
- Steepness. r says how tightly the dots follow a line, not how steep the line is. An r of 0.72 could go with 1 extra minute per km or with 10.
- Shape. In 1973 the statistician Francis Anscombe published four small datasets, now called Anscombe's quartet. All four have the same mean of x (9) and of y (7.5), the same correlation (0.82) and the same best-fit line, y = 3 + 0.5x. Plotted, they look nothing alike.
The lesson of the quartet is a habit: plot first, then compute. A correlation or a line is a summary of a picture, and a summary of the wrong picture is wrong.
Quick check
Across 30 stores, Tomás finds r = 0.05 between the hour a store opens and its daily app orders. What can he conclude?
r near 0 rules out a straight-line pattern only. Stores that open very early and very late might both do well, which is a curve with r close to 0. "Weak effect" is also wrong: r is not an effect size, and it says nothing about cause.