Training and comparing models
You have a framed problem and honest features. This lesson trains four kinds of model on them with scikit-learn (a regularised linear model, a decision tree, a random forest and gradient boosting), shows you how to read under- and overfitting from two numbers, and walks through a comparison against the baseline that you could defend in front of the people who will order stock from it.
By the end you can
- Explain what training does: a model with adjustable parameters, a loss, and a search for the parameters that make the loss small
- Train Ridge, random forest and gradient boosting models with scikit-learn pipelines, and say how each one predicts
- Read underfitting and overfitting from training error and validation error, and choose a model setting from them
- Tune a few hyperparameters with a time-aware search, without touching the test period
- Compare every model with the baseline on the same backtest, by segment and by bias, and pick one with a written reason
The everyday picture
Think of three students preparing for a maths exam with ten past papers. The first learns one trick, "the answer is usually around 50", and uses it on every question. She does badly on the past papers and badly on the real exam. The second memorises every answer on all ten papers. He scores 100% when he re-does them, then fails the real exam, because its questions are new. The third works through the papers, checks her mistakes, and learns the methods behind the answers. She scores a little lower on the past papers than the memoriser, and much higher on the real exam.
A good teacher does two more things. She keeps one past paper aside as a mock exam that nobody studies from, and uses the mock score to decide who is ready. And she never shows anyone the real exam before the day, because a score on a paper you have already seen tells you nothing. Training models works the same way. The past papers are your training data, the mock exam is your validation period, the real exam is the test period, and "the answer is usually around 50" is the baseline every model has to beat.
What training does: a loss and a search
A model is a formula with knobs. Training means turning the knobs until the formula's predictions on the training rows are as close as possible to the real values. "As close as possible" is measured by a loss function. For regression, the most common loss is squared error: each row's error is squared and the squares are averaged. The knobs the model learns are its parameters. The settings you choose before training, such as how deep a tree may grow, are hyperparameters.
The loss a model trains on and the metric you report do not have to match. Mangrove's forecasts are judged on WAPE and bias, as you set up in Framing a problem as a model. Most models train on squared error, which punishes big misses more than WAPE does. That is usually fine, as long as you always compare models on the metric the business cares about.
Linear regression is the simplest model worth training. It predicts a weighted sum of the features. For one store-product-day it might learn:
units = 2 + 0.45 x lag_7 + 0.50 x avg_last_28 + 6 x is_weekend
Saturday, lag_7 = 40, avg_last_28 = 36:
units = 2 + 18 + 18 + 6 = 44Training finds the weights (0.45, 0.50, 6) and the intercept (2) that make the squared error on the training rows smallest. With hundreds of features, some weights grow large because they happen to fit noise in the training data. Regularisation adds a penalty for large weights. scikit-learn's Ridge adds alpha times the sum of the squared weights to the loss. A bigger alpha shrinks the weights more; the default is 1.0.
Two preparation steps matter for linear models. First, feature scaling: the Ridge penalty treats every weight the same, so a feature measured in hundreds of units and a 0/1 weekend flag get penalised unfairly unless you rescale them first, for example with StandardScaler. Second, one-hot encoding for categories. Store 300 is not "thirty times" store 10, so store_id becomes one 0/1 column per store (about 320 columns), and the same for day of week and product category.
A linear model extrapolates. If last week's sales doubled before a festival, its forecast doubles too, which tree models cannot do. The same property is a risk: a data error that makes lag_7 4,000 gives a forecast in the thousands, and a slow seller can get a negative forecast. Clip predictions at zero and check inputs for impossible values.
Quick check
Tomás passes store_id (1 to 320) and lag_7 to Ridge as plain numbers, without scaling or one-hot encoding. What is the main problem?
A linear model multiplies each feature by one weight, so an id used as a number becomes a meaningless slope. One-hot encoding gives each store its own 0/1 column and its own weight. Ridge accepts integers without complaint, which is exactly why the mistake goes unnoticed.