Explaining and choosing models
A model that beats the baseline on average can still fail the stores that need it most, or be the wrong tool for the job. This lesson shows you how to explain what a model learned, find where it goes wrong, check who carries its errors, and decide, in writing, whether a model, a rule, an LLM or nothing should ship.
By the end you can
- Rank features with permutation importance on validation data, and say why impurity-based importance misleads
- Explain one prediction with SHAP values and read a partial dependence plot, without reading them as cause and effect
- Run an error analysis by segment and decide what to do about the worst slice
- Check error and benefit across groups such as store size band and city
- Choose between ML, a rule, an LLM or not building, and write a model choice memo the supply chain team can act on
The everyday picture
Picture a baker who has served the same neighbourhood for twenty years. Every evening she decides how many loaves to bake for tomorrow, and she is rarely far off. Ask her why she is baking 140 tomorrow instead of the usual 90, and she will tell you: the school fair is on Saturday, that adds a lot; there is a discount on rolls, that adds a little; it rained all week and people stayed home, that takes some off. That is an explanation of one decision.
Ask her what matters most in general, and she says last week's sales and the calendar. Ask her where she gets it wrong, and she names two things: new kinds of bread, and the small kiosk across town she supplies, which she never quite gets right. And for some jobs she does not use her judgement at all. If a café that usually orders 20 loaves suddenly orders 200, she has a plain rule: phone them before baking. This lesson is about getting the same four answers from a model: why this prediction, what matters overall, where it fails and for whom, and when a plain rule is the better tool.
Which features matter: permutation importance
By Forecasting demand, Tomás has a gradient boosting model that beats the 28-day baseline: WAPE 24% against 34%, and 35% against 58% in the festival week. Before the supply chain team trusts it, they ask a fair question: what is it looking at? A global importance ranking answers that, and there are two common ways to get one. One of them misleads.
Tree models in scikit-learn, such as RandomForestRegressor, have a feature_importances_ attribute. It is impurity-based importance: it adds up how much each feature reduced the error at the splits during training. That has two problems. It is measured on the training data, so a feature the model used to memorise rows looks important even if it is useless on new days. And it favours high-cardinality features, because a column with 1,000 distinct values offers many more possible splits than a yes or no column. When Tomás added a column called store_article_key (1,000 values for 20 stores × 50 products), the random forest ranked it second. HistGradientBoostingRegressor does not offer feature_importances_ at all, which is a hint.
Permutation importance asks a better question: if I scramble this column, how much worse does the model get? You take the validation fold, shuffle one column so its values no longer line up with the right rows, predict again and measure the error. Repeat 10 times and average. A feature the model really relies on makes the error jump; a useless one changes nothing. It works for any model, and because it runs on validation data, it measures what helps on days the model has not seen. On Tomás's model, shuffling avg_last_28 raises WAPE by about 6 points, days_to_festival by about 4, and store_article_key by nothing. A cheap sanity check: add a column of random numbers. Any feature that scores below it is noise.
One trap remains: correlated features share their importance. The model has lag_7 (same day last week) and lag_14 (same day two weeks ago). Shuffle lag_7 alone and the model leans on lag_14, so each looks minor, about 2 and 1.4 points. Shuffle both together and WAPE rises by about 8 points. Read a low score as "the model can do without this column, given the others", never as "this information does not matter".
Quick check
The random forest's feature_importances_ ranks store_article_key second, but its permutation importance on the validation weeks is about zero. What is the most likely explanation?
The key lets the forest carve the training data into tiny groups, which counts as error reduction during training. On new days that memorisation does not help, and shuffling the key on validation data shows it. The two methods measure different things: training splits versus validation error.