Monitoring, drift and retraining
A model that passed every test on the backtest can still go wrong on an ordinary Tuesday, because the world and the data that feed it keep changing. This lesson shows you what to watch at four layers, how to measure drift with PSI, how to tell a festival from a broken feed, and how to retrain and roll back without panic.
By the end you can
- Monitor four layers (pipeline health, input data, predictions, outcomes and business metrics) and know when each signal arrives
- Measure data drift with the population stability index (PSI) and tell data drift from concept drift
- Tell expected seasonal change, such as a festival, from real breakage, with separate festival thresholds
- Write alerts that have an owner, a severity and a runbook
- Retrain on a schedule and on triggers through the same promotion gate, and roll back in minutes
The everyday picture
Think of a baker who sells bread from a market stall. She checks different things at different times. Before dawn: is the oven hot, has the dough risen? If not, there is no bread at all. While she mixes: do the flour, water and yeast look and weigh as usual? When the loaves come out, she counts them against a normal morning. Only at closing time does she learn how good her plan was: how much bread is left, and how many customers she sent away.
Before a festival she bakes twice as much and expects some odd days, so she doesn't panic when sales jump. One morning, though, every loaf comes out flat. Her recipe has not changed. The supplier switched from one-kilogram bags to 500-gram bags, and she still counts bags. Writing a new recipe would be the wrong fix; she needs to fix the measuring. And when a new recipe she tried last week turns out worse, she takes the old card out of the drawer. Monitoring a model works the same way: checks that arrive at different times, a sense of what normal looks like on each day of the calendar, and the habit of fixing the cause before changing the recipe.
Four layers, four clocks
In Serving: batch and online, the nightly run published about 10 million predictions by 01:30. How do you know they were any good? Actual sales arrive the next day, and for the day-4 forecast four days later. This wait is called label delay, and it is why you also watch layers that answer sooner, each on its own clock.
- Pipeline health (minutes). Did the run finish and publish by 01:30, inside its 90-minute budget, with about 10 million rows? It all lives in
forecast_run(status,started_at,finished_at,row_count). If this layer is red, the stores need the fallback forecast from ml-04 tonight. - Input data (during the run). Do tonight's features look like the training features? Freshness, null rates, ranges and the shift in distribution, which the next section measures. The validation gate in Data and feature pipelines stops obvious breaks such as an empty table; monitoring catches the slow ones, like a feature that moves a little every night.
- Predictions (before publish). Does the output look like a normal night? Compare total p50 per DC with the same weekday last week, and watch the share of zero forecasts and the pairs forecast at more than 5 times their 28-day average. These checks run before the publish step, so they can stop wrong numbers reaching the stores.
- Outcomes and business metrics (next day, then days). WAPE and bias per DC, product group and horizon from the next day's sales (Forecasting demand), then the out-of-stock rate, waste and how often managers change the suggested order (Responsible ML in a business). Slow and noisy, but the truth.
Watch the early layers because they are fast: a broken feed found at 00:10 can be fixed before the 02:00 order cut-off, while the same break found in tomorrow's WAPE has already become wrong orders on trucks. Watch the late layers because the early ones can all be green while the model is wrong: if a competitor opens next to six stores, every input looks normal and only those stores' WAPE goes up.
Two details keep outcome numbers fair. Yesterday's WAPE is provisional: about 2% of transactions arrive after midnight from offline tills, so recompute the last 3 days every night, as the extract does in ml-02. And report stock-out days on their own, because on a day the shelf was empty from 3 pm, sales were lower than demand and a good forecast looks wrong.