Experiments, versions and the model registry

35 minAdvanced

Experiments, versions and the model registry

A model in production is a chain of choices: code, data, settings, libraries and a random seed. This lesson shows you how to record every link of that chain for every training run, rebuild a run months later, keep models in a registry with versions and aliases, and promote a new version only after it passes a written gate, a shadow week and a sign-off.

By the end you can

Log every training run with its parameters, metrics, code version, data snapshot, environment and model files
Reproduce a run months later, and say which input most often breaks reproduction
Keep models in a registry with versions and aliases such as champion and challenger, and trace tonight's forecast back to the code and data that made it
Compare a candidate with the champion fairly and promote it through an automated gate, a shadow week and a human sign-off
Roll back by moving an alias, in minutes, without a code deploy

The everyday picture

Pick up a jar of peanut butter and look at the lid. There is a batch code printed on it, something like L2614 0847. To you it means nothing. To the factory it means everything: which lot of peanuts went in, which production line, which version of the recipe, which shift, which day. If customers in one city get sick, the factory reads the code, finds the batch, and recalls only those jars. Without the code, they would have to recall everything they made that month.

A good factory also never puts a new recipe straight on the shelves. It is tried in the test kitchen against the current recipe, then sold for a while in a few shops, and the head of quality signs it off. The old recipe card stays in the binder, so if the new one causes complaints, the line can switch back the same afternoon. Running a model in production needs the same three habits: a code on every output that leads back to how it was made, a trial before a new version replaces the old one, and the old version kept ready to switch back to.

Track every training run

Once a model runs every night for 320 stores, questions start to arrive that a notebook can't answer. The supply chain lead asks which model made last week's festival forecast. Tomás wants to know why this Sunday's model is worse than August's. An auditor asks for the exact model that set orders on 3 October. Without records, each answer takes days of digging through notebooks, chat messages and files called model_final_v3.pkl, and some answers are never found.

Experiment tracking fixes this. A tracking server, such as a self-hosted MLflow server, stores one record per run. MLflow-style runs hold four kinds of information:

Parameters: the settings you chose, such as learning_rate, max_iter, the training window and the random seed.
Metrics: the results, such as backtest WAPE overall, in festival weeks and per product group, and bias.
Tags: free-form labels, such as the git commit, the data snapshot id, the image digest, who started the run and why.
Artifacts: files, such as the trained model, the package lock file, the list of features and a chart of errors by week.

The parameters and metrics are the part most teams remember. The part they forget is the record of inputs, which is what you need when something goes wrong. For Mangrove's demand forecast, Tomás logs this for every run:

What	Example	Answers the question
Git commit	`7c1e2d9`	Which feature code and training code made it?
Data snapshot id	`snap-2026-10-11`	Exactly which rows did it learn from?
Image digest and lock file	`sha256:4b1f…`, `requirements.lock`	Which library versions?
Parameters and seed	`learning_rate=0.05`, `seed=42`	Which settings?
Backtest folds and metrics	WAPE 24.1%, festival 35.0%, bias −1.2%	How good was it, on which weeks?
Trigger and owner	`scheduled`, `tomas`	Was it the weekly job or a manual try?

Two rules keep the records honest. First, the pipeline logs, people don't: the weekly training step from From notebook to pipeline writes every field above automatically, so a run that isn't logged doesn't exist and can't be promoted. Second, the training step refuses to start if the code has changes that are not committed, because a git commit only describes the code if the code matches it. The cost is small next to the questions it answers: Mangrove's tracking server and its storage cost about $400 a month (approximate), less than a tenth of an engineer.

Quick check

Tomás logs parameters, metrics and the model file for every run, but not the git commit. The feature code has changed 40 times since August. What can he not answer about August's model?