From notebook to pipeline
Tomás's demand forecast beats the baseline, but it lives in a notebook that one person runs by hand. This lesson turns it into a pipeline: small steps with clear inputs and outputs, safe to re-run, scheduled as a DAG with retries and a deadline, packaged in Docker and tested before it ships. Then 320 stores get a forecast every night, whoever is on leave.
By the end you can
- Name the five things that make a notebook unsafe to run every night
- Split a model's nightly work into steps with a contract: inputs, one output partition and a run date
- Make every step idempotent, so a retry or a backfill overwrites instead of doubling
- Schedule the steps as a DAG with retries, timeouts, a deadline and alerts, and say when cron is enough
- Package the pipeline in a Docker image with pinned versions, and test it in CI on a small sample
The everyday picture
Think of a neighbourhood bakery. For years the owner baked every morning from memory: a pinch of this, the oven "hot enough", the dough left until it looked right. The bread was good. Then she fell ill in the busiest week of the year, nobody else could bake it, and the shop opened at 7:00 with empty shelves.
A bakery that supplies fifty shops works differently. Each recipe is written on a card with exact weights and oven temperatures. The work is split into stations (mixing, proving, baking, packing), and each station takes a known input and hands on a known output. A board on the wall says when each station must start so the vans leave at 6:00. When a tray burns, the baker bakes that tray again and bins the burnt one; nobody puts both on the shelf. And loaves reach the van only after someone checks them. A production pipeline is that bakery: written recipes, stations with inputs and outputs, a timetable with a deadline, redo instead of add, and a check before anything leaves.
Why the notebook breaks at night
In Forecasting demand Tomás built a gradient boosting forecast that beat the 28-day baseline on the backtest: WAPE 24% against 34%, and 35% against 58% in the festival week. In the 20-store pilot he ran it himself: at about 22:00 he ran the notebook top to bottom (mostly) and copied a CSV file into a shared folder that the ordering system read.
Production is a different job. About 320 stores each range roughly 8,000 of the 25,000 products, so there are about 2.5 million active store-product pairs (approximate). Forecasting 4 days ahead means about 10 million predictions a night, ready before the distribution centres (DCs) close their order intake. A notebook run by a person fails that job in five predictable ways:
- Hidden state. A DataFrame from a cell Tomás has since deleted is still in memory, so the notebook works today and fails on a fresh start.
- Out-of-order cells. The result depends on the order cells were clicked. Cell 14 ran before cell 9 last Tuesday, and nobody wrote that down.
- Manual steps. Download the extract, rename it, copy the output to the shared folder. Each step lives only in one person's habits.
- Hard-coded dates and paths.
TODAY = "2026-03-14"andC:/Users/tomas/sales.csv. Forget to edit the date and the system orders tomorrow's stock from yesterday's forecast with today's label. - Unpinned packages. Whatever pandas and scikit-learn are installed today. scikit-learn warns when you load a model saved with a different version, and does not promise it behaves the same.
Google's MLOps guide calls this stage "level 0": every step manual, run by the data scientist. Sculley and colleagues ("Hidden Technical Debt in Machine Learning Systems") point out that the model code is a small part of a real ML system; the data handling, checks, configuration and monitoring around it hold most of the work. Notebooks stay excellent for exploring. The job is to move the logic into code that runs the same way every night without a person.
Quick check
Tomás's notebook gives the right forecast on his laptop, but "Restart and run all" fails at cell 12 with "name promo_flags is not defined". What is the problem?
A NameError on a fresh start means the code never creates that variable; it only existed in the old session's memory. A changed package or a moved file gives a different error (an AttributeError or a file-not-found), and both would also fail without a restart.
Steps with a contract
The first move is to split the notebook into a pipeline of steps. For the nightly forecast there are six, plus one that runs weekly:
| Step | Reads (for run date D) | Writes | Typical time |
|---|---|---|---|
| extract | daily_sales, stock_snapshot, promo, festival up to D | raw partition for D | 10 min |
| validate | raw partition for D | a pass or fail report | 5 min |
| build features | raw partition for D | store_article_features rows with as_of_date = D | 20 min |
| predict | features for D, the champion model | forecast rows with this night's run_id | 40 min |
| evaluate | tonight's forecast; yesterday's forecast and sales | checks and WAPE on forecast_run | 10 min |
| publish | a run that passed evaluate | forecast_run.status = published | under 5 min |
| train (Sundays) | features for the last two years | a candidate model version | about 2 hours |
Validate is a gate that checks the data before the model sees it (Data and feature pipelines builds those checks). Evaluate checks tonight's output (one row per store, product and day, no negative forecasts, a chain total close to last week's) and scores yesterday's forecast against actual sales. Train only registers a candidate; the nightly predict always uses the current champion, and Experiments, versions and the model registry covers how a candidate is promoted.
Each step gets a written contract with four parts:
- Inputs: named tables or partitions, for one run date and nothing else.
- Output: exactly one partition, or rows tagged with one
run_id. A step never edits other steps' outputs. - Parameter: the run date, passed in as
--run-date 2026-10-20. The step never asks the clock what day it is. - Exit code: 0 for success, anything else for failure. A step that hits a problem fails loudly; it never writes an empty file and exits 0.
The run date parameter is what makes a backfill possible. Suppose the promo feature was wrong from 1 to 7 October. You fix the code and run build features, predict and evaluate for each of those seven dates, with the same code that runs tonight.
Small steps also make failures cheaper: if predict fails, the retry starts at predict, and the extract is already in its partition. Put a boundary where the resources change (predict needs far more CPU), at a natural checkpoint (validated data) or where the schedule differs (weekly training). Don't wrap every function in a step: each boundary costs a write and a read.
Idempotent steps: re-run without fear
Retries are normal in a nightly pipeline. A warehouse connection drops, a machine is reclaimed, a disk fills. The scheduler will run some steps twice, sometimes after the first attempt has already written part of its output. So every step must be idempotent: running it twice for the same run date leaves the same result as running it once. Three rules get you there.
- Write to a place named by the run. Features go to the partition for
as_of_date = 2026-10-20; forecast rows carryrun_id = 'fc-2026-10-20'. A retry writes to the same place as the attempt that failed. - Overwrite on re-run. For files, write to a temporary name and rename it over the old file. For tables, delete the run's rows and insert the new ones in one transaction, or replace the whole partition. A half-written attempt is wiped by the next one.
- Publish last. Readers (the DC ordering system, the Store app's ordering screen) never read a partition while it is being written. They read the newest run marked
publishedinforecast_run, and publish is the last, tiny step that sets that status; until then everyone keeps last night's forecast. Serving: batch and online builds this read and its fallback.
The opposite is "append and hope". Suppose predict crashes at 00:02 after inserting 1.2 million of tonight's 10 million rows, and the retry inserts all 10 million again. Now 1.2 million store-product-days appear twice, and anything that sums forecasts per store will order those products double. Nothing errors, so nobody notices until shelves overflow.
Two traps look idempotent and aren't. "Skip if the partition exists" turns a half-written partition into a permanent one, so check for a completion marker instead. And side effects outside your storage, such as sending the order file to a DC, also run twice on a retry. Give them a key built from the run_id so the receiver can ignore a repeat, the idempotency key from Queues, background jobs and pub/sub.
Quick check
The extract step appends tonight's sales to a raw_sales table. It failed after loading 60% of the rows, and the scheduler retried it successfully. What do you expect tomorrow?
An append adds rows; it never replaces them, and schedulers don't roll back what a step wrote. The fix is to load into the partition for the run date with an overwrite, or delete the run date's rows and insert in one transaction. Warehouses rarely reject duplicates unless you add a unique key.
Orchestration: a DAG with a deadline
An orchestrator runs the steps. You describe them as a DAG, a graph of steps where each arrow says "this step needs that one" and there are no loops, and the orchestrator does the rest:
- Schedule. The nightly DAG starts at 23:15, after the POS sales sync to the warehouse by about 23:00; training runs on Sunday evening. About 2% of a day's transactions arrive after midnight from offline tills (approximate), and the next night's extract picks them up (ml-02).
- Dependencies. A step starts only when every step it needs has succeeded. If validate fails, nothing downstream runs, and readers keep last night's forecast.
- Retries with backoff. Two retries per step with exponential backoff, 5 minutes and then 10. Never unlimited: a third failure is a real problem.
- Timeouts. Predict normally takes 40 minutes, so it gets a 50-minute timeout. A step that hangs without an error is the worst kind of failure, because it eats the slack in silence.
- A deadline and alerts. The SLA is "published by 01:30". If a step fails after its retries, or 01:30 passes with nothing published, the on-call engineer is paged with a link to the runbook. Monitoring, drift and retraining covers which alerts deserve a page.
- Backfills and history. Run the DAG for a range of past dates with one command, and see every run, attempt and log afterwards (Observability: logs, metrics and traces).
Do the time arithmetic before the first night. The run's budget is 90 minutes, 23:15 to about 00:45. That leaves 45 minutes before the 01:30 deadline and 75 before the DCs' 02:00 cut-off, roughly an hour of slack for one retry. A step that fails fast fits easily: extract losing its connection at 23:17 costs about 7 minutes. A worst case does not fit. If predict hangs until its 50-minute timeout at 00:40, the retry starts at 00:45 and predict finishes around 01:25; evaluate and publish land near 01:40, after the deadline. Plan that night in advance: the deadline alert fires at 01:30, the ordering system falls back to last night's forecast shifted by a day, and a person decides whether to wait. Splitting predict across four workers (ml-04) shrinks the worst case.
Do you need an orchestrator at all? For one script with no dependencies, cron is enough: it runs a command at a set time, and a small wrapper sends an alert when the command exits with an error. Once you have several steps with dependencies, per-step retries, a deadline and backfills, building that around cron means rebuilding an orchestrator, badly. Apache Airflow, Dagster and Prefect are common open-source choices, with managed equivalents in each cloud; the ideas above apply to all of them. Keep the business logic in the step code and only "when" and "in what order" in the DAG, so any step can also be run by hand with the same arguments.
Quick check
Predict usually takes 40 minutes. One night it hangs without an error. Why does a 50-minute timeout on that step help?
Retries only start after a step fails, and a hang never fails by itself. The timeout makes it fail early enough to retry and to warn the on-call engineer before the 01:30 deadline. Publishing a partial run is exactly what the publish-last rule prevents, and a timeout gives no extra resources.
One environment, tested before it ships
The steps run inside one Docker image per pipeline. Every step, including weekly training, uses the same image, so the feature code that trains the model is the code that predicts with it. Each run records the image tag (for example forecast-pipeline:1.4.2), so you can say which code made any night's forecast; the registry lesson adds the image digest and data snapshot. Shipping safely: containers, CI/CD and releases covers images in general.
Pin everything that could change under you. A lock file lists exact versions of every package, including the ones your packages pull in; tools such as pip-tools or uv generate it from the short list you asked for. Pin the base image too (python:3.12-slim, or its digest). The test: rebuild next month from the same commit and get the same packages.
Keep three kinds of setting apart. Code goes in the image. Config that changes between environments (the warehouse host, the bucket, "all stores" or "5 sample stores") comes from a config file or environment variables. Secrets, such as the warehouse password, are injected at run time from a secret store, never put in the image or in Git: anyone who can pull an image can read every file in it.
Then test at three levels:
- Unit tests for the feature functions. Keep them pure (a DataFrame in, a DataFrame out, no reading or writing), so a test can feed in five rows. The leak checks from Features and leakage become permanent tests.
- An end-to-end run on a sample in CI. Two weeks of data for 5 stores and 200 products sits in the repository; on every pull request, GitHub Actions runs every step on it in a few minutes, checks the row counts, then runs it again and checks that nothing changed. That second run is your idempotency test.
- Data checks at run time. The code can be right while tonight's data is wrong. That is the validate step, built in the next lesson.
# tests/unit/test_features.py
from datetime import date
import pandas as pd
from pipeline.features import add_lag_features
def test_lag_uses_only_sales_up_to_the_run_date():
sales = pd.DataFrame({
"store_id": [14, 14, 14],
"article_id": [501, 501, 501],
"sale_date": pd.to_datetime(["2026-10-18", "2026-10-19", "2026-10-20"]),
"units": [5, 7, 99],
})
out = add_lag_features(sales, as_of=date(2026, 10, 19))
# the 99 units sold on the 20th must not leak into features as of the 19th
assert out.loc[0, "units_last_7d"] == 12
Build it
Turn the forecast notebook into a pipeline that runs in Docker, is scheduled as a DAG and is tested on every pull request.
Lay out the project
One repository for the pipeline. Feature logic lives in pure functions, reading and writing partitions in one module, and each step in its own small script.
Outputforecast-pipeline/ Dockerfile requirements.in # what we ask for: pandas, scikit-learn, pyarrow requirements.lock # what we get: every package, exact versions config/ prod.yaml # warehouse schema, bucket, all stores sample.yaml # 5 stores, local folder, for CI dags/ nightly_forecast.yaml # when and in what order, nothing else pipeline/ features.py # pure functions: DataFrame in, DataFrame out io.py # the one place that reads and writes partitions run_all.py # runs steps in order, for CI and backfills steps/ extract.py validate.py build_features.py predict.py evaluate.py publish.py train.py # weekly tests/ unit/test_features.py sample/ # 2 weeks of data, 5 stores x 200 productsWrite each step as a script with a run date
The step takes
--run-date, reads only that date's input, writes only that date's partition, and replaces the file atomically. It raises an error instead of writing nothing.Python# pipeline/steps/build_features.py import argparse import os from datetime import date from pathlib import Path import pandas as pd from pipeline.features import add_lag_features # pure, unit-tested DATA_ROOT = Path(os.environ.get("DATA_ROOT", "data")) # config, not code def run(run_date: date) -> Path: src = DATA_ROOT / "raw" / f"run_date={run_date}" / "sales.parquet" out_dir = DATA_ROOT / "features" / f"as_of_date={run_date}" out_dir.mkdir(parents=True, exist_ok=True) sales = pd.read_parquet(src) # input: this run date's extract only features = add_lag_features(sales, as_of=run_date) if features.empty: raise ValueError(f"no feature rows for {run_date}") # fail loudly tmp = out_dir / "part.parquet.tmp" features.to_parquet(tmp, index=False) os.replace(tmp, out_dir / "part.parquet") # atomic swap on one disk return out_dir if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--run-date", required=True, type=date.fromisoformat) args = parser.parse_args() print(run(args.run_date))On object storage, one uploaded object replaces the old one whole, so the same rule holds. For a partition made of many files, write them under a new prefix and publish a pointer to it last.
Pin the environment in one image
Install from the lock file first so Docker caches that layer, then copy the code. No secrets in the image.
Dockerfile# One image for every step of the forecast pipeline FROM python:3.12-slim WORKDIR /app # Locked versions first: this layer is rebuilt only when the lock changes COPY requirements.lock . RUN pip install --no-cache-dir -r requirements.lock COPY config/ config/ COPY pipeline/ pipeline/ # The warehouse password arrives as an environment variable at run time ENV PYTHONUNBUFFERED=1 ENTRYPOINT ["python", "-m"]Output# requirements.lock (excerpt): every package, exact version numpy==2.1.3 pandas==2.2.3 pyarrow==17.0.0 scikit-learn==1.5.2 scipy==1.14.1 joblib==1.4.2Terminaldocker build -t forecast-pipeline:1.4.2 . docker run --rm -e DATA_ROOT=/data -v /mnt/forecast:/data forecast-pipeline:1.4.2 pipeline.steps.build_features --run-date 2026-10-20Describe the DAG
This is a generic sketch, not any one tool's format. Every orchestrator has these settings under its own names; in Airflow, for example,
retries,retry_delayandexecution_timeoutare task arguments.YAML# dags/nightly_forecast.yaml: a generic sketch pipeline: nightly-forecast image: forecast-pipeline:1.4.2 schedule: "15 23 * * *" # 23:15 every night timezone: store-local # set it explicitly, never trust the server clock params: run_date: logical-date # the business date, passed to every step defaults: retries: 2 retry_delay: 5m retry_backoff: exponential # 5 minutes, then 10 timeout: 30m deadline: publish_by: "01:30" on_miss: page ml-oncall # with a link to the runbook steps: extract: { run: pipeline.steps.extract, needs: [] } validate: { run: pipeline.steps.validate, needs: [extract] } build_features: { run: pipeline.steps.build_features, needs: [validate] } predict: { run: pipeline.steps.predict, needs: [build_features], timeout: 50m } evaluate: { run: pipeline.steps.evaluate, needs: [predict] } publish: { run: pipeline.steps.publish, needs: [evaluate] } # a second DAG, weekly-train, runs pipeline.steps.train on Sundays at 19:00Test on every pull request
Unit tests, then the whole pipeline on the sample, twice, and a check that the second run changed nothing. Last, prove the image still builds.
YAML# .github/workflows/pipeline-tests.yml name: pipeline-tests on: [pull_request] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: pip install -r requirements.lock pytest==8.3.3 - name: Unit tests for the feature functions run: pytest tests/unit - name: End-to-end on the 5-store sample, run twice env: DATA_ROOT: tests/sample run: | python -m pipeline.run_all --run-date 2026-03-10 --from validate cp -r tests/sample/forecast first_run python -m pipeline.run_all --run-date 2026-03-10 --from validate python -m tests.compare_partitions first_run tests/sample/forecast - name: The image builds run: docker build -t forecast-pipeline:ci .Shadow, then switch
Run the pipeline next to the old process for a week. Compare the outputs every morning, row counts and totals per store first. Switch the readers only when they match and every night finished on time, and keep the old process ready for one more week.
Bring it to your team
Ask these in the review before a model's first night in production.
- If this step runs twice for the same date, what changes? (The right answer is "nothing".)
- Where does each step get its date from, and what happens on a retry after midnight?
- What happens at 01:30 if nothing is published, and who is told?
- If we rebuild this image next month from the same commit, do we get the same packages?
- Where do the warehouse credentials live, and who can read them?
- Can someone other than the model's author run, re-run and backfill this from the runbook?
Red flags
- "It ran fine on my machine" as the test plan.
- An output table that steps append to, with no run date or run id.
date.today()ornow()inside step code.- A requirements file with no versions, or
FROM python:latest. - Unlimited retries, or no timeouts.
- Alerts that go to a channel nobody owns at 01:30.
Practice
Order the steps, and find what to re-run
Write two functions. runOrder(steps) takes an object where each key is a step name and each value is the list of steps it needs. Return every step name in an order where each step comes after everything it needs. When several steps are ready at the same time, take the alphabetically first one, so the answer is unique. If the steps can't all run (a cycle, such as a needs c, c needs b, b needs a), throw an Error. stepsToRerun(steps, failed) returns the failed step and every step downstream of it (that needs it directly or through other steps), in the order runOrder gives. Test data: { publish: ['evaluate', 'report'], evaluate: ['predict'], report: ['predict'], predict: ['features', 'stock'], features: ['validate'], validate: ['extract'], stock: ['extract'], extract: [] }.
function runOrder(steps) {
// steps: { stepName: [names of the steps it needs] }
// Return all step names in a valid order. Among ready steps,
// take the alphabetically first. Throw an Error on a cycle.
return Object.keys(steps);
}
function stepsToRerun(steps, failed) {
// Return the failed step and everything downstream of it, in run order.
return [failed];
}const nightlySteps = () => ({
publish: ['evaluate', 'report'], evaluate: ['predict'], report: ['predict'],
predict: ['features', 'stock'], features: ['validate'], validate: ['extract'],
stock: ['extract'], extract: []
});
test('runOrder puts every step after what it needs, alphabetical among ready steps', () => {
expect(runOrder(nightlySteps())).toEqual(['extract', 'stock', 'validate', 'features', 'predict', 'evaluate', 'report', 'publish']);
});
test('runOrder throws an Error when the steps form a cycle', () => {
expect(() => runOrder({ a: ['c'], b: ['a'], c: ['b'], d: [] })).toThrow();
});
test('stepsToRerun returns the failed step and everything downstream, in run order', () => {
expect(stepsToRerun(nightlySteps(), 'features')).toEqual(['features', 'predict', 'evaluate', 'report', 'publish']);
});
test('stepsToRerun follows every path down from a side branch', () => {
expect(stepsToRerun(nightlySteps(), 'stock')).toEqual(['stock', 'predict', 'evaluate', 'report', 'publish']);
});
test('stepsToRerun for the last step is only that step', () => {
expect(stepsToRerun(nightlySteps(), 'publish')).toEqual(['publish']);
});
Show a solution
function runOrder(steps) {
const names = Object.keys(steps).sort(); // alphabetical, so find() picks the first ready step
const done = new Set();
const order = [];
while (order.length < names.length) {
const ready = names.find((n) => !done.has(n) && steps[n].every((d) => done.has(d)));
if (ready === undefined) {
const stuck = names.filter((n) => !done.has(n));
throw new Error('cycle or missing step among: ' + stuck.join(', '));
}
done.add(ready);
order.push(ready);
}
return order;
}
function stepsToRerun(steps, failed) {
const rerun = new Set([failed]);
let grew = true;
while (grew) { // keep adding steps that need something already in the set
grew = false;
for (const [name, needs] of Object.entries(steps)) {
if (!rerun.has(name) && needs.some((d) => rerun.has(d))) {
rerun.add(name);
grew = true;
}
}
}
return runOrder(steps).filter((n) => rerun.has(n));
}Review a notebook-to-script plan
A teammate sends you this plan for the weekly churn model from Classification and churn, which picks customers for coupons every Monday. List the five biggest risks, and for each one say what you would change.
- Export the notebook to one script,
churn.py, and run it with cron on the analytics server every Monday at 06:00. - The script reads
customer_activity WHERE activity_date >= '2026-01-01'. - It appends the scored customers to the
churn_scoretable, which the coupon tool reads at 09:00. - Packages: whatever is on the server. "If something breaks I'll pip install the latest."
- The warehouse password is in the script so cron can run it.
- If it fails I'll see it in the cron log when I check on Monday afternoon.
- It loads
model_final_v3.pklfrom my home folder. - Testing: I ran it once and the numbers matched my notebook.
- You flag the append into
churn_score: a re-run doubles customers, and the fix is a run date or run id with overwrite, read by the coupon tool only when published - You flag the hard-coded date and the missing run-date parameter, which block re-runs and backfills
- You flag the silent failure: nobody is alerted before the 09:00 coupon run, and there is no fallback or owner
- You flag the environment: unpinned packages and a model file in a home folder, fixed with a lock file, an image and a model loaded from a registry or shared storage by version
- You flag the password in the script and say where it should come from instead
Show a model answer
1. Append without a run key (item 3). A retry or a manual re-run adds every customer again, and the coupon tool sends two coupons. Write scores with a run date, delete and insert that run's rows in one transaction, and have the coupon tool read only the latest run marked published.
2. Hard-coded date, no run-date parameter (item 2). The window grows every week and the script can't be re-run for last Monday. Take --run-date and compute the window from it, for example the 90 days before the run date.
3. Silent failure (item 6). If it fails at 06:00, the coupon tool runs at 09:00 on old or missing scores, and the author finds out in the afternoon. Alert on a non-zero exit and on "no published run by 08:00", name an owner, and decide the fallback (skip this week's coupons or reuse last week's list).
4. Unpinned environment and a model in a home folder (items 4 and 7). A package upgrade or a deleted file changes or stops the scores without any code change. Build an image from a lock file, and load the model by version from shared storage or a registry (ml-03).
5. Password in the script (item 5). Anyone who can read the script or a copy of it has warehouse access. Inject it at run time as an environment variable from a secret store, and rotate the current one.
Also worth raising: one big script with no steps (a failure re-runs everything), and a test plan of one manual run (add unit tests for the features and a sample run in CI). Cron is fine here: one weekly script with an alert does not need an orchestrator yet.
Check yourself
Predict crashed at 00:02 after inserting 1.2 million of tonight's 10 million rows into forecast with a plain INSERT. The orchestrator retried at 00:07 and it succeeded. The ordering system reads every row for tomorrow's date. Which design would have made this night safe?
Overwrite by run id plus publish-last makes the retry harmless and keeps readers off half-written data. Turning off retries turns every blip into a missed night. Deduplicating in every reader spreads the fix around and breaks the first time a reader forgets. Random table names leave readers with no way to know which table is complete.
A buyer's dashboard needs one SQL extract refreshed at 06:00 each day. It is one script with no dependencies, and a late refresh only means a stale chart for an hour. A teammate wants to install a DAG orchestrator for it. What do you suggest?
The orchestrator earns its cost when there are dependencies, per-step retries, deadlines and backfills to manage. One independent script needs a schedule and an alert, which cron and a small wrapper give. Running it by hand brings back the notebook problem, and tying it to the forecast DAG makes an unrelated job wait on, and possibly break, the forecast.
Monday's forecasts are about 3% higher than Sunday's across most stores. The code, the model file and the input data did not change. On Sunday the image was rebuilt from a Dockerfile containing RUN pip install pandas scikit-learn. What is the most likely cause and fix?
The only thing that changed was the environment, and an unpinned install picks up whatever is newest on the day it runs. Drift and overfitting would not appear overnight with identical inputs and an identical model file. There is no notebook kernel in a containerised run.
Flashcards
Key takeaways
- A notebook fails nightly work through hidden state, out-of-order cells, manual steps, hard-coded dates and paths, and unpinned packages.
- Split the work into steps with a contract: inputs for one run date, one output partition or run id, the date passed in, and a clear exit code. The same code then runs tonight and in backfills.
- Make every step idempotent: write to a place named by the run, overwrite on re-run, publish last. Never append and hope.
- Schedule the steps as a DAG with retries, backoff, timeouts, a deadline and an alert, do the worst-case time arithmetic, and decide the fallback in advance. Cron is enough for one script.
- Run everything in one image built from a lock file, keep config and secrets out of the code, and test with unit tests and a twice-run sample in CI.
Sources and further reading
- MLOps: Continuous delivery and automation pipelines in machine learning, Google Cloud Architecture Center
- Hidden Technical Debt in Machine Learning Systems, D. Sculley and colleagues, NeurIPS 2015
- Apache Airflow documentation, Apache Software Foundation
- Docker documentation, Docker
- GitHub Actions documentation, GitHub