Features and leakage

35 minIntermediate

Features and leakage

A model can only use what you know at the moment you predict. This lesson shows you how to build demand features from sales, stock, promotions and the festival calendar, how to find the leaks that make a model look brilliant on your laptop and fail in the stores, and how to test it on time the way it will be used.

By the end you can

Build lag, rolling, calendar, promotion and price features that follow an "as of tonight" rule
Find and remove four kinds of leakage: target, time, preprocessing and duplicate rows
Split data by time, with a gap equal to the forecast lead time
Run a rolling-origin backtest with scikit-learn's TimeSeriesSplit and a Pipeline
Treat a result that is too good as a bug until you have proved otherwise

The everyday picture

Picture a student preparing for a maths exam with old papers. A friend lends her a stack, and some of them have the answers pencilled faintly on the back. She doesn't mean to cheat. But when she is stuck, her eye drifts, and her practice scores climb to 95%. In the exam hall the papers are blank on the back. She scores 60%.

Two things went wrong. Her practice used information she would not have in the exam hall. And she practised on papers she had partly seen before, so her scores measured memory more than skill. A forecasting model fails in the same two ways. A leaked feature is the pencil on the back of the paper: a column that quietly contains the answer. A random split is practising on papers you have seen: the model is tested on days that sit between days it trained on. This lesson is about using only what you will have in the exam hall, and practising on papers you have never seen.

Features are what you know at 22:00 tonight

In Framing a problem as a model you framed Mangrove's out-of-stock problem. Every night at 22:00 each store's order goes to its distribution centre (DC), and the forecast behind it covers tomorrow to day 4. The label is units_sold for one store, one product and one day. In Mangrove's tables a store is a row in site (a site is a store or a DC) and a product is a row in article (one SKU). A feature is any input column the model uses to predict the label.

Every feature has to pass one test, the as-of rule: a feature for a prediction made at 22:00 tonight may only use data that exists, in its final form, at 22:00 tonight. For each column, ask two questions. When does this value become known? Is it final at that moment, or will it change later? The column name never tells you. The same stock column is safe for today and leaky for tomorrow.

Candidate feature	Becomes known	Safe tonight for tomorrow?
`lag_7`: units sold on the same weekday last week	6 days ago	Yes
`avg_last_28`: mean daily sales of the 28 days ending today	Tonight, after stores close	Yes
`days_to_festival` from the festival table	About a year ahead	Yes
`discount_pct` of a promotion planned for tomorrow	When the promotion is planned, weeks ahead	Yes
`on_hand_close` for today	Tonight, after stores close	Yes
`on_hand_close` for tomorrow	Tomorrow night	No: it is partly caused by tomorrow's sales
Promotion uplift measured after the promotion ran	After the promotion ends	No

The useful feature families for grocery demand are few, and each has a reason to exist:

Lags. A lag feature copies sales from an earlier day. Grocery demand has a strong weekly ripple (busier weekends), so lag_7, the same weekday last week, often carries more signal than yesterday.
Rolling means that end yesterday. A rolling mean smooths the noise of single days. The 28-day mean is Mangrove's baseline itself, so giving it to the model as a feature lets the model start from today's rule and learn corrections.
Calendar. Day of week, day of month (month-end salary days lift baskets) and days to the next festival. Festival dates are fixed a year ahead, so a feature can describe the future and still be safe. Forecasting demand goes deeper into festivals.
Promotions and price. The planned discount from the promo table and the planned shelf price. The plan is known weeks ahead; the measured effect is not.
Stock and store. Today's closing stock, the number of days in the last week the item ran out, and the store's size band from site. Out-of-stock days matter twice: they hold sales down (the censored demand from ds-01), and a flag tells the model that a low lag was a shortage, not low demand.

The lead time changes what is safe. For tomorrow (day 1), the day before the target day is today, which you know tonight. For day 3, the day before the target day is day 2, which hasn't happened yet. So a feature defined as "sales on the day before the target day" is safe for day 1 and leaky for days 2 to 4. Two fixes work. Build every feature relative to the forecast date, so all windows end today whatever the horizon. Or train one model per horizon and make every lag at least as long as the horizon: a day-3 model may use lag_7 but not lag_1 or lag_2.

Quick check

Tonight's run forecasts day 3, the day after tomorrow. Which feature is safe?

Four ways the future leaks in

Leakage is information in the training data that will not exist when the model runs. It always moves the offline score the same way: better. Then the real forecast is worse, often much worse, because the model leaned on a column that is missing or different at 22:00. Leaks come in four kinds.

Target leakage: a feature that is an effect of the label. on_hand_close for the target day is low when a lot sold, so the model reads the answer from the stock column. "App order lines cut for this item" on the target day is the same: lines are cut because demand ran past the stock on the shelf. Both columns are real data. They are recorded after the fact.
Time leakage: a window that reaches the target day. In pandas, rolling(7).mean() without shift(1) includes today's own sales, so one seventh of the feature is the label. In SQL, a frame that ends at CURRENT ROW does the same. A random train and validation split is time leakage across the whole dataset, covered in the next section.
Preprocessing leakage: a step that learns from rows it should not see. A scaler fitted on all data before the split learns the validation period's mean and spread, a small leak. Target encoding computed over 2024 to 2026, validation weeks included, puts each store's average sales during the test period straight into a feature, a large leak.
Duplicate entities: the same thing on both sides of the split. A join to promo where two promotions overlap turns one store-product-day into two identical rows. With a random split, one copy trains and the other validates, and the model "predicts" it perfectly. In Classification and churn the same customer in training and validation is the classic case.

How do you notice? The loudest signal is a score that is too good. Here is a quick sense check. A product in one store that sells about 20 units a day moves by around 3 or 4 units a day for reasons nobody can know: who happened to walk in. If sales behave like random arrivals, even a model that knows the true average perfectly has a WAPE of roughly 18% on that item. WAPE, from ds-01, is the sum of absolute errors divided by the sum of actual sales. Mangrove's baseline scores 34% and the best honest model about 24%. A first try at 6% means the model has seen the answer.

Three other signals point to the guilty column. One feature carries most of the model's skill when you measure importance (Explaining and choosing models shows how). A column name says close, final, total or after. And the canary test fails: copy the data, change one day's units_sold to 99,999, rebuild the features, and compare. Any feature of that day, or of an earlier day, that changed is reading the label.

Everything left of the as-of line is known at 22:00 and may become a feature. Plans made in advance (promotions, festival dates) may describe future days because they are fixed tonight. Tomorrow's closing stock sits right of the line; using it as a feature pulls part of the answer back into the inputs.

Quick check

A feature counts the app order lines cut for this product at this store on the target day. What kind of problem is it?

Split by time, with a gap

The model will be used on days after the days it learned from, so test it that way. Put the data in date order and cut it into three blocks: training (oldest), validation (to choose features and settings) and test (newest, touched once at the end, as Training and comparing models explains). Never shuffle rows before the cut.

Why not the usual K-fold cross-validation? With random folds, 14 July lands in validation while 13 July and 15 July sit in training. Their lags and rolling means are nearly the same numbers, and half of the festival week is in training, so the model fills in gaps instead of forecasting forward. When Tomás ran random 5-fold on his gradient boosting model, WAPE came out at about 17%. The same model on time-ordered folds scored 24%. The 7 points were never real.

A gap handles the edge between blocks. Tonight's order covers day 1 to day 4, so a training row dated 28 June has a target window of 29 June to 2 July, and that window reaches into a validation block starting 1 July. Drop the days in between. The rule: the gap equals the forecast lead time, here 4 days.

One split gives you one score from one season. A rolling-origin backtest repeats the split with the cut-off moving forward. Each fold trains on everything up to its cut-off (an expanding window), skips the gap, and validates on the next 8 weeks. The FPP3 textbook calls this evaluation on a rolling forecasting origin. You get three scores instead of one, and you see whether a model is steady across seasons or lucky in one of them.

A rolling-origin backtest (not to scale). Every fold starts training in January 2024 and ends 4 days before its 8-week validation block. Fold 3 validates on 1 July to 25 August 2026, the backtest window this course uses. Three steady scores are worth more than one good one.

scikit-learn's TimeSeriesSplit builds these folds: n_splits=3, test_size=56 for 8 weeks, and gap=4. It has one trap. It counts rows, not days. Tomás's backtest has 1,000 rows per day (50 products in 20 stores), so a row-based split cuts a day in half and gap=4 skips four rows, not four days. Split the sorted list of unique dates instead, then select the rows for each fold's dates. The Build it section does exactly that.

Quick check

Validation starts on 1 July and the forecast covers day 1 to day 4. Training runs to 30 June with no gap. Which training row leaks?

Fit preprocessing inside the folds

Any step that learns numbers from data is part of the model. A median imputer learns the median. A scaler learns a mean and a standard deviation. A one-hot encoder learns the list of categories. A target encoder learns the average label per category. Each of these must be fitted on the training fold only, then applied unchanged to the validation fold, exactly as tonight's run will apply them to data they have never seen.

scikit-learn's Pipeline makes this automatic. You put the imputer, the scaler, the encoder and the model in one object. fit learns every step from the rows you pass it, and predict applies them in order. A ColumnTransformer sends numeric columns to one chain and categorical columns to another. Then each fold refits the whole pipeline, and nothing about the validation weeks reaches training.

For target encoding, scikit-learn 1.3 and later has TargetEncoder. Its fit_transform uses cross fitting: each training row is encoded with averages computed without that row's own fold, so rows don't see their own label. Inside a Pipeline it is fitted per fold like everything else.

The leaks that slip past Pipelines happen in pandas before the split. Watch for these: filling missing sales with the column mean of the whole table, dropping "outliers" using limits computed from all three years, and choosing features by their correlation with the label on all data. Each one lets the validation weeks shape the training data. The rule is short: if a step is computed from many rows, it belongs inside the split.

Keep features honest in production

A backtest proves the features were honest in the past. Tonight's run has to compute them the same way, at the same moment, or the model meets inputs it never saw in training. That mismatch is called training-serving skew, and it looks like leakage from the other side.

Same code. One function builds features for the backtest and for tonight. When training features come from a SQL query in the warehouse and nightly features from a separate Python job, the two drift apart within months: a different window, a different way of filling missing days.
Same timing. Mangrove's tills keep selling when the internet drops and sync later. At 22:00, a few stores' sales for today may be incomplete. In the warehouse a month later the same day is complete. A model trained on complete "today" values gets partial ones at night and under-forecasts in exactly the stores with the worst connection. Fix it by keeping a load timestamp on every sales row and rebuilding training features from the rows that had arrived by 22:00, or by using only lags that are always complete by then.
Same shape. Check every night that tonight's features look like the training features: share of missing values, ranges, the mix of stores. A course on running models in production picks this up later.

At Mangrove Market

Tomás trains his first gradient boosting model on the backtest set, 50 fast-moving products in 20 stores, and gets a WAPE of 6% on the 8 validation weeks. The 28-day baseline scores 34%. Imran sees the number and asks whether they can start a store pilot next week.

Tomás says not yet. One product in one store moves by several units a day for reasons nobody can know, so 6% is too good to be real. He measures feature importance: one column, on_hand_close, carries most of the skill. It is the closing stock of the target day, known only the next night, and the model has learned that low stock at close means high sales. Without it, WAPE rises to about 15%, still suspicious. The canary test finds the second leak: avg_last_7 was built with rolling(7) and no shift(1), so it included the target day's own sales.

With both fixed, and today's closing stock kept as an honest feature, the model scores 24% on the 8 weeks and about the same on the two earlier folds. That is the number in the course's backtest table, and it still beats the baseline by 10 points. Tomás adds the canary test to the feature code, so the next leak fails a test instead of reaching a meeting. His note to Imran ends with the lesson: be suspicious of great results, and check when every feature becomes known before you celebrate.

Build it

Build honest features for the backtest set in SQL and in pandas, then run a rolling-origin backtest that refits preprocessing in every fold.

Compute features in SQL

Window functions do the lags and rolling means (see SQL you'll actually use). The frame 7 PRECEDING AND 1 PRECEDING is the 7 days ending yesterday, relative to each row's date. Compute windows first, then join the calendar and promotions, so a join can't duplicate rows under the windows.

SQL

WITH f AS (
  SELECT store_id, article_id, sale_date, units_sold,
         lag(units_sold, 7) OVER w AS lag_7,              -- same weekday last week
         avg(units_sold) OVER (w ROWS BETWEEN 7 PRECEDING
                                 AND 1 PRECEDING) AS avg_last_7,   -- ends the day before
         avg(units_sold) OVER (w ROWS BETWEEN 28 PRECEDING
                                 AND 1 PRECEDING) AS avg_last_28   -- the baseline
  FROM daily_sales
  WINDOW w AS (PARTITION BY store_id, article_id ORDER BY sale_date)
)
SELECT f.*,
       extract(isodow FROM f.sale_date) AS day_of_week,
       (SELECT min(festival_date) FROM festival
         WHERE festival_date >= f.sale_date) - f.sale_date AS days_to_festival,
       -- planned promotion; max() keeps one row even if two promos overlap
       coalesce((SELECT max(discount_pct) FROM promo p
                  WHERE p.article_id = f.article_id
                    AND f.sale_date BETWEEN p.start_date AND p.end_date), 0) AS discount_pct
FROM f;

These windows end the day before each row's date, which is right for a day-1 model. For a day-3 model, shift every window so it ends 3 days before the row's date.

Build the same features in pandas

Sort, group by store and article, and shift(1) before every rolling window. The festival feature uses merge_asof to find the next festival on or after each date.

Python

import pandas as pd

df = df.sort_values(["store_id", "article_id", "sale_date"])
g = df.groupby(["store_id", "article_id"])["units_sold"]

df["lag_7"] = g.shift(7)                                    # same weekday last week
df["avg_last_7"] = g.transform(lambda s: s.shift(1).rolling(7).mean())    # ends yesterday
df["avg_last_28"] = g.transform(lambda s: s.shift(1).rolling(28).mean())  # the baseline
df["day_of_week"] = df["sale_date"].dt.dayofweek

# days to the next festival; festival dates are known a year ahead
fest = festival[["festival_date"]].sort_values("festival_date")
df = pd.merge_asof(df.sort_values("sale_date"), fest,
                   left_on="sale_date", right_on="festival_date",
                   direction="forward")
df["days_to_festival"] = (df["festival_date"] - df["sale_date"]).dt.days

Put preprocessing inside a Pipeline

Every step that learns from data goes in one object, so each fold fits it on training rows only.

Python

import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import Ridge
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

NUM = ["lag_7", "avg_last_7", "avg_last_28", "days_to_festival", "discount_pct"]
CAT = ["day_of_week", "size_band"]
prep = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), NUM),
    ("cat", OneHotEncoder(handle_unknown="ignore"), CAT),
])
model = Pipeline([("prep", prep), ("ridge", Ridge(alpha=1.0))])

def wape(actual, forecast):
    return np.abs(actual - forecast).sum() / actual.sum()

Run a rolling-origin backtest on dates

Split the unique dates, not the rows. test_size=56 gives 8-week validation blocks and gap=4 drops the 4 days before each block.

Python

from sklearn.model_selection import TimeSeriesSplit

dates = pd.DatetimeIndex(df["sale_date"].unique()).sort_values()   # one entry per day
tscv = TimeSeriesSplit(n_splits=3, test_size=56, gap=4)

for k, (tr_i, va_i) in enumerate(tscv.split(np.arange(len(dates))), start=1):
    tr = df[df["sale_date"].isin(dates[tr_i])]
    va = df[df["sale_date"].isin(dates[va_i])]
    model.fit(tr[NUM + CAT], tr["units_sold"])     # imputer, scaler, encoder: training rows only
    pred = model.predict(va[NUM + CAT])
    print(f"fold {k}: train to {dates[tr_i][-1]:%Y-%m-%d}, "
          f"validate {dates[va_i][0]:%Y-%m-%d} to {dates[va_i][-1]:%Y-%m-%d}, "
          f"WAPE {wape(va['units_sold'].to_numpy(), pred):.1%}")

Output

fold 1: train to 2026-03-06, validate 2026-03-11 to 2026-05-05, WAPE 31.2%
fold 2: train to 2026-05-01, validate 2026-05-06 to 2026-06-30, WAPE 29.4%
fold 3: train to 2026-06-26, validate 2026-07-01 to 2026-08-25, WAPE 30.1%

Your numbers will differ. Fold 3 is the course's backtest window, where the linear model scores about 30%. Notice that the gap moves the end of training from 30 June to 26 June.

Run the leak checks before you believe a number
Compare the result with the baseline (34%) and with the noise floor (roughly 18% for an item selling 20 a day). Look at the most important feature and ask when it becomes known. Run the canary test: change one day's sales to 99,999, rebuild features, and confirm that no feature of that day or any earlier day changed. Keep the canary test in the feature code so it runs on every change.

Bring it to your team

Ask these when a model's results are presented.

At what time is the prediction made, and is every feature known and final at that time?
How was the data split, and how big was the gap compared with the forecast lead time?
How many folds, and how much did the score move between them?
Is every scaler, encoder and imputer fitted inside the folds?
Which feature matters most, and when does it become known?
Will tonight's run compute features with the same code, from data as complete as in training?

Red flags

A first model that beats the baseline by a huge margin, presented without a leak check.
"We used 5-fold cross-validation" for a forecast.
Features with close, final or total in the name for the target day.
Missing values filled, or outliers removed, before the split.
Training features built by one query and production features by a different job.

Practice

Fix three leaks in a feature query

The table daily_sales(store_id, article_id, sale_date, units_sold) has 10 days (1 to 10 July 2026) for three store-product series: store 1 article 101, store 1 article 205, and store 2 article 101. Return store_id, article_id, sale_date, units_sold, lag_7 (units sold 7 days earlier in the same series) and prev_3_avg (the mean of the 3 days before this day in the same series, excluding this day, rounded to 1 decimal). Return only rows from 8 July on, ordered by store, article and date. The starter has three mistakes: the windows ignore the series, the average includes today, and the filter runs before the windows.

SQL exerciseRuns in a real PostgreSQL inside your browser

CREATE TABLE daily_sales (store_id int, article_id int, sale_date date, units_sold int);
INSERT INTO daily_sales VALUES
(1, 101, '2026-07-01', 20), (1, 101, '2026-07-02', 22), (1, 101, '2026-07-03', 19),
(1, 101, '2026-07-04', 25), (1, 101, '2026-07-05', 30), (1, 101, '2026-07-06', 34),
(1, 101, '2026-07-07', 21), (1, 101, '2026-07-08', 23), (1, 101, '2026-07-09', 24),
(1, 101, '2026-07-10', 20),
(1, 205, '2026-07-01', 8), (1, 205, '2026-07-02', 9), (1, 205, '2026-07-03', 7),
(1, 205, '2026-07-04', 10), (1, 205, '2026-07-05', 12), (1, 205, '2026-07-06', 15),
(1, 205, '2026-07-07', 9), (1, 205, '2026-07-08', 8), (1, 205, '2026-07-09', 11),
(1, 205, '2026-07-10', 10),
(2, 101, '2026-07-01', 40), (2, 101, '2026-07-02', 38), (2, 101, '2026-07-03', 45),
(2, 101, '2026-07-04', 50), (2, 101, '2026-07-05', 61), (2, 101, '2026-07-06', 70),
(2, 101, '2026-07-07', 44), (2, 101, '2026-07-08', 41), (2, 101, '2026-07-09', 47),
(2, 101, '2026-07-10', 43);

-- lag_7 and the mean of the 3 days before each day, per store and article
SELECT store_id, article_id, sale_date, units_sold,
       lag(units_sold, 7) OVER (ORDER BY sale_date) AS lag_7,
       round(avg(units_sold) OVER (ORDER BY sale_date
             ROWS BETWEEN 3 PRECEDING AND CURRENT ROW)::numeric, 1) AS prev_3_avg
FROM daily_sales
WHERE sale_date >= '2026-07-08'
ORDER BY store_id, article_id, sale_date;

Ctrl + Enter runs

[
{"store_id": 1, "article_id": 101, "sale_date": "2026-07-08", "units_sold": 23, "lag_7": 20, "prev_3_avg": 28.3},
{"store_id": 1, "article_id": 101, "sale_date": "2026-07-09", "units_sold": 24, "lag_7": 22, "prev_3_avg": 26.0},
{"store_id": 1, "article_id": 101, "sale_date": "2026-07-10", "units_sold": 20, "lag_7": 19, "prev_3_avg": 22.7},
{"store_id": 1, "article_id": 205, "sale_date": "2026-07-08", "units_sold": 8, "lag_7": 8, "prev_3_avg": 12.0},
{"store_id": 1, "article_id": 205, "sale_date": "2026-07-09", "units_sold": 11, "lag_7": 9, "prev_3_avg": 10.7},
{"store_id": 1, "article_id": 205, "sale_date": "2026-07-10", "units_sold": 10, "lag_7": 7, "prev_3_avg": 9.3},
{"store_id": 2, "article_id": 101, "sale_date": "2026-07-08", "units_sold": 41, "lag_7": 40, "prev_3_avg": 58.3},
{"store_id": 2, "article_id": 101, "sale_date": "2026-07-09", "units_sold": 47, "lag_7": 38, "prev_3_avg": 51.7},
{"store_id": 2, "article_id": 101, "sale_date": "2026-07-10", "units_sold": 43, "lag_7": 45, "prev_3_avg": 44.0}
]

Show a solution

SQL

WITH f AS (
  SELECT store_id, article_id, sale_date, units_sold,
         lag(units_sold, 7) OVER (PARTITION BY store_id, article_id
                                  ORDER BY sale_date) AS lag_7,
         round(avg(units_sold) OVER (PARTITION BY store_id, article_id
                                     ORDER BY sale_date
                                     ROWS BETWEEN 3 PRECEDING AND 1 PRECEDING)::numeric, 1) AS prev_3_avg
  FROM daily_sales
)
SELECT *
FROM f
WHERE sale_date >= '2026-07-08'   -- filter after the windows have seen the full history
ORDER BY store_id, article_id, sale_date;

Pass the exercise to finish this task.

Safe or leaky: eight candidate features

Tomás's nightly run at 22:00 forecasts units sold per store and product for day 1 to day 4. For each candidate below, say whether it is safe or leaky, and why. If it is leaky, name the kind (target, time, preprocessing or duplicate) and say how to make it safe if you can.

Units sold on the same weekday last week, for a day-1 to day-4 forecast
on_hand_close on the target day
Units sold on the day before the target day, used for every horizon from 1 to 4
App order lines cut for this item at this store on the target day
discount_pct of a promotion planned three weeks ago for the target day
The sales uplift the promotion actually produced, measured after it ended
Each store-article's average daily sales, computed over all of 2024 to 2026 and added before the split
Days to the next festival, from the festival table

You mark 1, 5 and 8 safe, each with when the value becomes known
You mark 2, 4 and 6 as target leakage because each is an effect of the target day's sales
You see that 3 is safe for day 1 only, and fix it with a lag of at least the horizon or a window ending today
You mark 7 as preprocessing leakage and move the encoding inside the folds (or use a cross-fitted target encoder in a Pipeline)

Show a model answer

Output

1 safe     lag_7 is known 6 days ago; a 7-day lag covers horizons up to 7
2 leaky    target: closing stock on the target day is low because a lot sold
3 mixed    safe for day 1 (it is today); time leak for day 2 to 4.
           Fix: one model per horizon with lags >= horizon, or windows ending today
4 leaky    target: lines are cut because demand ran past stock that day
5 safe     the plan is fixed weeks ahead; check the plan is not edited afterwards
6 leaky    target: uplift is computed from the sales you are predicting
           Fix: use the planned discount, or uplift from earlier promotions
7 leaky    preprocessing: the average includes the validation weeks.
           Fix: compute it inside each training fold (TargetEncoder in a Pipeline)
8 safe     festival dates are published about a year ahead

The pattern: the column name never decides. Ask when each value becomes known and whether it is final at 22:00. Item 5 hides one more trap: if buyers edit promotion plans after the fact (for example, extending a promotion that sold well), the stored plan is partly an outcome. Keep the version of the plan that existed the night before.

Check yourself

A colleague reports WAPE 18% from random 5-fold cross-validation and 25% from a 3-fold rolling-origin backtest, for the same model on the same data. Which number should go in the review, and why?

An analyst fills missing lag_7 values with the median of the whole table, then splits by time. The linear model's validation WAPE barely changes when this is fixed. Should it still be fixed?

A model trained on the warehouse's complete history uses "today's sales" as a feature. In production, at 22:00, about 1 store in 20 hasn't synced today's till sales yet. What happens, and what is the best fix?

Flashcards

Add these 8 cards to my review when I finish the lessonOpen my review

Key takeaways

A feature is safe only if it is known and final at the moment of prediction. Ask when each value becomes known; the column name never decides.
Leaks come as target leakage, time leakage, preprocessing leakage and duplicate rows. All of them make offline scores better and real forecasts worse.
Split by time, leave a gap equal to the lead time, and use a rolling-origin backtest with several folds. Split dates, not rows.
Fit every imputer, scaler and encoder inside the folds with a Pipeline.
A result far better than the baseline and the noise floor is a bug until a leak check proves otherwise.

Sources and further reading

Common pitfalls and recommended practices, scikit-learn documentation (data leakage and Pipelines)
TimeSeriesSplit, scikit-learn documentation
Cross-validation: evaluating estimator performance, scikit-learn documentation
Time series cross-validation, Rob J Hyndman and George Athanasopoulos, Forecasting: Principles and Practice (3rd edition)

Episode transcriptThe six percent that was never real · about 12 minutes · tap any line to play from there

NadiaImran, Tomás sent you his first forecast results last month. Do you remember the number?
ImranSix percent error. I was ready to book a store pilot for the next week. The old rule sits at thirty-four percent.
NadiaAnd what happened to the six?
ImranIt became twenty-four. Still a good model. But I'd already told two people about the six.
NadiaToday is about where that six came from, and how to catch it before it reaches a meeting.
NadiaStart with a student preparing for a maths exam with old papers. Some of the papers she borrowed have the answers pencilled faintly on the back.
ImranAnd when she gets stuck, she peeks.
NadiaWithout really meaning to. Her practice scores climb to ninety-five percent. In the exam hall the backs are blank, and she scores sixty.
ImranSo two problems. She used information she won't have in the exam. And she practised on papers she'd partly seen.
NadiaExactly the two ways a forecasting model fools you. The pencil on the back is called leakage. Practising on seen papers is a random split. We'll take them one at a time.
ImranFirst, remind me what the model is predicting.
NadiaUnits sold, for one product in one store on one day. Every night at ten pm each store's order goes to its distribution centre, and the forecast behind it covers tomorrow to day four.
ImranAnd the features are the inputs. Last week's sales, the festival calendar, promotions.
NadiaRight. And every feature has to pass one test. We call it the as-of rule. A feature for a prediction made at ten pm tonight may only use data that exists, in final form, at ten pm tonight.
ImranThat sounds obvious.
NadiaIt is obvious until you look at a real table. Take closing stock. Is it a safe feature?
ImranI'd guess yes. We know our stock. It's a fact about the store, not about sales.
NadiaThat's the trap. Today's closing stock is safe, because the nightly job reads it after the stores close. Tomorrow's closing stock is leaky. It only exists tomorrow night, and it's low exactly when a lot sold.
ImranSo the same column is safe for one day and leaky for the next.
NadiaYes. The column name never decides. For every feature you ask two questions. When does this value become known? And is it final at that moment?
NadiaIf you can see your screen, the first diagram shows this as a timeline. A vertical line marks ten pm tonight. Everything to its left is known. The target window, day one to day four, sits to the right.
ImranAnd there's a red box on the right with an arrow pointing back left. Closing stock for tomorrow, leaking into the features.
NadiaAnd notice one long green box that crosses the line. Planned promotions and festival dates. They describe future days, but they're safe, because the plan is fixed tonight.
ImranSo a feature can talk about the future, as long as it's known now.
NadiaWell put. A promotion's planned discount is fine. The uplift you measured after the promotion ran is not.
ImranWhich features actually matter for groceries?
NadiaLags first. Sales on the same weekday last week, called lag seven. Grocery demand has a weekly ripple, busier weekends, so last week's Saturday often says more about this Saturday than yesterday does.
NadiaThen rolling means that end yesterday. The twenty-eight day average is our baseline itself, so giving it to the model lets the model start from today's rule and learn corrections.
ImranPlus day of week, salary days at month end, days to the next festival, the planned discount, and store size.
NadiaAnd one about stock outs. How many days last week the item ran out. That tells the model a low lag was a shortage, not low demand.
ImranYou mentioned the lead time matters. Why?
NadiaTake a feature called sales on the day before the target day. For a tomorrow forecast, that's today. Known. For day three, it's day two, which hasn't happened yet.
ImranSo it's safe for day one and leaky for days two to four. That's sneaky.
NadiaThe fix is to end every window at today, whatever the horizon. Or train one model per horizon, with lags at least as long as the horizon.
ImranSo back to Tomás and the six percent. What exactly leaked?
NadiaTwo things. First, closing stock for the target day was in the feature list. When he measured which feature mattered most, that one column carried most of the skill. The model had learned that low stock at close means high sales.
ImranAnd taking it out fixed it?
NadiaIt went to about fifteen percent. Still suspicious. Then he found the second leak. His seven day rolling average was built without a shift, so it included the target day's own sales.
ImranOne seventh of the answer, sitting inside a feature.
NadiaWith both fixed, and today's closing stock kept as an honest feature, the model scored twenty-four percent. That's the number in the backtest table, and it still beats the baseline by ten points.
ImranHow was he supposed to know six was too good? It's a new model. Maybe it was brilliant.
NadiaHere's a quick sense check. A product that sells about twenty a day in one store moves by three or four units a day for reasons nobody can know. Who happened to walk in.
NadiaIf sales behave like random arrivals, even a perfect model of the average has an error of roughly eighteen percent on that item. Six percent is below the noise. The model has seen the answer.
ImranSo there's a floor. You can't beat randomness.
NadiaRight. And there's a test he now runs every time. The canary test. Copy the data, change one day's sales to ninety-nine thousand, rebuild the features, and compare. If any feature for that day or an earlier day changed, it's reading the answer.
ImranI like that. It turns a leak into a failing test instead of an embarrassing meeting.
NadiaThere are two more kinds of leak. Preprocessing leakage, when a step that learns from data is fitted on all of it before the split. Say, each store's average sales over three years, validation weeks included.
ImranAnd the fourth?
NadiaDuplicates. A join to the promotions table where two promotions overlap turns one store day into two identical rows. With a random split, one copy trains and the other is tested. Perfect prediction, for free.
ImranWhich brings us to the second problem. The papers she'd already seen.
NadiaThe usual method in machine learning courses is K fold cross validation. Shuffle the rows into groups, hold one group out at a time. For a forecast, that lies.
ImranWhy? It's the standard method.
NadiaBecause the fourteenth of July lands in validation while the thirteenth and fifteenth sit in training. Their lags and averages are almost the same numbers. The model fills the gap instead of forecasting forward.
ImranHow big is the difference?
NadiaTomás tried it. Random five fold gave about seventeen percent. The same model on time ordered folds gave twenty-four. Seven points that were never real.
ImranSo split by date. Oldest for training, then validation, then a test period at the end.
NadiaWith a gap in between. The order covers four days. So a training row dated the twenty-eighth of June has a target running to the second of July, inside validation. Drop the days in between. The gap equals the lead time, four days.
NadiaThe second diagram shows a rolling origin backtest. Three rows, one per fold. Each training bar starts in January twenty twenty-four and gets longer. Then a small gap, then an eight week validation block that moves forward.
ImranSo three scores instead of one. And you can see whether the model is steady across seasons or got lucky once.
NadiaExactly. The last fold validates on the first of July to the twenty-fifth of August, the backtest window the whole course uses.
ImranScikit-learn has a tool for this, doesn't it? Time series split.
NadiaIt does, with a gap setting. But it counts rows, not days. Tomás has a thousand rows per day, fifty products in twenty stores. A gap of four skips four rows. So you split the list of dates, then pick the rows for each fold's dates.
ImranWalk me through the build, then.
NadiaStep one, the features in SQL. Window functions per store and product. A lag of seven, and averages over the seven and twenty-eight rows before each day, ending the day before. Compute the windows first, then join the festival and promotion tables.
NadiaStep two, the same features in pandas. Sort, group by store and product, and shift by one before every rolling average. The festival feature finds the next festival on or after each date.
ImranStep three?
NadiaA scikit-learn Pipeline. The imputer, the scaler, the encoder for categories, and the model, all in one object. So each fold fits every one of them on training rows only.
ImranAnd the last steps are the backtest over the three folds, and the leak checks before anyone believes a number.
NadiaCompare with the baseline and the noise floor, ask when the top feature becomes known, and keep the canary test in the code.
ImranOne worry from my side. Does a model that passes all this still fail in the stores?
NadiaIt can, from the other direction. Our tills keep selling when the internet drops and sync later. At ten pm a few stores' sales for today may be incomplete. A month later in the warehouse, the same day is complete.
ImranSo the model trained on complete days, and at night it sees partial ones.
NadiaAnd under-forecasts in exactly the stores with the worst connection. That's called training serving skew. The fix is to build training features from the rows that had arrived by ten pm, and to use one piece of code for training and for tonight's run.
NadiaTime for a quick one. A feature counts the app order lines cut for this product on the target day. Safe or leaky? Take a second.
ImranLeaky. Lines get cut because the shelf ran empty, which happens on high demand days. It's an effect of the answer. But the same count for today would be fine for tomorrow.
NadiaExactly right. Timing, not the table it lives in.
ImranMy summary. One, a feature is safe only if it's known and final at ten pm tonight, and the column name never tells you. Two, split by date with a gap the size of the lead time, and use several folds. Three, a result far better than the baseline is a bug until you've proved it isn't.
NadiaGood. I'd add one. Put every step that learns from data inside the Pipeline, even the boring median fill. The habit protects you on the day it matters.
ImranSo next time we finally train the models properly.
NadiaNext time. Linear models, trees and gradient boosting, all on the same honest backtest, and how to pick one with a reason.

Finished reading?Mark it complete to fill in its stop on your course line.

Use it at work

Data-quality checks

Duplicates, missing values, orphans and impossible values, each as one row with a count.

Open in the SQL Lab

Next stopTraining and comparing models

Was this lesson helpful?

Questions and discussion

Loading the discussion…