Data science and ML cheat sheet

9 minReference

Data science and ML cheat sheet

Machine learning from framing to production on a few screens: how to frame the problem, split without leaking, pick a model and a metric, forecast, explain a prediction, and keep a model healthy once it is live. Code is scikit-learn. Keep it open while you plan or review a model.

Frame it before you fit it

Most failed models answer the wrong question. Write these down first (Framing).

Decision: what changes when the model speaks? "Send a win-back voucher to customers likely to stop ordering in the next 30 days."
Unit and moment: one prediction per customer, made every Monday with data up to Sunday night.
Target: exactly defined. "No app order in the 30 days after the prediction date."
Baseline: the rule you must beat, such as "no order in the last 21 days".
Cost of errors: a wasted voucher costs $2; a lost customer costs about $40 of margin a year.
Success metric: a business result (customers kept per dollar), not only AUC.

Problem type: "I want to..."

Start with the simplest model that could work, and keep it as the baseline the complex one must beat (Models).

I want to...	Problem	Start with	Lesson
Predict yes or no (will churn, is fraud)	Classification	Logistic regression, then gradient boosted trees	Models, Churn
Predict a number (basket size, delivery minutes)	Regression	Linear regression, then gradient boosted trees	Models
Predict demand per store per day	Forecasting	Seasonal naive (same day last week), then a model with calendar features	Forecasting
Group similar stores or customers	Clustering	k-means on scaled features	Models
Order items by relevance	Ranking	Popularity per segment, then a learned ranker	Models
Flag unusual till totals	Anomaly detection	Robust z-score per store, then isolation forest	Models
Classify or extract from free text	NLP	An LLM prompt with evals, or TF-IDF plus logistic regression	LLM APIs

Splits and leakage

Leakage is when the model sees, during training, information it will not have at prediction time. It gives great test scores and useless predictions (Leakage).

Situation	Split this way	Lesson
Rows are independent	`train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)`	Leakage
Anything with time	Train on the past, test on a later period; `TimeSeriesSplit` for cross-validation	Leakage, Forecasting
Many rows per customer or store	Keep each group on one side: `GroupKFold`	Leakage
Tuning hyperparameters	Train, validation and a test set you touch once at the end	Models

Leakage you will meet

A feature computed after the prediction moment: "refund issued" when predicting a bad delivery.
Scaling or imputing on all the data before the split. Fit those steps on training data only, inside a Pipeline.
Target encoding computed on the full dataset, which lets each row see its own label.
Duplicates of one order on both sides of the split.
A feature that is a proxy for the label, such as "account closed date" for churn.

Rule: for every feature, ask "would I know this value at 23:59 on Sunday, when the prediction is made?" (Leakage).

A pipeline that cannot leak

Put every preprocessing step inside the pipeline, so cross-validation refits it on each training fold (scikit-learn 1.2 and later; Models, Pipelines).

Python

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import TimeSeriesSplit, cross_val_score

num = ["orders_90d", "days_since_last", "avg_basket_cents"]
cat = ["city", "home_store"]

prep = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                      ("sc", StandardScaler())]), num),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), cat),
])
model = Pipeline([("prep", prep), ("clf", HistGradientBoostingClassifier())])

scores = cross_val_score(model, X, y, cv=TimeSeriesSplit(n_splits=5),
                         scoring="average_precision")
print(scores.mean(), scores.std())

Which metric

Pick the metric from the cost of each kind of mistake, then report it on the test set beside the baseline (Models, Churn).

Metric	Means	Use when	scikit-learn
Precision	Of those flagged, the share that were right	False alarms are costly (vouchers, fraud holds)	`precision_score`
Recall	Of the real cases, the share you caught	Misses are costly (food safety, fraud losses)	`recall_score`
F1	Balance of precision and recall	You need one number and both matter	`f1_score`
ROC AUC	Chance a random positive ranks above a random negative	Comparing rankers, classes not too rare	`roc_auc_score`
PR AUC (average precision)	Precision across all recall levels	Rare positives, such as 2% fraud	`average_precision_score`
Accuracy	Share of all predictions that were right	Balanced classes only; 98% is easy with 2% fraud	`accuracy_score`
MAE	Average absolute error, in real units	Easy to explain: "off by 4 minutes"	`mean_absolute_error`
RMSE	Punishes big errors more	Large misses hurt more than small ones	`root_mean_squared_error` (1.4+)
MAPE	Average error as a percentage	Comparing stores of different sizes; breaks near zero	`mean_absolute_percentage_error`
Calibration	Do 30% predictions come true 30% of the time?	Probabilities feed a cost decision	`CalibrationDisplay`

If you see X, reach for Y

The symptom usually tells you which side to fix: the data, the model or the setup (Models, Leakage).

You see	Likely cause	Try
Test score far above anything you expected	Leakage	Check feature timestamps; split by time or group
Train score high, validation low	Overfitting	Simpler model, regularisation, fewer features, more data
Both scores low and close	Underfitting or weak features	Better features, a more flexible model
Great in the notebook, poor in production	Training and serving compute features differently	One feature code path for both
Scores swing a lot between folds	Small or uneven data	More folds, report the spread, simpler model
Rare class never predicted	Class imbalance with a default 0.5 threshold	Choose the threshold from costs; `class_weight="balanced"`
One feature dominates importance	A proxy for the label, or an ID	Remove it and see what is left
Accuracy slowly drops month by month	Data or concept drift	Monitor inputs, retrain on a schedule

Forecasting

Forecast at the grain the decision needs: Mangrove orders stock per store per article per day, so a city-level weekly forecast does not help the buyer (Forecasting).

Always beat a naive baseline first: same day last week for daily retail data.
Features that usually matter: day of week, holidays and festival dates, paydays at month end, promotions, weather, lagged sales (last week, last year).
Backtest: train up to week N, forecast week N+1, move forward, repeat. Report error per week, not one average.
Give a range, not a single number. Stock decisions need the 90th percentile, not the mean.
Out-of-stock days hide true demand: sales of zero do not mean demand of zero. Flag or remove them.
Lag features must use only data available on the forecast day.

Explaining a model

Explain for the person who acts on it: a buyer needs "why this store, this week", a reviewer needs global importance and fairness checks (Explaining).

Question	Method	Lesson
Which features matter overall?	Permutation importance on the test set: `sklearn.inspection.permutation_importance`	Explaining
How does the prediction move with one feature?	Partial dependence: `PartialDependenceDisplay`	Explaining
Why this one prediction?	SHAP values (the `shap` package)	Explaining
Simple enough to read whole?	Logistic regression coefficients, or a shallow tree	Explaining
Does it treat groups fairly?	Compare error rates and outcomes per group	Responsible ML

ML in production

A model is a product with an owner, a version, inputs that drift and users who depend on it (Pipelines, Serving).

Concern	Do this	Lesson
Reproducible training	Pin code, data snapshot, library versions and random seed; train from a pipeline, not a notebook	Pipelines
Same features in training and serving	One feature definition, computed by shared code or a feature store, with point-in-time joins	Features
Knowing what is live	A model registry: version, metrics, training data, approver, stage	Registry
Batch or real time	Batch scoring overnight if the decision is daily; an API only when an answer is needed in the request	Serving
Safe rollout	Shadow mode first, then a canary on a share of traffic, with a rollback to the previous version	Serving
Input drift	Compare live feature distributions with training (PSI, or simple percentiles) and alert	Monitoring
Performance drift	Join predictions to outcomes when labels arrive; track the metric weekly	Monitoring
Silent failures	Alert on missing features, prediction volume and the share of default values	Monitoring
Harm and accountability	A model card: purpose, data, limits, fairness checks, owner, how to appeal a decision	Responsible ML

Numbers and rules of thumb

Test set: 10% to 20% of the data, or the most recent weeks for anything with time.
Population stability index (PSI): below 0.1 is usually stable, above 0.25 is a large shift. Treat these as conventions, not laws.
A model must beat the baseline by enough to pay for running it: monitoring, retraining and on-call.
Small test sets give noisy metrics: with 600 customers flagged in the test set, a precision of 40% has an interval of roughly ±4 points.
Retraining cadence follows how fast the world changes: weekly for demand, monthly or quarterly for churn.

Glossary

Plain definitions, with the lesson that explains each one.

Baseline: A simple rule or model that a new model must beat to be worth it. Framing
Backtesting: Testing a forecast by replaying the past: train up to a date, forecast after it, repeat. Forecasting
Calibration: How well predicted probabilities match how often things really happen. Churn
Class imbalance: When one outcome is much rarer than the other, such as fraud. Churn
Concept drift: The link between inputs and outcome changes, so an old model gets less accurate. Monitoring
Cross-validation: Training and testing several times on different splits to get a steadier score. Models
Data drift: Live inputs no longer look like the training data. Monitoring
Feature: An input column the model learns from. Features
Feature store: A shared place that computes and serves features the same way for training and serving. Features
Gradient boosting: Many small decision trees, each correcting the errors of the ones before; strong on tables. Models
Hyperparameter: A setting chosen before training, such as tree depth, tuned on validation data. Models
Label (target): The outcome the model learns to predict. Framing
Leakage: Training information that will not be available at prediction time. Leakage
Model card: A short document of a model's purpose, data, performance, limits and owner. Responsible ML
Model registry: A catalogue of trained model versions with their metrics and approval stage. Registry
Overfitting: Learning the training data's noise, so the model does worse on new data. Models
Pipeline: Preprocessing and model chained as one object, so the same steps run every time. Pipelines
Point-in-time join: Joining each training row to feature values as they were at that row's time, never later. Features
Precision and recall: Share of flagged cases that were right, and share of real cases that were caught. Churn
Regularisation: A penalty on model complexity that reduces overfitting. Models
Shadow mode: Running a new model on live traffic without acting on its output, to compare it safely. Serving
SHAP: A method that splits one prediction into contributions from each feature. Explaining
Threshold: The score above which you act; choose it from the cost of each error. Churn
Training-serving skew: A difference between how features are built in training and in production. Features

Courses on Kolom

Data Science and Machine Learning: framing, leakage, models, forecasting, churn and explaining.
Machine Learning in Production: pipelines, features, registry, serving, monitoring and responsible ML.
Statistics for Data Work: the inference underneath model evaluation.
LLM Apps: Prompts, RAG and Evals: when a language model is the better tool.