Serving: batch and online
A forecast helps nobody until its numbers reach the person or system that makes the decision, in time. This lesson shows you how to score about 10 million predictions inside a 90-minute night window, publish them so nobody ever reads half a run, and answer the Store app's ordering screen in milliseconds from those results, keeping live model calls for the few cases that need them.
By the end you can
- Choose batch scoring, an online lookup or real-time inference from when the decision is made and how fresh its inputs must be
- Run nightly batch scoring for 2.5 million store-product pairs inside the window with partitions, parallel workers and chunks
- Publish a run atomically with a
run_idand a complete flag, and fall back safely when a run fails - Build an online endpoint that meets a p95 budget of 150 ms by reading precomputed results, with timeouts and a fallback answer
- Package batch and online in one Docker image, pin the model version for each run, and keep the cost in proportion
The everyday picture
Think of a busy restaurant kitchen. In the afternoon, before anyone arrives, the cooks do the prep: they chop onions, make the sauces, portion the rice and cook the stock. At 8 pm, when the dining room is full, an order for a curry takes four minutes, because the cook only assembles and heats things that are already made. Nobody chops onions while a guest waits.
A few dishes can't be prepped, such as a steak cooked the way one guest likes it. Those are cooked to order, and the kitchen keeps that list short, because cooking to order is slow and needs the best cook at the stove. And when the afternoon prep goes wrong, the head chef doesn't serve half-made sauce. She uses yesterday's sauce that still keeps, tells the waiters, and takes one dish off the menu. Model serving works the same way: do the heavy work before service (batch), answer each request by assembling prepared results (online lookup), cook to order only what must be fresh (real-time inference), and have a safe plan for when the prep fails.
When the decision happens decides the serving style
Before you pick any tool, ask one question: when does someone act on the prediction, and how fresh must the inputs be at that moment? The answer puts each use case into one of four styles.
- Batch scoring. Mangrove's nightly order is the textbook case. The distribution centres order at 02:00 for every store-product pair, using sales up to yesterday. Every pair is known in advance and the inputs don't change between 23:00 and 02:00, so all about 10 million predictions (2.5 million pairs × 4 days) can be made in one run and stored.
- Online lookup. A request at any time reads a stored prediction and adjusts it with light arithmetic. When Ama opens the ordering screen at 08:00, the forecast hasn't changed since the night. What changes is her stock count and whether there is a local event. Those can be combined with the stored numbers in microseconds.
- Real-time inference. The model runs per request because the inputs can't be listed in advance. A rider's arrival time (ETA) on the Mangrove app depends on where the rider is right now, the traffic now and the stops left. A card fraud check depends on a payment made a second ago. You can't precompute every GPS point.
- Streaming. Features themselves must be seconds fresh, such as the number of payments from this card in the last 10 minutes. It needs a stream platform, a second code path for features and people who can run both. It is the most expensive style to build and to keep correct (the training/serving skew risks from Data and feature pipelines multiply).
Three tests settle most cases. Can you list every input in advance? How old may the inputs be when the decision is made? How many requests come, and how fast must each be answered? If the answers are "yes", "hours" and "a modest number", use batch, plus a lookup for screens. Most business decisions land there: orders, staffing rosters, weekly coupons, price reviews. Start with batch, and add an online part only where one decision needs fresher inputs than last night's.
Quick check
Buyers want a price-sensitivity estimate for each of 25,000 products, refreshed weekly and viewed in a back-office dashboard. Which style fits?
All 25,000 products are known in advance and a weekly refresh is fresh enough, so a batch job is the cheapest correct answer. Calling the model on each page view adds latency and a service to run for no gain, and streaming updates answer a question nobody asked.