Evals that never stop
The eval suite you built before launch tells you whether a change is safe to merge. After launch, the questions change, the judge can drift, and the provider can change the model under you. This lesson turns evals into a loop that runs every day: a golden set that grows from production, a judge you have checked against people, a sample of real traffic scored daily, user signals as pointers, and a gate that every model, prompt and retrieval change must pass, category by category.
By the end you can
- Keep a golden set healthy after launch: categories, must-pass cases, weekly cases from production failures, and a held-out slice
- Write a judge rubric per category, know the judge's biases, and calibrate it against about 200 human labels
- Sample production for online evals, size the sample so a drop is visible, and combine judge scores with user signals
- Catch provider model updates with pinned versions, a pre-switch suite, a shadow run and a 5% canary
- Gate model, prompt and retrieval changes per category in CI, with no drop allowed in must-pass categories
The everyday picture
Think of a busy restaurant that passed its health inspection on opening day. Passing once didn't make it safe forever. The head chef still tastes every sauce before service, from a fixed list of dishes that must be perfect. A server writes down every complaint, and once a week the chef cooks the dishes that drew complaints and adds them to the tasting list. When a new supplier offers cheaper cream, the chef doesn't switch on a Friday night: she cooks the menu with the new cream on a quiet afternoon, then serves it at two tables, then everywhere, and keeps the old supplier's number.
One more habit matters. The chef knows her own palate drifts when she is tired, so once a month she tastes blind with two other cooks and checks they agree. That is the whole lesson: a fixed tasting list that grows from complaints, a taster you have checked, a sample of real plates every day, and a careful switch for every new ingredient.
The golden set after launch
Before launch you built a golden set of 120 cases, stored as data with frozen fixtures, run three times each, with a gate per category in CI (Evals: knowing if it got better, with the product view in Evals: measuring AI quality). It measured the questions the team could imagine in June. Production keeps asking new ones. New products arrive every week, a festival brings questions about gift boxes and opening hours, a heatwave brings questions about cold-chain delivery, and customers find phrasings nobody wrote down. This is drift, and a golden set that never changes measures a shrinking share of real traffic.
Three months later Mangrove's set has about 300 cases, re-cut into six categories that match how traffic and risk split:
| Category | Cases | Gate |
|---|---|---|
| Allergens | 40 | Must-pass: no drop, at least 97% |
| Must-refuse (medical advice, other customers' data, injections) | 30 | Must-pass: no drop, at least 90% |
| Substitutes | 50 | At most a 2-point drop |
| Delivery | 60 | At most a 2-point drop |
| Order status | 50 | At most a 2-point drop |
| Product facts | 70 | At most a 2-point drop |
A must-pass category allows no drop at all. At launch the allergen gate was 100% of runs on 25 easy cases. As the set filled with the hardest real questions (every new case is one the system once got wrong), no model reached 100% on all of them. The team changed the rule on purpose: no drop against the pinned version, a floor of 97%, a person reads every failing allergen case, and the allergen output check blocks unsupported claims at runtime. Write a rule change like this down with its reason; a gate that quietly loosens is worse than none.
Keep the set alive with a weekly routine of about an hour:
- Add. Read the week's thumbs-down answers and judge failures (from the online sample below). Every confirmed failure becomes a case, with the catalogue snapshot and tool results from that moment saved as fixtures. Aim for 5 to 15 new cases a week.
- Balance. Watch the share of each category in traffic. If delivery questions double in the monsoon season, the delivery category should grow too.
- Retire. A case about a product that left the catalogue tests nothing. Move it to an archive file with the reason, never delete it silently.
- Hold out. Keep about 15% of cases that nobody looks at while tuning prompts. If the tuned set improves and the held-out slice doesn't, you tuned to the test.
Quick check
The suite has passed every release for two months, yet thumbs-down on substitute answers rose from 6% to 10%. The substitute category is 50 cases from June. What is the best first move?
A suite that passes while users complain is measuring old questions. New cases from real failures let the suite see what users see; then the gate means something again. A stricter threshold on stale cases still can't see the new failures, and changing the model before you know the failure is a guess.
A judge you can trust
Most of the 300 cases need reading, so a judge model grades them, and in production there is no answer key at all. An LLM judge is only as good as its rubric and only as trusted as its calibration.
A rubric per category. One generic "is this a good answer?" prompt grades everything a little and nothing well. Allergen answers need criteria such as GROUNDED (every allergen fact appears in the retrieved catalogue text for the product asked about), RIGHT_PRODUCT, and the NO_SAFETY_PROMISE criterion from launch. Delivery answers need a slot and fee check against the tool result. Each criterion is pass or fail with one sentence of reason, never a 1 to 10 score.
Absolute or pairwise. Absolute grading asks "does this answer meet the rubric?" and works for the golden set and for daily online scores. Pairwise grading asks "which of these two answers is better?" It is more sensitive to small differences, so it suits comparing an old and a new model version on the same traffic.
Known biases. The MT-Bench study by Zheng and colleagues found that strong judge models agreed with human experts about as often as the experts agreed with each other, and also found three biases you must design around:
- Position bias: in pairwise grading, judges tend to prefer the answer in a particular position. Run every pair twice with the order swapped, and count a win only when both orders agree; otherwise call it a tie.
- Verbosity bias: judges tend to prefer longer answers. Say in the rubric that length is not a criterion, and grade facts, not style.
- Self-preference: a model may favour answers written by itself or its own family. Where you can, use a judge model that is not the model being judged, and hold the judge fixed when you compare candidates.
Calibration. Two people label about 200 real answers pass or fail against the rubric, and settle disagreements. Then the judge grades the same 200. Mangrove's judge agrees on 184 of 200, about 92%. Agreement alone can hide the failure that matters, so look at the failures. Humans marked 40 answers as failing; the judge caught 34 of them (85%) and wrongly failed 10 of the 160 good ones. A judge that misses one failure in seven is good enough to watch trends and choose what people read; it is not good enough to be the only guard on allergen answers, which is why the runtime output check exists.
Pin the judge model and version the rubric, exactly as you pin the assistant. Re-run the calibration set when either changes, and once a month anyway. If the provider moves the judge, every trend line in your dashboard moves with it, and you would read the judge's drift as the assistant's.
Quick check
In a pairwise comparison, the new model version wins 64% of pairs. You notice its answer was always shown first, and its answers are about 40% longer. What should you do before trusting the result?
A fixed order and longer answers are the two biases that inflate a win. Swapping order removes position bias, and a rubric that grades facts instead of length reduces verbosity bias. A 1 to 10 score keeps both biases and adds noise, and editing one side's answers changes what you are testing.
Online evals and user signals
The golden set tests questions you chose. Online evals test the questions customers actually asked yesterday. Each day, take a 2% sample of the assistant's conversations, about 300, from the redacted content capture you set up in Observability for LLM systems. Classify each turn into the six categories, run the judge with that category's rubric, and store one row per verdict.
There is no reference answer online, so the online rubric is reference-free: is every claim supported by the retrieved text and the tool results, is it about the right product, did it refuse what it must refuse. The trace gives the judge what it needs: the question, the retrieved chunks by id, the tool results and the answer.
Size the sample per category, because that is where the decisions are. At a 90% pass rate, the rough 95% range is about ±2 × √(0.9 × 0.1 ÷ n). For 300 conversations a day that is about ±3.5 points, fine for the overall line. Allergen questions are about 8% of traffic, so a plain 2% sample holds about 24 a day, and the range is about ±12 points: a nine-point drop would be invisible for days. So stratify: judge every allergen conversation up to 150 a day, and pool small categories over 7 days for the trend. Alert on a 7-day pass rate that falls more than 3 points below the previous four weeks, and on 3 or more allergen failures in one day, which go straight to a person.
The judge's verdicts are themselves a model's output. Guard against judge drift: each week a person grades 30 conversations the judge passed and 30 it failed. If people disagree with the judge more than in calibration, fix the rubric before you trust the trend.
User signals add a second, cheaper view, joined to traces by trace id:
- Thumbs up and down. Only about 3% of conversations get a rating, and angry people vote more often than satisfied ones. A late rider earns a thumbs down on a perfect answer about delivery slots.
- Re-ask within 10 minutes. The customer asks nearly the same question again, which usually means the answer didn't help.
- Hand-off to a human agent. The customer taps "talk to someone" or the assistant hands over.
- Abandonment. The customer leaves within seconds of the answer, without reading, ordering or replying.
Signals are biased and noisy, so use them as pointers, not scores. They choose which conversations a person or the judge reads first, and a change in their trend per category is a reason to look. They never prove quality on their own.
Quick check
The online allergen pass rate reads 83% today and 96% yesterday, from a plain 2% sample. What do you conclude?
On two dozen conversations, 3 extra failures move the rate by about 12 points. Reading the failures and judging every allergen conversation tells you quickly whether it is real. Rolling back on noise trains the team to ignore alerts, and blaming the judge without checking is the same guess in the other direction.
When the provider changes the model
Most providers offer two kinds of model names: an alias that moves to the newest version (such as "latest") and exact, dated versions that don't change. They also announce when old versions will be retired. In production, never call an alias. Pin an exact version for the assistant and for the judge, record both on every span, and treat a new version like any other release.
When a new version appears, run four steps, each one a gate for the next:
- Pre-switch suite. Run the full golden set on the new version, three runs per case, and compare per category with the pinned version. This takes about two hours and costs tens of dollars.
- Shadow run. Send one day of real traffic (a sample of a few thousand conversations is enough) to the new version as well, without showing its answers. The judge compares old and new pairwise, both orders. Only read-only tools run in the shadow; anything that writes or sends uses recorded results, so the shadow can't create a second order or a second task.
- Canary. Route 5% of conversations to the new version for three to seven days. Watch the online judge per category, user signals and cost per conversation for the canary against the rest.
- Switch, with a way back. Move all traffic, keep the old version configured as the rollback, and remember that you can only roll back while the provider still serves it. Put the retirement date in the team calendar.
Why per category? Because an overall number hides exactly the regression that matters. When "latest" moved on 12 September, the overall score fell four points. Allergens fell nine.
Read the September version's row by row and the gate's answer is clear: allergens and must-refuse dropped (no drop allowed), product facts fell five points and substitutes three (more than the 2-point tolerance), while delivery and order status stayed within tolerance. When a new version fails, report the failing cases to the provider with examples, stay pinned, and test again when they release the next version.
One gate for every change
A model update is one kind of change. Prompt edits, retrieval changes (chunk size, the number of chunks, a new index, a reranker), tool definitions, router rules and the judge's rubric all change answers too. Every one of them goes through the same gate, triggered in CI by the files that hold them, as you set up in Evals: knowing if it got better. What changes after launch is what the gate compares against: the baseline is the pinned production configuration's latest run on the current golden set, rerun whenever the set changes, so you never compare a candidate on 300 cases with a baseline from 280.
The gate's rules, per category:
- Must-pass categories (allergens, must-refuse): the candidate may not score below the baseline, and must reach the floor (97% and 90%).
- Other categories: the candidate may drop at most 2 points against the baseline.
- Missing data fails. A category in the baseline with no results in the candidate run fails, so renaming or deleting cases can't open the gate.
Two points is strict. In a 50-case category with three runs each, one case that fails all three runs costs exactly 2 points, so the tolerance allows about one fully broken case. That is deliberate: the gate is a tripwire, and the paired flip list from llm-05 tells a person whether a 2.7-point drop is a few flaky runs or one real break. A person may override a failed tolerance category with a written reason. Nobody overrides a must-pass category; you fix the change.
Finally, the gate needs owners. The AI engineer on duty reads the online dashboard every morning, the weekly case review has a named owner (at Mangrove, Tomás with a support lead), and the gate's rules live in the repository, so loosening them is a reviewed pull request.
Build it
Write the eval config, a judge rubric, the CI gate script and the online dashboard query for the app assistant.
Put categories, gates and the pinned judge in one config
The config lives in the repository, so any change to a threshold is a reviewed pull request.
YAML# evals/assistant.yaml suite: app-assistant cases: evals/cases/*.jsonl # about 300 cases, one per line runs_per_case: 3 assistant_model: large-2026-06 # pinned; never an alias judge: model: judge-2026-06 # pinned too rubric_version: judge-rubric-v7 calibration: set: evals/calibration-200.jsonl # 200 answers labelled by two people min_agreement: 0.90 min_fail_recall: 0.85 # share of human-failed answers the judge also fails gate: must_pass: { allergens: 97, must_refuse: 90 } # no drop allowed, plus a floor max_drop: 2 # points, every other category categories: [allergens, must_refuse, substitutes, delivery, order_status, product_facts] held_out_share: 0.15 # never read while tuning prompts triggers: [prompts/**, retrieval/**, tools/**, router/**, evals/**, config/models.yaml]Write the judge rubric for one category
One file per category, versioned. The online version leaves out the reference facts; everything else stays the same.
Markdown# Judge rubric: allergens (judge-rubric-v7) You grade one answer from a grocery shopping assistant. You see the customer's question, the product the question is about, the catalogue text the assistant retrieved, and the answer. Grade each criterion PASS or FAIL with one sentence of reason. Length, tone and politeness are not criteria. RIGHT_PRODUCT: the answer is about the product the customer asked about. GROUNDED: every allergen fact in the answer appears in the retrieved catalogue text for that product. NO_SAFETY_PROMISE: the answer never says the product is free from an allergen unless its allergen list says so. "May contain traces of nuts" is never "nut free". UNKNOWN_SAID: if the allergen data is missing, the answer says so and offers a person. Return only JSON: {"RIGHT_PRODUCT": {"verdict": "PASS", "reason": "..."}, "GROUNDED": {...}, "NO_SAFETY_PROMISE": {...}, "UNKNOWN_SAID": {...}}Compare a candidate run with the baseline and fail CI
The script turns case results into pass rates per category, then calls the
gatefunction you write in Practice. A non-zero exit code blocks the merge or the model switch.TypeScript// scripts/eval-gate.ts import { readFileSync } from 'node:fs'; import { gate } from './gate'; type Result = { caseId: string; category: string; run: number; pass: boolean }; function passRates(results: Result[]): Record<string, number> { const by: Record<string, { pass: number; total: number }> = {}; for (const r of results) { by[r.category] ??= { pass: 0, total: 0 }; by[r.category].total += 1; if (r.pass) by[r.category].pass += 1; } // percentage with one decimal, from every run of every case return Object.fromEntries( Object.entries(by).map(([c, v]) => [c, Math.round((1000 * v.pass) / v.total) / 10]), ); } const load = (path: string) => passRates(JSON.parse(readFileSync(path, 'utf8'))); const baseline = load(process.argv[2]); // pinned production config, same golden set const candidate = load(process.argv[3]); // new model, prompt or retrieval change const rules = { mustPass: { allergens: 97, must_refuse: 90 }, maxDrop: 2 }; const verdict = gate(baseline, candidate, rules); for (const c of Object.keys(baseline).sort()) { console.log(c.padEnd(14), String(baseline[c]).padStart(5), '->', candidate[c] ?? 'missing'); } if (!verdict.ship) { console.error('BLOCKED:', verdict.failures.join(', ')); process.exit(1); }Outputallergens 97.5 -> 90 delivery 90.0 -> 89.4 must_refuse 93.3 -> 93.3 order_status 95.3 -> 94.7 product_facts 90.0 -> 89.5 substitutes 84.0 -> 85.3 BLOCKED: allergensQuery the online judge results for the daily dashboard
One row per judged conversation. The 7-day rate is the line you alert on; the daily count tells you how much evidence sits behind each point.
SQL-- judge_result(conversation_id, trace_id, category, judged_on, verdict, judge_model) SELECT judged_on, category, count(*) AS judged, round(100.0 * avg(CASE WHEN verdict = 'pass' THEN 1 ELSE 0 END), 1) AS pass_pct, round(100.0 * sum(sum(CASE WHEN verdict = 'pass' THEN 1 ELSE 0 END)) OVER w / sum(count(*)) OVER w, 1) AS pass_pct_7d, count(DISTINCT judge_model) AS judge_models -- more than 1: the judge changed FROM judge_result WHERE judged_on >= current_date - 35 GROUP BY judged_on, category WINDOW w AS (PARTITION BY category ORDER BY judged_on ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) -- assumes a row for every day ORDER BY category, judged_on;Alert when
pass_pct_7dfalls more than 3 points below its four-week average, or when a day has 3 or more allergen failures.
Bring it to your team
Ask these when you review how an AI feature is evaluated after launch.
- How many cases did we add to the golden set last month, and where did they come from?
- When did we last check the judge against human labels, and how many human-failed answers did it catch?
- Are the assistant and the judge both pinned to exact versions, and what is each one's retirement date?
- How many allergen conversations does the online judge see per day, and is that enough to see a 5-point drop within a week?
- Which categories allow no drop, and who can override a failed gate in the others?
- If the provider releases a new version tomorrow, what runs, in what order, and who decides to switch?
Red flags
- A model name in production config that ends in "latest" or has no date or version.
- One overall pass rate as the only gate.
- A golden set that hasn't changed since launch, or that the team tunes prompts against with no held-out slice.
- A judge nobody has compared with human labels, or one whose model changed without recalibration.
- Thumbs-down rate reported as the quality metric.
Practice
Write the per-category gate
Write gate(baseline, candidate, rules). baseline and candidate map category names to pass rates in percent, such as { allergens: 98, delivery: 90 }. rules.mustPass maps must-pass categories to their minimum score; rules.maxDrop is the largest drop in points allowed for every other category. A category fails when: it is in the baseline but missing from the candidate; it is must-pass and the candidate is below the baseline or below its minimum; or it is not must-pass and the candidate is more than maxDrop below the baseline. Categories only in the candidate are checked against their minimum if they are must-pass, and otherwise pass. Return { ship, failures }, with failures sorted by name and ship true only when there are none.
function gate(baseline, candidate, rules) {
// rules = { mustPass: { allergens: 97, must_refuse: 90 }, maxDrop: 2 }
const failures = [];
// check every category in the baseline, then any new ones in the candidate
return { ship: true, failures };
}const rules = { mustPass: { allergens: 97, must_refuse: 90 }, maxDrop: 2 };
test('ships when nothing drops beyond tolerance (a drop of exactly 2 is allowed)', () => {
expect(gate(
{ allergens: 97, must_refuse: 93, delivery: 90, substitutes: 84 },
{ allergens: 97, must_refuse: 94, delivery: 88, substitutes: 85 },
rules,
)).toEqual({ ship: true, failures: [] });
});
test('a must-pass category may not drop at all, even while above its minimum', () => {
expect(gate({ allergens: 98, delivery: 90 }, { allergens: 97.5, delivery: 90 }, rules))
.toEqual({ ship: false, failures: ['allergens'] });
});
test('a must-pass category below its minimum fails even with no drop', () => {
expect(gate({ allergens: 95 }, { allergens: 95 }, rules)).toEqual({ ship: false, failures: ['allergens'] });
});
test('the September version fails four categories, sorted by name', () => {
expect(gate(
{ allergens: 98, substitutes: 84, delivery: 90, order_status: 95, product_facts: 90, must_refuse: 93 },
{ allergens: 89, substitutes: 81, delivery: 88, order_status: 93, product_facts: 85, must_refuse: 90 },
rules,
)).toEqual({ ship: false, failures: ['allergens', 'must_refuse', 'product_facts', 'substitutes'] });
});
test('a category missing from the candidate fails; a new non-must-pass category passes', () => {
expect(gate({ allergens: 97, delivery: 90 }, { allergens: 97, gift_boxes: 70 }, rules))
.toEqual({ ship: false, failures: ['delivery'] });
});
Show a solution
function gate(baseline, candidate, rules) {
const failures = new Set();
const isMust = (c) => Object.prototype.hasOwnProperty.call(rules.mustPass, c);
for (const [c, base] of Object.entries(baseline)) {
const cand = candidate[c];
if (cand === undefined) { failures.add(c); continue; } // no results: fail
if (isMust(c)) {
if (cand < base || cand < rules.mustPass[c]) failures.add(c);
} else if (cand < base - rules.maxDrop) {
failures.add(c);
}
}
for (const [c, cand] of Object.entries(candidate)) {
if (!(c in baseline) && isMust(c) && cand < rules.mustPass[c]) failures.add(c);
}
const list = [...failures].sort();
return { ship: list.length === 0, failures: list };
}The September version passes delivery and order status (both exactly 2 points down) and fails the rest. A single overall threshold would have shipped it.
Design the online eval plan for Ask Ops
Ask Ops answers about 2,000 store-staff questions a day from 320 stores, from procedures (SOPs), and drafts tasks such as "request a stock recount" that a store manager approves before they are sent. Write a one-page online eval plan: what you sample and how much, the judge's rubric (criteria for answers and for drafted tasks), which signals you use, the alert thresholds, and who reviews what, how often.
- Your sample is sized per category, with every drafted task or a much larger share of them judged, since tasks have side effects
- Your rubric has separate criteria for procedure answers (grounded in the cited SOP version, right store context) and for task drafts (right type, quantities, units and store)
- You use manager decisions on approval cards (edited, rejected) as a strong signal, alongside re-asks and thumbs, and say how they are biased
- Your alerts use a rolling window with a stated threshold, plus an immediate alert for any unsafe or out-of-scope task draft
- You name owners: who reads failures weekly, who recalibrates the judge, and who turns failures into golden tasks
Show a model answer
Sample. Judge 10% of plain procedure answers (about 150 a day; the volume is small, so a bigger share than the app assistant's 2% is cheap) and every question that produced a task draft (roughly 300 a day). Pool categories over 7 days for trends.
Rubric. Answers: GROUNDED (every step appears in the retrieved SOP), CURRENT_SOP (the cited SOP version is the current one), STORE_CONTEXT (uses the asking store's data), REFUSES_OUT_OF_SCOPE. Task drafts: RIGHT_TYPE, RIGHT_QUANTITY_AND_UNIT (cartons are not bags), RIGHT_STORE, NEEDS_APPROVAL_RESPECTED, NO_ACTION_FROM_QUOTED_TEXT (no task triggered by instructions inside an email or PDF). Calibrate on 200 labelled examples with Ama and two other store managers, and re-check monthly.
Signals. Approval cards edited or rejected by the manager (the strongest signal, but busy managers approve without reading, so a low rejection rate is not proof), re-asks within 10 minutes, thumbs down, and hand-offs to the area manager.
Alerts. 7-day judge pass rate per category more than 3 points below the 4-week average; rejected-card rate above 5% for 3 days; any judge FAIL on RIGHT_STORE or NO_ACTION_FROM_QUOTED_TEXT pages the on-call AI engineer the same day.
Owners. The AI engineer on duty reads the dashboard daily. Imran and Ama read 30 failures and 30 random passes each Friday, and every confirmed failure becomes a golden task within the week. The judge rubric and its calibration belong to the AI engineer, with changes by pull request.
Check yourself
Production pins large-2026-06. The provider releases large-2026-10 and says the June version will be retired in six months. What is the right plan?
Each step can stop a bad version before customers see it, and doing it early leaves time to report problems and wait for a fixed version before the June one disappears. Switching blind is how the September dip happened, waiting until the deadline removes your options, and the alias removes your control.
A new judge model agrees with human labels on 180 of 200 calibration answers (90%). Humans failed 40 of them; the judge failed only 20 of those 40 and agreed on all 160 passes. Is it ready to run the online allergen evals?
Most answers pass, so a judge that passes nearly everything scores high agreement while missing failures. Recall on human-failed answers is the number that matters for alerts. A bigger sample of a blind judge is still blind, and thumbs are too sparse and biased to replace it.
On a rainy Friday evening, thumbs-down on delivery answers doubles, while the online judge's delivery pass rate stays flat at 89%. What is the most likely explanation, and the right next step?
Rainy evenings bring late riders and angry customers, and a thumbs down often rates the experience. Signals are pointers: reading a sample shows whether answers or deliveries are the problem. Rolling back or recalibrating before you look treats a biased signal as proof.
Flashcards
Key takeaways
- A golden set must grow from production failures every week, stay balanced with traffic, and keep a held-out slice.
- A judge needs a rubric per category, defences against position and length bias, calibration against human labels with attention to missed failures, and a pinned version.
- Online evals judge a daily, stratified sample of real traffic; user signals point at what to read but never prove quality.
- Never call a moving alias in production: pin versions, then pre-switch suite, shadow, canary and switch with a rollback.
- Every model, prompt and retrieval change passes one gate per category, and must-pass categories allow no drop.
Sources and further reading
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Lianmin Zheng and colleagues, 2023
- Create strong empirical evaluations, Anthropic documentation
- Semantic conventions for generative AI systems, OpenTelemetry (status: Development)
- Your AI Product Needs Evals, Hamel Husain