Evals: knowing if it got better

40 minIntermediate

Evals: knowing if it got better

You changed the prompt and three answers you tried look better. That tells you almost nothing. This lesson builds the engineering side of evals for the product-question assistant: eval cases stored as data, checks written in code, a model judge with a written rubric, retrieval scored on its own, honest numbers despite randomness, and a gate in CI that blocks a pull request when a category gets worse.

By the end you can

Write eval cases as data, with frozen context, a category and the checks that apply
Build a golden set from real customer questions by sampling each question type from the logs
Write deterministic checks, reference checks and a rubric judge, and score retrieval separately with recall@k
Compare two versions case by case, with repeated runs and counts, instead of trusting a total
Run the suite in GitHub Actions on prompt and retrieval changes, with per-category gates

The everyday picture

Think of a yearly vehicle inspection. Every car goes through the same checklist. Some items are measured by a machine: the rollers measure brake force and print a number, and the number is either above the limit or not. Other items need the inspector's eye, so each one has a written rule, such as "rust that weakens the frame", and the inspector marks it pass or fail. Nobody gives the car "7 out of 10".

Two more rules make it work. Brakes must pass, whatever else is true: shiny paint and new tyres don't buy back a failed brake. And after a mechanic fixes the exhaust, the car goes through the whole checklist again, because a repair in one place can loosen something in another. An eval suite works the same way. The cases are the checklist, code checks are the rollers, the model judge is the inspector with written rules, the allergen category is the brakes, and every pull request is a car coming back from the garage.

An eval case is data, not a sentence in a doc

In Evals: measuring AI quality you saw the product view: read traces, count failures, give the golden set an owner, calibrate the judge. Here you build the machine that runs it. Its smallest unit is the eval case, and it must be data a program can read, because your runner will execute it hundreds of times a week.

The assistant from Prompts and structured output returns JSON with a question_type, the answer text, a list of sources, an allergens list where each entry's status may be "unknown", a substitute_id, and a needs_human flag. A case describes what one question should produce:

JSON

{
  "id": "alg-014",
  "category": "allergen",
  "tags": ["nuts", "child", "from-thumbs-down"],
  "input": { "question": "do the oat cookies have nuts?? my son is allergic", "zone": "north-3" },
  "fixtures": { "snapshot": "catalog-2026-10-01", "products": ["art-88412"] },
  "expect": { "sources": ["art-88412"] },
  "checks": ["schema", "cites_retrieved", "allergen_unknown", "judge"],
  "origin": "app log 2026-09-03, thumbs down"
}

Each field has a job:

input is what the customer sent, in their own words and spelling, plus the context the app adds, such as the delivery zone. The same question in two zones can have two right answers.
fixtures freeze everything that changes in real life. Retrieval runs against a pinned catalogue snapshot, and tools such as check_stock and list_delivery_slots from Tools and MCP return recorded results. Without this, a supplier updating a label or a slot filling up makes a case fail with no code change.
expect holds the answer key: the source ids retrieval should find, facts the answer must contain, and phrases it must never contain.
checks lists which graders apply. A delivery case needs a slot-time check; an out-of-scope case needs a refusal check. Naming them per case keeps the runner generic.
category and tags let you report results per slice, and origin tells the next engineer why the case exists.

Store one case per line in a JSONL file in the repository, next to the code. Adding a case is then a one-line diff that a reviewer can read.

Quick check

A delivery case expects "the earliest slot is 19:30". It passed all week, then failed last night with no code change. What is the most likely fix?