Evals: measuring AI quality
Without evals, "the new prompt seems better" is a feeling. With them, it is a number you can check per category, before and after every change. This lesson shows the method strong AI teams use: read real outputs and count the failures, build a golden set, grade with code, a model judge and people, check the judge against human labels, and link the scores to the product metrics leadership cares about.
By the end you can
- Run error analysis on real outputs and turn notes into counted failure categories
- Build a golden set with the right cases, labels and owner, and keep it fresh
- Choose between code checks, an LLM judge and people for each criterion
- Check an LLM judge against human labels before trusting it, using catch rate as well as agreement
- Read results by category, set release gates, and link eval scores to product metrics
The everyday picture
Before service in a good restaurant, the head chef tastes every sauce against how it should taste. She doesn't score it seven out of ten. She says "too salty, send it back" or "good", and she can always say why. When the restaurant grows, she trains a sous-chef to taste for her. Before she lets him taste alone, they taste the same sauces side by side for a week and she checks that they agree, above all on the sauces she would send back. A taster who passes everything is worse than no taster, because the kitchen stops worrying.
When a new recipe for the menu arrives, she doesn't only ask whether the menu is better overall. She tastes every dish, because a better soup doesn't excuse a ruined dessert. Evals work the same way. A domain expert labels outputs pass or fail with a reason. A golden set holds the dishes she knows by heart. An LLM judge is the sous-chef, trusted only after it has been checked against her. And every change is tasted dish by dish, category by category.
Start with error analysis, not a score
An eval is a repeatable test of output quality. The most common mistake is to start with a score from a generic tool before anyone has read the outputs. Start by reading. Hamel Husain and Shreya Shankar, who teach evals to many product teams, recommend reviewing about 100 real traces to begin with, and continuing while you are still finding new kinds of failure.
- Sample. Take about 100 traces from real use: mostly random, plus some the users reported. A trace shows the question, who asked, the retrieved procedure pieces, the tool results and the answer.
- Write a note on each failure, about the first thing that went wrong, in plain words: "said $500 threshold, ST-4 says $100". This is called open coding. Don't use categories yet.
- Group the notes into categories and count them (axial coding). The counts tell you what to fix first.
- Fix the biggest category in the part that caused it, then look again.
Label each trace pass or fail with a reason, not on a 1 to 5 scale. Binary labels force a decision and are far more consistent between people; a "3" means something different to everyone. Let one domain expert own the final call on what counts as good; Husain and Shankar call this person a "benevolent dictator". At Mangrove that is the ops lead at head office, who wrote half the procedures.
Expect your criteria to move as you read. A 2024 study by Shankar and colleagues, Who Validates the Validators?, found that people need criteria to grade outputs, but grading outputs is how they discover their criteria. That is normal. Write the criteria down as they settle, and re-label the early traces.
This is what Mangrove's first round found, from 120 pilot traces (100 random and 20 reported):
| Failure category | Count | Example note | Fix, in the part that failed |
|---|---|---|---|
| Wrong number from a procedure | 7 | "$500 threshold; ST-4 says $100" | Fewer, more precise pieces; code check that numbers appear in the cited section |
| Used an out-of-date procedure | 5 | "quoted the old RF-2 limit" | Ingestion retires old versions |
| Wrong procedure cited, right action | 4 | "expiry answer cites RC-1, should be EX-3" | Procedure titles added to each piece |
| Made-up policy | 3 | "invented a 15% service discount" | Prompt rule and a judge criterion; new golden cases |
| Didn't answer the question asked | 2 | "answered about counts, asked about who to tell" | Prompt example of restating the question |
| Unsafe with customer data | 1 | "suggested sending a phone number to a rider" | Must-pass golden cases; output check |
Twenty-two failures in 120. Notice that only one of the six fixes is "change the prompt". Most AI quality problems live in the data, the retrieval or the missing checks, which is why reading traces beats tweaking prompts.
Quick check
Your team wants to start measuring Ask Ops quality. What should come first?
Reading real traces shows what actually goes wrong, which decides what to measure and fix. A generic score may not track your failures at all. 1 to 5 ratings are inconsistent between people. Synthetic questions can help later, but they don't tell you how real users phrase things or where the product really fails.