Prompting, retrieval or fine-tuning?
After launch, every quality complaint arrives with a proposed fix, and the most expensive one is often "let's fine-tune our own model". This lesson gives you a way to diagnose a failure as missing knowledge, wrong behaviour or not enough capacity, what each fix really changes and costs to keep, how fine-tuning works and what data it needs, and a bake-off on your own evals that makes the decision for you.
By the end you can
- Diagnose failures from traces as missing knowledge, wrong behaviour or not enough capacity
- Say what prompting, retrieval, fine-tuning and distillation each fix, and what each costs to build and to keep
- Prepare fine-tuning data from production with consent, redaction, labels and a clean held-out test set
- Run a fair bake-off: quality per category, cost per 1,000 requests, p95 latency and upkeep
- Write the decision down with the triggers that should reopen it
The everyday picture
Picture a new assistant at a pharmacy counter who keeps making mistakes. If they quote last month's price, you don't send them on a two-week course; you give them the current price list, because prices will change again next month. If they ramble and forget to ask about allergies, you give them a short script and a few examples of good conversations. If the shop needs someone to sort hundreds of prescriptions a day into twelve trays, you might train a junior colleague properly on that one job, so they do it fast, cheaply and the same way every time.
A good manager tells these three problems apart before choosing a fix, and tests whoever does the job on the same set of real customer cases before deciding. In AI terms: the price list is retrieval, the script is prompting, and the trained junior is a fine-tuned small model. The real cases are your eval suite.
Diagnose before you choose a fix
"The answers are bad" is not a diagnosis. Read 30 to 50 failing traces from the online sample (Evals that never stop) and, for each, mark the first step that went wrong, the way you labelled agent failures in Evaluating agents. Almost every failure falls into one of three kinds:
- Knowledge: the fact needed for a correct answer was not in the model's context. The trace's retrieved document ids show the wrong page, an old version, or a similar product's chunk. No amount of instruction fixes this, because the model can't use what it never saw.
- Behaviour: the fact was in the context, and the model still answered in the wrong way: too long, the wrong format, a missing safety line, a tone that doesn't fit, a rule ignored.
- Capacity: the facts were there and the instructions were clear, and the model still got the reasoning wrong, for example comparing the sugar per 100 g of two cereals whose packs list different serving sizes. Bigger models, or splitting the task into steps, help here.
The label decides the fix, and mixed-up labels waste months. Three complaints reached Mangrove's AI team in one week. Customers were quoted the old delivery fee after a price change: the traces showed the retired help page still in the index, a knowledge problem. Answers to simple questions ran to 180 words: a behaviour problem. And Ask Ops left required fields empty in about 6% of drafted tasks: a behaviour problem first, which schema-constrained output from Prompts and structured output should fix before anything heavier.
Quick check
A trace shows the assistant was given the correct allergen record for the granola, and still wrote "nut free" when the record says "may contain nuts". What kind of failure is this?
The right fact reached the model, so retrieval did its job. The model turned "may contain" into "free", which is how it used the fact. Clear rules, an example of a "may contain" case and the claim check from Guardrails and red-teaming target that directly. A bigger model is a guess, not a diagnosis.