System instructions that change behaviour

Most model APIs take a system prompt separately from the conversation. You wrote one line of it in How LLM APIs work. For a feature people depend on, it needs structure. A system prompt that works well usually has five parts, in this order:

1. Role and audience. Who the model is acting as and who reads the answer. "You answer questions from Mangrove customers in the Mangrove app. They read on a phone, and many read English as a second language."
2. The task. What to do with the input. "Answer the question using only the documents provided."
3. Rules and limits. Length, tone, what to quote, what never to do. Each rule is one line you could test.
4. What to do when unsure. The most important part, and the one most prompts leave out. If you don't say what to do when the documents have no answer, the model will fill the gap with something fluent.
5. Output format. The exact shape of the reply. Later in this lesson that becomes a JSON Schema.

Rules must be concrete. "Be accurate" or "be helpful" changes almost nothing, because the model already tries to do both, and there is nothing in those words it could check. A concrete rule names a behaviour. Compare "be accurate about prices" with "quote a price only if it appears in a product record, in dollars and cents, and never estimate one". The second rule tells the model what to look for and what to do when it is missing. It also gives you something to test in Evals: you can count answers that contain a price not found in any record.

The second habit is to keep instructions and data apart. For each question, the assistant receives product records and help pages (from retrieval, which you build in Retrieval-augmented generation) and the customer's own words. Put each of these in a labelled block with clear delimiters, such as <documents> and <question>, and say in the system prompt that text inside those blocks is data to read, never instructions to follow. Give every document an id the model can cite. Delimiters make it much easier for the model to tell what you asked from what a supplier wrote in a product description. They lower the risk of prompt injection without removing it; the full defence is in Cost, latency and safety.

Examples, and reasoning where it helps

Rules describe the behaviour. Few-shot examples show it. A few worked pairs of question, documents and the correct reply are often the fastest way to fix the format and tone. Provider guides commonly suggest about three to five. Keep them in the prompt file, wrapped in their own tags such as <example>, so the model does not mistake them for the real question.

Choose them for coverage. One easy product question teaches little. Better: one plain answer with a source, one allergen question where the record is silent and the right status is unknown, one substitute suggestion, and one question the documents cannot answer. The hard cases are where the model needs the most help.

<example>
Question: Is the Mangrove oat granola 500 g peanut free?
Documents: art-20417 says "Ingredients: oats, honey, sunflower oil. Allergens: gluten (oats)."
Reply: {"question_type": "allergen",
  "answer": "The label lists gluten from oats. It does not say whether it is peanut free, so we can't confirm that. Please check the pack or chat with our team.",
  "sources": ["art-20417"], "allergens": [{"name": "peanut", "status": "unknown"}],
  "substitute_id": null, "needs_human": true}
</example>

Examples teach more than you intend. The model copies their length, their opening words and their habits. If all three examples are two sentences long, most answers will be two sentences. If every example ends with "Happy shopping!", so will every reply. If every allergen example says "contains", answers lean that way. Facts leak too: a price in an example can turn up in a real answer, which is why the example's product is one you would not mind being quoted. Vary the examples on purpose, and keep them in version control with the prompt, because changing an example changes behaviour as much as changing a rule. Each example is also paid for on every call. Three examples of about 150 tokens add about 450 input tokens per question. At 9,000 questions a day and an illustrative $3 per million input tokens (check your provider's current price list), that is about $12 a day.

Reasoning is the other lever. Asking the model to work through steps before it answers helps on multi-step questions, such as "which of these three rice bags is cheapest per kilo?" or a delivery fee that depends on basket size and slot. For a plain lookup it adds tokens and seconds and changes nothing. Many current models have a built-in reasoning mode that you switch on through the API instead of asking for it in the prompt; how its output is returned or hidden differs by provider. Two rules hold either way. Keep any reasoning out of what the customer sees: if you add a field for it, your server logs it and removes it before the reply reaches the app. And don't treat written reasoning as proof of how the model decided. It is useful for debugging and can still be wrong.

Free text, JSON mode and schema-constrained output

The app needs more than a paragraph. It shows the answer in a chat bubble, turns sources into links, shows an allergen badge, offers a substitute as a product card with an Add to cart button, and opens a chat with a person when the assistant can't help. So the reply has to be data. There are three levels of help from the provider.

JSON mode guarantees syntax only. The reply will parse, but it may call a field allergen_status instead of allergens, or leave one out. Schema-constrained output, which several providers now offer (one calls it Structured Outputs), goes further. You send a JSON Schema, and the provider restricts the model's choice of each next token so that the text can only follow that schema. This is called constrained decoding. Support and limits differ by provider and change often: each supports only part of JSON Schema, several require "additionalProperties": false and every field listed as required (an optional field becomes a required field that can be null), some reject keywords such as string length limits, and the first call with a new schema can be slower while it is prepared. Read your provider's page before you design around it.

Even with the strongest setting, you still validate on your side, for four reasons. First, provider documentation itself lists cases where the output does not match: a refusal, or a reply cut off when it hits the output token limit. Second, some rules can't be written in the schema your provider accepts, such as "needs_human must be true when any allergen is unknown". Third, your code may one day run against a different model or a fallback provider with weaker support. Fourth, a valid shape says nothing about truth. {"name": "peanut", "status": "free"} is perfectly valid JSON, and it is a dangerous answer if no record says so.

Designing the schema the app renders

Design the schema from the screen backwards. Each field exists because the app does something with it. Leila's design for the answer card needs six:

{
  "type": "object",
  "properties": {
    "question_type": { "type": "string",
      "enum": ["product_info", "allergen", "substitute", "delivery", "other"] },
    "answer": { "type": "string", "description": "Shown to the customer. At most 3 short sentences." },
    "sources": { "type": "array", "items": { "type": "string" },
      "description": "Ids of the documents used, such as art-20417 or help-12." },
    "allergens": { "type": "array", "items": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "status": { "type": "string", "enum": ["contains", "free", "unknown"] }
      },
      "required": ["name", "status"], "additionalProperties": false } },
    "substitute_id": { "type": ["string", "null"], "description": "Article id of a suggested product, or null." },
    "needs_human": { "type": "boolean" }
  },
  "required": ["question_type", "answer", "sources", "allergens", "substitute_id", "needs_human"],
  "additionalProperties": false
}

A few design choices carry most of the weight. Enums for anything the app branches on. An enum turns question_type into five values your code can switch on and Tomás can count in reports. Free text such as "allergy question" or "Allergens" would need guessing to group. Sources as ids. The model returns art-20417, never a link it made up; the app builds the link from the id, and your validator checks that the id was one you sent. Nullable instead of empty strings. substitute_id is nullable, and null means "no substitute offered". An empty string would mean the same thing with more room for bugs.

Unknown is a value. This is the rule that matters most. If the allergen status could only be "contains" or "free", the model would be forced to pick one every time, and a silent label would turn into "free" because nothing was listed. With "unknown" in the enum, the honest answer has a place to go. The prompt says "free" is allowed only when a product record says so in words, and your code adds the hard rule: any unknown allergen sets needs_human to true, and the app shows "We can't confirm this. Please check the pack or chat with our team" with a button to a person. Allergen answers are never guessed, and the schema plus the validator enforce that even when the prompt is ignored. AI features people can trust covers the design side of this handoff.

Keep the schema small. Every field costs output tokens, is one more thing the model can get wrong, and is one more thing to validate. Six fields is plenty for the first version; add a field when the app has a use for it.

Validate, retry once, then fall back

Every reply goes through the same pass. Parse it in a function that catches errors. Validate it twice: against the schema (shape and enums) and against your own rules (sources were sent, unknown means needs_human, the answer is not too long). If it fails, retry once: send the reply back to the model with the exact errors, such as "needs_human: must be true when an allergen is unknown", and ask for corrected JSON. Models fix a named error well. If the second reply also fails, stop. Return a fallback written in advance, which says the assistant can't answer and offers a person, and log the question, both replies and the errors.

Why only one retry? Each retry costs a full call in money and in seconds while the customer waits, and a model that failed twice on the same input usually fails a third time. The log is what fixes the cause. Tomás reviews failures each week, and a pattern such as "substitute questions return an id we did not send" points to a prompt rule or an example to add. A call that times out or errors goes straight to the fallback, since retrying an outage inside the request only makes the wait longer (Observability covers alerting on fallback rate).

Version the prompt like code. The prompt and its examples live as files in git, for example prompts/product-assistant/v3.md, reviewed in pull requests like any other change (Pull requests and code review). Every log line carries the prompt id and the schema version, so when a complaint arrives you know exactly which instructions produced the answer. Change one thing at a time. If v4 rewrites the allergen rule, adds two examples and shortens the answer limit together, and allergen answers get worse, you cannot tell which change did it. In Evals you will run each version against a golden set of real questions before release, so "v4 seems better" becomes a number.