Guardrails and red-teaming
At launch you built layered defences and a set of injection cases. Three months later the systems read more content, serve far more people and have tools that change real things, so new holes open that no launch test covered. This lesson turns guardrails into something you operate: a threat model for both systems, guards you measure, attack signals you watch, and a red-team exercise whose findings become fixes and eval cases.
By the end you can
- Build a threat model for the app assistant and Ask Ops with the OWASP Top 10 for LLM Applications, ranked by harm and likelihood
- Measure each guardrail by catch rate, false-positive rate and added latency, and decide whether it should block, degrade or only log
- Spot attacks in production from traces, audit logs and budgets
- Plan and run a two-day internal red-team exercise and score its findings by severity
- Turn every finding into a fix in the right layer, an owner and an eval case, or a written, accepted risk
The everyday picture
Think about how a supermarket deals with theft. It doesn't hope that a sign saying "please don't steal" will work. It puts expensive items such as razor blades and baby formula behind the counter, so a thief can take little even if nobody is watching. It has tags that beep at the door, cameras, and staff who know what odd behaviour looks like. And every few months, the loss-prevention team pays a "mystery shopper" to try to walk out with goods, then fixes whatever let them through.
Two things make this work. Nobody believes any single measure stops all theft, so the measures are layered and the most valuable things are the hardest to reach. And the store keeps testing itself, because thieves learn and the store changes: a new aisle, a new self-checkout, a new supplier. AI features are the same. Your prompt is the sign on the wall. Least-privilege tools are the counter. Output checks are the door tags. Traces and alerts are the cameras. The red team is the mystery shopper.
A threat model for two systems after launch
You met prompt injection, sensitive data and the OWASP list in Cost, latency and safety, and tools that run as the user with approvals for writes in Designing tools and permissions. Those defences were designed for the systems as they launched. Since then three things changed at Mangrove, and each opened a new path:
- More content is read. Version 1 of the app assistant didn't index customer reviews. Version 2 does, so it can answer "what do people say about this rice cooker?". Every review is text written by a stranger that now reaches the model.
- More tools do things. Ask Ops can now draft stock write-offs and recount requests from supplier delivery PDFs, not only answer questions from procedures.
- Far more people use it. About 15,000 conversations a day on the app and about 2,000 Ask Ops questions a day. Rare attacks become daily events at that volume, and some users actively probe.
A threat model keeps this manageable. List the threats per system, map each to the OWASP Top 10 for LLM Applications (2025 edition), score harm and likelihood, and name the control you rely on. Then work from the top.
| Threat | OWASP | Harm | Likely | Main control |
|---|---|---|---|---|
| Instructions hidden in a supplier PDF make Ask Ops draft a write-off | LLM01, LLM06 | High (stock, money) | Medium | Approval card built from system data |
| The assistant claims a product is free from an allergen | LLM09 | High (health) | Medium | Claim check against the catalogue field |
| A customer gets another customer's order details | LLM02 | High (privacy) | Low | Order tool scoped to the session |
| A review steers answers towards another shop | LLM01 | Medium (trust) | High | Reviews quoted as data, judge criterion |
| A script runs up the bill with long questions | LLM10 | Medium (money) | Medium | Per-user caps, feature budget |
| Someone extracts the system prompt | LLM07 | Low (no secrets in it) | High | Accepted risk, canary token |
Two habits make the table useful. First, score harm by what happens to a person, money or someone else's data, not by how embarrassing a screenshot would be. A leaked system prompt makes a funny social media post, but it hurts nobody if the prompt holds no secrets. A wrong allergen answer can put someone in hospital. Second, the "main control" column should name a control that still holds when the model is fooled. "The prompt tells it not to" is not a main control; it is a sign on the wall.
Quick check
Which threat should the team work on first?
Rank by harm first, then likelihood. A write-off moves stock and money, and only the approval step stands between an injected instruction and the change. The prompt leak is frequent but harmless when the prompt holds no secrets, and the review problem damages trust rather than money or health.