Shipping an agent safely
Your store-manager operations assistant passes its evals. This lesson takes it to a pilot in five Mangrove stores: what one task really costs and how long it takes, the budgets that stop it in time, the traces and audit trail that tell you what happened, the switches that turn it off in seconds, and the staged plan that earns each next step with numbers.
By the end you can
- Work out cost and latency per task from steps, tokens and tool time, and set budgets per task, per user and per store
- Add a budget guard to the agent loop that stops cleanly and hands over with a summary
- Trace every task with one span per step, and keep an audit trail that is separate from the debug traces
- Put feature flags and a kill switch in front of every write tool, and pin model and prompt versions
- Write a pilot plan with stages, entry and exit gates, stop rules and an incident playbook
The everyday picture
Think about a new cashier's first month at a busy supermarket. On day one she watches an experienced colleague and says what she would do, while the colleague does the work. In week two she runs a till, but a supervisor signs every refund. Later she approves small refunds herself, and only big ones still need a signature.
Around her there are quiet controls. Her till holds a limited cash float, so one mistake can't empty the safe. The shift manager has a key that locks every drawer at once. Every sale and refund prints on the till roll with her name and the time, and the store manager or an auditor can read it a year later. Nobody gives her the whole store on day one. Shipping an agent works the same way: a limited float (budgets), one key that stops everything (the kill switch), a till roll (the audit trail) and a supervisor who steps back one stage at a time (the staged pilot).
What one task costs
In What makes an agent you built the loop: the model reads the conversation, asks for a tool, reads the result, and repeats until it can answer. Each pass is one model call. A store manager doesn't care about calls. She asks one thing, such as "why is the yoghurt delivery short, and can store 114 send me two cartons?", and that is one task. Budget and measure per task.
The cost of a task grows faster than its number of steps. Most model APIs are stateless: on every call you send the whole context again. So step 1 sends the system prompt and tool definitions, step 2 sends those plus step 1's request and result, and so on. If the fixed part is F tokens and each step adds g tokens, the input for a task of n steps is n × F + g × n × (n − 1) ÷ 2. The second term grows with the square of the steps.
Put numbers on the Mangrove assistant. The prices here are illustrative: say $3 per million input tokens and $15 per million output tokens. Check your provider's current price list, because prices differ by model and change often.
Fixed context F (system prompt + 9 tool definitions) 4,000 tokens
Added per step g (tool result + model's request) 1,500 tokens
Output per step 300 tokens
Average task, 6 steps:
input 6 x 4,000 + 1,500 x 15 = 46,500 tokens x $3/M = $0.140
output 6 x 300 = 1,800 tokens x $15/M = $0.027
total about 17 cents
p95 task, 14 steps:
input 14 x 4,000 + 1,500 x 91 = 192,500 tokens x $3/M = $0.578
output 14 x 300 = 4,200 tokens x $15/M = $0.063
total about 64 centsThe p95 task has 2.3 times the steps of the average one and costs nearly 4 times as much. A few long tasks can decide the bill. Two levers from Cost, latency and safety help a lot here: prompt caching for the fixed part, and a smaller model for easy steps. This lesson stays with the agent-specific part: limiting and watching cost per task.
Budgets at three levels. A per-task limit stops one runaway loop. A per-user daily limit stops one person, or one script using their account, from running up a bill. A per-store monthly limit keeps the pilot inside what Ken agreed to spend. OWASP lists this risk for LLM apps as unbounded consumption. For the pilot, with about 15 tasks per manager per day and an average near 20 cents (the long tasks pull the average above the 17-cent six-step task):
- Per task: 16 steps, 250,000 input tokens, 80 cents, 90 seconds of wall-clock time. Each limit sits just above the p95 task, so normal work never hits it.
- Per user per day: $5, against about $3 of normal use.
- Per store per month: $120, against about 15 × 30 × 20 cents = $90 of normal use. Five stores cost about $450 a month. All 320 stores would cost about $28,800 a month, which is why cost is a gate before you go wide.
The guard runs before each model call, using an estimate of that call, so it never spends past the limit. And when a limit is reached, the agent stops, saves its state (as in Memory, state and multi-step plans) and hands over to the person with a plain summary: what it checked, what it did, what is left and why it stopped. It never loops quietly, never retries forever, and never switches to a weaker model without saying so. A handover is a normal outcome you count, like a test that is skipped with a reason.
Quick check
With the same numbers (4,000 fixed, 1,500 added per step, 300 output per step, the illustrative prices), the 6-step task costs about 17 cents. Roughly what does a 12-step task cost?
Input is 12 × 4,000 + 1,500 × 66 = 147,000 tokens, about 44 cents, plus 3,600 output tokens, about 5 cents. Doubling the steps roughly triples the cost here, because the resent history grows with the square of the steps. It is not a cube, so $2 is far too high.