Cost and latency at scale
Three months after launch, the app assistant costs about twice its budget and its slowest answers take 6.5 seconds. This lesson shows you how to find where the money and the time go from your traces, fix them with prompt caching, routing and smaller prompts, prove with evals that quality held, and put a budget and a kill switch around every AI feature.
By the end you can
- Compute cost per call, per conversation and per feature from token counts, and see which lever moves the bill most
- Lay out prompts so the provider's prompt cache hits, and decide where answer caching and batch jobs are safe
- Route between a small and a large model with rules, a classifier and escalation, and prove with per-category evals that quality held
- Set latency budgets for the first word and the full answer, and meet them with fewer round trips, fewer tokens and streaming
- Give each AI feature a monthly budget, early alerts and a kill switch that degrades in steps
The everyday picture
Picture a busy restaurant kitchen. Every dish starts from the same base sauce, so a good kitchen makes it once in the morning and ladles from the pot all day. Simple dishes, toast and salads, go to the junior cook. The head chef, who costs three times as much, cooks the hard dishes and checks anything the junior is unsure about.
One night a dish goes out with the wrong allergen label, and the manager says "the head chef cooks everything, to be safe". Weeks later the rule is still there: the wage bill has doubled and customers wait longer. Meanwhile someone started writing each order number into the sauce pot, so the sauce is made fresh for every plate. The fix is a taste test: the junior cooks a batch of each simple dish, the head chef scores them, and only dishes that pass go back to the junior. Allergen dishes stay with the head chef by rule. That is this lesson: a stable prefix the provider can reuse, a router with checks, and evidence before every switch.
Where the money goes at 15,000 conversations a day
In How LLM APIs work you priced tokens before launch, and in Cost, latency and safety you added an exact answer cache and a router that sent about 60% of calls to the small model. Since then the traces from Observability for LLM systems have been landing in the trace store. Every model-call span carries the model, the input, cached and output tokens, and a cost computed from a price table. So the first step of a cost review is a query, not a guess.
Start from one average call, then multiply. Prices here are illustrative (large model $3 per million input tokens and $15 per million output tokens; small model $0.80 and $4); check your provider's current price page.
One app-assistant call today, all on the large model
system prompt + tool definitions 3,000 tokens
8 retrieved chunks 1,500 tokens
conversation so far 500 tokens
input 5,000 x $3 / 1M = $0.01500
output 250 x $15 / 1M = $0.00375
per call $0.01875
per conversation (2.5 calls) about $0.047
per day (15,000) about $700
per month (30 days) about $21,000 budget $10,000Two facts jump out. First, input is 80% of the call ($0.015 of $0.01875), even though output tokens cost five times more each, because you send twenty times more of them. Second, 3,000 of the 5,000 input tokens are the same on every call. The text you choose to send is where the bill lives, and it is fully under your control.
Three changes since launch explain the jump from about $5,000 a month, each made for a good reason. The prompt grew from about 4,000 to 5,000 input tokens (more tools and rules, 8 chunks instead of 5, longer history). During the allergen incident the team switched the router off "to be safe". And a request id and timestamp were added at the top of the system prompt for debugging, which broke the provider's prompt cache. Nobody was measuring cost per conversation each week.
Tag every span with a feature attribute, because one provider bill covers the app assistant, Ask Ops and the nightly catalogue jobs. And watch conversations, not only calls: each call resends the prefix and the whole history, so a conversation's cost grows faster than its length. A cap on turns and a trimmed history belong in a cost plan.
Rank the levers by how many tokens they touch. The 3,000-token prefix is on every call, so it comes first. Routing changes the price of every token on most calls, so it comes second. Fewer chunks trims about 560 tokens a call. The first diagram shows the plan.
Quick check
A call sends 5,000 input tokens at $3 per million and gets 250 output tokens at $15 per million. Roughly what share of the call's cost is input?
Input is 5,000 × $3 ÷ 1M = $0.015 and output is 250 × $15 ÷ 1M = $0.00375, so input is $0.015 of $0.01875, which is 80%. The higher price per output token is outweighed by sending twenty times more input tokens.