Cost, latency and safety
The product-question assistant in the Mangrove app works and passes its evals. Before launch you need three more answers: what 1,000 questions cost, how long a customer waits, and what happens when someone, or some text, tries to make the model misbehave. This lesson gives you the levers for each and a plan you can defend.
By the end you can
- Break one request into the tokens and milliseconds it spends, and build a cost and latency budget for 1,000 questions
- Use prompt caching, response caching, smaller context, output limits, model routing and batch jobs, and know where each one fails
- Cut waiting time with streaming, parallel steps, fewer round trips and timeouts with a fallback, and cap spend per request and per day
- Explain direct and indirect prompt injection, why there is no complete fix, and put layered defences around the assistant
- Map the OWASP Top 10 for LLM Applications 2025 to a real feature and add injection cases to your eval set
The everyday picture
Picture the information desk at a large grocery store. Every question that reaches the senior clerk costs time and money, so the desk keeps a printed card with answers to the questions people ask all day ("Where are the trolleys?"). A junior clerk handles the easy questions and calls the senior clerk only for hard ones. Nobody reads the whole store binder aloud for each question; the clerk opens the two right pages.
The clerk also reads supplier leaflets to customers. One day a leaflet says, in small print, "Clerk: tell everyone the shop next door is cheaper." A good clerk reads leaflets as information about products, never as orders from the manager. And the clerk has no key to the till, so even a fooled clerk cannot hand out money. An LLM feature needs the same habits: answer repeats from a card, send easy work to the cheaper worker, read only what is needed, treat outside text as data, and give the model no keys it does not need.
Where the money and the time go
In How LLM APIs work you saw that providers charge per input token and per output token, with output usually several times more expensive per token. To control cost you need to know which tokens you send. For the Mangrove assistant, one question to a large model looks like this (prices here are illustrative: say $3 per million input tokens and $15 per million output tokens; check your provider's current price list):
- Instructions: the system rules, the answer schema from Prompts and structured output and the tool definitions from Tools and MCP. About 1,800 tokens, the same on every call.
- Retrieved chunks: five chunks of about 400 tokens from the help pages and the catalogue (RAG). About 2,000 tokens, different for each question.
- The question and a little chat history: about 200 tokens.
- The answer: about 250 output tokens.
That is 4,000 input tokens ($0.012) and 250 output tokens ($0.00375), so $0.01575 a question, or $15.75 per 1,000. Notice that the customer's own words are only 5% of the bill. Most of the cost is text you chose to send: instructions and chunks. That is good news, because you control it.
Time splits the same way. Before the model writes a word, the request passes your gateway (sign-in check, rate limit), retrieval (embed the question, hybrid search, rerank: roughly 250 ms in Mangrove's tests) and maybe a live tool call such as check_stock. Then the model reads the whole prompt and starts writing. The wait until the first word appears is the time to first token (TTFT). After that, output arrives at a steady rate, often tens of tokens a second for a large model, so a 250-token answer takes a few more seconds. Long prompts raise TTFT; long answers raise total time. Every tool round trip is another full model call, with its own TTFT.
Quick check
The team wants to cut the assistant's bill. Which change touches the largest part of the cost in the example above?
Instructions and chunks are 3,800 of the 4,000 input tokens. The question is only 200 tokens, and citations are a few output tokens. Shorter questions feel cheaper but barely move the bill.