What happens between a question and an answer

This course follows one product from idea to agent. Mangrove's store managers send head office about 3,000 questions a week through the ops desk's chat groups and phone line: "What markdown goes on yoghurt two days before expiry?", "Why is GRN 8812 still open?", "Why was my app order cut?". Eight people on the ops desk spend about 300 hours a week answering them, and a typical answer arrives about three hours later. Imran's team is building Ask Ops, an assistant inside the Store app and the back office that answers these questions from Mangrove's procedures and live data. Later lessons decide which questions it should take (pm-47), design it (pm-48), measure it (pm-49) and let it act (pm-50).

Here is what happens when Ama, a store manager, types "Why is GRN 8812 still open?". A GRN is a goods received note, the record of a delivery.

1. The app sends the question with who is asking: Ama, store manager, store 112.
2. Retrieval finds the few pieces of the procedures that match the question, here parts of RC-1, the receiving procedure, from the procedures Ama's role may read.
3. A tool call reads the live record of GRN 8812 from the Store app's API, with Ama's permissions.
4. The context is assembled: the instructions, the procedure pieces, the GRN record, the earlier messages in this chat and the question.
5. The model writes the answer a word or so at a time, streamed to Ama's screen, with the procedure it used and the time the GRN was checked.

The model is a large language model (LLM). It reads and writes tokens, which are pieces of words; in English one token is roughly three quarters of a word. Everything the model can see in one call must fit in its context window. The fixed instructions at the start of every call are the system prompt.

Three facts follow, and each one shapes the product.

• The model has no memory between calls. A follow-up question works only because the app sends the earlier messages again. Long chats therefore cost more per message.
• It knows nothing about Mangrove. Its knowledge was fixed when it was trained. Procedures and live data must arrive in the context of each call.
• It is not deterministic. The same question can get different answers, and sometimes a fluent, wrong one, called a hallucination. You can't judge quality from one try, which is why the course spends a whole lesson on evals.

Engineers who want the details of tokens, embeddings and vector search will find them in Designing AI features: LLMs, RAG and vector search. You need the picture and the trade-offs.

Four ways to make a model useful at your company

A general model knows a lot about the world and nothing about your stores. There are four ways to close the gap. Each costs something different and fixes something different.

Retrieval, often called RAG, is how Ask Ops knows the procedures. When the refund rules change, the team updates the document and the next answer uses the new text. Answers can cite the piece they came from, so a manager can check in a tap. Tool calling is how it sees live data. The model never touches the database. It asks for get_grn_status with a GRN number, and Mangrove's code checks that Ama may see that GRN, runs the query and returns the result as text.

Fine-tuning is the technique people ask for first and need last. It is good at habits and poor at facts that change: train the model on this month's refund limit and it will repeat it after the limit changes, with no source to check. It also needs hundreds of good examples and has to be redone when you move to a newer base model. Reach for it only when evals show that prompts, retrieval and tools can't reach your quality bar.

Cost per answer, and a cost budget

With an API, every call is billed by tokens: cost per answer = input tokens × input price + output tokens × output price. Providers publish a price per million tokens for each model, and output tokens usually cost several times more than input tokens. Prices change often, so this lesson counts tokens and leaves the price to your provider's current price page. If you run the model yourself, the bill is servers and people instead (see the last section).

Most of the tokens in an answer are ones the user never sees. Here is one Ask Ops answer, measured in the pilot:

System prompt (rules, format, examples)    1,500 tokens in
Procedure pieces (5 pieces of ~400)        2,000 tokens in
Tool result (GRN 8812 record and lines)    1,500 tokens in
Earlier messages in this chat              1,000 tokens in
Question                                      50 tokens in
------------------------------------------------------
Input                                     ~6,000 tokens
Output (the answer)                         ~300 tokens

Two things push the count up quickly. Every follow-up resends the chat so far. And every extra tool round is a whole new model call that reads everything again plus the new result: an answer that needs three rounds reads about 6,000, then 7,000, then 8,000 tokens, roughly 21,000 in total. Agents, which work in loops, multiply this (pm-50).

Then multiply by volume. Today's 3,000 questions a week is the floor: when asking becomes easy, people ask more. Mangrove plans for about 6,000 a week, about 26,000 a month, and twice that in a festival week. That is roughly 26,000 × 6,300 ≈ 164 million tokens a month. At the price Mangrove paid in the pilot this came to about 1.5 cents an answer, or about $390 a month. Compare that with the ops desk: 300 hours a week at about $8 an hour is over $10,000 a month. The model bill is rarely the biggest cost of an AI feature. Engineering time, evals and people reviewing the output usually are. It still needs a budget, because it grows with every new feature and every runaway loop.

The levers, roughly in order of effort:

• Send less. Three good procedure pieces instead of five; the last two chat turns plus a short summary instead of the whole chat.
• Route by difficulty. Send easy questions to a smaller, cheaper model and hard ones to the strongest. Check both with evals.
• Cache. Many providers charge less when the start of a prompt is identical to a recent call (prompt caching, explained in Claude: how Anthropic serves Claude at scale). Keep the fixed system prompt first. Identical questions, such as "when is the weekly stock count?", can reuse a stored answer keyed by role and procedure version.
• Batch what can wait. Overnight work, such as tomorrow's suggested tasks, can use slower, cheaper batch processing where your provider offers it.
• Cap it. A cost alert per day and a limit per store, so a loop that goes wrong can't spend the month's budget in an afternoon.

Response time, and a latency budget

Two numbers describe how fast an AI answer feels. Time to first token is how long until the first word appears; it grows with the amount of input the model must read. Then the model writes at some tens of tokens per second, so a long answer takes several more seconds. Streaming shows the words as they come, which makes the wait feel much shorter. Judge speed at the p95, the time 95 of 100 questions beat, never the average.

Write a budget the way you would for money. Ask Ops promises first words within 3 seconds and the full answer within 10 seconds, for 95% of questions. The pilot measured where the time goes:

Step                                       Time (approx.)
App to Ask Ops API, check role             0.1 s
Retrieval: find RC-1 pieces                0.2 s
Tool: read GRN 8812                        0.4 s
Model reads 6,000 tokens, first word       1.0 s
-------------------------------------------------
First words on screen                     ~1.7 s   budget 3 s
Model writes 300 tokens at ~50 a second    6.0 s
-------------------------------------------------
Full answer                               ~7.7 s   budget 10 s

What spends the budget: a longer input delays the first word, a longer answer delays the end, each extra tool round adds a full model call, and peak load adds queueing. If you want to know why reading and writing behave so differently inside the chip, read ChatGPT: serving a large language model to hundreds of millions. The product fixes are ordinary: run lookups at the same time instead of one after another, ask for short answers, show progress ("Checking GRN 8812…"), set a timeout with a fallback (show the matching procedure pieces if the model is slow), and move work that doesn't need a person waiting into the background, with a notification when it is done.

Your own server or an API

There are two ways to run the model. You can call a provider's API, where a hosted model runs on the provider's machines. Or you can run an open-weight model on servers you control, in your data centre or on rented cloud GPUs, using serving software such as vLLM, Ollama or llama.cpp. Open-weight families include Llama, Mistral, Qwen, Gemma and, since August 2025, OpenAI's gpt-oss, whose larger version runs on a single 80 GB GPU. Treat every name here as an example: the list changes every few months. Judge the choice on four questions, then on who will run it.

The cost comparison has a shape worth drawing. API cost rises with every answer; your own server is a flat monthly cost that steps up when you add machines. The point where they cross is the break-even: fixed monthly cost ÷ API cost per answer.

Privacy is often the real reason companies self-host. In April 2023 staff at Samsung accidentally leaked internal data into ChatGPT; on 1 May Samsung restricted generative AI tools on company devices until it had safeguards, and was reported to be building in-house tools. There are middle paths too: an API under a business agreement with limited retention and a processing region you approve; an open-weight model hosted for you by a cloud provider in your region; or a hybrid that sends the few sensitive question types to your own server. Whatever you choose, keep the model behind one interface in your code and keep your test set, so switching later is a test you run, not a rewrite (sd-34 shows the engineering).