What one call looks like

Most model providers offer a chat-style API: you send one HTTPS request and get one response. The field names differ by provider, but the parts are the same everywhere. The request carries system instructions (the rules: who the assistant serves, what it may say, what format to use), a list of messages, each tagged with a role (user for the customer, assistant for the model's earlier replies), and parameters such as the model name, the most tokens the answer may use, and sampling settings. Some APIs take the system instructions as a separate field; others take them as the first message with a system role.

Here is the first raw call Nadia's team made for the product-question assistant in the Mangrove app. A customer is looking at a granola and asks about nuts:

{
  "model": "your-model-id",
  "system": "You answer Mangrove Market customers' questions about one product.\nUse ONLY the product facts below. If they do not answer the question, say so.\n\nProduct facts:\nOat Crunch Granola 500 g. Ingredients: oats, honey, almonds (8%), sunflower oil.\nAllergens: contains nuts (almonds). Made in a factory that also handles peanuts.",
  "messages": [
    { "role": "user", "content": "Does this granola have nuts? My son is allergic." }
  ],
  "max_tokens": 300,
  "temperature": 0.2
}

The response carries three things your code must read, not only the text:

{
  "content": [{ "type": "text", "text": "Yes. Oat Crunch Granola contains almonds, and it is made in a factory that also handles peanuts. It is not safe for a nut allergy." }],
  "stop_reason": "end_turn",
  "usage": { "input_tokens": 1540, "output_tokens": 38 }
}

The text is the answer. The stop reason says why the model stopped: it finished, it ran out of output tokens, or (in Tools and the Model Context Protocol) it wants to call a tool. The usage block counts input and output tokens, which is exactly what you are billed for. This example follows one provider's documented shape; another returns a list of choices, each with a finish_reason. Your wrapper maps whichever shape you get into one of your own, so the rest of the app never cares.

The most important fact about the API is that it is stateless. The model keeps nothing between calls. When the customer follows up with "what about the honey one?", the app must send the system instructions, the first question, the first answer and the new question again. A conversation is a list your app keeps and resends, longer each turn. So the bill for one turn grows with the length of the conversation so far.

Tokens and the context budget

Models read and write tokens, pieces of words chosen by the model's tokenizer. In English, one token is roughly 4 characters, or about three quarters of a word, so 1,000 words is roughly 1,300 tokens. That rule of thumb breaks for other scripts. Text in Bengali, Hindi, Thai or Amharic often needs several times more tokens for the same meaning, because tokenizers were built mostly from English-heavy text. The ratio depends on the tokenizer, so never plan with the English rule. Take a sample of real customer questions in each language you serve, send them, and read the usage numbers (or use your provider's token-counting endpoint, if it has one).

The context window is the most the model can handle in one call. Current models range from tens of thousands of tokens to more than a million, depending on the model. Think of it as a budget shared by four users: your instructions, the conversation history, any data you attach (product facts now, retrieved help pages in Retrieval-augmented generation), and the answer itself. The window is the hard limit, but your own budget should usually be far smaller. Every input token is paid on every call, a longer input makes the first word slower to arrive, and models tend to use a key fact less reliably when it is buried in a very long input.

The answer has its own, separate limit: max output tokens. You set it on every request, and models also have a ceiling of their own, usually much smaller than the window. If input plus the output you ask for does not fit in the window, depending on the provider the request is rejected or the answer is cut short. If the model reaches your max output tokens, it stops mid-sentence, and the stop reason says so (length in one API, max_tokens in another). A truncated sentence looks bad. Truncated JSON breaks the app. Check the stop reason on every call before you use the text.

When history outgrows its share, you have three choices. You can drop the oldest turns (cheap and simple, and what you will write in the Practice section), summarise old turns into a short note with an extra model call, or start a fresh conversation. For a product-question assistant, dropping old turns is usually fine: questions about the granola rarely need what was said twenty messages ago. Always keep the system instructions, and never cut a turn in half, because an answer without its question confuses the model.

Sampling: temperature and top-p

A model writes one token at a time. At each step it computes a probability for every token it knows, then samples one. After "Oat Crunch Granola contains", the token "almonds" might get 70%, "oats" 20%, and thousands of others share the rest. Two settings shape that choice.

Temperature rescales the probabilities before sampling. Low temperature sharpens them, so the top token wins almost every time. High temperature flattens them, so less likely tokens get picked more often, which reads as more varied and sometimes more creative, and also more often wrong. Top-p cuts the tail instead: with top-p 0.9, the model keeps only the most likely tokens that together make up 90% of the probability and samples among them. Ranges differ by provider (temperature is 0 to 1 in some APIs and 0 to 2 in others), providers usually advise changing one of the two settings and leaving the other alone, and some newer models fix these settings or reject them, so check the model's documentation.

Pick settings by task. For extraction and classification, such as pulling allergens out of a supplier's spec sheet or labelling a question as "delivery" or "product", use temperature 0 or close to it: you want the most likely reading every time. For writing, such as a friendly reply or three recipe ideas for leftover rice, a moderate temperature gives natural, varied wording. For answers that must be correct, low settings help but are never enough on their own; the checks in Prompts and structured output and the evals in Evals: knowing if it got better do that job.

Temperature 0 is not a promise of identical output. At temperature 0 the model picks the top token, but the probabilities it computes can differ slightly from run to run. Providers process many requests together in batches, and the batch size changes the order of floating-point arithmetic, which can flip a near-tie between two tokens. One flipped token changes everything written after it. Providers also update models and serving software behind the same name. So never write a test that expects the exact same sentence twice, and never cache an answer on the assumption that a re-run would match it.

Streaming, timeouts and retries

Two numbers describe how fast an answer feels. Time to first token (TTFT) covers the network, any queue at the provider and the model reading your input; longer inputs make it longer. After that, output arrives at some tens of tokens per second, depending on the model and how busy the provider is. In Mangrove's first test the TTFT was about 0.8 seconds and output came at about 40 tokens per second, so a 200-token answer took about 0.8 + 200 ÷ 40 = 5.8 seconds in total.

Without streaming, the customer stares at a spinner for all 5.8 seconds. With streaming, the provider sends the answer in small pieces as it writes them, using server-sent events (SSE): the response stays open and carries lines starting with data:, each event ended by a blank line. The first words appear after 0.8 seconds. The total time is the same; the wait feels far shorter, because reading starts almost at once.

Streaming has costs. Structured output is not usable until it is complete: half a JSON object cannot be parsed, so the app either waits for the end or uses a parser that handles partial input. Checks on the whole answer, such as "does this allergen answer match the product facts?", cannot run before the customer has already read the first sentences. For answers where a wrong word is dangerous, buffer the reply, check it, then show it. And usage arrives at the end of the stream, if at all; some providers include it only when you ask, so make sure you still log it.

Calls also fail. Providers enforce rate limits, usually requests per minute and tokens per minute, and answer 429 Too Many Requests when you go over, often with a retry-after header saying how long to wait (Rate limiting explains the algorithms behind them). Servers also return 5xx errors when overloaded, and a call can hang. The rules:

• Set a timeout on every call, so a hung connection does not hold a customer's request forever.
• Retry only what can succeed next time: 429, 5xx, timeouts and dropped connections. A 400 (bad request, input too long) fails the same way every time.
• Back off exponentially, with jitter. Wait about 0.5, 1, 2, then 4 seconds, picking a random time up to that limit, and stop after a few attempts. Without jitter, every app server that hit the limit at 20:00:00 retries at exactly 20:00:01, and the limit is hit again. Respect retry-after when it is present.
• Make retries safe. A model call changes nothing on the provider's side except your bill and your limits, so retrying it is idempotent for data, but not free. A call that timed out may have finished and been billed anyway. The danger is in your own code: if the handler saves the answer or counts the customer's daily questions before the call succeeds, a retry does it twice. Key that work by a request ID for the question. And once streamed text is on screen, do not retry silently; a second attempt writes different words.

What a call really costs

Providers price input and output tokens separately, per million tokens, and output usually costs several times more than input. Prices change often and differ by model, so this lesson uses illustrative prices only: say $3 per million input tokens and $15 per million output tokens for a large hosted model, and $0.25 and $1.25 for a small fast model. Check your provider's current price list before you plan with real numbers.

cost of one call = input tokens x input price + output tokens x output price

Turn 1 (1,540 in, 200 out):   1,540 x $3/M + 200 x $15/M  = $0.00462 + $0.00300 = $0.00762
Three-question conversation:  input 1,540 + 1,780 + 2,020 = 5,340 tokens, output 600
                              5,340 x $3/M + 600 x $15/M  = $0.01602 + $0.00900 = $0.02502
Per question (average):       $0.02502 / 3                = $0.00834
Per 1,000 questions:                                        about $8.34

Three things stand out. Most of the input is tokens the customer never sees: rules and product facts, paid on every turn. Output is a small share of the tokens but a large share of the money, because each output token costs five times an input token here. And history makes later turns dearer, so a long chat costs much more per question than a short one.

Add what the simple formula leaves out: retries (a timed-out call may be billed), tool rounds in Tools and the Model Context Protocol (each round is a full new call), eval runs before every release, and festival peaks. Discounts for cached prompt prefixes and cheaper models for easy questions come in Cost, latency and safety. For the product manager's view of the same numbers, see How AI features work, for PMs; for the big picture of an AI feature with retrieval and evals, see Designing AI features.