One answer is many steps

Three months after launch, you take over Mangrove's app assistant. It answers about 15,000 customer conversations a day (approximate) about products, allergens, substitutes and delivery, with about 2.5 model calls per conversation. When a parent complains that it called a cereal bar nut free, the team today has log lines: one from the API ("POST /assistant 200 4.1 s"), one from the retrieval service, one from the model client with token counts. Nothing reliably says which lines belong to that one answer. The same customer asked three questions in a minute, and thousands of other customers were asking at the same time.

You met the fix in Observability: logs, metrics and traces and used it for Ask Ops in Shipping an agent safely. A trace gives every step of one piece of work the same trace id. Each step is a span with a parent, so the steps form a tree. Context propagation carries the trace id across services, so the retrieval service's span joins the API's trace.

LLM features need traces more than most code, for four reasons:

• Failures look like success. A wrong answer comes back with HTTP 200 and a normal finish reason. Status codes can't tell you about quality; only the facts about what went in can: which documents, which prompt version, which model.
• Cost varies per call. One call reads 3,000 tokens, the next 40,000. Cost lives on the call, so it must be recorded on the call.
• The same input can give a different output. You can't always reproduce a bad answer later. The trace of the real one is often all you have.
• Your most important dependency can change behind a name. If you call a model alias such as "latest", the provider can move it to a new model version and your code never changes.

Choose the unit first. For a chat assistant, use one trace per conversation turn: one customer message and everything that happens until the answer is shown. A whole conversation can last twenty minutes while the customer cooks dinner; a trace that long is hard to view, and most tools sample whole traces, so you would keep or drop a whole conversation at once. Put a conversation id on every turn's root span so you can line the turns up again.

The span tree of one turn

A customer asks "Is the oat and honey bar in stock, and does it have nuts?" Here is the turn as spans. The root span turn is the server span for the request, with the feature name and conversation id. Under it, in order:

1. embed query, about 40 ms: the question becomes a vector.
2. vector search, about 80 ms: the 8 closest catalogue and help-page chunks come back. The span records their document ids.
3. model call 1, about 1.9 s: the large model reads about 5,000 input tokens (system prompt and tool definitions 3,000, chunks 1,500, conversation 500) and asks for the check_stock tool.
4. check_stock, about 120 ms: your code runs the tool. It is a child of the turn, a sibling of the model calls, because the model only asked; your code did the work.
5. model call 2, about 1.6 s: the model reads everything again plus the tool result and writes the answer, about 250 tokens.
6. output check, about 30 ms: code checks the answer, for example that any allergen claim matches the product's catalogue record. It records pass or blocked.

Read the waterfall from the top. About 92% of the time is in the two model calls, and a turn that uses a tool pays for the 5,000-token prompt twice. Retrieval is fast; making it faster would save almost nothing. Within a model call, the time until the first token grows with the input length and any queue at the provider, and the rest grows with the output length. When you stream, record the time to first token as its own attribute, because that is what the customer feels first.

Two details decide whether the waterfall tells the truth. First, with streaming, the span must end when the last chunk arrives. A common bug ends the span when the client returns the stream object, so every model call looks like 300 ms. Second, every retry is its own child span. When the provider answers 429 (too many requests) and your client waits two seconds and tries again, you want to see two model-call spans, the first with error status, instead of one mysterious 4-second call. Tool timeouts work the same way: the check_stock span ends with status error and an attribute saying timeout.

Attributes: the questions you can ask later

A span is only as useful as its attributes. OpenTelemetry's semantic conventions for generative AI give the common ones agreed names, so any tool can read them. Their status is still Development: names have changed between versions and can change again. For example, recent versions use gen_ai.provider.name where older ones used gen_ai.system. Pin the version you follow, write it down, and put your own fields under your own prefix.

Record both models. The request model is what your code asked for; most providers return the exact version that answered in every response. If you record only the request, a provider moving an alias to a new version is invisible. With both, one chart of gen_ai.response.model per day shows the hour it changed.

Compute cost on the span. Multiply tokens by the price of the model that answered, from a price table kept in the repository. The illustrative prices in this course are $3 per million input tokens and $15 per million output tokens for the large model, $0.80 and $4 for the small one, with cached input reads at about 10% of the input price (illustrative; check your provider's current price page). One of today's calls costs 5,000 × $3/M + 250 × $15/M ≈ $0.0188. If the answering model has no row in the price table, record that, and alert on it: a model you have never priced is a model you have never evaluated. Providers report cached tokens differently (some include them in the input count, some report them separately), so normalise them in your client wrapper and document what input_tokens means.

Ids, not text. Document ids such as art-88412#allergens, the prompt template version and the catalogue snapshot date let you rebuild most of what the model saw without storing any of it in the trace. Keep prompts in the repository with versions, as you did for the eval suite in Evals: knowing if it got better, so "it got worse on Tuesday" leads to what changed on Tuesday.

Feedback arrives later. A thumbs down comes minutes after the answer, in a different request. Send the trace id to the app with each answer; when feedback arrives, record it with that trace id (a small span or a row in a feedback table). Do the same for a re-ask within 10 minutes and a hand-off to a human agent. These are the user signals the online evals in Evals that never stop start from.

One rule about cardinality: ids with millions of values (conversation id, document ids) belong on spans, never as labels on metrics. Each metric label value becomes its own time series, and 15,000 new conversation ids a day will break most metrics systems and their bill.

Prompt and answer text: capture with care

Customers type more than you expect: "my son Arif is allergic to peanuts, can you deliver to 14 Lake Road, call me on…". A message about a child's allergy is health information about a named person. Traces are read by many engineers, kept for weeks, and often sent to a separate system. So treat content as the most sensitive thing in the pipeline:

• Off by default. The GenAI conventions treat message content (in recent versions, attributes such as gen_ai.input.messages and gen_ai.output.messages) as opt-in, and instrumentation libraries don't record it unless you turn it on. Keep it off in production.
• Ids first. Most debugging needs only ids, versions, token counts and statuses. With the document ids, the catalogue snapshot and the template version you can rebuild the prompt except the customer's own words.
• When you need text, capture a sample, redacted twice. Redact in the service before export (phone numbers, email addresses, card-like numbers, street addresses), then again in the collector as a safety net. Keep it in a separate store with stricter access (named people, every read logged).
• Short retention. For example, content 7 days, spans without content 30 days, the slim per-call rows without text 13 months for trends. Decide with whoever owns data protection, and write it down.

A collector is where those rules live. Every service's OpenTelemetry SDK batches its spans and sends them over OTLP to the collector you run. The collector deletes or masks sensitive attributes, decides which traces to keep, and exports them to the trace store. Putting the rules in one place means a new service or a third-party library can't forget them, and you can change trace store without touching application code.

Sampling, and the first dashboards

The assistant makes about 37,500 model calls a day (15,000 conversations × 2.5). With retrieval, tool and check spans that is roughly 150,000 spans a day, and if a sample carries prompt text, each model-call span holds about 20 KB. You don't need all of it, but you need the right part of it.

Head sampling decides at the start of the turn, for example a random 10%. It is cheap and blind: it drops 90% of the errors too. Tail sampling waits until the turn's spans have arrived, then decides. For an LLM feature, keep every trace that has an error, every slow turn (over 8 s), every expensive one (over 12,000 input tokens), every answer the output check blocked, plus 10% of the rest. Tail sampling needs all of a trace's spans at the same collector instance, so with several instances you route spans by trace id first (the collector has a load-balancing exporter for this).

A thumbs down arrives after the sampler has decided. Two answers work. The slim per-call rows keep the trace id, document ids, model and prompt version for every turn, which explains most bad answers even when the full trace is gone. Some teams also keep every full trace for a day or two before sampling down, so a turn with negative feedback can still be pinned.

Now the first dashboards. Build them from the unsampled per-call rows, never from the sampled trace store:

• Cost per feature per day against the budget line. A $10,000 monthly budget is about $330 a day.
• Input tokens per call as a histogram, with the cache hit rate (cached input tokens divided by input tokens). A floor where no call is below about 3,000 tokens is the stable prefix showing up on every call.
• p95 by step: embed, search, each model call, each tool, and the full answer, plus p95 time to first token.
• Errors and refusals: provider errors and timeouts per hour, answers cut off (finish reason length), tool errors, output-check blocks, and refusals flagged by the output check.
• Model mix: calls per gen_ai.response.model per day, and the share sent to each model by the router.

Then add alerts that someone will act on: a new value of the answering model, a call with no price, daily cost above 120% of the budget line, p95 full answer over 8 s for 15 minutes.

When one answer is wrong, debug it from its trace in a fixed order. Start from the thumbs down or complaint, open the trace by its id, and read top down. What did retrieval return, and were those the right products? Which prompt version and which model answered? What did each tool return? Did the output check run, and what did it check? Find the first wrong step, as you did for agents in Evaluating agents; everything after it is a consequence. If no attribute can tell you whether a step was right, that missing attribute is your next instrumentation task.