Questions you did not plan for

In Observability: logs, metrics and traces you met the three signals and RED dashboards. Those dashboards answer the questions you thought of in advance. An outage asks new ones: is it only card payments, only app version 5.2, only stores in one city, only orders with more than 40 items? You are observable when you can answer such a question with the data you already send, without writing and deploying new code in the middle of the incident.

That needs high-cardinality context: fields with many possible values, such as store id, order id or app version. The trap is where you put it. In a metric, every new combination of label values creates a new time series, and the metrics system pays for each one all the time. On a span or a log line, the same field costs a few bytes on that one record.

A quick count shows the difference. Mangrove's order service has a latency histogram with about 40 route and method pairs, about 6 status codes, and 12 series per histogram (10 buckets, a sum and a count). That is 40 × 6 × 12 = 2,880 series per pod, or 34,560 across 12 pods. Add a payment_method label with 3 values (card, wallet, cash on delivery) and you have about 104,000. Add store_id with 320 values instead and you have about 11 million series. The first is fine, the second is expensive, and the third can take down a Prometheus server.

So the team sets a cardinality budget: for example, at most 50,000 series per service, checked in code review. Prometheus can enforce it per target with sample_limit in the scrape config. The OpenTelemetry metrics SDK specification also defines a per-metric cardinality limit (2,000 by default) that folds extra combinations into one overflow series; support differs by language, so check yours. Everything above the budget goes on spans and logs, where you can group by store id later in the trace tool.

API, SDK and instrumentation

OpenTelemetry splits into two layers. The API (@opentelemetry/api in Node) is what code calls: get a tracer, start a span, set an attribute. On its own it does nothing, and every call is a cheap no-op. The SDK is what the application installs at startup. It samples, batches and exports. A shared library, such as Mangrove's payment client package, depends only on the API. The service that uses it installs the SDK and decides where data goes. So a library never forces an exporter or a vendor on the services that use it.

Instrumentation comes in two kinds. Auto instrumentation (@opentelemetry/auto-instrumentations-node) patches http, Express, NestJS, pg, ioredis and pino at startup. You get a span for every incoming request, every outgoing call and every query, with context passed along for you. It does not know your business. It cannot tell you that this request was a cash-on-delivery order with 43 items that needed two payment attempts. For that you add manual spans and attributes in the code that knows.

A span carries three kinds of detail. Attributes describe the whole operation: mangrove.payment.method = "wallet". Events mark moments inside it: "stock reserved" at 112 ms, "retry scheduled" at 2,040 ms. The status says whether it failed. Keep span names low-cardinality, such as checkout.place_order, and put ids in attributes. A name with an order id in it makes every span a group of one.

Two kinds of shared naming make data from different teams line up. Semantic conventions are standard attribute names: http.request.method, http.route, http.response.status_code. Auto instrumentation uses them, so one query works across every service. Your own attributes get your own prefix, such as mangrove., so they never collide with a future standard name. Resource attributes describe the process itself and ride on every span, metric and log it sends: service.name, service.version and the environment. The environment key was deployment.environment and is deployment.environment.name in newer convention versions; pick one and use it everywhere. Without these, staging traffic mixes into production dashboards and you cannot tell which release a slow span came from.

Following one checkout across services and the queue

A trace only helps if every service adds its spans to the same trace. That hand-off is context propagation. Over HTTP it uses the W3C traceparent header, which has four parts separated by dashes:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             |  |                                |                |
             |  trace id: 32 hex, whole journey  parent span id:  flags: bit 0 set
             version 00                          16 hex, caller   means "sampled"

The ids are lowercase hex, and an id of all zeros is invalid. The flags are a bit field. Only the lowest bit means "sampled", so 01 and 03 both mean the caller recorded this trace. With a parentbased sampler (sd-31 set one up), each service follows that bit, so a trace is never recorded in the gateway and dropped in the payment service.

HTTP and gRPC are covered by auto instrumentation. A job queue usually is not. When the order service puts a "send receipt" job on the queue, nothing carries the context unless you put it into the job yourself: propagation.inject writes a traceparent into the job data, and the worker calls propagation.extract to read it back.

Then you choose how the worker's span relates to the order. The job may run seconds or minutes later, and a batch job may handle 500 orders at once. Making it a child would stretch the checkout trace over hours, or give one span 500 parents. So the worker starts a new trace and adds a span link to the producer's span context. The trace tool shows the link as a jump between the two traces, and each trace keeps a sensible length.

Baggage is a second header that carries your own key-value pairs to every downstream service, such as mangrove.channel=app. It is not added to spans automatically, and it travels in plain headers to every service you call, including outside providers.

The Collector pipeline, sampling and cost

Services can send telemetry straight to a backend, but most teams put an OpenTelemetry Collector in between. It speaks OTLP, the standard protocol, on port 4317 for gRPC and 4318 for HTTP. Each pipeline has three stages. Receivers take data in. Processors change it in order. Exporters send it to the backends. Because the apps only know "send OTLP to the Collector", you can change sampling, scrub data or switch vendors without redeploying a single service.

Four processors matter most for checkout, and their order matters too:

• memory_limiter goes first. When the Collector nears its memory limit it refuses new data, so senders back off and retry, instead of the Collector crashing and losing everything it holds.
• attributes deletes or hashes fields that must not be stored, such as an authorization header or a raw customer id. One rule here protects every service.
• tail_sampling decides which traces to keep, once they are complete.
• batch goes near the end and groups spans into fewer, bigger sends.

Head sampling decides at the first span, in the SDK. It is cheap, but blind: at that moment nobody knows whether the checkout will fail. Sample 10% at the head and you throw away 90% of your failed checkouts too. Tail sampling waits until the trace is complete, for example 10 seconds after its first span, then applies policies: keep every trace with an error, every trace slower than 2 seconds, every trace with more than one payment attempt, and 5% of the rest.

The cost difference is large. Say the checkout path sends roughly 10,000 spans a second at the festival peak. Keeping all of them is about 36 million spans an hour. If roughly 1% of traces have errors and 1% are slow, the policies above keep about 7%, around 2.5 million spans an hour, and that 7% includes every trace you will want during an incident. The price is memory: a 10-second wait holds about 100,000 spans in the Collector, which is why memory_limiter runs before it.

Tail sampling has one hard rule: all spans of a trace must reach the same Collector, or each copy sees half a trace and decides wrongly. With several Collectors behind a round-robin load balancer, failed traces arrive with missing spans. The fix is two tiers. A first tier of Collectors uses the loadbalancing exporter with routing_key: traceID, so every span of one trace goes to the same second-tier Collector, which runs tail_sampling.

Joining the signals: exemplars and trace ids in logs

During an incident you move between signals: the metric shows that something is wrong, a trace shows where, and logs show the exact error. Two features make each jump one click.

An exemplar is a trace id attached to a metric sample. When a histogram counts a 4.2-second checkout into its slowest bucket, it can keep that request's trace id next to the count. In the OpenMetrics text format it looks like this:

http_server_request_duration_seconds_bucket{route="/v1/orders",le="5.0"} 1873 # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 4.21 1760985000.120

Grafana draws exemplars as dots on the latency panel, and clicking a dot opens that trace. One detail: with tail sampling, an exemplar may point at a trace that was dropped. Slow and failed requests are the ones you click, and your policies keep exactly those, so in practice the links work where they matter. In Prometheus, storing exemplars is a feature flag (version dependent), and SDK support differs by language.

The other jump goes from a trace to its logs. OpenTelemetry's pino instrumentation adds trace_id and span_id to every log line written while a span is active. Your log tool can then link a trace to all of its log lines, across every service, even when the request id from sd-31 was lost at a queue boundary.