Observability: logs, metrics and traces
When a store manager says "the app is slow today", you need to answer three questions fast: is it really slow, for whom, and where does the time go. Logs, metrics and traces each answer one of them. This lesson wires all three into a NestJS API and adds an alert that wakes someone only when users are hurting.
By the end you can
- Say which question logs, metrics and traces each answer, and pick the right one
- Write structured JSON logs with nestjs-pino and one request id across every line
- Expose RED metrics with a prom-client histogram and read them in Prometheus
- Add OpenTelemetry tracing and read a trace waterfall
- Write an alert rule tied to your SLO, and avoid alert fatigue
The everyday picture
A store has three ways to know what happened today. The daily sales report gives numbers: 1,240 bills, average basket $16, sales down 30% after 6 pm. It is small and cheap, and it tells you something is wrong, but not why. The receipt trail has one paper for every sale: time, cashier, items, payment method. It has every detail, but there are thousands, so you only read the few you need. The CCTV lets you follow one customer from the door to the counter and see where they waited: 2 minutes picking rice, 9 minutes in the queue at counter 3, 4 minutes waiting for a mobile wallet payment to confirm.
Software has the same three tools. Metrics are the sales report. Logs are the receipts. Traces are the CCTV that follows one request. A good team uses all three together: the report says "something is wrong since 6 pm", the CCTV shows "people wait at payment", and the receipts show "wallet timeouts on these bills".
Monitoring and observability
Monitoring answers questions you thought of in advance: is the API up, is the disk full, is the error rate above 2%. Observability is about questions you did not plan for: "why are goods received note (GRN) submits slow only for stores in one city, only since Tuesday?" You can answer that without shipping new code only if the system already sends out rich data. Monitoring is part of observability. You need both.
The data comes as three signals. They differ in what they hold and what they cost:
| Signal | What one item is | Answers | Cost |
|---|---|---|---|
| Metrics | A number over time, such as requests per second | Is something wrong? Since when? How much? | Low and fixed. Does not grow with traffic |
| Logs | One event with details, such as "GRN 8812 saved, 42 lines" | What exactly happened to this request or this user? | Grows with traffic. Usually the biggest bill |
| Traces | One request's path, split into timed steps | Where did the time go? Which service failed? | Medium. Usually you keep a sample |
Quick check
Store managers say "the app has been slow since lunch". Which signal tells you fastest whether that is true, and since when?
A latency metric over time shows at a glance whether it got worse and exactly when. Logs and traces hold the detail for single requests, so they are the next step once you know what to look for.
Logs: structured, with one request id
A log line like Saved GRN for store is useless at 3 a.m. Which GRN? Which store? Which request? A structured log writes one JSON object per line, with named fields: {"level":"info","msg":"GRN saved","grnId":8812,"siteId":27,"reqId":"7f3c…"}. A log tool can then filter and count by any field, the way you would query a table.
The most important field is the request id. In a typical setup (see Load balancers), Nginx adds an X-Request-Id header to every request. The API puts that id on every log line it writes for the request, sends it on to any service it calls, and returns it in the response. One search for that id then shows the whole story, across all API copies.
Rules that keep logs useful and safe:
- Use levels.
errormeans someone may need to act.warnis unusual but handled.infois a normal business event.debugis off in production. If every 404 is an error, nobody reads errors. - Write to stdout. The app prints JSON lines. Docker or the platform collects them and ships them to a log store such as Grafana Loki or OpenSearch. The app never writes log files or talks to the log store itself.
- Never log secrets or personal data. No passwords, tokens, OTPs or full card numbers. Customer phone numbers only if you truly need them, and masked. Use the logger's redact option so a mistake in one place cannot leak.
- Log events, not loops. One line per GRN saved, not one per GRN line. Logs are often the largest observability cost, and they grow with traffic.
Metrics: RED for services, USE for resources
A metric is a number tracked over time. There are three basic kinds. A counter only goes up (total requests). A gauge goes up and down (jobs waiting in a queue). A histogram counts values into buckets: how many requests finished under 50 ms, under 100 ms, under 300 ms, and so on.
Two short checklists tell you which metrics to have. For every service that answers requests, use RED: Rate (requests per second), Errors (share that fail), Duration (latency as a histogram). For every resource, such as CPU, a database connection pool, a disk or a queue, use USE. Utilization is how busy it is, such as 70% CPU. Saturation is how much work is waiting, such as 40 requests queued for a database connection. Errors are failures. Saturation is the one people forget, and it is often the first warning. A pool at 100% use with nobody waiting is fine. A pool with 200 requests waiting is an outage starting.
Why a histogram for latency, and not the p95 computed inside each server? Because you cannot combine percentiles. If three API copies report p95 values of 120, 140 and 900 ms, the average of those (387 ms) is not the p95 of all requests. Histogram buckets are plain counts, so Prometheus can add the buckets from all copies and then compute one true p95. Choose bucket edges around your SLO (see Measuring speed and reliability: p99, SLOs and nines). If the target is "95% under 300 ms", put a bucket edge at exactly 0.3 seconds, and the share of good requests is one division.
Each different set of labels creates a separate time series. route="/v1/grns/:grnId" and status_code="200" are fine: a few dozen routes times a few status codes. A label with the order id, user id or full URL creates millions of series and can bring Prometheus down. This is called a cardinality explosion. Put ids in logs and traces, never in metric labels.
Prometheus collects metrics by scraping: every 15 seconds it calls GET /metrics on each API copy and stores the numbers. Grafana draws dashboards from it.
Quick check
The database connection pool is 100% in use and 200 requests are waiting for a connection. Which USE number is the alarm here?
Work piling up in a queue is saturation, and it is often the first sign of an outage. Full utilization alone can be healthy: a busy pool with nobody waiting is doing its job. There may be no errors yet, because requests are still waiting rather than failing.
Traces: one request, step by step
A trace is the journey of one request. It is made of spans. Each span is one step with a start time, a duration, a name and a parent. The first span covers the whole request. Each database query, Redis call or outgoing HTTP call becomes a child span. Drawn on a timeline, a trace is a waterfall, and the slow step is the longest bar.
POST /v1/orders. The top span is the whole request; the five spans below are its children. Without the trace, the team might add database indexes. The waterfall shows that 1,632 of the 1,840 ms is spent waiting for the payment API.When the API calls another of your services, the trace must follow. The caller adds a traceparent header (the W3C Trace Context standard) with the trace id and its own span id. The next service reads it and makes its spans children of that span. This passing-along is called context propagation, and the instrumentation libraries do it for you.
OpenTelemetry (OTel) is the open standard and set of libraries for producing traces, metrics and logs. Its Node.js auto-instrumentation patches http, Express, NestJS, pg, ioredis and pino when the process starts, so you get spans without touching your services. It sends data in the OTLP format to any backend: Jaeger or Grafana Tempo if you host it yourself, or a managed service. Because the format is standard, you can change the backend later without changing app code.
Storing every trace of a busy API is expensive. Sampling keeps a share, such as 10%. The simple kind decides at the start of the request. Keeping every failed or slow trace needs tail sampling in an OpenTelemetry Collector, which waits until the trace ends before deciding. Start with head sampling at 10 to 100% while traffic is small.
The best part comes when the signals link. With the pino instrumentation, every log line written during a traced request gets trace_id and span_id fields automatically. From a slow bar in the waterfall you jump to its logs, and from an error log you jump to its trace.
Quick check
A trace shows a 1.8-second checkout request, and 1.6 seconds of it is one child span for the call to the payment provider. What do you look at next?
The waterfall already says where the time goes, so fix that step. The database spans add up to about 100 ms, so indexes would barely help. More API copies do not make one slow outside call any faster.
Alerts that mean something
An alert interrupts a person, maybe at 3 a.m. So every alert must mean "users are hurting now, or soon will be, and a human can do something". Alert on symptoms users feel, tied to your SLO: error rate, latency, freshness of synced sales. Do not page on causes like CPU at 85%. High CPU with happy users is a good use of a server you pay for. Put causes on dashboards, where they help you find the reason once a symptom alert fires.
Too many alerts cause alert fatigue: people mute the channel, and then miss the real one. Rules that prevent it:
- Two levels only. Page (phone, now) for user-facing problems. Ticket (look tomorrow) for slow trends such as a disk that will be full in 10 days.
- Wait before firing. A
for: 5mclause means the condition must stay true for 5 minutes, so one bad scrape does not wake anyone. - Guard low traffic. At 2 a.m., 1 failure out of 3 requests is a 33% error rate. Require a minimum request rate before paging.
- Every alert has a runbook. A runbook link says what it means and what to check first.
- Review alerts monthly. Delete any alert that fired and needed no action.
A more advanced form alerts on the error budget's burn rate: page when the budget is being used up 10 or more times faster than the allowed pace. The Google SRE workbook describes it well. A plain threshold with for: is a good start.
Quick check
At 2 a.m. the API gets 3 requests a minute, and 1 of them fails. Which rule avoids waking someone for that?
At low traffic one failure is a huge percentage, and it can stay that way for 5 minutes, so the hold alone does not help. A minimum request rate makes the alert ignore the quiet hours. Paging on every error line is the fastest route to alert fatigue.
Build it
Add JSON logs with a request id, RED metrics, OpenTelemetry traces and a Prometheus alert to a NestJS 10 API, then run Prometheus, Grafana and Jaeger next to it. The steps need Node and Docker on your machine. The two ideas at the core, computing a percentile and deciding when an alert should page, are exercises you can run in the browser in the Practice section below.
-
Install the packages
nestjs-pino 4.x supports NestJS 8 to 11. In 2026 prom-client was republished as
@prometheus-io/client(Node 22 or later). The parts used below (Histogram,register,collectDefaultMetrics) work the same in both; read its changelog before switching.Terminalnpm i nestjs-pino pino-http pino prom-client npm i @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \ @opentelemetry/exporter-trace-otlp-http @prisma/instrumentation -
Log JSON with one request id per request
genReqIdreuses the id Nginx sent, or makes a new one, and returns it to the client.redactreplaces secrets with[Redacted]wherever they appear. Health checks and scrapes are not logged, because they would be most of the volume.TypeScript// src/app.module.ts import { randomUUID } from 'node:crypto'; import { Module } from '@nestjs/common'; import { LoggerModule } from 'nestjs-pino'; import { stdTimeFunctions } from 'pino'; @Module({ imports: [ LoggerModule.forRoot({ pinoHttp: { level: process.env.LOG_LEVEL ?? 'info', timestamp: stdTimeFunctions.isoTime, // "2026-10-06T20:59:16.471Z" genReqId: (req, res) => { const incoming = req.headers['x-request-id']; const id = typeof incoming === 'string' && incoming.length <= 64 ? incoming : randomUUID(); res.setHeader('X-Request-Id', id); // the app can show it on errors return id; }, customProps: () => ({ service: 'ops-api', env: process.env.NODE_ENV ?? 'dev' }), customLogLevel: (_req, res, err) => err || res.statusCode >= 500 ? 'error' : res.statusCode >= 400 ? 'warn' : 'info', serializers: { // keep request logs short req: (req) => ({ id: req.id, method: req.method, url: req.url }), res: (res) => ({ statusCode: res.statusCode }), }, redact: ['req.headers.authorization', 'req.headers.cookie', '*.password', '*.otp'], autoLogging: { ignore: (req) => req.url === '/metrics' || !!req.url?.startsWith('/health') }, }, }), ], }) export class AppModule {}Make Nest's own logger go through pino, so startup messages are JSON too:
TypeScript// src/main.ts import { NestFactory } from '@nestjs/core'; import { Logger } from 'nestjs-pino'; import { AppModule } from './app.module'; async function bootstrap() { const app = await NestFactory.create(AppModule, { bufferLogs: true }); app.useLogger(app.get(Logger)); app.enableShutdownHooks(); await app.listen(3000); } void bootstrap(); -
Log business events with fields
Keep using Nest's normal
Logger. Pass an object first and the message second. The request id is added for you, even deep inside a service, because nestjs-pino tracks the current request with Node's AsyncLocalStorage.TypeScript// src/grn/grn.service.ts import { Injectable, Logger } from '@nestjs/common'; @Injectable() export class GrnService { private readonly logger = new Logger(GrnService.name); async submit(grnId: number, siteId: number, lines: number) { // ... save the GRN ... this.logger.log({ grnId, siteId, lines }, 'GRN submitted'); } }Output, one JSON object per line. The
trace_idfields appear once tracing is on (step 5):Output{"level":30,"time":"2026-10-06T20:59:16.471Z","req":{"id":"7f3c9a2e-1b4d-4c8e-9a51-2f6d0b8e4c11","method":"POST","url":"/v1/grns/8812/submit"},"service":"ops-api","env":"dev","trace_id":"f2e2fc693b8b609819e923a12da4981f","span_id":"5394a3a85723a81b","context":"GrnService","grnId":8812,"siteId":27,"lines":42,"msg":"GRN submitted"} {"level":50,"time":"2026-10-06T20:59:16.477Z","req":{"id":"7f3c9a2e-1b4d-4c8e-9a51-2f6d0b8e4c11","method":"POST","url":"/v1/grns/8812/submit"},"service":"ops-api","env":"dev","trace_id":"f2e2fc693b8b609819e923a12da4981f","res":{"statusCode":503},"responseTime":8,"msg":"request errored"}Pino writes the level as a number: 30 is info, 40 warn, 50 error. In development, pipe the output through
npx pino-prettyto read it as text. -
Expose RED metrics with a histogram
One histogram gives all three RED numbers: its
_countis the rate, filtering_countby status gives errors, and its buckets give duration. Label by the route pattern, never the real URL, and useunmatchedfor 404s so scanners hitting random paths cannot create new series.TypeScript// src/metrics/metrics.ts import { Controller, Get, Header, Injectable, NestMiddleware } from '@nestjs/common'; import type { NextFunction, Request, Response } from 'express'; import { collectDefaultMetrics, Histogram, register } from 'prom-client'; collectDefaultMetrics(); // process CPU, memory, event-loop lag, GC export const httpDuration = new Histogram({ name: 'http_request_duration_seconds', help: 'HTTP request duration in seconds', labelNames: ['method', 'route', 'status_code'] as const, buckets: [0.025, 0.05, 0.1, 0.2, 0.3, 0.5, 1, 2.5, 5], // 0.3 = our SLO edge }); @Injectable() export class MetricsMiddleware implements NestMiddleware { use(req: Request, res: Response, next: NextFunction) { const end = httpDuration.startTimer(); res.once('finish', () => { end({ method: req.method, route: req.route?.path ?? 'unmatched', // '/v1/grns/:grnId/submit' status_code: String(res.statusCode), }); }); next(); } } @Controller('metrics') export class MetricsController { @Get() @Header('Content-Type', register.contentType) scrape(): Promise<string> { return register.metrics(); } }Register the controller, and apply the middleware to every route by having
AppModuleimplementNestModule:TypeScript// in AppModule: controllers: [MetricsController], and export class AppModule implements NestModule { configure(consumer: MiddlewareConsumer) { consumer.apply(MetricsMiddleware).forRoutes('*'); } }Terminalcurl -s localhost:3000/metrics | grep 'grnId/submit",status_code="200"'Outputhttp_request_duration_seconds_bucket{le="0.1",method="POST",route="/v1/grns/:grnId/submit",status_code="200"} 1810 http_request_duration_seconds_bucket{le="0.3",method="POST",route="/v1/grns/:grnId/submit",status_code="200"} 1967 http_request_duration_seconds_bucket{le="+Inf",method="POST",route="/v1/grns/:grnId/submit",status_code="200"} 2000 http_request_duration_seconds_sum{method="POST",route="/v1/grns/:grnId/submit",status_code="200"} 183.4 http_request_duration_seconds_count{method="POST",route="/v1/grns/:grnId/submit",status_code="200"} 2000 # 1967 of 2000 under 0.3 s: 98.35% good on latencyDo not put
/metricson the public internet. Block it in Nginx, or serve it on a separate internal port. -
Turn on OpenTelemetry tracing before the app loads
This file must run first. Prisma does not use the
pgdriver by default, so it needs its own instrumentation. On Prisma 6.1 and later that is all; on Prisma 5, also addpreviewFeatures = ["tracing"]to the generator block.TypeScript// src/tracing.ts import { NodeSDK } from '@opentelemetry/sdk-node'; import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'; import { PrismaInstrumentation } from '@prisma/instrumentation'; const sdk = new NodeSDK({ serviceName: process.env.OTEL_SERVICE_NAME ?? 'ops-api', traceExporter: new OTLPTraceExporter(), // reads OTEL_EXPORTER_OTLP_ENDPOINT instrumentations: [ getNodeAutoInstrumentations(), // http, express, nestjs, pg, ioredis, pino... new PrismaInstrumentation(), // one span per Prisma query ], }); sdk.start(); process.on('SIGTERM', () => { // send the last spans before exit sdk.shutdown().finally(() => process.exit(0)); });Terminal# package.json "start:prod": load tracing first with -r node -r ./dist/tracing.js dist/main.js # settings, read by the SDK from the environment OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318 # /v1/traces is added for you OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.2 # keep 20% of new tracesparentbasedmeans: if the caller already decided to sample this trace, follow that decision, so a trace is never half-recorded across services. -
Run Prometheus, Grafana and Jaeger next to the API
Add these services to the Docker Compose file that runs your API (the one from Scaling up vs scaling out works). The Jaeger image name and tag come from the Jaeger getting-started page; check it for the current version.
YAML# docker-compose.yml (added to api, nginx, redis, db) services: api: command: ["node", "-r", "./dist/tracing.js", "dist/main.js"] environment: OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4318 OTEL_SERVICE_NAME: ops-api prometheus: image: prom/prometheus:v3.5.0 volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro - ./alerts:/etc/prometheus/alerts:ro ports: ["127.0.0.1:9090:9090"] # UI on this machine only grafana: image: grafana/grafana:12.0.0 ports: ["127.0.0.1:3001:3000"] # add Prometheus at http://prometheus:9090 jaeger: image: cr.jaegertracing.io/jaegertracing/jaeger:2.21.0 ports: ["127.0.0.1:16686:16686"] # trace UI; OTLP 4318 stays internalYAML# prometheus.yml global: scrape_interval: 15s evaluation_interval: 15s rule_files: - /etc/prometheus/alerts/*.yml scrape_configs: - job_name: ops-api metrics_path: /metrics dns_sd_configs: # "api" resolves to every copy of the API - names: ['api'] type: A port: 3000 -
Write the dashboard queries
These PromQL queries give the RED row of a service dashboard, plus two USE numbers for the Node process.
rate(...[5m])means "per second, averaged over the last 5 minutes". Buckets are summed across copies before the percentile is computed.Output# Rate: requests per second, per route sum by (route) (rate(http_request_duration_seconds_count{job="ops-api"}[5m])) # Errors: share of requests that ended in 5xx sum(rate(http_request_duration_seconds_count{job="ops-api",status_code=~"5.."}[5m])) / sum(rate(http_request_duration_seconds_count{job="ops-api"}[5m])) # Duration: p95 per route, across all copies histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket{job="ops-api"}[5m]))) # SLI: share of requests under 300 ms sum(rate(http_request_duration_seconds_bucket{job="ops-api",le="0.3"}[5m])) / sum(rate(http_request_duration_seconds_count{job="ops-api"}[5m])) # USE for the process: CPU cores used, and event-loop lag (saturation) rate(process_cpu_seconds_total{job="ops-api"}[5m]) nodejs_eventloop_lag_p99_seconds{job="ops-api"} -
Alert when more than 2% of requests fail for 5 minutes
The
andline is the low-traffic guard: no page unless there is at least one request every 2 seconds.YAML# alerts/ops-api.yml groups: - name: ops-api rules: - alert: OpsApiHigh5xxRate expr: | ( sum(rate(http_request_duration_seconds_count{job="ops-api", status_code=~"5.."}[5m])) / sum(rate(http_request_duration_seconds_count{job="ops-api"}[5m])) ) > 0.02 and sum(rate(http_request_duration_seconds_count{job="ops-api"}[5m])) > 0.5 for: 5m # must stay true for 5 minutes labels: severity: page # Alertmanager sends "page" to the on-call phone annotations: summary: "ops-api: {{ $value | humanizePercentage }} of requests failing" runbook_url: "https://wiki.example.com/runbooks/ops-api-5xx" - alert: OpsApiSlowP95 expr: | histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="ops-api"}[5m])) ) > 0.3 for: 15m labels: severity: ticket # look at it in working hours annotations: summary: "ops-api p95 is {{ $value | humanizeDuration }} (SLO 300 ms)"Check the file before you load it.
promtoolships with Prometheus;promtool test rulescan also replay fake series against your rules, which is how you prove the low-traffic guard works.Terminaldocker compose exec prometheus promtool check rules /etc/prometheus/alerts/ops-api.ymlOutputChecking /etc/prometheus/alerts/ops-api.yml SUCCESS: 2 rules foundPrometheus only decides that an alert is firing. A separate Alertmanager service groups alerts and sends them to Slack, email, SMS or a phone app; add it when the rules are stable.
| Need | Self-hosted option | What it is good at |
|---|---|---|
| Metrics and alerts | Prometheus + Alertmanager, Grafana for dashboards | Cheap, standard, PromQL everywhere |
| Logs | Grafana Loki, or OpenSearch | Loki indexes only labels, so it is cheap; OpenSearch has full-text search |
| Traces | Jaeger or Grafana Tempo, fed by OpenTelemetry | Waterfalls, service maps |
| Errors and crashes | Sentry (also offered as a hosted service) | Grouped stack traces, release tracking, React Native crash reports |
Bring it to your team
Ask these before a feature goes live, not after the first outage.
- If this endpoint gets slow tomorrow, which dashboard shows it, and which trace shows why?
- Do every log line and every error screen carry the request id, so support can find one store's failed request?
- What are the RED numbers for this service, and the USE numbers for the database pool and the queue?
- Which alert fires if users are hurting, and does it link to a runbook?
- What personal data could end up in these logs, and where is it redacted?
- How long do we keep logs and traces, and what does that cost per month at festival-peak traffic?
Red flags
- "We will find out from the stores" as the plan for detecting outages.
- Alerts on CPU, memory or every error log line, with nobody acting on most of them.
- User ids, order ids or full URLs as metric labels.
- Averages on the latency dashboard and no percentiles.
- Logs that contain tokens, passwords or OTPs.
Practice
Compute a percentile
Write percentile(values, p) using the nearest-rank method: sort a copy of the latencies from fastest to slowest, compute rank = ceil(p × n / 100), where n is the number of values, and return the value at that rank (counting from 1). Return null for an empty list, and do not change the array you were given. The starter returns the average, which is the number this lesson warns you not to put on a latency dashboard.
function percentile(values, p) {
// The average hides the slow requests. Replace it with nearest-rank.
return values.reduce((sum, v) => sum + v, 0) / values.length;
}test('p50 of three values is the middle one', () => {
expect(percentile([300, 100, 200], 50)).toBe(200);
});
test('p95 of 1..100 ms is 95 ms', () => {
const ms = Array.from({ length: 100 }, (_, i) => i + 1);
expect(percentile(ms, 95)).toBe(95);
});
test('p99 of 20 requests is the slowest one', () => {
const ms = [80, 95, 110, 90, 85, 100, 120, 105, 99, 88, 92, 101, 97, 115, 91, 89, 2400, 94, 96, 98];
expect(percentile(ms, 99)).toBe(2400);
});
test('does not reorder the input array', () => {
const ms = [300, 100, 200];
percentile(ms, 50);
expect(ms).toEqual([300, 100, 200]);
});
test('an empty list gives null', () => {
expect(percentile([], 95)).toBe(null);
});
Show a solution
function percentile(values, p) {
if (values.length === 0) return null;
const sorted = [...values].sort((a, b) => a - b); // copy, then sort as numbers
const rank = Math.ceil((p * sorted.length) / 100); // nearest rank, 1-based
return sorted[Math.max(rank, 1) - 1];
}Two traps: sort() without a compare function sorts as text, so 2400 would come before 300; and sort() changes the array in place, which is why the solution copies it first.
Decide when an alert should page
Write shouldPage(minutes, options), a simplified version of the lesson's alert rule. minutes is a list of one-minute samples, oldest first, each { requests, errors }. options has maxErrorRate (default 0.02), minRps (default 0.5 requests per second) and forMinutes (default 5). Return true only when each of the last forMinutes minutes has an error rate strictly above maxErrorRate and at least minRps requests per second. With fewer samples than forMinutes, return false.
function shouldPage(minutes, { maxErrorRate = 0.02, minRps = 0.5, forMinutes = 5 } = {}) {
// Pages on one bad minute, even at 2 a.m. with almost no traffic.
const last = minutes[minutes.length - 1];
return last.errors / last.requests > maxErrorRate;
}const bad = { requests: 600, errors: 30 }; // 10 requests a second, 5% failing
const ok = { requests: 600, errors: 3 };
test('pages after 5 bad minutes with real traffic', () => {
expect(shouldPage([bad, bad, bad, bad, bad])).toBe(true);
});
test('4 bad minutes are not enough', () => {
expect(shouldPage([ok, bad, bad, bad, bad])).toBe(false);
expect(shouldPage([bad, bad, bad, bad])).toBe(false);
});
test('a good minute resets the clock', () => {
expect(shouldPage([bad, bad, bad, ok, bad, bad, bad, bad])).toBe(false);
});
test('a quiet night with 1 failure in 3 requests does not page', () => {
expect(shouldPage(Array(10).fill({ requests: 3, errors: 1 }))).toBe(false);
});
test('exactly 2% is not above the line', () => {
expect(shouldPage(Array(5).fill({ requests: 600, errors: 12 }))).toBe(false);
});
test('uses the options you pass in', () => {
const m = { requests: 120, errors: 20 };
expect(shouldPage([m, m], { maxErrorRate: 0.1, minRps: 1, forMinutes: 2 })).toBe(true);
});
Show a solution
function shouldPage(minutes, { maxErrorRate = 0.02, minRps = 0.5, forMinutes = 5 } = {}) {
if (minutes.length < forMinutes) return false; // not held long enough yet
return minutes.slice(-forMinutes).every((m) =>
m.requests / 60 >= minRps && // the low-traffic guard
m.errors / m.requests > maxErrorRate); // the symptom
}Check yourself
Three API copies report p95 latencies of 120, 140 and 900 ms. How do you get the true p95 across all requests?
Percentiles cannot be averaged or combined, because each copy served a different number of requests with a different spread. Bucket counts are plain counts, so they add up, and histogram_quantile then gives one correct p95. The maximum only tells you the worst copy.
Which of these should page the on-call engineer at 3 a.m.?
Page on symptoms users feel, held for a few minutes, with a traffic guard. High CPU with fast responses is not a problem. One error log is normal noise. A disk at 70% is a ticket for working hours, not a phone call.
A developer adds a user_id label to the request histogram "to find slow users". What happens?
Each unique label combination is its own series. With 10 buckets, 20 routes and 5,000 users, that is about a million series. Find slow users through traces and logs, which are built for per-request detail.
Flashcards
Key takeaways
- Metrics tell you something is wrong, traces show where the time goes, logs show what happened to one request.
- Log JSON to stdout, with one request id on every line and secrets redacted.
- Track RED for services and USE for resources. Use histograms for latency and keep labels low-cardinality.
- Load OpenTelemetry before the app, and link traces and logs by
trace_id. - Page only on user-facing symptoms tied to the SLO, held for a few minutes, with a runbook.
Sources
- Monitoring distributed systems, Site Reliability Engineering, Google
- Alerting on SLOs, The Site Reliability Workbook, Google
- Histograms and summaries, Prometheus documentation
- OpenTelemetry for JavaScript, OpenTelemetry documentation
- The USE method, Brendan Gregg