1. Courses
  2. System Design
  3. Distributed Systems
  4. Stop 2

40 minAdvanced

Failure is normal: timeouts, retries, backoff and circuit breakers

Every call your service makes to another machine can be slow, fail, or never answer. This lesson gives you the five tools that stop one sick dependency from freezing everything: timeouts, careful retries, backoff with jitter, circuit breakers and bulkheads, plus the fallbacks that keep the user moving.

By the end you can

  • Put a timeout on every outgoing call, and pick its value from real latency
  • Decide which calls are safe to retry, and retry them with exponential backoff and full jitter
  • Explain how retry storms happen and stop them with retry limits and budgets
  • Draw the circuit breaker states and configure one with opossum
  • Design a fallback so checkout keeps working when the SMS or payment API is slow

The everyday picture

A store manager needs a delivery update from a supplier. He calls, and nobody picks up. He does not stand there with the phone ringing for an hour. After about thirty seconds he hangs up (a timeout). He tries again in a few minutes, not every second, because calling non-stop only jams the supplier's line further (backoff). If ten other stores are also calling, it helps if they don't all call at exactly 10:00 (jitter).

If the supplier has not answered all morning, he stops calling for a while and uses another plan: he orders from the second supplier, or puts a note on the shelf saying "coming tomorrow" (circuit breaker and fallback). After lunch, he tries one call to see if they are back. None of this is clever. It is what a sensible person does, written down as code.

The network will let you down

In 1994, engineers at Sun Microsystems listed the : wrong beliefs that people bring from single-machine programming. The first three cause most of the trouble:

  • The network is reliable. Packets get lost, connections reset, a Wi-Fi router reboots, a mobile operator drops the store's 4G for twenty seconds.
  • Latency is zero. A function call takes nanoseconds. A call to the payment API takes 100 ms on a good day and 10 seconds on a bad one (see Speed numbers and back-of-the-envelope math).
  • Bandwidth is infinite. A 5 MB response that is instant in the office takes a long time on a weak connection.

The others are: the network is secure, the topology does not change, there is one administrator, transport cost is zero, and the network is the same everywhere. The lesson for your code is short. Every remote call has three possible results: success, failure, or "I don't know". The third one is the dangerous one. A request that timed out may have done nothing, or it may have charged the customer. Your design has to handle all three.

Timeouts: never wait forever

A call with no can wait forever. Each waiting request holds memory, a connection from the pool and maybe a database transaction. When a dependency hangs, the waiting requests pile up until the whole service stops responding. One slow SMS gateway can take down checkout this way, even though checkout does not need SMS at all.

Axios has no timeout by default (timeout: 0), and Node's fetch has none either, so you must set one on every client. There are two kinds of wait to limit:

  • Connect timeout: how long to wait to open the connection. If the server is up, this takes milliseconds, so a short limit (1 to 3 seconds) catches dead hosts quickly.
  • Read timeout: how long to wait for the response once connected. This depends on the work the server does.

In Node, axios's timeout covers both, roughly: it fires if the connection is not ready in time, or if the socket is silent for that long. It is not a limit on the total time. A server that sends one byte every second never goes silent for long, so it can keep the call open far beyond the timeout. For a hard limit on the whole call, also pass signal: AbortSignal.timeout(ms). Other clients have their own options: ioredis has connectTimeout and commandTimeout, and Postgres has statement_timeout (see Running Postgres in production).

How long should the timeout be? Look at the dependency's real latency, not a guess. Start a little above its p99 on a normal day (see Measuring speed and reliability): if the payment API's p99 is 1.2 seconds, a 2 or 3 second timeout catches real problems without cutting off slow but healthy calls. Then check the budget from the user's side. If the user gives up after 5 seconds, a 10-second timeout to a dependency means you are still waiting after nobody cares about the answer.

Quick check

The loyalty service's p99 is 800 ms on a normal day, and customers at the till give up after about 4 seconds. Which timeout fits best?

Deadlines across calls

Timeouts add up through a chain of calls. Say the app gives checkout 5 seconds. Checkout calls the stock service with a 3-second timeout, and the stock service calls Postgres with its own 3-second limit and retries once. In the worst case, the stock service is still working at second 6, long after the app has given up and the user has tapped "Pay" again.

The fix is a : an absolute end time for the whole request, passed along with each call. Every service computes "time left = deadline minus now", uses that as its timeout, and refuses to start work it cannot finish in time. gRPC has deadlines built in. Over HTTP you pass a header that you define yourself, such as X-Request-Deadline, and the Build-it section shows how to use it.

Retries: only when it is safe

Many failures are temporary: a dropped connection, a server restarting during a deploy, a 503 from an overloaded proxy. Trying again often works. But a retry sends the same request twice, so first ask: if the first attempt actually worked, is a second one harmless?

  • GET, HEAD, PUT and DELETE are idempotent by HTTP's rules, if your server implements them properly (see APIs done right). Retry freely.
  • POST is not. If the first "create order" or "charge" request reached the server and only the response was lost, a retry creates a second order or a second charge. Retry a POST only when it carries an that the server checks.

Then ask whether the error can be fixed by waiting. A timeout, a reset connection, 502, 503, 504 and 429 often can. A 400, 401, 403, 404 or 422 will fail the same way every time, so retrying only wastes time. A connection that was refused never reached the server, so it is safe to retry even for a POST.

Finally, limit the number of tries. Three attempts in total is a common default for calls made while a user waits. Background jobs can try more often over a longer time, as you saw with BullMQ in Queues, background jobs and pub/sub.

Quick check

Which failed call is safe to retry exactly as it is?

Backoff and jitter

If a service is down because it is overloaded, instant retries make the overload worse. waits longer after each failure: 200 ms, then 400, then 800, up to a cap. That gives the service room to recover.

Backoff alone has a hidden problem. When a dependency fails, it fails for everyone at the same moment. Every client then waits exactly 200 ms, retries at the same instant, fails, waits exactly 400 ms, and retries together again. The retries arrive in waves, and each wave is as big as the first one. The fix is : a random part in the wait. The version that Marc Brooker tested at AWS and recommends is called full jitter:

Output
wait = random(0, min(cap, base × 2^attempt))

attempt 0: random between 0 and 200 ms
attempt 1: random between 0 and 400 ms
attempt 2: random between 0 and 800 ms ... never above cap

Each client picks a random point inside a growing window, so the retries spread out and the service sees a steady trickle instead of a wall. If the server sends Retry-After (see Rate limiting and protecting your system), wait at least that long.

Quick check

With full jitter, a base of 200 ms and a cap of 5 seconds, how long does a client wait before its retry after attempt 3 (counting from 0)?

Plain backoff all 50 stay in step service down room for 20 only 10 of 50 served Full jitter same 50, spread out service down room for 20 0 s 2 s 4 s 6 s all 50 served by 4.5 s
Attempts per 200 ms when 50 clients hit a service that is down for 2 seconds and then serves 10 requests per 100 ms (room for 20 per bar). With plain exponential backoff the clients stay in step: each wave holds all 50, so most attempts fail even after the service is back. With full jitter they spread out, and after the outage every bar fits under the line. Bars come from a small seeded simulation with the same setup as the simulator below.

Retry storms and retry budgets

Retries multiply across layers. The app retries 3 times, the API it calls retries 3 times, and that API's call to the database retries 3 times: one tap can become 3 × 3 × 3 = 27 database calls. When the database is slow, this turns a small problem into a that keeps it down.

Three rules keep this under control. First, retry at one layer only, usually the one closest to the failure. Google's SRE book describes exactly this: only the layer directly above the failing one retries, and the others pass the error up. Second, use a retry budget: allow retries only while they are less than about 10% of all requests from this client. When more than that are failing, the dependency is not having a blip, and more retries only add load. Third, stop calling when it is clearly down, which is the circuit breaker's job.

Watch the effect in the simulator. Start with "Retry immediately" and look at the peak, the wasted attempts and how many clients gave up. Then step through the other strategies, ending with "Exponential + full jitter", and compare the time until every client succeeded.

Simulator

Loading the simulator…

  • Immediate retries use up all 8 attempts in under a second, while the service is still down, so every client gives up. Fixed-delay retries arrive in waves far above capacity, so many attempts are wasted even after the service is back.
  • Plain exponential backoff spaces the waves out but keeps them synchronised. Full jitter turns the waves into a spread, and the service recovers with far fewer rejected attempts.

Circuit breakers: stop calling a dependency that is down

A watches the results of calls to one dependency, like an electrical breaker in your house watches current. It has three states:

  • Closed (normal): calls go through. The breaker counts failures over a recent window, such as the last 10 seconds.
  • Open: too many calls failed, for example more than 50% of at least 10 calls. The breaker now refuses calls at once, without touching the network, and your fallback runs. The sick service gets time to recover, and your users get a fast answer instead of a 3-second wait.
  • Half-open: after a pause (the reset timeout, 30 seconds by default in opossum), the breaker lets one trial call through. If it works, the breaker closes. If it fails, the breaker opens again for another pause.

Two details matter in practice. Errors that are your fault, such as a 400 for an invalid phone number, should not count as failures, or one bad input could open the breaker for everyone. And each dependency gets its own breaker, so a broken SMS gateway does not block payment calls.

Quick check

The loyalty breaker opens when over 50% of at least 10 calls in the last 10 seconds failed, and it ignores 400 errors. In the window there were 12 calls: 5 succeeded, 4 got 503 and 3 got 400 for a bad card number. What does the breaker do?

Closed calls go through Open fail fast, fallback Half-open one trial call errors over 50% of 10+ calls after 30 s trial call works trial fails: open again old failures age out
The three states of a circuit breaker. While closed it counts failures. When too many recent calls fail it opens and answers instantly with the fallback. After the reset timeout it lets one trial call through: success closes it, failure opens it again.

Bulkheads and fallbacks

A ship is split into watertight compartments called bulkheads, so one hole does not sink the whole ship. In software, a is a limit on how much of your shared resources one dependency may use. In a Node service it usually means a cap on calls in flight to each dependency (opossum's capacity option), a separate connection pool per database, or a separate worker process for heavy jobs. If the SMS gateway hangs, at most 20 requests wait for it, and the other requests still have memory, sockets and pool connections.

A is the plan B. Good fallbacks are product decisions, and each one should be written down:

  • Queue it for later. The receipt SMS goes to a queue and is sent when the gateway is back.
  • Use a cached or default value. Show yesterday's recommended products, or the stock count from the cache with "as of 10:40".
  • Hide the feature. The loyalty points box disappears from the receipt screen; checkout still works.
  • Fail clearly and fast. For money, sometimes the right fallback is an honest error in 300 ms: "Mobile wallet payments are not responding. Pay by cash or card, or try again."

Together these give : when a part fails, the system loses a feature instead of falling over.

Build it

Give each outside service a client with timeouts, add a retry helper with full jitter that only retries safe calls, wrap the SMS gateway in an opossum circuit breaker with a fallback, and pass a deadline through NestJS's HttpModule. The steps run in a Node project on your machine. The two core ideas, the full-jitter delay and the breaker's state machine, are exercises you can try in the browser in the Practice section below.

  1. One axios client per dependency, each with a timeout

    Install npm i axios opossum and npm i -D @types/opossum. Separate clients let each dependency have its own timeout and its own breaker.

    TypeScript
    // http-client.ts
    import axios from 'axios';
    
    // One instance per dependency, each with its own timeout
    export const supplierApi = axios.create({
      baseURL: process.env.SUPPLIER_API_URL,
      timeout: 2000,          // no response within 2 s, or 2 s of silence: give up
    });
    
    export const smsApi = axios.create({
      baseURL: process.env.SMS_GATEWAY_URL,
      timeout: 2500,
    });
    
    // timeout alone does not stop a server that sends a byte every second.
    // AbortSignal.timeout() is a hard limit on the whole call, body included.
    export async function getPrice(articleId: number) {
      const res = await supplierApi.get<{ price: string }>(`/articles/${articleId}/price`, {
        signal: AbortSignal.timeout(3000),
      });
      return res.data.price;
    }

    A timed-out axios call throws an error with code: 'ECONNABORTED', or 'ETIMEDOUT' if you set transitional: { clarifyTimeoutError: true }. An aborted signal throws a CanceledError with code: 'ERR_CANCELED'.

  2. A retry helper with full jitter

    It retries only errors that waiting can fix, only for safe methods or requests with an idempotency key, at most 3 attempts in total, and it respects Retry-After.

    TypeScript
    // retry.ts
    import axios from 'axios';
    
    const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
    const RETRY_STATUS = new Set([429, 502, 503, 504]);
    const RETRY_CODES = new Set(['ECONNABORTED', 'ETIMEDOUT', 'ECONNRESET', 'EAI_AGAIN']);
    const SAFE_METHODS = new Set(['get', 'head', 'options', 'put', 'delete']);
    
    export function isRetryable(err: unknown): boolean {
      if (!axios.isAxiosError(err)) return false;
      if (err.code === 'ECONNREFUSED') return true;            // never reached the server
      const method = (err.config?.method ?? 'get').toLowerCase();
      const safe = SAFE_METHODS.has(method) || !!err.config?.headers?.has('Idempotency-Key');
      if (!safe) return false;                                  // a POST may have worked already
      if (err.response) return RETRY_STATUS.has(err.response.status);
      return RETRY_CODES.has(err.code ?? '');                   // timeout or dropped connection
    }
    
    function retryAfterMs(err: unknown): number {
      const h = axios.isAxiosError(err) ? err.response?.headers['retry-after'] : undefined;
      if (!h) return 0;
      const secs = Number(h);                                   // "Retry-After: 2"
      if (!Number.isNaN(secs)) return secs * 1000;
      const at = Date.parse(String(h));                         // or an HTTP date
      return Number.isNaN(at) ? 0 : Math.max(0, at - Date.now());
    }
    
    export async function withRetry<T>(
      fn: () => Promise<T>,
      { attempts = 3, baseMs = 200, capMs = 5000 } = {},
    ): Promise<T> {
      for (let attempt = 0; ; attempt++) {
        try {
          return await fn();
        } catch (err) {
          if (attempt + 1 >= attempts || !isRetryable(err)) throw err;
          const ceiling = Math.min(capMs, baseMs * 2 ** attempt);   // 200, 400, 800 ... capped
          const wait = Math.random() * ceiling;                      // full jitter
          await sleep(Math.max(wait, retryAfterMs(err)));
        }
      }
    }
    TypeScript
    // usage
    const price = await withRetry(() => getPrice(1001));            // GET: safe to retry
    
    await withRetry(() =>                                            // POST: only with a key
      supplierApi.post('/purchase-orders', po, { headers: { 'Idempotency-Key': po.clientRef } }),
    );
  3. See which calls get retried

    Against a test server that always answers 503, or never answers, the helper behaves like this (attempts counted on the server):

    Output
    GET  /x          -> 503 each time      attempts: 3, then throws
    POST /x, no key  -> 503                attempts: 1 (not safe to repeat)
    POST /x, key     -> 503 each time      attempts: 3
    GET  /hang       -> times out          attempts: 3, code ECONNABORTED
  4. Wrap the SMS gateway in a circuit breaker with a fallback

    The fallback queues the receipt for later with BullMQ (see Queues, background jobs and pub/sub). The jobId is the same reference the gateway receives, so a receipt is never queued twice. Opossum's own timeout is a safety net above the axios timeout.

    TypeScript
    // sms.breaker.ts
    import axios from 'axios';
    import CircuitBreaker from 'opossum';
    import { Queue } from 'bullmq';
    import { Logger } from '@nestjs/common';
    import { smsApi } from './http-client';
    
    const log = new Logger('SmsBreaker');
    const smsRetryQueue = new Queue('sms', { connection: { host: 'localhost', port: 6380 } });
    
    async function sendSms(to: string, text: string, ref: string): Promise<{ id: string }> {
      const res = await smsApi.post('/messages', { to, text, clientRef: ref }, {
        headers: { 'Idempotency-Key': ref },
      });
      return res.data;
    }
    
    export const smsBreaker = new CircuitBreaker(sendSms, {
      timeout: 3000,                // opossum stops waiting after 3 s (axios gives up at 2.5 s)
      errorThresholdPercentage: 50, // open when half of the recent calls failed...
      volumeThreshold: 10,          // ...and there were at least 10 calls in the window
      rollingCountTimeout: 10_000,  // the window: the last 10 seconds
      resetTimeout: 30_000,         // stay open 30 s, then let one trial call through
      capacity: 20,                 // bulkhead: at most 20 calls in flight at once
      // A rejected phone number is our problem, not an outage: don't count it
      errorFilter: (err) => axios.isAxiosError(err) && err.response?.status === 400,
    });
    
    // Fallback: runs when the call fails, times out, or the circuit is open
    smsBreaker.fallback(async (to: string, text: string, ref: string) => {
      await smsRetryQueue.add('receipt', { to, text, ref }, { jobId: ref, delay: 60_000 });
      return { id: 'queued' };
    });
    
    smsBreaker.on('open', () => log.warn('SMS gateway circuit OPEN: failing fast'));
    smsBreaker.on('halfOpen', () => log.log('SMS gateway circuit HALF-OPEN: one trial call'));
    smsBreaker.on('close', () => log.log('SMS gateway circuit CLOSED: back to normal'));

    Checkout calls await smsBreaker.fire(phone, text, `receipt-${saleId}`) after the sale is saved. With a fallback set, fire() resolves with the fallback's value instead of rejecting, so checkout carries on. A filtered 400 is not counted against the breaker and does not trigger the fallback: fire() rejects with it, so handle it where you call fire(). In a test with the breaker set to open after 4 calls, the gateway received exactly 4 calls; the 5th and 6th were rejected with EOPENBREAKER in under a millisecond.

  5. Set timeouts in NestJS's HttpModule and pass a deadline

    @nestjs/axios passes its options to axios, so timeout works the same way. The service turns the request's deadline into a per-call timeout, so a call never waits past the moment the user stops caring.

    TypeScript
    // payments.module.ts
    import { Module } from '@nestjs/common';
    import { HttpModule } from '@nestjs/axios';
    import { ConfigModule, ConfigService } from '@nestjs/config';
    import { PaymentsService } from './payments.service';
    
    @Module({
      imports: [
        HttpModule.registerAsync({
          imports: [ConfigModule],
          inject: [ConfigService],
          useFactory: (config: ConfigService) => ({
            baseURL: config.getOrThrow<string>('PAYMENT_API_URL'),
            timeout: 4000,            // passed straight to axios
            maxRedirects: 0,
          }),
        }),
      ],
      providers: [PaymentsService],
      exports: [PaymentsService],
    })
    export class PaymentsModule {}
    TypeScript
    // payments.service.ts
    import { Injectable, ServiceUnavailableException } from '@nestjs/common';
    import { HttpService } from '@nestjs/axios';
    import { firstValueFrom } from 'rxjs';
    
    export interface PaymentStatus { paymentId: string; status: 'PENDING' | 'PAID' | 'FAILED' }
    
    @Injectable()
    export class PaymentsService {
      constructor(private readonly http: HttpService) {}
    
      /** `deadline` is the time (ms since epoch) by which the whole checkout must answer. */
      async status(paymentId: string, deadline: number): Promise<PaymentStatus> {
        const left = deadline - Date.now();
        if (left < 200) throw new ServiceUnavailableException('No time left to ask the payment API');
        const res = await firstValueFrom(
          this.http.get<PaymentStatus>(`/payments/${paymentId}`, {
            timeout: Math.min(4000, left - 100),            // never wait past the caller's deadline
            headers: { 'X-Request-Deadline': String(deadline) },
          }),
        );
        return res.data;
      }
    }

    The controller sets the deadline once, for example const deadline = Date.now() + 5000, and passes it to every service call it makes. The called service can read X-Request-Deadline and do the same with the time that is left.

ToolProtects againstWatch out
TimeoutWaiting forever on a hung dependencyToo long is useless, too short cuts off healthy slow calls; base it on p99
RetryBrief blips: resets, restarts, 503Only for idempotent calls; limit attempts; retry at one layer
Backoff + jitterSynchronised retry wavesCap the wait; respect Retry-After
Circuit breakerHammering a service that is clearly downOne per dependency; don't count your own 4xx errors
BulkheadOne dependency using up all connections or memorySize limits from real concurrency, not guesses
FallbackA failed dependency breaking the user's taskIt is a product decision; test it, or it fails when you need it

Bring it to your team

Walk through each outside call in the design with these questions.

  • What is the timeout on this call, and how did we choose it? What is the dependency's p99 today?
  • If this call times out, could the work have happened anyway? How do we find out before retrying?
  • Which layer retries, how many times, with what backoff? Does any other layer also retry the same call?
  • What does the user see when this dependency is down for 10 minutes? Is that fallback written down and tested?
  • Does this dependency have its own breaker and its own concurrency limit, so it cannot take everything else down?
  • Do we alert when a breaker opens, and do we have a dashboard of timeouts per dependency?

Red flags

  • An HTTP client created with no timeout, or with axios's default of 0.
  • "We retry everything 5 times", including POSTs, at every layer.
  • Fixed retry delays with no jitter in a mobile app with thousands of users.
  • Checkout that waits for SMS, loyalty points or analytics before it finishes.
  • A fallback that has never been run, because "the gateway never goes down".

Practice

Compute a full-jitter retry delay

Write retryDelay(attempt, random, options). attempt counts from 0. random is a function that returns a number from 0 up to (but not including) 1, like Math.random; the tests pass in fixed values so the answer is predictable. options has baseMs (default 200) and capMs (default 5000). Return the full-jitter wait: a random point between 0 and min(cap, base × 2^attempt). The starter returns plain exponential backoff, which is the bug this lesson warns about.

JavaScript exerciseRuns in your browser with tests
function retryDelay(attempt, random, { baseMs = 200, capMs = 5000 } = {}) {
  // Plain backoff: every client waits exactly the same time.
  // Add the cap, then full jitter.
  return baseMs * 2 ** attempt;
}
Ctrl + Enter runs
Show a solution
JavaScript
function retryDelay(attempt, random, { baseMs = 200, capMs = 5000 } = {}) {
  const ceiling = Math.min(capMs, baseMs * 2 ** attempt); // 200, 400, 800 ... capped
  return random() * ceiling;                              // full jitter: 0 to ceiling
}
Pass the exercise to finish this task.

Build a circuit breaker state machine

Write createBreaker({ failureThreshold, resetMs }). It returns an object with three methods. state() returns 'closed', 'open' or 'half-open'. canCall(now) says whether a call may go out at time now (in milliseconds). record(ok, now) reports how a call ended. The rules: it starts closed. In closed, failureThreshold failures in a row open it, and a success resets the count. In open, calls are refused until resetMs have passed since it opened; the first canCall after that moves it to half-open and allows exactly one trial call. In half-open, a success closes it and a failure opens it again from that moment. To keep the exercise small, this breaker counts failures in a row instead of a percentage over a time window.

JavaScript exerciseRuns in your browser with tests
function createBreaker({ failureThreshold = 3, resetMs = 30000 } = {}) {
  let current = 'closed';
  return {
    state: () => current,
    canCall(now) {
      return true; // always calls: this is the bug
    },
    record(ok, now) {
      // count failures, open, half-open, close...
    },
  };
}
Ctrl + Enter runs
Show a solution
JavaScript
function createBreaker({ failureThreshold = 3, resetMs = 30000 } = {}) {
  let current = 'closed';
  let failures = 0;
  let openedAt = 0;
  const open = (now) => { current = 'open'; openedAt = now; };
  return {
    state: () => current,
    canCall(now) {
      if (current === 'closed') return true;
      if (current === 'open' && now - openedAt >= resetMs) {
        current = 'half-open';               // let one trial call through
        return true;
      }
      return false;                          // open, or a trial is already out
    },
    record(ok, now) {
      if (current === 'half-open') {
        if (ok) { current = 'closed'; failures = 0; } else open(now);
        return;
      }
      if (ok) { failures = 0; return; }
      failures += 1;
      if (failures >= failureThreshold) open(now);
    },
  };
}
Pass the exercise to finish this task.

Check yourself

A POST /payments call to a wallet provider times out after 5 seconds. What should the code do?

Thousands of handscanners lose their connection when a server restarts, and all retry with exponential backoff of exactly 1, 2, 4 and 8 seconds. What happens?

A circuit breaker for the SMS gateway is open. What happens to a new call?

Flashcards

Open my review

Key takeaways

  • Every remote call can succeed, fail, or leave you not knowing. Design for all three.
  • Set a timeout on every client, based on the dependency's p99 and the user's patience, and pass deadlines through chains of calls.
  • Retry only errors that waiting can fix, only idempotent calls, a few times, at one layer, with full jitter: random(0, min(cap, base × 2^attempt)).
  • A circuit breaker stops calls to a dependency that is clearly down; a bulkhead stops one dependency from using up everything.
  • Fallbacks are product decisions. Decide them, write them down and test them before the festival rush finds them for you.

Sources

Episode transcriptWhen the SMS gateway freezes checkout · about 12 minutes · tap any line to play from there
  1. NadiaImran, when you phone a supplier and nobody picks up, how long do you let it ring?
  2. ImranMaybe thirty seconds. Then I hang up and try again later. I'm not standing there for an hour.
  3. NadiaGood. Code only has the common sense we write into it, and left to its defaults, a lot of code will let that phone ring forever.
  4. NadiaThe lesson opens with that picture. A store manager calls a supplier, nobody answers, so after about thirty seconds he hangs up. That's a timeout.
  5. ImranThen he waits a few minutes before calling again, instead of redialling every second and jamming their line. That's backoff?
  6. NadiaYes. And if ten other stores are calling too, it helps if they don't all call at exactly ten o'clock. Spreading them out is called jitter.
  7. ImranAnd if the supplier hasn't answered all morning?
  8. NadiaHe stops calling and uses another plan. He orders from the second supplier, or puts a note on the shelf saying coming tomorrow. After lunch he tries one call to see if they're back. That's a circuit breaker with a fallback.
  9. ImranSo the whole lesson is a sensible store manager, written down as code.
  10. NadiaThat's all it is. It starts from a list made at Sun Microsystems in 1994, of wrong beliefs people bring from single-machine programming. The first one is: the network is reliable.
  11. ImranWhich anyone who has watched a store's mobile data drop for twenty seconds knows is false.
  12. NadiaSo every remote call has three possible results. Success, failure, or I don't know.
  13. ImranI don't know? Surely it either worked or it didn't.
  14. NadiaOn the server, yes. On your side, a timeout tells you nothing. The request may have done nothing, or it may have charged the customer and only the reply got lost. Hold on to that one.
  15. ImranSo, timeouts. If a call has none, it just waits. One waiting request can't hurt much, can it?
  16. NadiaOne can't. But each one holds memory, a pool connection, maybe a database transaction. When a dependency hangs, hundreds pile up, until the whole service stops answering.
  17. ImranSo one slow SMS gateway could freeze checkout, even though a sale doesn't need the SMS.
  18. NadiaExactly. And the trap is that axios, the HTTP client most Node teams use, has a timeout of zero by default. Zero there means no limit at all. Set one on every client.
  19. ImranHow long? I'd say ten seconds, to be safe.
  20. NadiaThat's the usual guess, and it's usually too long. Start a little above the dependency's p ninety-nine on a normal day. If the payment API's p ninety-nine is one point two seconds, a two or three second timeout catches real problems without cutting off healthy slow calls.
  21. NadiaThen check from the user's side. If the cashier gives up after five seconds, a ten second timeout means you're still waiting after nobody cares.
  22. ImranWhat about chains, where checkout calls stock and stock calls the database?
  23. NadiaTimeouts add up, and stock can still be working at second six, after the cashier has tapped Pay again. So you pass a deadline: one end time for the whole request. Each service works out the time left and uses that as its timeout.
  24. ImranNow retries. My instinct says, if it fails, try again.
  25. NadiaOften that works. But first ask: if the first attempt actually worked, is a second one harmless? Reads are safe. A POST, like create order or charge, is not. Retry a POST only with an idempotency key the server checks.
  26. ImranThe unique value that lets the server say, I've seen this one, here's the first answer.
  27. NadiaYes. Then ask whether waiting can fix it. A timeout, a dropped connection or a five oh three often can. A four hundred or a four oh four will fail the same way every time. And three attempts in total is a common default while a person waits.
  28. ImranThen backoff. Two hundred milliseconds, then four hundred, then eight hundred.
  29. NadiaUp to a cap. But when a dependency fails, it fails for everyone at the same moment. What happens next?
  30. ImranEveryone waits two hundred milliseconds, and then everyone retries at once.
  31. NadiaRight. They come back together, in waves, and each wave is as big as the first.
  32. NadiaIf you can see your screen, the first diagram shows two bar charts. Fifty clients hit a service that's down for two seconds. With plain backoff the bars stay tall spikes, and by six seconds only ten of the fifty are served.
  33. ImranAnd the second chart?
  34. NadiaSame fifty clients with full jitter. Short bars, spread out, all under the capacity line, and all fifty served by about four and a half seconds. Full jitter means each client picks a random wait between zero and its backoff limit. Marc Brooker tested it at AWS.
  35. ImranSo instead of every store calling at ten sharp, each one calls at a random minute inside the window.
  36. NadiaYes. Now multiply across layers. The app retries three times, the API retries three times, and its database call retries three times. One tap becomes twenty-seven database calls.
  37. ImranFrom one cashier pressing one button. That's the retry storm.
  38. NadiaSo retry at one layer only, the one closest to the failure. Keep a retry budget: retries under about ten percent of requests. And stop calling when it's clearly down.
  39. NadiaThe lesson has a simulator for this. When you're off the bus, start with retry immediately and watch every client give up while the service is still down. Then step through to exponential plus full jitter and compare.
  40. ImranOK, circuit breakers. Like the fuse box in my flat.
  41. NadiaSame idea. The second diagram shows three states in a loop. Closed is normal: calls go through and the breaker counts failures. When more than half of at least ten recent calls fail, it opens.
  42. NadiaOpen means it refuses calls straight away, without touching the network, and your fallback runs. After thirty seconds by default in opossum, the library the lesson uses, it goes half-open and lets one trial call through. Success closes it. Failure opens it again.
  43. ImranA cashier types a wrong phone number and the gateway answers four hundred. That's a failure, surely?
  44. NadiaIt shouldn't count. That's our mistake, not an outage, and one bad input could open the breaker for everyone. Also, each dependency gets its own breaker, so a broken SMS gateway never blocks payments.
  45. ImranAnd the bulkhead?
  46. NadiaA ship is split into watertight compartments so one hole doesn't sink it. In software, it's a cap. At most twenty requests waiting on the SMS gateway, so the rest of checkout still has connections to work with.
  47. ImranAnd the fallback is plan B.
  48. NadiaWhich is a product decision, so partly yours, Imran. Queue the receipt SMS for later. Show stock as of ten forty from the cache. Hide the loyalty box. Or fail fast and clearly: mobile wallet payments are not responding, pay by cash or card, or try again.
  49. ImranLet me play it out at Mangrove. The evening before a festival, every till is busy, and the SMS gateway slows to fifteen seconds a message.
  50. NadiaWithout timeouts, every checkout waits for the SMS, the counter queues grow, and cashiers decide the system is down. With these tools, the SMS call has a two and a half second timeout and its own breaker. After ten slow calls it opens, and receipts go to a queue.
  51. ImranAnd the customers in line never notice.
  52. NadiaNow the classic mistake. A wallet payment times out, and the code retries it without an idempotency key.
  53. ImranThe I don't know case. The first one may have charged the customer, so the retry charges them twice.
  54. NadiaSo send a key the provider checks, or after a timeout, ask the provider for the payment's status before trying again.
  55. NadiaThe Build-it, in words. First, one axios client per outside service, each with its own timeout: two seconds for the supplier, two and a half for SMS. Plus an abort signal as a hard three second limit, because a server sending one byte a second never trips the normal timeout.
  56. ImranThen the retry helper.
  57. NadiaIt retries only errors waiting can fix, only safe calls or calls with a key, three attempts, full jitter, and it respects Retry-After. Against a test server that always fails, a GET tries three times and a POST without a key tries once.
  58. ImranThen the breaker around the SMS gateway.
  59. NadiaOpossum wraps the send, with twenty calls in flight at most and four hundred errors filtered out. The fallback queues the receipt on BullMQ, with the receipt reference as the job ID so it's never queued twice. In the test, after the breaker opened, extra calls were rejected in under a millisecond.
  60. NadiaLast, NestJS. The payment service turns the request's deadline into a per-call timeout and passes the deadline on in a header.
  61. ImranMy design review questions, then. What's the timeout on this call, and what's the dependency's p ninety-nine today? If it times out, could the work have happened anyway?
  62. NadiaGood. And which layer retries? Does any other layer retry the same call?
  63. ImranAnd if this dependency is down for ten minutes, what does the user see, and has anyone tested that fallback?
  64. NadiaIf you want to practise, the lesson has two small coding tasks you can run in the page. One computes a full jitter delay. The other builds the breaker's three states and the rules that move between them.
  65. NadiaHere's one to think about. Thousands of handscanners lose their connection when a server restarts. They all retry with backoff of exactly one, two, four and eight seconds. What happens? Take a second.
  66. ImranI'd guess that's fine. The backoff spreads them out.
  67. NadiaThat's the trap. They failed at the same moment, so the same fixed waits bring them back at the same moments. Synchronised waves, each one able to knock the server over again. Only jitter breaks them up.
  68. ImranMy three takeaways. One: every call can succeed, fail, or leave me not knowing, and payments need a plan for not knowing. Two: every client gets a timeout from the p ninety-nine and the user's patience.
  69. ImranThree: retry only safe calls, a few times, at one layer, with jitter, and put a breaker and a fallback around anything that isn't essential.
  70. NadiaOne nuance. Write the fallback down and test it, before the festival rush tests it for you. Next time, your team's favourite argument: should we split this into microservices?
Finished reading?Mark it complete to fill in its stop on your course line.

Use it at work

API client with retries

A fetch wrapper with a timeout, retries with growing waits, and errors that say what failed.

Open in the Code Lab
Next stopMonolith, modular monolith, microservices
Was this lesson helpful?
Questions and discussion

Loading the discussion…