Failure is normal: timeouts, retries, backoff and circuit breakers
Every call your service makes to another machine can be slow, fail, or never answer. This lesson gives you the five tools that stop one sick dependency from freezing everything: timeouts, careful retries, backoff with jitter, circuit breakers and bulkheads, plus the fallbacks that keep the user moving.
By the end you can
- Put a timeout on every outgoing call, and pick its value from real latency
- Decide which calls are safe to retry, and retry them with exponential backoff and full jitter
- Explain how retry storms happen and stop them with retry limits and budgets
- Draw the circuit breaker states and configure one with opossum
- Design a fallback so checkout keeps working when the SMS or payment API is slow
The everyday picture
A store manager needs a delivery update from a supplier. He calls, and nobody picks up. He does not stand there with the phone ringing for an hour. After about thirty seconds he hangs up (a timeout). He tries again in a few minutes, not every second, because calling non-stop only jams the supplier's line further (backoff). If ten other stores are also calling, it helps if they don't all call at exactly 10:00 (jitter).
If the supplier has not answered all morning, he stops calling for a while and uses another plan: he orders from the second supplier, or puts a note on the shelf saying "coming tomorrow" (circuit breaker and fallback). After lunch, he tries one call to see if they are back. None of this is clever. It is what a sensible person does, written down as code.
The network will let you down
In 1994, engineers at Sun Microsystems listed the fallacies of distributed computing: wrong beliefs that people bring from single-machine programming. The first three cause most of the trouble:
- The network is reliable. Packets get lost, connections reset, a Wi-Fi router reboots, a mobile operator drops the store's 4G for twenty seconds.
- Latency is zero. A function call takes nanoseconds. A call to the payment API takes 100 ms on a good day and 10 seconds on a bad one (see Speed numbers and back-of-the-envelope math).
- Bandwidth is infinite. A 5 MB response that is instant in the office takes a long time on a weak connection.
The others are: the network is secure, the topology does not change, there is one administrator, transport cost is zero, and the network is the same everywhere. The lesson for your code is short. Every remote call has three possible results: success, failure, or "I don't know". The third one is the dangerous one. A request that timed out may have done nothing, or it may have charged the customer. Your design has to handle all three.
Timeouts: never wait forever
A call with no timeout can wait forever. Each waiting request holds memory, a connection from the pool and maybe a database transaction. When a dependency hangs, the waiting requests pile up until the whole service stops responding. One slow SMS gateway can take down checkout this way, even though checkout does not need SMS at all.
Axios has no timeout by default (timeout: 0), and Node's fetch has none either, so you must set one on every client. There are two kinds of wait to limit:
- Connect timeout: how long to wait to open the connection. If the server is up, this takes milliseconds, so a short limit (1 to 3 seconds) catches dead hosts quickly.
- Read timeout: how long to wait for the response once connected. This depends on the work the server does.
In Node, axios's timeout covers both, roughly: it fires if the connection is not ready in time, or if the socket is silent for that long. It is not a limit on the total time. A server that sends one byte every second never goes silent for long, so it can keep the call open far beyond the timeout. For a hard limit on the whole call, also pass signal: AbortSignal.timeout(ms). Other clients have their own options: ioredis has connectTimeout and commandTimeout, and Postgres has statement_timeout (see Running Postgres in production).
How long should the timeout be? Look at the dependency's real latency, not a guess. Start a little above its p99 on a normal day (see Measuring speed and reliability): if the payment API's p99 is 1.2 seconds, a 2 or 3 second timeout catches real problems without cutting off slow but healthy calls. Then check the budget from the user's side. If the user gives up after 5 seconds, a 10-second timeout to a dependency means you are still waiting after nobody cares about the answer.
Quick check
The loyalty service's p99 is 800 ms on a normal day, and customers at the till give up after about 4 seconds. Which timeout fits best?
A little above p99 lets healthy slow calls finish and still leaves time inside the customer's 4 seconds. 300 ms is below p99, so it would cut off many healthy calls. No timeout and 30 seconds both keep requests waiting long after the customer has stopped caring.
Deadlines across calls
Timeouts add up through a chain of calls. Say the app gives checkout 5 seconds. Checkout calls the stock service with a 3-second timeout, and the stock service calls Postgres with its own 3-second limit and retries once. In the worst case, the stock service is still working at second 6, long after the app has given up and the user has tapped "Pay" again.
The fix is a deadline: an absolute end time for the whole request, passed along with each call. Every service computes "time left = deadline minus now", uses that as its timeout, and refuses to start work it cannot finish in time. gRPC has deadlines built in. Over HTTP you pass a header that you define yourself, such as X-Request-Deadline, and the Build-it section shows how to use it.
Retries: only when it is safe
Many failures are temporary: a dropped connection, a server restarting during a deploy, a 503 from an overloaded proxy. Trying again often works. But a retry sends the same request twice, so first ask: if the first attempt actually worked, is a second one harmless?
- GET, HEAD, PUT and DELETE are idempotent by HTTP's rules, if your server implements them properly (see APIs done right). Retry freely.
- POST is not. If the first "create order" or "charge" request reached the server and only the response was lost, a retry creates a second order or a second charge. Retry a POST only when it carries an idempotency key that the server checks.
Then ask whether the error can be fixed by waiting. A timeout, a reset connection, 502, 503, 504 and 429 often can. A 400, 401, 403, 404 or 422 will fail the same way every time, so retrying only wastes time. A connection that was refused never reached the server, so it is safe to retry even for a POST.
Finally, limit the number of tries. Three attempts in total is a common default for calls made while a user waits. Background jobs can try more often over a longer time, as you saw with BullMQ in Queues, background jobs and pub/sub.
Quick check
Which failed call is safe to retry exactly as it is?
A GET is idempotent and a 503 often clears when you wait. The POST may already have created the order, so a blind retry can make a second one. The 422 says the request itself is invalid, so it will fail the same way every time.
Backoff and jitter
If a service is down because it is overloaded, instant retries make the overload worse. Exponential backoff waits longer after each failure: 200 ms, then 400, then 800, up to a cap. That gives the service room to recover.
Backoff alone has a hidden problem. When a dependency fails, it fails for everyone at the same moment. Every client then waits exactly 200 ms, retries at the same instant, fails, waits exactly 400 ms, and retries together again. The retries arrive in waves, and each wave is as big as the first one. The fix is jitter: a random part in the wait. The version that Marc Brooker tested at AWS and recommends is called full jitter:
wait = random(0, min(cap, base × 2^attempt))
attempt 0: random between 0 and 200 ms
attempt 1: random between 0 and 400 ms
attempt 2: random between 0 and 800 ms ... never above capEach client picks a random point inside a growing window, so the retries spread out and the service sees a steady trickle instead of a wall. If the server sends Retry-After (see Rate limiting and protecting your system), wait at least that long.
Quick check
With full jitter, a base of 200 ms and a cap of 5 seconds, how long does a client wait before its retry after attempt 3 (counting from 0)?
The window is min(5,000, 200 × 2³) = 1,600 ms, and full jitter picks anywhere from 0 to the top of the window. "Exactly 1,600" is plain backoff with no jitter. The cap only matters once 200 × 2^attempt grows past 5 seconds.
Retry storms and retry budgets
Retries multiply across layers. The app retries 3 times, the API it calls retries 3 times, and that API's call to the database retries 3 times: one tap can become 3 × 3 × 3 = 27 database calls. When the database is slow, this turns a small problem into a retry storm that keeps it down.
Three rules keep this under control. First, retry at one layer only, usually the one closest to the failure. Google's SRE book describes exactly this: only the layer directly above the failing one retries, and the others pass the error up. Second, use a retry budget: allow retries only while they are less than about 10% of all requests from this client. When more than that are failing, the dependency is not having a blip, and more retries only add load. Third, stop calling when it is clearly down, which is the circuit breaker's job.
Watch the effect in the simulator. Start with "Retry immediately" and look at the peak, the wasted attempts and how many clients gave up. Then step through the other strategies, ending with "Exponential + full jitter", and compare the time until every client succeeded.
- Immediate retries use up all 8 attempts in under a second, while the service is still down, so every client gives up. Fixed-delay retries arrive in waves far above capacity, so many attempts are wasted even after the service is back.
- Plain exponential backoff spaces the waves out but keeps them synchronised. Full jitter turns the waves into a spread, and the service recovers with far fewer rejected attempts.
Circuit breakers: stop calling a dependency that is down
A circuit breaker watches the results of calls to one dependency, like an electrical breaker in your house watches current. It has three states:
- Closed (normal): calls go through. The breaker counts failures over a recent window, such as the last 10 seconds.
- Open: too many calls failed, for example more than 50% of at least 10 calls. The breaker now refuses calls at once, without touching the network, and your fallback runs. The sick service gets time to recover, and your users get a fast answer instead of a 3-second wait.
- Half-open: after a pause (the reset timeout, 30 seconds by default in opossum), the breaker lets one trial call through. If it works, the breaker closes. If it fails, the breaker opens again for another pause.
Two details matter in practice. Errors that are your fault, such as a 400 for an invalid phone number, should not count as failures, or one bad input could open the breaker for everyone. And each dependency gets its own breaker, so a broken SMS gateway does not block payment calls.
Quick check
The loyalty breaker opens when over 50% of at least 10 calls in the last 10 seconds failed, and it ignores 400 errors. In the window there were 12 calls: 5 succeeded, 4 got 503 and 3 got 400 for a bad card number. What does the breaker do?
The 400s are the caller's mistake, so they are filtered out and not counted as failures. Four failures out of twelve calls is about 33%, under the 50% threshold. Half-open only happens after the breaker has been open for the reset timeout.
Bulkheads and fallbacks
A ship is split into watertight compartments called bulkheads, so one hole does not sink the whole ship. In software, a bulkhead is a limit on how much of your shared resources one dependency may use. In a Node service it usually means a cap on calls in flight to each dependency (opossum's capacity option), a separate connection pool per database, or a separate worker process for heavy jobs. If the SMS gateway hangs, at most 20 requests wait for it, and the other requests still have memory, sockets and pool connections.
A fallback is the plan B. Good fallbacks are product decisions, and each one should be written down:
- Queue it for later. The receipt SMS goes to a queue and is sent when the gateway is back.
- Use a cached or default value. Show yesterday's recommended products, or the stock count from the cache with "as of 10:40".
- Hide the feature. The loyalty points box disappears from the receipt screen; checkout still works.
- Fail clearly and fast. For money, sometimes the right fallback is an honest error in 300 ms: "Mobile wallet payments are not responding. Pay by cash or card, or try again."
Together these give graceful degradation: when a part fails, the system loses a feature instead of falling over.
Build it
Give each outside service a client with timeouts, add a retry helper with full jitter that only retries safe calls, wrap the SMS gateway in an opossum circuit breaker with a fallback, and pass a deadline through NestJS's HttpModule. The steps run in a Node project on your machine. The two core ideas, the full-jitter delay and the breaker's state machine, are exercises you can try in the browser in the Practice section below.
-
One axios client per dependency, each with a timeout
Install
npm i axios opossumandnpm i -D @types/opossum. Separate clients let each dependency have its own timeout and its own breaker.TypeScript// http-client.ts import axios from 'axios'; // One instance per dependency, each with its own timeout export const supplierApi = axios.create({ baseURL: process.env.SUPPLIER_API_URL, timeout: 2000, // no response within 2 s, or 2 s of silence: give up }); export const smsApi = axios.create({ baseURL: process.env.SMS_GATEWAY_URL, timeout: 2500, }); // timeout alone does not stop a server that sends a byte every second. // AbortSignal.timeout() is a hard limit on the whole call, body included. export async function getPrice(articleId: number) { const res = await supplierApi.get<{ price: string }>(`/articles/${articleId}/price`, { signal: AbortSignal.timeout(3000), }); return res.data.price; }A timed-out axios call throws an error with
code: 'ECONNABORTED', or'ETIMEDOUT'if you settransitional: { clarifyTimeoutError: true }. An aborted signal throws aCanceledErrorwithcode: 'ERR_CANCELED'. -
A retry helper with full jitter
It retries only errors that waiting can fix, only for safe methods or requests with an idempotency key, at most 3 attempts in total, and it respects
Retry-After.TypeScript// retry.ts import axios from 'axios'; const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); const RETRY_STATUS = new Set([429, 502, 503, 504]); const RETRY_CODES = new Set(['ECONNABORTED', 'ETIMEDOUT', 'ECONNRESET', 'EAI_AGAIN']); const SAFE_METHODS = new Set(['get', 'head', 'options', 'put', 'delete']); export function isRetryable(err: unknown): boolean { if (!axios.isAxiosError(err)) return false; if (err.code === 'ECONNREFUSED') return true; // never reached the server const method = (err.config?.method ?? 'get').toLowerCase(); const safe = SAFE_METHODS.has(method) || !!err.config?.headers?.has('Idempotency-Key'); if (!safe) return false; // a POST may have worked already if (err.response) return RETRY_STATUS.has(err.response.status); return RETRY_CODES.has(err.code ?? ''); // timeout or dropped connection } function retryAfterMs(err: unknown): number { const h = axios.isAxiosError(err) ? err.response?.headers['retry-after'] : undefined; if (!h) return 0; const secs = Number(h); // "Retry-After: 2" if (!Number.isNaN(secs)) return secs * 1000; const at = Date.parse(String(h)); // or an HTTP date return Number.isNaN(at) ? 0 : Math.max(0, at - Date.now()); } export async function withRetry<T>( fn: () => Promise<T>, { attempts = 3, baseMs = 200, capMs = 5000 } = {}, ): Promise<T> { for (let attempt = 0; ; attempt++) { try { return await fn(); } catch (err) { if (attempt + 1 >= attempts || !isRetryable(err)) throw err; const ceiling = Math.min(capMs, baseMs * 2 ** attempt); // 200, 400, 800 ... capped const wait = Math.random() * ceiling; // full jitter await sleep(Math.max(wait, retryAfterMs(err))); } } }TypeScript// usage const price = await withRetry(() => getPrice(1001)); // GET: safe to retry await withRetry(() => // POST: only with a key supplierApi.post('/purchase-orders', po, { headers: { 'Idempotency-Key': po.clientRef } }), ); -
See which calls get retried
Against a test server that always answers
503, or never answers, the helper behaves like this (attempts counted on the server):OutputGET /x -> 503 each time attempts: 3, then throws POST /x, no key -> 503 attempts: 1 (not safe to repeat) POST /x, key -> 503 each time attempts: 3 GET /hang -> times out attempts: 3, code ECONNABORTED -
Wrap the SMS gateway in a circuit breaker with a fallback
The fallback queues the receipt for later with BullMQ (see Queues, background jobs and pub/sub). The
jobIdis the same reference the gateway receives, so a receipt is never queued twice. Opossum's owntimeoutis a safety net above the axios timeout.TypeScript// sms.breaker.ts import axios from 'axios'; import CircuitBreaker from 'opossum'; import { Queue } from 'bullmq'; import { Logger } from '@nestjs/common'; import { smsApi } from './http-client'; const log = new Logger('SmsBreaker'); const smsRetryQueue = new Queue('sms', { connection: { host: 'localhost', port: 6380 } }); async function sendSms(to: string, text: string, ref: string): Promise<{ id: string }> { const res = await smsApi.post('/messages', { to, text, clientRef: ref }, { headers: { 'Idempotency-Key': ref }, }); return res.data; } export const smsBreaker = new CircuitBreaker(sendSms, { timeout: 3000, // opossum stops waiting after 3 s (axios gives up at 2.5 s) errorThresholdPercentage: 50, // open when half of the recent calls failed... volumeThreshold: 10, // ...and there were at least 10 calls in the window rollingCountTimeout: 10_000, // the window: the last 10 seconds resetTimeout: 30_000, // stay open 30 s, then let one trial call through capacity: 20, // bulkhead: at most 20 calls in flight at once // A rejected phone number is our problem, not an outage: don't count it errorFilter: (err) => axios.isAxiosError(err) && err.response?.status === 400, }); // Fallback: runs when the call fails, times out, or the circuit is open smsBreaker.fallback(async (to: string, text: string, ref: string) => { await smsRetryQueue.add('receipt', { to, text, ref }, { jobId: ref, delay: 60_000 }); return { id: 'queued' }; }); smsBreaker.on('open', () => log.warn('SMS gateway circuit OPEN: failing fast')); smsBreaker.on('halfOpen', () => log.log('SMS gateway circuit HALF-OPEN: one trial call')); smsBreaker.on('close', () => log.log('SMS gateway circuit CLOSED: back to normal'));Checkout calls
await smsBreaker.fire(phone, text, `receipt-${saleId}`)after the sale is saved. With a fallback set,fire()resolves with the fallback's value instead of rejecting, so checkout carries on. A filtered400is not counted against the breaker and does not trigger the fallback:fire()rejects with it, so handle it where you callfire(). In a test with the breaker set to open after 4 calls, the gateway received exactly 4 calls; the 5th and 6th were rejected withEOPENBREAKERin under a millisecond. -
Set timeouts in NestJS's HttpModule and pass a deadline
@nestjs/axiospasses its options to axios, sotimeoutworks the same way. The service turns the request's deadline into a per-call timeout, so a call never waits past the moment the user stops caring.TypeScript// payments.module.ts import { Module } from '@nestjs/common'; import { HttpModule } from '@nestjs/axios'; import { ConfigModule, ConfigService } from '@nestjs/config'; import { PaymentsService } from './payments.service'; @Module({ imports: [ HttpModule.registerAsync({ imports: [ConfigModule], inject: [ConfigService], useFactory: (config: ConfigService) => ({ baseURL: config.getOrThrow<string>('PAYMENT_API_URL'), timeout: 4000, // passed straight to axios maxRedirects: 0, }), }), ], providers: [PaymentsService], exports: [PaymentsService], }) export class PaymentsModule {}TypeScript// payments.service.ts import { Injectable, ServiceUnavailableException } from '@nestjs/common'; import { HttpService } from '@nestjs/axios'; import { firstValueFrom } from 'rxjs'; export interface PaymentStatus { paymentId: string; status: 'PENDING' | 'PAID' | 'FAILED' } @Injectable() export class PaymentsService { constructor(private readonly http: HttpService) {} /** `deadline` is the time (ms since epoch) by which the whole checkout must answer. */ async status(paymentId: string, deadline: number): Promise<PaymentStatus> { const left = deadline - Date.now(); if (left < 200) throw new ServiceUnavailableException('No time left to ask the payment API'); const res = await firstValueFrom( this.http.get<PaymentStatus>(`/payments/${paymentId}`, { timeout: Math.min(4000, left - 100), // never wait past the caller's deadline headers: { 'X-Request-Deadline': String(deadline) }, }), ); return res.data; } }The controller sets the deadline once, for example
const deadline = Date.now() + 5000, and passes it to every service call it makes. The called service can readX-Request-Deadlineand do the same with the time that is left.
| Tool | Protects against | Watch out |
|---|---|---|
| Timeout | Waiting forever on a hung dependency | Too long is useless, too short cuts off healthy slow calls; base it on p99 |
| Retry | Brief blips: resets, restarts, 503 | Only for idempotent calls; limit attempts; retry at one layer |
| Backoff + jitter | Synchronised retry waves | Cap the wait; respect Retry-After |
| Circuit breaker | Hammering a service that is clearly down | One per dependency; don't count your own 4xx errors |
| Bulkhead | One dependency using up all connections or memory | Size limits from real concurrency, not guesses |
| Fallback | A failed dependency breaking the user's task | It is a product decision; test it, or it fails when you need it |
Bring it to your team
Walk through each outside call in the design with these questions.
- What is the timeout on this call, and how did we choose it? What is the dependency's p99 today?
- If this call times out, could the work have happened anyway? How do we find out before retrying?
- Which layer retries, how many times, with what backoff? Does any other layer also retry the same call?
- What does the user see when this dependency is down for 10 minutes? Is that fallback written down and tested?
- Does this dependency have its own breaker and its own concurrency limit, so it cannot take everything else down?
- Do we alert when a breaker opens, and do we have a dashboard of timeouts per dependency?
Red flags
- An HTTP client created with no timeout, or with axios's default of 0.
- "We retry everything 5 times", including POSTs, at every layer.
- Fixed retry delays with no jitter in a mobile app with thousands of users.
- Checkout that waits for SMS, loyalty points or analytics before it finishes.
- A fallback that has never been run, because "the gateway never goes down".
Practice
Compute a full-jitter retry delay
Write retryDelay(attempt, random, options). attempt counts from 0. random is a function that returns a number from 0 up to (but not including) 1, like Math.random; the tests pass in fixed values so the answer is predictable. options has baseMs (default 200) and capMs (default 5000). Return the full-jitter wait: a random point between 0 and min(cap, base × 2^attempt). The starter returns plain exponential backoff, which is the bug this lesson warns about.
function retryDelay(attempt, random, { baseMs = 200, capMs = 5000 } = {}) {
// Plain backoff: every client waits exactly the same time.
// Add the cap, then full jitter.
return baseMs * 2 ** attempt;
}
test('attempt 0 waits a random part of 200 ms', () => {
expect(retryDelay(0, () => 0.5)).toBe(100);
});
test('attempt 3 picks inside a 1,600 ms window', () => {
expect(retryDelay(3, () => 0.25)).toBe(400);
});
test('the window never grows past the cap', () => {
expect(retryDelay(10, () => 0.5)).toBe(2500);
});
test('random 0 means retry at once', () => {
expect(retryDelay(2, () => 0)).toBe(0);
});
test('uses the base and cap you pass in', () => {
expect(retryDelay(4, () => 0.5, { baseMs: 100, capMs: 1000 })).toBe(500);
});
Show a solution
function retryDelay(attempt, random, { baseMs = 200, capMs = 5000 } = {}) {
const ceiling = Math.min(capMs, baseMs * 2 ** attempt); // 200, 400, 800 ... capped
return random() * ceiling; // full jitter: 0 to ceiling
}Build a circuit breaker state machine
Write createBreaker({ failureThreshold, resetMs }). It returns an object with three methods. state() returns 'closed', 'open' or 'half-open'. canCall(now) says whether a call may go out at time now (in milliseconds). record(ok, now) reports how a call ended. The rules: it starts closed. In closed, failureThreshold failures in a row open it, and a success resets the count. In open, calls are refused until resetMs have passed since it opened; the first canCall after that moves it to half-open and allows exactly one trial call. In half-open, a success closes it and a failure opens it again from that moment. To keep the exercise small, this breaker counts failures in a row instead of a percentage over a time window.
function createBreaker({ failureThreshold = 3, resetMs = 30000 } = {}) {
let current = 'closed';
return {
state: () => current,
canCall(now) {
return true; // always calls: this is the bug
},
record(ok, now) {
// count failures, open, half-open, close...
},
};
}
test('starts closed and lets calls through', () => {
const b = createBreaker();
expect(b.state()).toBe('closed');
expect(b.canCall(0)).toBe(true);
});
test('opens after 3 failures in a row and refuses calls', () => {
const b = createBreaker({ failureThreshold: 3, resetMs: 30000 });
b.record(false, 1); b.record(false, 2); b.record(false, 3);
expect(b.state()).toBe('open');
expect(b.canCall(1000)).toBe(false);
});
test('a success resets the failure count', () => {
const b = createBreaker({ failureThreshold: 3 });
b.record(false, 1); b.record(false, 2); b.record(true, 3); b.record(false, 4);
expect(b.state()).toBe('closed');
});
test('after resetMs it allows exactly one trial call', () => {
const b = createBreaker({ failureThreshold: 1, resetMs: 30000 });
b.record(false, 0);
expect(b.canCall(29999)).toBe(false);
expect(b.canCall(30000)).toBe(true);
expect(b.state()).toBe('half-open');
expect(b.canCall(30001)).toBe(false);
});
test('trial success closes it, trial failure opens it again', () => {
const a = createBreaker({ failureThreshold: 1, resetMs: 100 });
a.record(false, 0); a.canCall(100); a.record(true, 150);
expect(a.state()).toBe('closed');
const b = createBreaker({ failureThreshold: 1, resetMs: 100 });
b.record(false, 0); b.canCall(100); b.record(false, 150);
expect(b.state()).toBe('open');
expect(b.canCall(200)).toBe(false);
expect(b.canCall(250)).toBe(true);
});
Show a solution
function createBreaker({ failureThreshold = 3, resetMs = 30000 } = {}) {
let current = 'closed';
let failures = 0;
let openedAt = 0;
const open = (now) => { current = 'open'; openedAt = now; };
return {
state: () => current,
canCall(now) {
if (current === 'closed') return true;
if (current === 'open' && now - openedAt >= resetMs) {
current = 'half-open'; // let one trial call through
return true;
}
return false; // open, or a trial is already out
},
record(ok, now) {
if (current === 'half-open') {
if (ok) { current = 'closed'; failures = 0; } else open(now);
return;
}
if (ok) { failures = 0; return; }
failures += 1;
if (failures >= failureThreshold) open(now);
},
};
}Check yourself
A POST /payments call to a wallet provider times out after 5 seconds. What should the code do?
A timeout means "I don't know". The charge may have gone through. Retrying without a key, or asking the customer to pay again, can charge them twice, and backoff does not change that. The safe options are a retry the provider will recognise as the same request, or checking the status before doing anything.
Thousands of handscanners lose their connection when a server restarts, and all retry with exponential backoff of exactly 1, 2, 4 and 8 seconds. What happens?
They all failed at the same moment, so the same fixed waits bring them all back at the same moments. Backoff without jitter spaces the waves out but does not break them up. A rate limiter can reject the excess, but the rejected devices then retry together again. Full jitter is what spreads them out.
A circuit breaker for the SMS gateway is open. What happens to a new call?
Failing fast is the whole point of the open state. Only after the reset timeout does the breaker go half-open and let a single trial call through. Waiting for the breaker to close would bring back the pile-up the breaker exists to prevent.
Flashcards
Key takeaways
- Every remote call can succeed, fail, or leave you not knowing. Design for all three.
- Set a timeout on every client, based on the dependency's p99 and the user's patience, and pass deadlines through chains of calls.
- Retry only errors that waiting can fix, only idempotent calls, a few times, at one layer, with full jitter:
random(0, min(cap, base × 2^attempt)). - A circuit breaker stops calls to a dependency that is clearly down; a bulkhead stops one dependency from using up everything.
- Fallbacks are product decisions. Decide them, write them down and test them before the festival rush finds them for you.
Sources
- Timeouts, retries and backoff with jitter, Marc Brooker, Amazon Builders' Library
- Exponential backoff and jitter, Marc Brooker, AWS Architecture Blog
- CircuitBreaker, Martin Fowler
- Addressing cascading failures, Site Reliability Engineering, Google