How reliable is reliable enough?
Last festival, Mangrove's checkout failed for 40 minutes at the evening peak, and nobody could say how much failure was acceptable or what should change afterwards. This lesson turns "checkout must be reliable" into numbers you can measure, a budget you can spend, and a written policy that decides what happens when the budget runs out.
By the end you can
- Choose user journeys and SLI types (availability, latency, correctness, freshness) for checkout
- Write an SLI as good events divided by valid events, and decide where to measure it
- Pick an SLO target from data, user needs and the cost of each extra nine
- Compute an error budget over a rolling window, in minutes and in failed requests
- Write an error budget policy that says who decides what when the budget is spent
The everyday picture
Think of a mobile data plan. You buy 10 GB a month. Nobody expects you to use zero data; the plan exists so you can use it. You can stream a football match, video-call your family, download a big game. The only rule is the total.
Three things make the plan work. You and the phone company agree on what counts as usage (Wi-Fi does not count, mobile data does). The meter counts every megabyte the same way, so you can check it at any time. And everyone knows what happens when it runs out: video stops, messages still go through, and the allowance refills on a known date.
Reliability works the same way. The SLI is the meter, the SLO is the plan size, the error budget is the data you are allowed to use, and the error budget policy is the rule for what stops when it runs out. You spend the budget on change: releases, migrations, experiments. A team that never touches its budget is moving too slowly.
Start from the user's journey, then pick the SLI type
Measuring speed and reliability introduced SLIs, SLOs, SLAs and the nines. Here you apply them to one service you own: the Mangrove app's checkout. The first decision is what to measure, and the answer starts with the user, not the servers. A user journey is a thing a customer is trying to get done. For checkout there are three that matter most:
- Place order: the customer taps "Place order" and the API gateway, the order service, the stock service and PostgreSQL create the order.
- Pay: the payment service charges a card, a mobile wallet like M-Pesa, UPI or GCash, or records cash on delivery.
- Track: after paying, the customer watches the order and the rider on the map.
Each journey needs a different kind of SLI. The SRE Workbook groups them by what the system does. For request-driven systems the usual types are availability and latency. For pipelines and background work they are freshness and correctness. Checkout has both:
| Journey | SLI type | A good event is… |
|---|---|---|
| Place order | Availability | a place-order request answered without a server error |
| Place order | Latency | a place-order request answered within 2 seconds |
| Pay | Correctness | a payment that reaches a final, correct state (paid, or declined by the bank) within 60 seconds, charged exactly once |
| Track | Freshness | a tracking screen load where the rider's location is less than 60 seconds old |
Notice that the payment SLI counts a card declined by the customer's bank as good. The system did its job: it gave a clear, correct answer. A payment that hangs in "pending" for ten minutes, or charges the customer twice, is bad, even if every HTTP request in it returned 200. Correctness SLIs often come from a reconciliation job that compares your records with the payment provider's, so they arrive minutes late. That is fine for an SLO measured over weeks.
Keep the list short. The Workbook suggests five or fewer SLIs for a service. Four SLIs across three journeys is enough for checkout; a fifth for browsing the catalogue would belong to a different service.
Quick check
Customers complain that the tracking map shows the rider standing still while the rider is already at the door. Which SLI type catches this?
The API can answer every request quickly and successfully while serving a location that is five minutes old. Only a freshness SLI, which checks the age of the data, sees the problem. Availability and latency both measure the request, not how recent its content is.
Good events over valid events, measured in the right place
Write every SLI as a ratio: good events ÷ valid events, over a window. The ratio is always between 0% and 100%, and the error budget falls straight out of it. The two halves need separate decisions.
Valid events are the ones you are responsible for. A customer who enters an expired coupon gets a 422 response. That is the system working, so it is not valid for the availability SLI: it counts as neither good nor bad. The same goes for health checks, synthetic probes and load tests, which are not customers. A 429 Too Many Requests is different. Your rate limiter refused a real customer (Rate limiting), so it is valid and it is bad.
Good events are valid events that went well enough: no 5xx, no 429, and for the latency SLI, an answer within the threshold. Write latency as a ratio too: "99% of place-order requests answered within 2 seconds", rather than "p99 under 2 seconds". Both say the same thing, but the ratio form gives you a count of bad events, and that count is what the budget is made of.
Next, decide where to count. The same journey looks different from different places, and each place has a blind spot:
- The load balancer or API gateway sees every request that reaches you, with status and duration, and you already have its metrics. It cannot see requests that never arrive: a broken CDN certificate, a DNS problem, a mobile network that drops the connection.
- The client, meaning telemetry from the Mangrove app itself, is closest to what the customer feels. It is also noisy (weak mobile signal looks like your failure), arrives late (phones upload when they reconnect), and needs code and infrastructure of its own.
- A synthetic probe places a fake order with a test card every minute, day and night. It catches a total outage at 4 a.m. when there is no real traffic, but it follows one path, so it misses failures that hit only some users.
- Service logs inside the order service are the easiest to add, and the worst for SLOs: when the gateway times out and returns
502, the order service may log a success.
A practical setup: compute the SLO from gateway metrics, because they are complete for everything that reaches you and cheap to keep for 30 days. Run a synthetic probe so a silent outage at night still shows up. Compare the gateway SLI with client telemetry every week. If the app sees 98.7% while the gateway sees 99.95%, something between the phone and the gateway is failing, and your SLO is blind to it.
Quick check
A customer's place-order request is refused with 429 Too Many Requests by Mangrove's own rate limiter during the festival rush. How does it count in the availability SLI?
Most 4xx codes mean the customer sent something wrong, so they are left out. A 429 from your own limiter means you chose not to serve a real customer, and the customer feels it as a failure. Shedding load can be the right decision during a rush, but it still spends budget.
Choosing the target: data, users and the cost of each nine
Start from data. Compute the SLI for the last 90 days with the exact definition you just wrote. Say place order was at 99.96% in normal weeks. A target of 99.9% sits a little below that: normal noise will not spend the whole budget, but a real regression will show up within days. A target above what you achieve today is a wish, and it puts the team in a freeze on day one.
Then check the target against your dependencies. Every request to pay goes through the payment provider. If its SLA promises 99.95% a month, your pay SLI cannot reliably beat 99.95% unless you add a fallback, such as a second provider or offering cash on delivery when card payments fail. Measuring speed and reliability shows why parts in series multiply. Read the SLA closely, too. SLAs often exclude planned maintenance, count only incidents you report, and use calendar months. Your SLO is your own internal target; their SLA is a contract about money.
The same logic runs the other way. If Mangrove ever signs an SLA, for example with partner shops, set it looser than the internal SLO. The SLO is the early warning: you notice and fix problems while you still owe nobody anything.
Finally, price each extra nine. The SRE book warns that each step up in reliability can cost far more than the one before. Over 30 days (43,200 minutes), the budget for a full outage shrinks fast, and what you must build changes with it:
| SLO | Full outage allowed in 30 days | What it takes |
|---|---|---|
| 99.9% | 43.2 minutes | A human on call can get paged, look, and roll back in time |
| 99.95% | 21.6 minutes | Fast burn-rate alerts (Alerting) and one-click rollback; no time for a long diagnosis |
| 99.99% | about 4.3 minutes | Automatic detection and failover, every release canaried, spare capacity in a second zone; a human cannot even acknowledge a page in time |
Users also set a ceiling. Many customers order over patchy mobile data, and their own connections fail more often than once in a thousand tries. They retry, and the order goes through. Going from 99.9% to 99.99% for them is mostly invisible, while the engineering cost is very visible. Pick the lowest target at which customers stay happy, and spend the rest on shipping.
Quick check
A director asks for a 99.99% availability SLO on place order. Your on-call engineer is expected to acknowledge a page within 5 minutes. What does that tell you?
At 99.99% over 30 days the budget is about 4.3 minutes of full outage. Acknowledging is only the start; diagnosis and rollback take longer still. A target that tight needs machines to detect and fail over by themselves. Faster humans do not close that gap, and the SLO matters with or without an SLA.
The window and the budget, in minutes and in requests
An SLO is always measured over a window. A calendar month matches business reports, but it resets on the 1st: an outage on the 30th vanishes two days later, and the team feels free to take big risks early in the month. A rolling window always covers the last 28 or 30 days, so a bad day keeps counting until it slides out the back. The SRE Workbook recommends a four-week rolling window as a good general choice. Twenty-eight days has one more advantage: it holds exactly four of every weekday, so busy Fridays and quiet Tuesdays weigh the same in every window. Mangrove uses 30 days because it matches how people talk; either is fine if you are consistent.
The error budget is 1 − SLO of the window. You can express it two ways, and they give different answers:
- Time-based: 0.1% of 30 days is 43.2 minutes. Easy to explain, and it matches how people remember outages.
- Request-based: 0.1% of the valid requests. At about 60,000 orders a day, 30 days is about 1.8 million attempts, so the budget is about 1,800 failed orders.
The request-based form is the one to use for checkout, because it weighs failures by how many customers felt them. An outage at 4 a.m. and one at 7 p.m. cost the same minutes, but the evening one hits far more customers. During a festival rush the difference becomes dramatic, as the box below shows. One more detail: count each checkout attempt once. During a retry storm, every failed attempt can turn into several requests, and counting raw requests multiplies both halves of the ratio.
The error budget policy: what happens when it is spent
A budget with no rule attached is a dashboard nobody acts on. The error budget policy is a page that the engineering manager, the product manager and the people on call agree on and sign before anything goes wrong. On the day the budget runs out, nobody negotiates. Everyone reads the page.
A good policy answers five questions:
- What happens at each level? Mangrove's checkout policy has four: more than 50% left, ship normally; 25–50%, risky changes (payment code, schema migrations, config on the checkout path) need an SRE reviewer and a tested rollback; under 25%, new features wait and only low-risk changes ship; budget spent, freeze everything except urgent fixes, security fixes and reliability work until the 30-day SLI is back above target. The example policy in the SRE Workbook uses the same idea: once the budget for the last four weeks is gone, it halts all changes except the most urgent fixes and security fixes.
- Who decides? Ken, the engineering manager, applies the policy. Imran, the product manager, can ask for an exception. Disagreements go one level up, and the Workbook's example sends them to the CTO. A name, not "the team".
- What are the exceptions? A release that fixes reliability always ships. A legal or security deadline ships with Ken's written approval and a rollback plan. A promotion booked with suppliers months ago can ship behind a feature flag, with someone watching the SLI during the rollout.
- What if someone else spent it? The Workbook's example lets a team keep shipping features when another team's service or a company-wide network problem caused the miss. But if the postmortem shows checkout depended too hard on that service, making checkout survive it becomes the reliability work. For the payment provider, that means timeouts, a circuit breaker and a fallback (Timeouts, retries and circuit breakers), not waiting for them to improve.
- When does one incident need more? The Workbook's example asks for a postmortem with at least one top-priority action whenever a single incident uses more than 20% of the four-week budget. Postmortems and learning covers that process.
The policy also changes the conversation with product. When the budget is healthy, the SLO is an argument for shipping: Imran can release the new wallet flow a week before the festival, because a few bad requests are affordable. When the budget is gone, he does not need to be persuaded; the page he signed already says what happens.
Build it
We will write checkout's SLO document, compute the place-order SLIs and the remaining budget with Prometheus recording rules, check them, and write the policy. The budget arithmetic itself is the SQL exercise in the Practice section.
-
Write the SLO document
One page per service, kept in the repository next to the code, reviewed like code. Every SLI says what is valid, what is good, and where it is measured.
Markdown# Checkout SLOs (owner: checkout team; approved: Ken, Imran) Window: rolling 30 days. Budgets are counted in failed attempts. ## Place order: availability, SLO 99.9% - Valid: POST /v1/orders at the API gateway, except 4xx other than 429, and except synthetic probes and load tests (header x-synthetic) - Good: valid requests with status 2xx or 3xx - Measured: gateway metrics (gateway_requests_total) ## Place order: latency, SLO 99% within 2 s - Valid: same as above. Good: answered in 2 s or less ## Pay: correctness, SLO 99.9% - Valid: payment attempts started by a customer - Good: final state (paid or declined by the bank) within 60 s, charged once - Measured: nightly reconciliation against the provider's report ## Track: freshness, SLO 99% - Good: tracking loads where the rider location is under 60 s old Review: every quarter, and after any festival. -
Count good and valid events with recording rules
The gateway exports a request counter with
route,methodandcodelabels and a latency histogram (Observability explains histograms). Metric names depend on your gateway; these are Mangrove's. The rules turn raw counters into an SLI and a remaining budget.YAML# checkout-slo.rules.yml: place-order SLIs over a rolling 30 days groups: - name: checkout-slo interval: 5m # 30-day queries are heavy; no need to run them every 15 s rules: # valid: 4xx are left out, except 429 (we refused a real customer) - record: checkout:place_order_valid:increase30d expr: sum(increase(gateway_requests_total{route="/v1/orders", method="POST", code=~"2..|3..|5..|429"}[30d])) - record: checkout:place_order_good:increase30d expr: sum(increase(gateway_requests_total{route="/v1/orders", method="POST", code=~"2..|3.."}[30d])) - record: checkout:place_order_availability:ratio30d expr: checkout:place_order_good:increase30d / checkout:place_order_valid:increase30d # latency: share answered within 2 s; 2 must be a histogram bucket boundary # (the le label reads 2 or 2.0 depending on the Prometheus version) - record: checkout:place_order_latency:ratio30d expr: | sum(increase(gateway_request_duration_seconds_bucket{route="/v1/orders", method="POST", le=~"2|2.0"}[30d])) / sum(increase(gateway_request_duration_seconds_count{route="/v1/orders", method="POST"}[30d])) # budget left: 1 = untouched, 0 = spent, below 0 = SLO missed - record: checkout:place_order_budget_remaining:ratio30d expr: 1 - (1 - checkout:place_order_availability:ratio30d) / (1 - 0.999)The last rule is the whole idea in one line. The failure ratio you had, divided by the failure ratio you were allowed, is the share of budget spent.
-
Check the rules before they ship
Run
promtoolin CI on every change to the rules file, so a typo never silently breaks the SLO.Terminalpromtool check rules checkout-slo.rules.ymlOutputChecking checkout-slo.rules.yml SUCCESS: 5 rules found -
Put the budget in words people use
Dashboards show ratios; meetings need counts. Turn the numbers into a short weekly line that Ken and Imran read:
OutputPlace order, rolling 30 days to 19 Oct valid attempts 1,800,000 allowed failures 1,800 (0.1%) failed so far 1,650 budget left 150 (8%) -> policy level: new features wait biggest spender 11 Oct, 1,150 failures (64% of budget): postmortem needed -
Write and sign the policy
Keep it short enough to read in a meeting. It lives next to the SLO document.
Markdown# Checkout error budget policy (signed: Ken, Imran, Nadia; review each quarter) | Budget left (30 days) | What we do | |---|---| | more than 50% | ship normally | | 25% to 50% | risky changes need an SRE reviewer and a tested rollback | | under 25% | new features wait; low-risk changes and reliability work only | | spent | freeze all but urgent fixes, security fixes, reliability work | - Ken applies the policy. Imran may request an exception in writing. Disagreements go to the head of engineering. - A single incident using more than 20% of the budget gets a postmortem with at least one top-priority action. - Budget spent by the payment provider still counts. Our work is then to make checkout survive it: timeouts, circuit breaker, cash-on-delivery fallback.
Bring it to your team
Ask these when a team proposes an SLO, or when nobody can say whether the last outage mattered.
- "Which user journey does this SLI stand for, and would a customer notice if it dropped?"
- "What exactly is a valid event and a good event? Where do 4xx, 429s, probes and load tests go?"
- "Where is it measured, and which failures can that point not see?"
- "Is the target based on our last 90 days and our dependencies' limits, or on a wish?"
- "Is the budget counted in failed requests, so a peak-hour outage weighs more than a quiet one?"
- "Who signed the error budget policy, and what exactly stops when the budget is spent?"
Red flags
- An SLO of 100%, or a target higher than the service has ever achieved
- An SLI measured from CPU or server uptime instead of customer requests
- A 30-day SLI made by averaging daily percentages
- An error budget with no written policy, so every freeze is a fresh argument
- A budget that is never spent, year after year: the target is too loose or the team ships too little
Practice
Compute the 30-day SLI and the budget left
The table checkout_daily has one row per day with the place-order good and valid counts from the gateway, from 15 September to 19 October 2026. The SLO is 99.9% over the rolling 30 days ending 19 October (included). Return one row with valid_requests, failed_requests, sli_pct (percentage, rounded to 3 decimals), allowed_failures, remaining_failures and budget_spent_pct (rounded to 1 decimal). The starter uses every row in the table and stops after the SLI.
CREATE TABLE checkout_daily (day date PRIMARY KEY, good int NOT NULL, valid int NOT NULL);
INSERT INTO checkout_daily (day, good, valid)
SELECT d::date,
60000 - CASE
WHEN d::date = '2026-09-17' THEN 4000 -- old incident
WHEN d::date = '2026-10-11' THEN 1150 -- bad deploy
ELSE 10 + (extract(day FROM d)::int % 4) * 5 -- normal noise
END,
60000
FROM generate_series('2026-09-15'::date, '2026-10-19'::date, interval '1 day') AS d;
-- 30-day SLI and error budget for place order (SLO 99.9%)
SELECT sum(valid) AS valid_requests,
sum(valid) - sum(good) AS failed_requests,
round(100.0 * sum(good) / sum(valid), 3) AS sli_pct
FROM checkout_daily;
[{"valid_requests": 1800000, "failed_requests": 1650, "sli_pct": 99.908, "allowed_failures": 1800, "remaining_failures": 150, "budget_spent_pct": 91.7}]
Show a solution
WITH w AS (
SELECT sum(good) AS good, sum(valid) AS valid
FROM checkout_daily
WHERE day > date '2026-10-19' - 30 -- 20 Sep to 19 Oct: exactly 30 days
AND day <= date '2026-10-19'
), b AS (
SELECT good, valid,
valid - good AS failed,
round(valid * (1 - 0.999)) AS allowed -- the budget, in failed requests
FROM w
)
SELECT valid AS valid_requests,
failed AS failed_requests,
round(100.0 * good / valid, 3) AS sli_pct, -- sum first, divide once
allowed AS allowed_failures,
allowed - failed AS remaining_failures,
round(100.0 * failed / allowed, 1) AS budget_spent_pct
FROM b;The window is 20 September to 19 October. The 4,000-failure incident on 17 September has slid out, so it no longer counts; with it, the budget would be spent several times over. The SLI of 99.908% looks comfortably above 99.9%, yet 91.7% of the budget is gone: under the policy, new features wait until the window recovers.
Decide on a release with 8% of the budget left
Nine days before the festival, place order has 150 failures left of an 1,800 budget (8%, the result above). The 11 October bad deploy used 1,150 of them. Imran wants to release a new "pay with wallet" flow that changes the payment service and adds a database column on the orders table. Using the policy in the Build it section, write what happens to this release, who decides, what must happen first, and what the team works on instead or as well.
- You read the level correctly: under 25% left, so new features wait and only low-risk changes ship
- You name Ken as the one who applies the policy, and say Imran can ask for an exception in writing
- If an exception is discussed, it comes with conditions: feature flag, small rollout, someone watching the SLI, a tested rollback
- You note that the 11 October incident used more than 20% of the budget, so it needs a postmortem with a top-priority action
- You point the team at reliability work for the festival, tied to last year's story (payment timeouts, pool limits, retries)
Show a model answer
With 8% of the budget left, checkout is at the "new features wait" level. The wallet flow changes payment code and the orders schema, so it is a risky change and does not ship under the normal process. Ken applies the policy. Imran can ask for an exception in writing; a reasonable one would be to ship the code dark behind a feature flag, add the column with a safe migration, and turn the flow on for 1% of customers only after the window recovers above 25%, with Nadia watching the SLI and a rollback tested in staging.
The 11 October deploy used 64% of the budget, well over the 20% line, so it needs a postmortem with at least one top-priority action. Until the budget recovers, the team spends its time on the festival risks: timeouts and a circuit breaker on the payment provider, retry limits with backoff, and connection pool limits, which together address what caused last year's 40-minute outage.
Check yourself
During one evening, the gateway SLI for place order reads 99.95%, and the synthetic probe succeeds every minute. Client telemetry from the Mangrove app shows only 98.7% of order attempts succeeding. What is the most likely explanation?
The gateway only counts requests that arrive, and the probe follows one path from one place. A gap that large between the client and the gateway points to failures in between: a CDN, DNS, TLS or network problem that hits some users. Client data is noisy, but a sudden drop of more than a point on one evening is a signal, not noise.
Mangrove switches the checkout SLO from a calendar month to a rolling 30-day window. A 30-minute outage happens on 30 October. What changes?
A calendar window resets on the 1st, so a late-month outage stops mattering two days later. A rolling window keeps it for 30 days, which matches what customers remember. It is counted once, in every window that contains it, and the budget size is the same 0.1%.
The payment provider had a two-hour slowdown and spent all of checkout's pay budget. Imran argues the freeze should not apply because "it wasn't our fault". What does a well-written policy do here?
Customers felt the failure whoever caused it, so the SLI keeps it. The policy decides the response ahead of time. The Workbook's example allows feature work when another team caused the miss, but a postmortem that shows checkout depended too hard on the provider makes timeouts, a circuit breaker and a fallback the priority. A vote on the day is exactly what the policy exists to avoid.
Flashcards
Key takeaways
- Start from user journeys (place order, pay, track) and pick the SLI type that matches each: availability, latency, correctness or freshness.
- Write every SLI as good events over valid events. Decide what is excluded, and know the blind spot of where you measure.
- Set the target from your history, your dependencies' limits and the cost of each nine; 99.99% means machines must fix outages, not people.
- Use a rolling window and count the budget in failed requests, so peak-hour outages weigh what customers felt.
- A signed error budget policy turns the budget into decisions: when to ship, when to freeze, who decides, and which exceptions are allowed.
Sources
- Service Level Objectives, Site Reliability Engineering (Google), 2016
- Implementing SLOs, The Site Reliability Workbook (Google), 2018
- Example Error Budget Policy, The Site Reliability Workbook (Google), 2018
- Embracing Risk, Site Reliability Engineering (Google), 2016