1. Courses
  2. Engineering Roles
  3. Site Reliability Engineering
  4. Stop 1

35 minAdvanced

How reliable is reliable enough?

Last festival, Mangrove's checkout failed for 40 minutes at the evening peak, and nobody could say how much failure was acceptable or what should change afterwards. This lesson turns "checkout must be reliable" into numbers you can measure, a budget you can spend, and a written policy that decides what happens when the budget runs out.

By the end you can

  • Choose user journeys and SLI types (availability, latency, correctness, freshness) for checkout
  • Write an SLI as good events divided by valid events, and decide where to measure it
  • Pick an SLO target from data, user needs and the cost of each extra nine
  • Compute an error budget over a rolling window, in minutes and in failed requests
  • Write an error budget policy that says who decides what when the budget is spent

The everyday picture

Think of a mobile data plan. You buy 10 GB a month. Nobody expects you to use zero data; the plan exists so you can use it. You can stream a football match, video-call your family, download a big game. The only rule is the total.

Three things make the plan work. You and the phone company agree on what counts as usage (Wi-Fi does not count, mobile data does). The meter counts every megabyte the same way, so you can check it at any time. And everyone knows what happens when it runs out: video stops, messages still go through, and the allowance refills on a known date.

Reliability works the same way. The SLI is the meter, the SLO is the plan size, the error budget is the data you are allowed to use, and the error budget policy is the rule for what stops when it runs out. You spend the budget on change: releases, migrations, experiments. A team that never touches its budget is moving too slowly.

Start from the user's journey, then pick the SLI type

Measuring speed and reliability introduced SLIs, SLOs, SLAs and the nines. Here you apply them to one service you own: the Mangrove app's checkout. The first decision is what to measure, and the answer starts with the user, not the servers. A is a thing a customer is trying to get done. For checkout there are three that matter most:

  • Place order: the customer taps "Place order" and the API gateway, the order service, the stock service and PostgreSQL create the order.
  • Pay: the payment service charges a card, a mobile wallet like M-Pesa, UPI or GCash, or records cash on delivery.
  • Track: after paying, the customer watches the order and the rider on the map.

Each journey needs a different kind of SLI. The SRE Workbook groups them by what the system does. For request-driven systems the usual types are availability and latency. For pipelines and background work they are freshness and correctness. Checkout has both:

JourneySLI typeA good event is…
Place orderAvailabilitya place-order request answered without a server error
Place orderLatencya place-order request answered within 2 seconds
Paya payment that reaches a final, correct state (paid, or declined by the bank) within 60 seconds, charged exactly once
Tracka tracking screen load where the rider's location is less than 60 seconds old

Notice that the payment SLI counts a card declined by the customer's bank as good. The system did its job: it gave a clear, correct answer. A payment that hangs in "pending" for ten minutes, or charges the customer twice, is bad, even if every HTTP request in it returned 200. Correctness SLIs often come from a reconciliation job that compares your records with the payment provider's, so they arrive minutes late. That is fine for an SLO measured over weeks.

Keep the list short. The Workbook suggests five or fewer SLIs for a service. Four SLIs across three journeys is enough for checkout; a fifth for browsing the catalogue would belong to a different service.

Quick check

Customers complain that the tracking map shows the rider standing still while the rider is already at the door. Which SLI type catches this?

Good events over valid events, measured in the right place

Write every SLI as a ratio: good events ÷ valid events, over a window. The ratio is always between 0% and 100%, and the error budget falls straight out of it. The two halves need separate decisions.

Valid events are the ones you are responsible for. A customer who enters an expired coupon gets a 422 response. That is the system working, so it is not valid for the availability SLI: it counts as neither good nor bad. The same goes for health checks, synthetic probes and load tests, which are not customers. A 429 Too Many Requests is different. Your rate limiter refused a real customer (Rate limiting), so it is valid and it is bad.

Good events are valid events that went well enough: no 5xx, no 429, and for the latency SLI, an answer within the threshold. Write latency as a ratio too: "99% of place-order requests answered within 2 seconds", rather than "p99 under 2 seconds". Both say the same thing, but the ratio form gives you a count of bad events, and that count is what the budget is made of.

All order requests 60,800 in one day Not valid: 800 bad coupon, probes Valid: 60,000 the denominator Bad: 60 5xx, 429, over 2 s Good: 59,940 the numerator SLI: 59,940 / 60,000 = 99.9% for the day
Valid events leave out what you are not responsible for. Good events are the valid ones that went well. One day of place-order requests gives an SLI of 99.9%.

Next, decide where to count. The same journey looks different from different places, and each place has a blind spot:

  • The load balancer or API gateway sees every request that reaches you, with status and duration, and you already have its metrics. It cannot see requests that never arrive: a broken CDN certificate, a DNS problem, a mobile network that drops the connection.
  • The client, meaning telemetry from the Mangrove app itself, is closest to what the customer feels. It is also noisy (weak mobile signal looks like your failure), arrives late (phones upload when they reconnect), and needs code and infrastructure of its own.
  • A synthetic probe places a fake order with a test card every minute, day and night. It catches a total outage at 4 a.m. when there is no real traffic, but it follows one path, so it misses failures that hit only some users.
  • Service logs inside the order service are the easiest to add, and the worst for SLOs: when the gateway times out and returns 502, the order service may log a success.
Phone app Mobile network API gateway Order service Payment provider Client telemetry closest to the user noisy, arrives late Synthetic probe every minute, all night one path, fake card Gateway metrics: SLO every request it gets blind to lost ones Service logs easy to add miss gateway 502s
Each measuring point sees part of the truth. Mangrove's SLO uses gateway metrics, the synthetic probe covers quiet hours, and client telemetry is checked weekly for failures the gateway never saw.

A practical setup: compute the SLO from gateway metrics, because they are complete for everything that reaches you and cheap to keep for 30 days. Run a synthetic probe so a silent outage at night still shows up. Compare the gateway SLI with client telemetry every week. If the app sees 98.7% while the gateway sees 99.95%, something between the phone and the gateway is failing, and your SLO is blind to it.

Quick check

A customer's place-order request is refused with 429 Too Many Requests by Mangrove's own rate limiter during the festival rush. How does it count in the availability SLI?

Choosing the target: data, users and the cost of each nine

Start from data. Compute the SLI for the last 90 days with the exact definition you just wrote. Say place order was at 99.96% in normal weeks. A target of 99.9% sits a little below that: normal noise will not spend the whole budget, but a real regression will show up within days. A target above what you achieve today is a wish, and it puts the team in a freeze on day one.

Then check the target against your dependencies. Every request to pay goes through the payment provider. If its promises 99.95% a month, your pay SLI cannot reliably beat 99.95% unless you add a fallback, such as a second provider or offering cash on delivery when card payments fail. Measuring speed and reliability shows why parts in series multiply. Read the SLA closely, too. SLAs often exclude planned maintenance, count only incidents you report, and use calendar months. Your SLO is your own internal target; their SLA is a contract about money.

The same logic runs the other way. If Mangrove ever signs an SLA, for example with partner shops, set it looser than the internal SLO. The SLO is the early warning: you notice and fix problems while you still owe nobody anything.

Finally, price each extra nine. The SRE book warns that each step up in reliability can cost far more than the one before. Over 30 days (43,200 minutes), the budget for a full outage shrinks fast, and what you must build changes with it:

SLOFull outage allowed in 30 daysWhat it takes
99.9%43.2 minutesA human on call can get paged, look, and roll back in time
99.95%21.6 minutesFast burn-rate alerts (Alerting) and one-click rollback; no time for a long diagnosis
99.99%about 4.3 minutesAutomatic detection and failover, every release canaried, spare capacity in a second zone; a human cannot even acknowledge a page in time

Users also set a ceiling. Many customers order over patchy mobile data, and their own connections fail more often than once in a thousand tries. They retry, and the order goes through. Going from 99.9% to 99.99% for them is mostly invisible, while the engineering cost is very visible. Pick the lowest target at which customers stay happy, and spend the rest on shipping.

Quick check

A director asks for a 99.99% availability SLO on place order. Your on-call engineer is expected to acknowledge a page within 5 minutes. What does that tell you?

The window and the budget, in minutes and in requests

An SLO is always measured over a window. A calendar month matches business reports, but it resets on the 1st: an outage on the 30th vanishes two days later, and the team feels free to take big risks early in the month. A always covers the last 28 or 30 days, so a bad day keeps counting until it slides out the back. The SRE Workbook recommends a four-week rolling window as a good general choice. Twenty-eight days has one more advantage: it holds exactly four of every weekday, so busy Fridays and quiet Tuesdays weigh the same in every window. Mangrove uses 30 days because it matches how people talk; either is fine if you are consistent.

The error budget is 1 − SLO of the window. You can express it two ways, and they give different answers:

  • Time-based: 0.1% of 30 days is 43.2 minutes. Easy to explain, and it matches how people remember outages.
  • Request-based: 0.1% of the valid requests. At about 60,000 orders a day, 30 days is about 1.8 million attempts, so the budget is about 1,800 failed orders.

The request-based form is the one to use for checkout, because it weighs failures by how many customers felt them. An outage at 4 a.m. and one at 7 p.m. cost the same minutes, but the evening one hits far more customers. During a festival rush the difference becomes dramatic, as the box below shows. One more detail: count each checkout attempt once. During a retry storm, every failed attempt can turn into several requests, and counting raw requests multiplies both halves of the ratio.

Budget left: place order 100% 50% 25% 0% day 19: bad deploy uses 45% at once ship normally extra review new features wait spent: freeze 22% left day 1 day 30
The budget line falls a little every day from normal failures, then drops when one bad deploy fails many orders. The policy lines decide, in advance, what the team does at each level.

The error budget policy: what happens when it is spent

A budget with no rule attached is a dashboard nobody acts on. The is a page that the engineering manager, the product manager and the people on call agree on and sign before anything goes wrong. On the day the budget runs out, nobody negotiates. Everyone reads the page.

A good policy answers five questions:

  1. What happens at each level? Mangrove's checkout policy has four: more than 50% left, ship normally; 25–50%, risky changes (payment code, schema migrations, config on the checkout path) need an SRE reviewer and a tested rollback; under 25%, new features wait and only low-risk changes ship; budget spent, freeze everything except urgent fixes, security fixes and reliability work until the 30-day SLI is back above target. The example policy in the SRE Workbook uses the same idea: once the budget for the last four weeks is gone, it halts all changes except the most urgent fixes and security fixes.
  2. Who decides? Ken, the engineering manager, applies the policy. Imran, the product manager, can ask for an exception. Disagreements go one level up, and the Workbook's example sends them to the CTO. A name, not "the team".
  3. What are the exceptions? A release that fixes reliability always ships. A legal or security deadline ships with Ken's written approval and a rollback plan. A promotion booked with suppliers months ago can ship behind a feature flag, with someone watching the SLI during the rollout.
  4. What if someone else spent it? The Workbook's example lets a team keep shipping features when another team's service or a company-wide network problem caused the miss. But if the postmortem shows checkout depended too hard on that service, making checkout survive it becomes the reliability work. For the payment provider, that means timeouts, a circuit breaker and a fallback (Timeouts, retries and circuit breakers), not waiting for them to improve.
  5. When does one incident need more? The Workbook's example asks for a postmortem with at least one top-priority action whenever a single incident uses more than 20% of the four-week budget. Postmortems and learning covers that process.

The policy also changes the conversation with product. When the budget is healthy, the SLO is an argument for shipping: Imran can release the new wallet flow a week before the festival, because a few bad requests are affordable. When the budget is gone, he does not need to be persuaded; the page he signed already says what happens.

Build it

We will write checkout's SLO document, compute the place-order SLIs and the remaining budget with Prometheus recording rules, check them, and write the policy. The budget arithmetic itself is the SQL exercise in the Practice section.

  1. Write the SLO document

    One page per service, kept in the repository next to the code, reviewed like code. Every SLI says what is valid, what is good, and where it is measured.

    Markdown
    # Checkout SLOs (owner: checkout team; approved: Ken, Imran)
    
    Window: rolling 30 days. Budgets are counted in failed attempts.
    
    ## Place order: availability, SLO 99.9%
    - Valid: POST /v1/orders at the API gateway, except 4xx other than 429,
      and except synthetic probes and load tests (header x-synthetic)
    - Good: valid requests with status 2xx or 3xx
    - Measured: gateway metrics (gateway_requests_total)
    
    ## Place order: latency, SLO 99% within 2 s
    - Valid: same as above. Good: answered in 2 s or less
    
    ## Pay: correctness, SLO 99.9%
    - Valid: payment attempts started by a customer
    - Good: final state (paid or declined by the bank) within 60 s, charged once
    - Measured: nightly reconciliation against the provider's report
    
    ## Track: freshness, SLO 99%
    - Good: tracking loads where the rider location is under 60 s old
    
    Review: every quarter, and after any festival.
  2. Count good and valid events with recording rules

    The gateway exports a request counter with route, method and code labels and a latency histogram (Observability explains histograms). Metric names depend on your gateway; these are Mangrove's. The rules turn raw counters into an SLI and a remaining budget.

    YAML
    # checkout-slo.rules.yml: place-order SLIs over a rolling 30 days
    groups:
      - name: checkout-slo
        interval: 5m                 # 30-day queries are heavy; no need to run them every 15 s
        rules:
          # valid: 4xx are left out, except 429 (we refused a real customer)
          - record: checkout:place_order_valid:increase30d
            expr: sum(increase(gateway_requests_total{route="/v1/orders", method="POST", code=~"2..|3..|5..|429"}[30d]))
          - record: checkout:place_order_good:increase30d
            expr: sum(increase(gateway_requests_total{route="/v1/orders", method="POST", code=~"2..|3.."}[30d]))
          - record: checkout:place_order_availability:ratio30d
            expr: checkout:place_order_good:increase30d / checkout:place_order_valid:increase30d
          # latency: share answered within 2 s; 2 must be a histogram bucket boundary
          # (the le label reads 2 or 2.0 depending on the Prometheus version)
          - record: checkout:place_order_latency:ratio30d
            expr: |
              sum(increase(gateway_request_duration_seconds_bucket{route="/v1/orders", method="POST", le=~"2|2.0"}[30d]))
              /
              sum(increase(gateway_request_duration_seconds_count{route="/v1/orders", method="POST"}[30d]))
          # budget left: 1 = untouched, 0 = spent, below 0 = SLO missed
          - record: checkout:place_order_budget_remaining:ratio30d
            expr: 1 - (1 - checkout:place_order_availability:ratio30d) / (1 - 0.999)

    The last rule is the whole idea in one line. The failure ratio you had, divided by the failure ratio you were allowed, is the share of budget spent.

  3. Check the rules before they ship

    Run promtool in CI on every change to the rules file, so a typo never silently breaks the SLO.

    Terminal
    promtool check rules checkout-slo.rules.yml
    Output
    Checking checkout-slo.rules.yml
      SUCCESS: 5 rules found
  4. Put the budget in words people use

    Dashboards show ratios; meetings need counts. Turn the numbers into a short weekly line that Ken and Imran read:

    Output
    Place order, rolling 30 days to 19 Oct
    valid attempts      1,800,000
    allowed failures        1,800   (0.1%)
    failed so far           1,650
    budget left               150   (8%)  -> policy level: new features wait
    biggest spender     11 Oct, 1,150 failures (64% of budget): postmortem needed
  5. Write and sign the policy

    Keep it short enough to read in a meeting. It lives next to the SLO document.

    Markdown
    # Checkout error budget policy (signed: Ken, Imran, Nadia; review each quarter)
    
    | Budget left (30 days) | What we do |
    |---|---|
    | more than 50% | ship normally |
    | 25% to 50% | risky changes need an SRE reviewer and a tested rollback |
    | under 25% | new features wait; low-risk changes and reliability work only |
    | spent | freeze all but urgent fixes, security fixes, reliability work |
    
    - Ken applies the policy. Imran may request an exception in writing.
      Disagreements go to the head of engineering.
    - A single incident using more than 20% of the budget gets a postmortem
      with at least one top-priority action.
    - Budget spent by the payment provider still counts. Our work is then to make
      checkout survive it: timeouts, circuit breaker, cash-on-delivery fallback.

Bring it to your team

Ask these when a team proposes an SLO, or when nobody can say whether the last outage mattered.

  • "Which user journey does this SLI stand for, and would a customer notice if it dropped?"
  • "What exactly is a valid event and a good event? Where do 4xx, 429s, probes and load tests go?"
  • "Where is it measured, and which failures can that point not see?"
  • "Is the target based on our last 90 days and our dependencies' limits, or on a wish?"
  • "Is the budget counted in failed requests, so a peak-hour outage weighs more than a quiet one?"
  • "Who signed the error budget policy, and what exactly stops when the budget is spent?"

Red flags

  • An SLO of 100%, or a target higher than the service has ever achieved
  • An SLI measured from CPU or server uptime instead of customer requests
  • A 30-day SLI made by averaging daily percentages
  • An error budget with no written policy, so every freeze is a fresh argument
  • A budget that is never spent, year after year: the target is too loose or the team ships too little

Practice

Compute the 30-day SLI and the budget left

The table checkout_daily has one row per day with the place-order good and valid counts from the gateway, from 15 September to 19 October 2026. The SLO is 99.9% over the rolling 30 days ending 19 October (included). Return one row with valid_requests, failed_requests, sli_pct (percentage, rounded to 3 decimals), allowed_failures, remaining_failures and budget_spent_pct (rounded to 1 decimal). The starter uses every row in the table and stops after the SLI.

SQL exerciseRuns in a real PostgreSQL inside your browser
-- 30-day SLI and error budget for place order (SLO 99.9%)
SELECT sum(valid) AS valid_requests,
       sum(valid) - sum(good) AS failed_requests,
       round(100.0 * sum(good) / sum(valid), 3) AS sli_pct
FROM checkout_daily;
Ctrl + Enter runs
Show a solution
SQL
WITH w AS (
  SELECT sum(good) AS good, sum(valid) AS valid
  FROM checkout_daily
  WHERE day > date '2026-10-19' - 30          -- 20 Sep to 19 Oct: exactly 30 days
    AND day <= date '2026-10-19'
), b AS (
  SELECT good, valid,
         valid - good AS failed,
         round(valid * (1 - 0.999)) AS allowed     -- the budget, in failed requests
  FROM w
)
SELECT valid AS valid_requests,
       failed AS failed_requests,
       round(100.0 * good / valid, 3) AS sli_pct,   -- sum first, divide once
       allowed AS allowed_failures,
       allowed - failed AS remaining_failures,
       round(100.0 * failed / allowed, 1) AS budget_spent_pct
FROM b;

The window is 20 September to 19 October. The 4,000-failure incident on 17 September has slid out, so it no longer counts; with it, the budget would be spent several times over. The SLI of 99.908% looks comfortably above 99.9%, yet 91.7% of the budget is gone: under the policy, new features wait until the window recovers.

Pass the exercise to finish this task.

Decide on a release with 8% of the budget left

Nine days before the festival, place order has 150 failures left of an 1,800 budget (8%, the result above). The 11 October bad deploy used 1,150 of them. Imran wants to release a new "pay with wallet" flow that changes the payment service and adds a database column on the orders table. Using the policy in the Build it section, write what happens to this release, who decides, what must happen first, and what the team works on instead or as well.

  • You read the level correctly: under 25% left, so new features wait and only low-risk changes ship
  • You name Ken as the one who applies the policy, and say Imran can ask for an exception in writing
  • If an exception is discussed, it comes with conditions: feature flag, small rollout, someone watching the SLI, a tested rollback
  • You note that the 11 October incident used more than 20% of the budget, so it needs a postmortem with a top-priority action
  • You point the team at reliability work for the festival, tied to last year's story (payment timeouts, pool limits, retries)
Show a model answer

With 8% of the budget left, checkout is at the "new features wait" level. The wallet flow changes payment code and the orders schema, so it is a risky change and does not ship under the normal process. Ken applies the policy. Imran can ask for an exception in writing; a reasonable one would be to ship the code dark behind a feature flag, add the column with a safe migration, and turn the flow on for 1% of customers only after the window recovers above 25%, with Nadia watching the SLI and a rollback tested in staging.

The 11 October deploy used 64% of the budget, well over the 20% line, so it needs a postmortem with at least one top-priority action. Until the budget recovers, the team spends its time on the festival risks: timeouts and a circuit breaker on the payment provider, retry limits with backoff, and connection pool limits, which together address what caused last year's 40-minute outage.

Check yourself

During one evening, the gateway SLI for place order reads 99.95%, and the synthetic probe succeeds every minute. Client telemetry from the Mangrove app shows only 98.7% of order attempts succeeding. What is the most likely explanation?

Mangrove switches the checkout SLO from a calendar month to a rolling 30-day window. A 30-minute outage happens on 30 October. What changes?

The payment provider had a two-hour slowdown and spent all of checkout's pay budget. Imran argues the freeze should not apply because "it wasn't our fault". What does a well-written policy do here?

Flashcards

Open my review

Key takeaways

  • Start from user journeys (place order, pay, track) and pick the SLI type that matches each: availability, latency, correctness or freshness.
  • Write every SLI as good events over valid events. Decide what is excluded, and know the blind spot of where you measure.
  • Set the target from your history, your dependencies' limits and the cost of each nine; 99.99% means machines must fix outages, not people.
  • Use a rolling window and count the budget in failed requests, so peak-hour outages weigh what customers felt.
  • A signed error budget policy turns the budget into decisions: when to ship, when to freeze, who decides, and which exceptions are allowed.

Sources

Episode transcriptThe data plan for failure · about 12 minutes · tap any line to play from there
  1. NadiaImran, last festival, checkout went down for forty minutes at the evening peak. If I asked you whether that broke our reliability target, what would you say?
  2. ImranI'd say yes, obviously. Forty minutes at the busiest hour of the year. But if you asked me what the target was, I couldn't tell you. I don't think we had one.
  3. NadiaWe didn't. And that's why the week after was all arguments. Engineers wanted a freeze, product wanted to keep shipping, and nobody had a number to point at.
  4. ImranSo today we fix that. How reliable is reliable enough, for checkout?
  5. NadiaLet's start with your phone. You buy a data plan, say ten gigabytes a month. Does the phone company expect you to use zero?
  6. ImranNo, the whole point is that I use it. Videos, calls, maps. As long as I stay under ten.
  7. NadiaRight. And three things make the plan work. You both agree what counts, mobile data yes, Wi-Fi no. There's a meter that counts it the same way every time. And everyone knows what happens when it runs out: video stops, messages still go through, and it refills on a known date.
  8. ImranAnd in reliability terms?
  9. NadiaThe meter is the SLI, the service level indicator. The plan size is the SLO, the objective. The data you're allowed to use is the error budget. And the rule for what stops when it runs out is the error budget policy.
  10. ImranSo the budget is something you're meant to spend.
  11. NadiaExactly. You spend it on change: releases, migrations, experiments. A team that never touches its budget is probably moving too slowly.
  12. ImranOkay. Where do we start? With the servers?
  13. NadiaWith the customer. What is someone actually trying to do in checkout? I see three journeys. Place the order. Pay for it, by card, by a mobile wallet like M-Pesa, UPI or GCash, or cash on delivery. And then track it, watching the rider on the map.
  14. ImranAnd each journey gets its own meter?
  15. NadiaIts own kind of meter. Place order is about availability and latency: did it work, and did it answer within two seconds. Pay is about correctness: did the payment end in a final, correct state, charged exactly once.
  16. ImranSo a card declined by the customer's bank counts as a failure?
  17. NadiaThat's the trap. No, a decline is the system doing its job. It gave a clear, correct answer. A payment stuck on pending for ten minutes, or a double charge, is bad, even if every request inside it returned a success code.
  18. ImranAnd tracking?
  19. NadiaFreshness. The tracking screen can load fast every time and still show a rider location from five minutes ago. So the good event is a screen where the location is less than a minute old.
  20. ImranNow, how do you actually count? You can't just count errors.
  21. NadiaYou write every SLI as a ratio: good events divided by valid events. If you can see your screen, the first diagram is a funnel. If not, picture sixty thousand eight hundred order requests in one day at the top.
  22. NadiaEight hundred drop out as not valid. A customer typed an expired coupon and got told so, or it was our own test probe. That's not our failure, so it's neither good nor bad. That leaves sixty thousand valid requests.
  23. ImranAnd of those, sixty were bad, so fifty-nine thousand nine hundred and forty good. Ninety-nine point nine percent.
  24. NadiaThat's it. Now a quick test for you. During the rush, our own rate limiter refuses a customer with a "too many requests" error. Valid or not?
  25. ImranIt's a four hundred something code, so a client error. Not valid?
  26. NadiaReasonable guess, but no. That one is valid and bad. A real customer wanted to order, and we chose not to serve them. Shedding load can be the right call in a rush, but it still spends budget.
  27. ImranFair. Next question: where do you count? The server, the app?
  28. NadiaEach place has a blind spot. The second diagram shows the request travelling down from the phone, through the mobile network, the API gateway, the order service and the payment provider, with a measuring point next to each.
  29. NadiaThe gateway sees every request that arrives, with status and duration, and it's cheap. But it can't see the ones that never arrive, like a broken CDN certificate or a mobile network dropping connections.
  30. ImranSo measure in the app instead?
  31. NadiaThe app is closest to the customer, but it's noisy, because weak signal looks like our failure, and the data arrives late, when phones reconnect. A synthetic probe, a fake order every minute, catches a total outage at four in the morning, but it only follows one path.
  32. ImranSo what does Mangrove do?
  33. NadiaThe SLO comes from gateway metrics. The probe covers the quiet hours. And every week we compare the gateway number with the app's own number. If the app says ninety-eight point seven and the gateway says ninety-nine point nine five, something between the phone and us is failing.
  34. ImranNow the target. Why not ninety-nine point nine nine? Customers deserve the best, right?
  35. NadiaLet's price it. Over thirty days, ninety-nine point nine percent allows about forty-three minutes of full outage. A person can get paged, look, and roll back in that time. Ninety-nine point nine nine allows about four minutes.
  36. ImranFour minutes. Our on-call engineer has five minutes just to acknowledge the page.
  37. NadiaSo one outage spends the whole month before a human even looks. At that level you need machines that detect and fail over by themselves, a second zone, every release tested on a small slice first. That's a lot of money and engineering.
  38. ImranAnd customers on patchy mobile data probably can't tell the difference anyway.
  39. NadiaRight. Their own connections fail more than once in a thousand, and they just retry.
  40. NadiaSo you start from data: what did place order achieve in the last ninety days? Say ninety-nine point nine six. A target of ninety-nine point nine sits just below.
  41. NadiaThen check the dependencies. If the payment provider's SLA promises ninety-nine point nine five, our pay target can't beat that without a fallback.
  42. ImranSLA is the contract, with money attached. SLO is our own internal target.
  43. NadiaYes, and if we ever sign an SLA ourselves, it should be looser than the SLO, so the SLO warns us first.
  44. ImranNow go back to last festival. Forty minutes out of forty-three. So technically we were just inside, weren't we?
  45. NadiaIn minutes, yes. That's exactly what worried Ken. So I counted it in requests instead. On festival day we had roughly four times normal orders, and the evening peak carries about a third of the day, so about twenty-seven thousand orders an hour.
  46. NadiaThe forty-minute outage, when the retry storm against the payment provider used up the connection pool, failed about eighteen thousand checkout attempts.
  47. ImranAnd the budget in requests?
  48. NadiaThe festival month had roughly two point six million valid attempts, so a ninety-nine point nine budget of about two thousand six hundred failures. The outage burned roughly seven times the whole budget, in forty minutes.
  49. ImranWow. Within target in minutes, seven budgets over in requests.
  50. NadiaBecause minutes treat four in the morning and seven in the evening the same. Requests weigh failures by how many customers felt them. Checkout's budget is counted in failed attempts now.
  51. ImranAnd the window, thirty days from the first of the month?
  52. NadiaA rolling thirty days. With a calendar month, an outage on the thirtieth disappears two days later. With a rolling window it keeps counting until it slides out.
  53. NadiaThe third diagram shows the budget line over a window: a slow slide, then a cliff when one bad deploy uses forty-five percent, ending around twenty-two percent left.
  54. ImranAnd the dashed lines across it?
  55. NadiaThat's the policy. Above half, ship normally. Between a quarter and a half, risky changes need an SRE reviewer and a tested rollback. Under a quarter, new features wait. Spent, everything freezes except urgent fixes, security fixes and reliability work.
  56. ImranWho decides? Because I can see myself arguing.
  57. NadiaKen applies it. You can ask for an exception in writing. Disagreements go one level up. And the whole point is that you, Ken and the on-call people sign it before the festival, so nobody negotiates on the day.
  58. ImranWhat if the payment provider burns the budget? That's not our fault.
  59. NadiaCustomers felt it anyway, so it stays in the SLI. The policy says in advance how that counts. For a hard dependency like the provider, the work becomes making checkout survive it: timeouts, a circuit breaker, falling back to cash on delivery.
  60. ImranSo what does the Build it section actually have us do?
  61. NadiaFive steps. Write a one-page SLO document in the repository, saying for each journey what's valid, what's good and where it's measured. Then write Prometheus recording rules that count good and valid order requests at the gateway over thirty days and turn them into an SLI and a remaining budget.
  62. NadiaThen check the rules with the Prometheus checking tool in CI, so a typo never breaks the SLO silently. Then turn the numbers into a weekly line people read: attempts, allowed failures, failures so far, budget left. And finally write and sign the policy.
  63. ImranAnd the practice exercise does the budget maths in SQL.
  64. NadiaFrom one row per day of good and valid counts, you compute the thirty-day SLI, the budget, and what's left. Watch the window edges. An old incident just outside the window shouldn't count.
  65. ImranFor my next review, I'm writing down three questions. Which customer journey does this SLI stand for? Where is it measured, and what can that point not see? And who signed the policy?
  66. NadiaGood list. Add one: is the budget counted in failed requests, so a peak-hour outage weighs more than a quiet one?
  67. NadiaNow a question to think about. The thirty-day SLI is ninety-nine point nine zero eight percent, against a target of ninety-nine point nine. Healthy, right? Take a second.
  68. ImranIt's above target, so I'd say yes.
  69. NadiaIt's above target, but look at the budget. The allowed failure rate is point one percent, and you've used point zero nine two of it. That's about ninety-two percent of the budget gone. Under our policy, new features wait.
  70. ImranLet me sum up. One: start from what the customer is doing, and pick availability, latency, correctness or freshness for each journey. Two: an SLI is good over valid, and every place you measure has a blind spot.
  71. ImranThree: count the budget in failed requests over a rolling window, and write down in advance what happens when it runs out.
  72. NadiaOne nuance on the target: don't pick it from ambition. Pick it from your history, your dependencies and what each extra nine costs. A tighter number means machines have to fix outages, not people.
  73. ImranNext time?
  74. NadiaNext time, observability with OpenTelemetry: how to answer a question about checkout during an outage that you never thought to ask before it started.
Finished reading?Mark it complete to fill in its stop on your course line.

Use it at work

Capacity plan

Expected peak traffic, a sized design and Find my limit, with headroom and cost per month as criteria.

Open in the System Design Studio
Next stopObservability with OpenTelemetry
Was this lesson helpful?
Questions and discussion

Loading the discussion…