1. Courses
  2. Engineering Roles
  3. Site Reliability Engineering
  4. Stop 5

35 minAdvanced

Postmortems and learning

An incident ends when checkout works again. The learning starts after that. This lesson shows how to turn last festival's 40-minute checkout outage into a blameless postmortem: impact in numbers, a timeline built from evidence, several contributing factors instead of one culprit, and action items that are actually finished before the next festival.

By the end you can

  • Decide which incidents and near misses need a postmortem, and how deep it should go
  • Run a blameless review: ask how, not who, and rewrite sentences that blame or use hindsight
  • Write the document: summary, impact with numbers, timeline, contributing factors, what went well and where you got lucky
  • Use 5 whys with its limits in mind, and map the several factors behind one outage
  • Write action items that are specific, owned, dated and prioritised, and track them to done with an SLO of their own

The everyday picture

A football team loses a match 3–0. The second goal came when a defender was caught far out of position. On Monday the coach plays the video. A bad coach stops the tape and shouts at the defender. Everyone learns one thing: hide your mistakes next time.

A good coach asks different questions. Why was the defender there? The tape shows the midfielder in front of him was injured and nobody covered. The plan never said who fills that gap. The pitch was wet, so the ball moved faster than in training. And the third goal was almost a fourth: a shot hit the post. That was luck, and luck is not a defence.

Then the coach changes things with names and dates: a new rule for who covers an injured player, a drill on wet grass on Thursday, a video session for the whole squad. A postmortem is that Monday video session for an outage.

When an incident earns a postmortem

A costs real time: a few hours to write, an hour of review with five or six people, then the work on its actions. So you need clear rules for when to write one, decided before the incident and written next to the SLO. If the rule is "when it feels serious", the embarrassing incidents get skipped, and those often teach the most.

The Google SRE book lists common triggers: user-visible downtime or degradation beyond a threshold, data loss of any kind, an on-call engineer having to step in (a rollback, rerouting traffic), a resolution time above a threshold, and a monitoring failure, which usually means a person found the problem before an alert did. Mangrove's checkout policy turns these into concrete rules:

  • Every SEV1 and SEV2 incident (the severity levels from Leading an incident).
  • Any single incident that uses more than 20% of the 30-day error budget, the rule from the error budget policy in How reliable is reliable enough?.
  • Any data loss or wrong charge, even one customer charged twice.
  • Any time on-call had to roll back, fail over or turn off a feature to stop harm.
  • Any incident that customers or support found before the alerts did.
  • that the on-call engineer flags as "this could have been a SEV1".

Near misses deserve attention because they are cheap lessons. Nobody was hurt, nobody is defensive, and the gap is still there. A replica disk that reached 96% at 02:00 and was cleaned up by hand is the same story as a full disk at 19:00 on festival night, minus the luck.

Match the depth to the event. A SEV1 gets the full template, a named owner, a review meeting and tracked actions. A near miss gets a short form: what happened, why it did not become an incident, and one or two actions. The incident commander names the postmortem owner as the incident closes, usually someone who was involved but not the commander. Mangrove aims for a draft within 5 working days, while chat logs, dashboards and memories are still fresh, and a review within 10.

Quick check

Which of these needs a postmortem under Mangrove's checkout policy?

Loading the rest of the lesson…