Leading an incident
When checkout breaks at the festival peak, about 450 orders fail every minute, and the first ten minutes decide how long the outage lasts. This lesson teaches you to run that response as incident commander: declare early, give everyone one job, keep people informed on a fixed rhythm, and stop the damage before anyone looks for the root cause.
By the end you can
- Declare an incident early and set its severity from concrete checkout criteria
- Staff the roles (incident commander, operations lead, communications lead, scribe, experts) and keep the commander out of debugging
- Write status updates on a fixed cadence for the status page and for the company
- Choose mitigations under uncertainty, reversible ones first, with a hypothesis and a time box
- Hand over command cleanly, end the incident and measure time to detect, mitigate and resolve
The everyday picture
Picture a busy restaurant kitchen on a Friday night when the main oven dies. If every cook runs to the oven, nobody cooks, nobody tells the waiters, and the guests stare at empty tables. A good kitchen works differently. The head chef stays at the pass, the counter where plates leave the kitchen. She does not crawl behind the oven. She sends one cook to deal with it, tells the others to move dishes to the grill and the second oven, and tells the waiters what to say to guests: "Pizzas are 20 minutes late, everything else is on time."
She decides fast and changes her mind fast. If the grill cannot keep up, she takes the slow dishes off the menu for the evening. Finding out why the oven died can wait until tomorrow; tonight's job is getting food to the tables.
Leading an incident is the head chef's job. The incident commander stays at the pass: she holds the whole picture, gives each person one task, keeps everyone informed, and makes the calls. Other people turn the screwdrivers.
Declare early, and set the severity from criteria
Incidents, problems and changes showed how a service desk runs a major incident. This lesson goes deeper on the engineering side: how one person runs the response to a fast, customer-facing outage like a checkout failure. It starts with one decision: is this an incident?
To declare an incident means you start the process: someone takes command, a channel opens, updates begin. The most common failure is declaring too late. Engineers want to be sure first, so they poke at dashboards for 15 minutes while customers fail. Make declaring cheap: anyone on call may declare, it costs one chat command, and nobody is ever blamed for a false alarm. Downgrading or closing an incident after five minutes is a normal outcome. Waiting 15 minutes at the festival peak costs about 6,700 failed orders.
Then set the severity. Severity decides the response, so it needs criteria anyone can check in a minute. Write them in terms of what customers feel, with checkout's own numbers:
| Level | Checkout criteria (any one is enough) | Response |
|---|---|---|
| SEV1 | More than 5% of checkout attempts failing for 5 minutes or more, in any city; customers charged wrongly (double charges, paid orders lost); a fast-burn page (14.4×) that is still rising | Commander and full roles now, status page within 15 minutes, internal updates every 15 minutes, postmortem required |
| SEV2 | One payment method down, for example a mobile wallet like M-Pesa, UPI or GCash, while cards and cash on delivery work; p99 latency of place order over 5 seconds; a 6× burn page | Commander and an operations lead, updates every 30 minutes, postmortem required |
| SEV3 | Degraded with a workaround and no SLO at risk today, for example order confirmation messages 20 minutes late | A ticket with an owner, handled in working hours, postmortem optional |
The burn rates come from Alerting that does not burn people out: a page means the error budget from How reliable is reliable enough? is going fast. When you are unsure between two levels, pick the higher one. You can always step down. Stepping up late means the extra people and the status page arrive late too.
Quick check
At 19:10 on a festival evening, wallet payments fail for about 40% of attempts. Card payments and cash on delivery work normally. Which severity fits the table above?
The table names this case directly: one payment method down while the others work is SEV2. The 40% is a share of wallet attempts, and many customers will retry with a card. It would become SEV1 if total checkout failures passed 5% for 5 minutes. SEV3 is wrong because some customers have no card, and the budget is burning now.