Alerting that does not burn people out
Last festival, Mangrove's on-call phone buzzed dozens of times a week for CPU and memory warnings, and the alert that mattered came late. This lesson replaces those alerts with a few that watch how fast checkout's error budget is burning, routes them to the right person, and keeps the people on call healthy enough to answer.
By the end you can
- Decide what deserves a page: symptoms users feel, not causes on a server
- Compute a burn rate and say how much budget a burn uses and how long the budget lasts
- Explain why a single threshold is too slow or too noisy, and build the multiwindow, multi-burn-rate rules
- Route alerts as pages or tickets, with runbooks, grouping and inhibition
- Measure on-call load and noise, design a rotation and handover, and delete alerts that do not earn their place
The everyday picture
Think of a smoke alarm that goes off every time you make toast. The first week you open a window. The second week you wave a towel at it. By the third week you take the battery out. Then one night there is a real fire, and the alarm is silent.
Now think of the warnings on your phone's data plan from the SLO lesson. A bad warning says "you used 100 MB in the last five minutes". That happens every time you watch a short video, so you learn to ignore it. A good warning says "at this pace your 10 GB runs out in two days". It looks at your pace against the whole month, so it stays quiet when you stream for ten minutes and speaks up when something is really draining your plan. It also stops warning once you stop streaming.
Good alerting is the second kind of warning. It measures how fast you are spending the error budget, checks that the spending is still happening right now, and only then wakes a person.
What deserves a page
Observability: logs, metrics and traces introduced the basic rule: alert on symptoms, not causes, and keep two levels, page and ticket. Here you apply that rule to a service you own, with an SLO behind it. A page is the most expensive message your company sends. It interrupts dinner, wakes a parent at 3 a.m., and pulls an engineer out of deep work. Every page should pass four tests:
- Users are hurt now, or will be soon. Failed or slow checkouts count. A busy CPU does not, if every order still goes through.
- It needs a human. If a script could fix it every time, write the script.
- It needs a human now. A disk that fills in nine days is a ticket for tomorrow.
- The person can act. They can roll back, fail over, shed load or call someone who can.
A symptom is what the customer feels: place-order requests failing or taking too long. A cause is a condition inside the system: CPU at 95%, Redis memory at 80%, a growing queue. There are many possible causes and only a few symptoms, so symptom alerts catch problems you never predicted, while cause alerts fire for problems users never feel. Causes belong on dashboards, where they help the person who was paged find the reason.
Your SLO from How reliable is reliable enough? is already a precise definition of the symptom. So the best page is "checkout is spending its error budget too fast". The rest of this lesson builds exactly that.
Quick check
Which of these should page the checkout on-call engineer at night?
Only the third is a symptom that customers feel now and that needs a human now. High CPU with a healthy SLI means the servers are working hard, which is what you pay them for. The disk is a real problem, but nine days away: a ticket for the morning.