Toil and automation
Every week, Mangrove's on-call engineers reset store price caches, push stuck payments through by hand and grow database disks, and every new store and every festival makes that pile bigger. This lesson shows you how to measure that work, decide which parts to automate first, and build automation and self-service that cannot make an outage worse.
By the end you can
- Tell toil apart from overhead and engineering work using the six properties from the SRE book
- Measure toil with ticket tags, on-call logs and a week of time tracking, and apply the 50% cap
- Rank automation candidates by frequency, time, risk and payback
- Climb the automation ladder from a runbook to self-service without skipping the safe rungs
- Add guard rails to runbook automation: dry run, idempotence, scope limits, rate limits, an audit log and a stop button
The everyday picture
Picture the kitchen of a busy restaurant. One cook washes every plate by hand. The job has to be done now, because the next table needs plates. It is the same movement every time. A machine could do it. And a clean plate has no lasting value: it is dirty again an hour later. When the restaurant adds ten tables, the washing grows by the same share.
The cook also does other things. The weekly staff meeting and the health-and-safety form are part of having a job, but they are not dishwashing. Redesigning the kitchen so plates flow from the tables to a machine is real improvement: it keeps paying back every night.
The owner can fix the dishwashing in steps. First a card on the wall that says how to wash. Then a sink with a sprayer. Then a dishwasher. Then trays that customers clear themselves. Each step only works if it is safe: the dishwasher must stop when the door opens. In SRE terms, the hand-washing is toil, the meeting is overhead, the kitchen redesign is engineering work, and the door switch is a guard rail.
What counts as toil, and what does not
The Google SRE book defines toil as work tied to running a production service that has six properties. Use them as a test, one at a time:
- Manual: a person runs it, even if the person is only typing a command into a terminal.
- Repetitive: you have done it before and you will do it again. Fixing something new for the first time is not toil.
- Automatable: a machine could do it as well as a person. If it needs human judgement each time, such as deciding whether a strange refund is fraud, it is not toil.
- Tactical: it is driven by an interrupt, a ticket or a page, not by a plan.
- No enduring value: after you finish, the service is in the same state as before. Nothing got better.
- Scales linearly with the service: twice the stores or twice the orders means twice the work.
Work does not need all six to be toil, but the more it has, the more toil it is. The book separates it from two other kinds of work. Overhead is administrative work not tied to running a service: team meetings, planning, interviews, training, performance reviews. Engineering work is novel and leaves something better behind: a new tool, a design change that removes a failure, an SLO that makes decisions easier.
Two common mistakes come up when teams first sort their work. The first is calling anything boring "toil". The book says plainly that toil is not "work I don't like". Writing a hard postmortem or cleaning up an alert rule file can be tedious, but it has lasting value, so it is engineering. The second mistake is thinking all toil is bad. Some is unavoidable, and a little of it keeps engineers in touch with how production behaves. The problem is toil that grows without limit, because it grows with the business while your team does not.
Quick check
Which of these is toil for the checkout team?
The payment checks are manual, repetitive, follow the same rules each time, leave nothing better behind and grow with orders. The postmortem is tedious but has lasting value, so it is engineering work. The planning meeting is overhead: needed, but not tied to running the service.