Running products in production
Once 320 stores depend on your app, a failure at 7 pm on the Friday before a festival costs sales, stock and trust in every store at once. This lesson shows you how to define severity in business terms, design support tiers that solve most problems before they reach engineers, run an incident with clear roles and regular updates, set reliability targets that protect time for keep-the-lights-on work, and write blameless postmortems that change something.
By the end you can
- Write an incident severity matrix in business terms, with who is told and how often
- Design support tiers from self-help to engineering, with clear escalation and the tools each tier needs
- Run an incident: declare it, assign roles, mitigate first, and send status updates on a fixed rhythm
- Use reliability targets and support data to set the capacity for keep-the-lights-on work
- Run a blameless postmortem and turn it into owned, dated actions
The everyday picture
When a fire is reported, the fire service doesn't argue about how bad it is. The call is graded by set rules (a kitchen fire in a flat, a warehouse with people inside), and the grade decides how many engines go. At the scene, one officer takes command. She doesn't hold a hose. She decides, assigns tasks and keeps track of who is where. Someone keeps the log. Someone talks to the residents standing in the street and tells them what is happening and what to do.
The first job is to put the fire out, or at least stop it spreading. Finding out why it started comes later. When it does, an investigator looks for causes without blaming whoever left the pan on, and sometimes the building rules change: smoke alarms, sprinklers, fire doors. Running a product in production works the same way: severity by rules, one commander, updates on a rhythm, mitigate first, then a review that changes the system.
Severity in business terms
An incident is an unplanned interruption, or a drop in quality, that hurts users or the business. Its severity decides how fast people respond, who is woken up, who is told and how often. Define severity in business terms, so a store manager, a support agent, an engineer and the vendor all reach the same level without debate. Three questions set it: what can't people do, how many sites or users are affected, and is there a workaround? Two rules adjust it: peak periods raise the level by one, and anything that makes money or stock data wrong is at least severity 2 even when every screen works.
Working with vendors and agencies wrote a severity table into a vendor contract. Your own matrix covers every team, inside and outside the company, so everyone uses the same words:
| Level | Definition at Mangrove | Examples | Response | Updates | Who is told |
|---|---|---|---|---|---|
| SEV1 Critical | A core task (sell, receive, deliver, order) is stopped or wrong in many sites, or customer data or a lot of money is at risk, with no workaround | Card payments fail in 30 stores; GRNs can't post anywhere in festival week; tills charge wrong prices chain-wide | Page on-call now; incident commander within 15 minutes, 24 hours a day | Every 30 minutes | Head of operations, MD, all affected stores, vendor |
| SEV2 High | A core task is degraded with a painful workaround, or one city is affected, or money or stock data is wrong in a way that costs money if not fixed today | Rider app can't show maps in one city; one DC's stores can't receive; prices wrong in 40 stores | Within 1 hour, 6:00 to 23:00 | Every hour | Head of the area affected, affected stores |
| SEV3 Medium | A non-core function is broken for some users; an easy workaround exists | A report shows yesterday's data; label printing fails in one store | Next working day | Daily, in the ticket | The requester |
| SEV4 Low | Cosmetic or minor | A typo; a misaligned button | Planned in the backlog | None | None |
Keep the rule pm-19 borrowed from PagerDuty's public guide: if you are unsure between two levels, choose the higher one, and lower it later if needed. Nobody is blamed for declaring an incident that turns out small; a team that hesitates to declare learns about big incidents from customers.
A failure in a background job can stop a whole chain. On a Friday evening in April 2015, Starbucks registers stopped working in about 7,000 company-operated stores in the US and about 1,000 in Canada. The company said the cause was "a failure during a daily system refresh". Many stores closed early, some gave drinks away, and the problem was fixed after several hours. The cause the company gave was a routine background job, not the tills themselves, which is why the matrix is about what people can't do rather than which component broke.
Quick check
On a normal Wednesday, the Store app can't print shelf labels in one store; staff can still receive deliveries and do price checks, and labels can be printed from the back-office PC. Which severity?
One store, a non-core function and a working alternative make it SEV3, handled the next working day. It isn't cosmetic, because labels matter for prices, so SEV4 is too low. SEV1 and SEV2 are for stopped or badly degraded core tasks, many sites or wrong money and stock data.