Tools · incident drills
Incident drills
Something in production is broken and the clock is running. Read the alert, change as little as you can, run the traffic and submit your fix. Every day there is a new drill for everyone, with a board for the best fixes.
Drill library
- StarterPlusAgents paid twiceAfter the gateway slowed down, some cash-outs went through twice.
- StarterPlusCache cut off by a bad deployRight after a deploy, every short-link click goes to the database and redirects time out.
- StarterPlusCache stampede after a restartA promo went out, the cache restarted, and short links started failing.
- StarterPlusLink lookups melt the databaseA promo SMS went out and short links crawl; the database is at 100%.
- StarterPlusPayout backlog after the rushCash-outs queue up during the evening rush and take minutes to arrive.
- StarterPlusSales counted twice after the outageThe internet came back, the stores synced, and head office shows more sales than the tills.
- StarterPlusThe queue that never drainsOrder updates arrive minutes late after a campaign launch.
- IntermediatePlusBlack Friday order backlogBlack Friday opens, orders arrive six times faster than usual and checkout starts timing out.
- IntermediatePlusFour servers, one doing all the workSomeone scaled the app to four servers, but checkout still falls over.
- IntermediatePlusImages clog the app serversEvery product photo goes through the app servers, and pages are slow for everyone.
- IntermediatePlusReplica lag breaks the receipts pageThe only read replica falls behind and is pulled out; the receipts page drowns the primary.
- IntermediatePlusSearch index falls behindNew prices take ten minutes to show in search: the indexing queue keeps growing.
- IntermediatePlusThe cloud bill doubledEverything was scaled up for the summer sale and nobody scaled it back down.
- IntermediatePlusTills stop selling when the line dropsEvery time a store loses internet, the queue at the tills grows and customers leave.
- AdvancedPlusA slow gateway blocks every cash-outWhen the payment gateway slows down, the whole app stops answering.
- AdvancedPlusCampaign messages go missingThe campaign went out and thousands of customers never got the message.
- AdvancedPlusFlash sale meltdownSeven times the traffic, two app servers and no caching. Fix it on a budget.
- AdvancedPlusHot partition on the loyalty tableDouble-points day: every sale updates the loyalty table and balance checks pile onto the same database.
- AdvancedPlusThundering herd after the outageThe app comes back after a short outage and every phone retries at once.
- AdvancedPlusZone failover drillA planned game day: one zone of the data centre goes dark, then the database primary is killed. Does checkout stay up?