Capacity, resilience and game days
The festival rush brings about four times Mangrove's normal orders, squeezed into three evening hours, and last year checkout gave way for 40 minutes. This lesson turns history into a peak forecast, finds the first thing that breaks, sizes every service to survive losing a zone, and proves the plan with load tests and a game day before the customers arrive.
By the end you can
- Forecast the festival's peak orders per second and API requests per second from last year's data
- Find each service's bottleneck with the USE method and size connection pools with Little's law
- Work out how many instances survive the loss of one instance or one zone, and why autoscaling alone is too slow
- Design a load test with an open arrival model, a realistic mix, a ramp and a soak
- Plan and run a game day with a steady-state hypothesis, a small blast radius and abort conditions
The everyday picture
Think of a busy restaurant before a festival night. The owner does not guess. She looks at last year's bills and finds the busiest half hour: 90 plates. She walks the kitchen and asks which station gives up first. It is the single oven, not the waiters. She plans for one cook calling in sick, because someone always does.
She also knows she cannot hire a cook at 8 pm when the queue reaches the door, so she books extra staff a week ahead. On a quiet afternoon she runs a practice service: the staff cook the festival menu at full speed while she switches off one burner to see what happens. And she prints a short festival menu, so that if the kitchen falls behind, it can still serve the dishes people came for.
Capacity work for checkout follows the same steps. Forecast from history, find the slowest station, plan for a failure, book capacity early, rehearse with controlled failure, and decide in advance what to switch off.
Forecast the peak, not the day
"Four times normal" is a daily number, and servers do not fail on daily averages. They fail in the busiest minute. So you walk the forecast down from the day to the second, using your own history at every step. Speed numbers and back-of-the-envelope math has the basic arithmetic; here you apply it to a real journey.
Mangrove takes about 60,000 app orders on a normal day, so about 240,000 on festival day. Last year's data shows the 18:00 to 21:00 evening peak carries roughly a third of the day: about 80,000 orders in 3 hours, or about 7.4 orders a second on average. Inside that hour the traffic is lumpy. A push notification or a TV advert sends a wave, and Tomás found that last festival's busiest minute had about twice the peak-hour average. That ratio is the peak factor. It brings you to about 15 orders a second.
Then you translate orders into the load each service sees. A customer browses, searches and edits the cart before paying, and Mangrove's gateway logs show about 40 API requests per completed order. About 35% of orders are cash on delivery, so roughly 65% call the payment provider. Finally you add a margin for forecast error, because this year's marketing may be louder than last year's. Mangrove uses 1.5×. The plan becomes 22 orders a second, about 900 API requests a second, and about 14 payment calls a second.
Quick check
A teammate sizes checkout for festival day as 240,000 orders ÷ 86,400 seconds ≈ 2.8 orders a second. What is the main problem?
Servers fail in the busiest minute. A third of the day lands in three hours, and the busiest minute is about twice that hour's average, so the real peak is roughly five times the daily average before any margin. Applying the multiplier twice would double-count the festival.