Replica lag breaks the receipts page

Incident drill · Intermediate

What is happening

Alert: receipts page p95 above 2 s and errors at 12%, while a big report runs on the read replica.

The replica fell more than 30 seconds behind, so health checks took it out of service. Every receipt read went to the primary, which also takes every sale.

Make the receipts page survive one lagging replica.

You have 2 changes and 6 minutes. A change is adding or removing a part or a connection, or changing a part’s settings. Fewer changes and less time score higher.