Fix the Monday morning till outage

Every Monday at about 8:00, the tills in 40 of Mangrove Market's 320 stores lose their connection to the price and payment services for around ten minutes, and the tills fall back to offline mode just as the stores open. The other 280 stores are fine. What you know: the 40 affected stores all got a new router model in a single wave four weeks ago, configured from one template; on those routers the WAN DHCP lease is 7 days and the site-to-site VPN rekeys its tunnel every 168 hours, both counted from the moment the routers were first switched on together, on a Monday at 8:00; router logs from store-0142 show 'DHCP: renewing lease on wan0' at 08:00:01 followed by 'IKE: rekey failed, peer unreachable' and the tunnel coming back at 08:09:40; the store box's journal shows the till sync queue growing from 08:00 to 08:10. Investigate as if you were on the team: write down the checks you would run from the till up to the data centre and what each one shows, prove the cause from the evidence, design the fix as a proper change with a test and a rollback plan, add a check that would catch it if it ever happened silently again, and write a short blameless incident report.

What you hand in

An investigation log: each check from the till to the data centre (commands or tools, expected result, what the evidence shows), ending with the proven cause
A change request for the fix: what changes, risk, test plan in one pilot store, schedule outside trading hours, rollout in waves and a rollback plan
A small script or monitoring check (with a short explanation) that alerts when a store's VPN tunnel drops or tills switch to offline mode
A one-page blameless incident report with impact in numbers, a timeline, contributing factors and actions with owners and dates

Check your work

Tick each item when your project meets it. Be strict: this is the rubric the reviewer uses.

The cause is proven from the evidence (the lease renewal and the VPN rekey falling at the same moment on the new router template), not guessed, and the 280 unaffected stores are explained
The investigation works layer by layer (device, local network, DHCP, DNS, routing, firewall, VPN, service) and records what each check showed
The change request has a pilot, a schedule outside trading hours, waves with a health gate, and a rollback plan that can be done in minutes
The fix changes the template so the timers can no longer line up (for example a different lease time, rekey lifetime or rekey margin), and the change is applied as configuration, not by hand on 40 routers
A script or check would raise an alert within minutes if tunnels drop or tills go offline again, and says who it alerts
The incident report is blameless, gives impact in numbers, and every action has an owner and a date

0 of 6 checked

Model answer

The model answer opens once your capstone passes the review, so you can compare your work with an outline of a strong submission.

Hand in your capstone

Your links are checked automatically, then the work is scored against the rubric above, item by item, with comments. A pass counts toward the course certificate and shows on your profile.

Loading…