Stable suites in CI

40 minIntermediate

Stable suites in CI

A test suite helps a team only while people believe a red result. This lesson shows you how to keep Mangrove's checkout suite trusted and under ten minutes: find and fix flaky tests, split the work across machines, publish reports people read, and review the tests an AI assistant drafts before they join the suite.

By the end you can

Trace a flaky test to one of seven causes and fix the cause instead of adding retries
Run a flaky-test policy: detect with retries, quarantine with an owner and a deadline, then fix or delete
Cut wall-clock time with workers, Playwright sharding balanced by duration, and fail-fast stages
Publish an HTML report with traces and JUnit annotations on the pull request, and track the suite's health
Review an AI-drafted test with a checklist that catches its usual defects

The everyday picture

The bakery corner of a Mangrove store has a smoke alarm. It goes off every time someone toasts bread. For a week, staff wave a towel at it. In the second week, someone takes the battery out. In the third week, a real fire starts behind the oven, and nobody hears a thing.

A flaky test is that alarm. Each false alarm costs only a minute, so ignoring it feels cheap. The real cost comes later: the team learns that red means "probably nothing", and a real bug walks through the same door. You can fix an alarm in three ways: move it away from the toaster, replace it, or take it down and say so. Leaving it on the ceiling with no battery is the worst choice, because everyone still thinks it protects them.

Why a test flakes: seven causes

A flaky test gives different results on the same commit. So its result depends on something besides the code. Almost every flake comes from one of seven sources, each with a fix:

Timing and fixed waits. waitForTimeout(2000) guesses how long the app needs. On a slow CI machine it is too short. Wait for a condition instead, with web-first assertions such as await expect(page.getByText('Order placed')).toBeVisible(), which retry until the condition holds or the timeout ends (End-to-end tests with Playwright).
Order dependence. Test B passes only if test A ran first and left a cart behind. Each test must create what it needs.
Shared data. Two tests use the same customer account. FRESH10 works only on a first order, so whichever test orders second fails. Give each test its own customer, made through the API with a unique id (Unit and API tests).
Time zones and dates. Mangrove's stores run on local time, and CI machines run on UTC. A test that asks for "today's delivery slots" fails every evening after the dates differ. Fix the clock and the zone in the test.
Animations. The cart drawer slides in for 300 ms. A click during the slide can miss. Playwright's actionability checks wait for an element to be stable, and you can turn motion off for tests.
External services. The wallet provider's test sandbox is slow or down. Mock it at the network edge with page.route in E2E tests, and check the real integration with a contract test (Contract tests).
Resource limits in CI. Your laptop has 10 cores and CI has a few. Too many workers on a small machine make tests hit timeouts. Use fewer workers per machine and more machines.

Two Playwright options remove two causes. timezoneId sets the browser's time zone, and page.clock.setFixedTime() fixes what Date.now() returns in the page:

TypeScript

// e2e/slots.spec.ts: the slot list depends on "now", so fix "now"
import { test, expect } from '@playwright/test';

test.use({ timezoneId: 'Asia/Jakarta' });          // the store's zone, on every machine

test('shows evening slots before 18:00', async ({ page }) => {
  await page.clock.setFixedTime(new Date('2026-10-20T17:30:00+07:00'));
  await page.goto('/checkout/slots');
  await expect(page.getByRole('radio', { name: '19:00 to 20:00' })).toBeVisible();
});

To find the cause, look at the pattern of failures. Fails only in the evening: dates or zones. Fails only when the whole suite runs, never alone: order or shared data. Fails more often when CI is busy: timing or resource limits. Always the same error at the same step: probably a real bug in the app. Open the trace of a failed attempt before you guess.

Quick check

A test for FRESH10 always passes when you run it alone, and fails about one time in five when the whole suite runs in parallel. Which cause fits best?

At Mangrove Market

Two months after the checkout suite went live, the E2E test "pays by wallet" failed about once in every 25 runs. The failure was always the same: the confirmation screen still said "Payment pending" when the assertion gave up. People pressed re-run, it passed, and they merged. Then Ama reported that some customers near her store had paid twice on a rainy evening. The cause was the same as the flake. The app asked the order service for the wallet result for 3 seconds and then stopped asking. When the wallet provider's callback came later, the screen stayed on "pending", and some customers paid again. A slow CI machine and slow mobile data had hit the same timing gap.

The team quarantined the test with an owner, Nadia, and a five-day deadline. She opened the trace from the first retry, saw the app stop polling at 3 seconds, and changed it to poll until the order reaches a final status. Then she ran the test 50 times in a row with no failure. Ken added the policy below to the team's test strategy, starting with one rule: a flake is a possible bug until someone proves otherwise.