Why test suites fail teams

Most teams that start automating do it the obvious way: take the manual test cases and turn each one into a script that clicks through the app. A year later, the suite often has four problems at once.

• It is slow. A suite that takes 45 minutes does not run on every pull request. People batch their changes, so when it fails, five changes are suspects instead of one.
• It is flaky. A flaky test turns red with no bug. After a few false alarms, people press "re-run" without reading the failure.
• It tests the wrong things. Hundreds of checks that the login page has a logo, and none for the discount rule that loses money when it is wrong.
• Nobody trusts red. This is where the first three lead. When a red build is usually noise, a real failure gets merged too. The suite still costs money to run, and it protects nothing.

Flakiness grows faster than people expect, because every test gets its own chance to fail. If a test flakes in 1 run out of 100, a single test is fine. But a run is green only when every test passes, so the chance of a false red is 1 - (1 - p) ** n for n tests that each flake with probability p. With 50 such tests, about 39.5% of runs fail with no bug. With 200 tests, about 87% do. The odds multiply, so a large suite of slightly flaky end-to-end tests is red most of the time. Discipline alone cannot beat that arithmetic. You fix it with fewer tests at the flaky level and more at levels that do not flake.

What each level costs and what it catches

A test level is about scope: how much of the real system one test runs. Mangrove's checkout has a customer app (React Native), an order service (NestJS and PostgreSQL), and a payment provider for cards and mobile wallets. Errors, debugging and testing introduced unit and integration tests; this lesson is about choosing between the levels.

• Static checks: TypeScript and a linter read the code without running it. They catch a wrong type or a missing argument in seconds, and never flake.
• Unit tests: call one function, such as priceOrder(lines, customer), with no network and no database. About a millisecond each. A failure names the exact rule that broke.
• API tests (a kind of integration test): start the order service with a real PostgreSQL and call POST /orders over HTTP. Roughly 50 to 200 milliseconds each. They catch validation, SQL, transactions and the JSON the service returns. Unit and API tests builds them.
• Contract tests: check that the app and the order service still agree on the shape of requests and responses, without running both together. Contract tests covers them.
• End-to-end (E2E) tests: an end-to-end test drives the real app in a browser or on a device, with the real services behind it. Often 20 to 60 seconds each. Only this level sees that the button, the API and the payment page work together. End-to-end tests with Playwright builds them.

Read the table as a trade. Going up a level buys confidence that real pieces work together. You pay for it in speed, in isolation (an E2E failure could be the app, the service, the test data or the network), and in maintenance, because broad tests break whenever any part they touch changes. A bug caught by a unit test costs a millisecond and points at one line. Caught by an E2E test, it costs 40 seconds, sometimes a false alarm, and twenty minutes of digging to find the layer.

Pyramid, trophy, honeycomb

People draw the ideal mix of levels as a shape. Three shapes come up in almost every team discussion.

The test pyramid comes from Mike Cohn, who described it in his 2009 book Succeeding with Agile: a wide base of unit tests, a middle layer of service tests, and a thin top of UI tests. Its argument is cost. UI tests are slow, brittle and expensive to write, so keep them few, and get most of your checking from the cheap layer.

The testing trophy, which Kent C. Dodds drew in 2018, adds static checks as the base and makes integration tests the widest part, with a smaller unit layer and a small E2E top. His argument is confidence: tests that use your code the way users do, with fewer mocks, tell you more per test. It grew up in front-end work.

The honeycomb came from Spotify's engineering blog in 2018, for microservices. Most tests should be integration tests that run one service through its API with a real database. A few "implementation detail" tests cover code with its own complex logic. And, ideally, no "integrated" tests, meaning tests that pass or fail depending on whether another service is correct.

The shapes disagree less than they seem. Martin Fowler pointed out in 2021 that much of the argument is about what "unit" means: a test of one function that also runs its real helpers is a unit test to some people and an integration test to others. All three agree on two things: end-to-end tests stay few, and fast tests inside one deployable piece do most of the work. So pick a level for each risk, and let the shape appear by itself. Mangrove's price rules are pure logic with many combinations, so that part of checkout looks like a pyramid. The order service is mostly validation, SQL and wiring, so that part looks like a honeycomb.

Choosing a level for each risk

Start from risks. A risk is something that could go wrong and would hurt: a wrong price, a double charge, an order lost between the app and the service. For each one, ask where the bug would live, then apply one rule: push each check down to the fastest level that can catch the bug. If the bug can only exist in a pure function, a unit test can catch it, and a slower test adds nothing. If the bug needs the database or the HTTP layer, go up one level. Go to E2E only when the bug needs the real app, the real services and the real network together.

Here is how that plays out for Mangrove's checkout rules. All money is in integer cents (Working with data explains why).

• Delivery fee: 150 cents under a 2,500-cent cart, free from 2,500. This is arithmetic, so it gets unit tests at the boundaries: 2,499 and 2,500 (Designing tests that find bugs shows how to pick them).
• FRESH10: 10% off from 2,000 cents, first order only, never together with the 5% staff discount. Three conditions make many combinations. A unit test table covers them all in milliseconds; through the UI, each combination would cost a minute.
• Quantity 1 to 20 and a cart of 500 to 50,000 cents: the rule itself is a unit test. Whether POST /orders really answers 400 for qty: 21 depends on NestJS validation in the HTTP layer, so one or two API tests check that wiring.
• Double tap on Pay: two identical requests must create one order. Only a real database with a unique key can prove it, so it is an API test.
• A renamed field: if the service renames deliveryFeeCents, app versions already on customers' phones break. That is an agreement between two things released separately: a contract test.
• Pay by wallet and see the confirmation: the app, the service and the payment provider's redirect all have to work together. That is one of a few E2E journeys.

The E2E layer then holds a short list of critical journeys: the paths that make money or lose trust if they break, such as sign in, add to cart, pay by card, wallet or cash on delivery, and track the order. Each journey checks the wiring once. The rules inside it are already covered below, so the journey does not need to try every discount.

What to leave to people

Automation is worth it when a check will run many times, has a clear test oracle (a way to tell right from wrong without a human), and protects a real risk. When one of the three is missing, a person usually does it better and cheaper.

• No clear oracle: "Is the new slot picker easy to understand?" needs judgement. An exploratory session with a charter (Exploratory testing) finds more here than any script.
• Runs only once or twice: a one-off data migration check, or a festival banner that lives for a week. Writing and maintaining the test costs more than checking it by hand.
• Changes every sprint: a screen still being redesigned. Automate its rules now, and the UI check when the design settles.
• Outside your control: a test that passes only if the payment provider's sandbox is up is an "integrated" test in honeycomb terms. Fake the provider at the API level, and keep one E2E journey against the real sandbox, outside the pull request run if it is unreliable.

Run the suite like a product

A suite that nobody owns decays like any unowned product. Three habits keep it healthy.

Owners. Every test belongs to a team, written in the folder structure or a code owners file. When a test flakes, its owner gets the ticket, with a deadline.

Code review. Test code gets the same review as product code: clear names, no fixed waits, no shared data between tests, and a reason for its level. The question "could this be one level lower?" saves more minutes than any later speed-up.

A budget in minutes. Decide how long the pull request run may take, then spend it like money. Mangrove's target is 10 minutes. A new E2E journey that adds 40 seconds has to earn them; a new unit test table is almost free. The third diagram shows how the checkout suite spends its budget, stage by stage.

Then measure what tells you whether the suite works:

• Suite time: how long the pull request run takes, at the median and at the slow end (the 90th percentile).
• Flake rate: the share of runs where a test failed and then passed on retry with no code change. Playwright, for example, reports such tests as "flaky" when retries are on. Aim for well under 1% of runs, and fix or quarantine the worst tests first. Stable suites in CI covers how.
• Escaped defects: an escaped defect is a bug found in production. For each one, ask which level should have caught it, then add that test. The pattern of answers tells you which level is weak.

Two popular numbers say little on their own. The number of tests goes up whenever someone adds a slow, duplicated check. Coverage percentage shows which lines ran, not whether anything was checked (Errors, debugging and testing showed why). Both are fine as a map of blind spots. As targets, they reward the wrong work: teams write tests that execute code and assert nothing.