Your test pyramid is a claim. Here's how to prove it

Every QA strategy deck has the pyramid on slide four. Wide unit base, narrower integration band, a few end-to-end tests on top. Everyone nods, because the picture is a statement of good intentions.

Then something ships broken, and the retro asks the only question that matters: which level should have caught this? Usually nobody knows. The suite is green, the coverage number is respectable, and the defect walked through every layer without a single test noticing. The pyramid was never wrong — it was just never checked.

I got tired of arguing about this in the abstract, so I built a small system to argue with instead: a phone-financing backend where a customer buys days of access one payment at a time, plus the complete testing strategy around it — 148 tests across seven levels, a traceability matrix, and a delivery pipeline that promotes one commit through three environments. It is a public repository, so everything below can be read, cloned and disagreed with: github.com/denisspunk/loan-platform-qa-lab.

Three things in it changed how I set this up for teams.

1. The shape of your pyramid is a finding, not a target

The textbook triangle says most tests should be unit tests. In the lab, the widest band is not unit (35 tests) — it is integration (59).

That looked wrong until I asked where the rules actually live. Validation and routing sit in the HTTP layer: "a payment of zero is refused", "an unknown loan is a 404 with a JSON body, never an HTML page", "limit is between 1 and 100". No unit test of the domain can reach any of them, because the domain never sees the request. Pushing those checks down a level would not have made them cheaper — it would have made them fictional.

So the shape is diagnostic. A fat integration band means logic has accumulated in the transport layer — worth knowing, and a conversation about the service's design rather than about the test strategy. What you must not do is redraw the chart to match the slide.

2. Break the code on purpose, then see who notices

The only way I know to test a test suite is to break the thing it guards.

The service ships with four defects seeded into it, off by default behind a flag: mvn test -Dlab.bugs=ROUNDING_UP turns one on. Then you run the pyramid and count red tests per level.

• Days rounded up — 8 unit, 2 component, 2 integration, 1 end-to-end test go red. First caught at unit, in under a minute.
• A repeated payment id applied twice — 1 unit, 1 component, 1 integration. First caught at unit.
• A timestamp sent to the device-lock partner as a number instead of ISO — 0 unit, 0 component, 2 contract, 1 integration, 2 end-to-end. Invisible to the domain, first caught at contract.
• A payment of zero accepted — 0 unit, 0 component, 0 contract, 1 integration, 0 end-to-end.

Two findings fall straight out of those runs.

The good one: no seeded defect survives. Each one turns at least one test red, and most are caught at the cheapest level that could possibly see them — arithmetic at unit, an outbound payload at contract. That is the claim a pyramid makes, and now it is measured rather than asserted.

The uncomfortable one: the zero-payment defect is caught by exactly one test, three levels up. A money rule hangs on a single thread. I did not discover that by staring at a coverage report — 100% line coverage of the validation method would have looked identical. I discovered it by breaking the rule and watching how little happened.

This is mutation testing's idea, applied by hand at the level of business rules rather than statements, and it costs an afternoon on an existing service: pick your five most expensive rules, break each one behind a flag, run the suite, write down what turns red. The rules where only one test fails — or none — are your real backlog.

3. One fact, stated once: the tag is the level is the CI stage

The thing that makes the above cheap to run is boring: every test carries exactly one level tag, one package holds one level, and the pipeline selects on that tag.

mvn test -Dgroups=unit                       # 35 tests, no Docker, seconds
mvn test -Dgroups="unit | component"         # what the first CI stage runs
mvn test -Dgroups=integration                # real Postgres via Testcontainers
mvn test -Dgroups=F-04                       # every test pinning one known defect

The directory layout, the tag and the pipeline stage are the same fact written down once. Stages run cheapest-first and each one starts only if the level below is green, so a broken arithmetic rule fails in under a minute instead of after Docker has finished pulling Postgres.

One detail that looks like plumbing and is actually about credibility: every stage resolves its dependencies in a separate step with retries, before any test runs. A registry outage used to turn the end-to-end stage red, and a suite that cries defect when the network hiccups is a suite people stop believing. Infrastructure failures must fail an infrastructure step.

The part that decides whether any of this is assertable: the 202

The payment endpoint answers 202 and applies the payment afterwards — HTTP in, event bus, processor, outbound call to the device-lock partner. Nothing a test wants to know is in that response. This is where most suites quietly rot: someone adds a sleep, the sleep is too short on a loaded runner, the test goes flaky, and within a month the team has learned to re-run red builds instead of reading them.

So the step waits, and the wait is a condition rather than a duration — read paid before sending, send, then poll the loan until paid has grown by the amount sent:

private LoanJson waitUntilPaidAtLeast(LoanJson loan, long expectedPaid) {
    return await("payments applied to " + loan.id())
            .atMost(Config.ASYNC_TIMEOUT)
            .pollInterval(Duration.ofMillis(50))
            .until(() -> loanSteps.current(loan), current -> current.paid() >= expectedPaid);
}

The interesting case is the one where that trick cannot work. To check at-least-once delivery you send the same payment id twice, and the second copy is supposed to change nothing — so there is no growth to wait for, and "wait a bit, then assert the balance" is exactly the race that makes a suite untrustworthy. What makes it assertable is a property of the bus: it handles events in order.

public LoanJson payTwiceWithSameId(LoanJson loan, long amount) {
    long paidBefore = loanSteps.current(loan).paid();
    String paymentId = uniquePaymentId();
    send(loan, paymentId, amount);
    send(loan, paymentId, amount);
    send(loan, uniquePaymentId(), 1);          // a marker, right behind both copies
    return waitUntilPaidAtLeast(loan, paidBefore + amount + 1);
}

Once the marker has been applied, both copies have already been processed. No sleep, no flake, and the comment in that method says out loud what it depends on: if the bus ever processes events in parallel, this step stops being a guarantee. That sentence is worth more than the code around it — it names the assumption the whole duplicate story rests on, so the day someone parallelises the consumer, the person reading this knows what broke.

Two more decisions follow from the same place. Time is an input: the journey tests move a clock by hand — clock.advance(Duration.ofHours(25)) — and assert that the phone relocked by itself, which is a state change no request triggers and no sleep could ever wait for. And every check of one result is soft, so a failing payment shows paid, balance, credit, status and device state together instead of stopping at the first mismatch.

None of that is framework taste. Unique device and payment ids per test are what make the same test body safe to point at a shared deployed stand; the clock is what keeps expected dates out of hardcoded strings; and "steps wait, tests assert" is what keeps the expected value in the test you are reading rather than in a helper two files away.

The pipeline: one commit, three gates, and a rollback that fires on the right things

Above the pyramid sits delivery. One commit — the same SHA the pyramid passed — is promoted to dev, then stage, then production, each behind a gate that deploys and then tests the deployed stand. Nothing is rebuilt per environment, and the tests that run against a stand are that commit's tests.

Four decisions in there have paid for themselves repeatedly:

Gates are asymmetric by design. Dev and stage get the full stand regression, because the loans it opens are disposable. Production gets the read-only half of the same suite: the business rules are still checked against prod, but no test suite creates real records. The read-only subset is a tag, not a separate copy of the tests.

The guard lives in the callee, not the caller. "Deploy a commit to a stand" has exactly one implementation, and the production rule lives inside it — so the escape hatch that skips every gate (hotfixes are real) inherits the same protection for free. If each workflow had to remember the rule itself, the rule would hold right up until someone wrote a sixth workflow.

Rollback fires when the deploy fails, not only when the gate fails. A deploy that times out is not a deploy that stopped; it can go live minutes after the pipeline has already declared failure, leaving production on a commit no gate ever looked at. That is not a thought experiment — it happened in this lab, and the fix was to cancel a timed-out deploy rather than abandon it.

Every run prints two commits: what the stand is running, and what the tests came from. When they differ, the run says so. That difference has explained more red stand runs than any actual defect.

Coverage answers a different question than you think

The lab measures line coverage with JaCoCo, merged across four CI machines into one number per pull request. It is useful, and it is not evidence of anything about risk: it tells you which lines ran.

The document that answers the other question is a traceability matrix — 57 written-down requirements, each mapped to the tests that defend it, with the gaps stated out loud: five requirements covered by nothing, and a sentence on each explaining why it is still open. A CI job compares that document against the suite and fails the build when they disagree — when a test references a finding the document does not list, or the document names a test that has been renamed. Without that check, a traceability matrix is accurate for about three weeks.

Five uncovered requirements, named, is a far more honest artefact than 80% coverage.

What to take from this on Monday

If you own quality for a service and want to know whether your pyramid is real:

1. Break your five most expensive business rules, one at a time, behind a flag. Count what turns red.
2. Any rule where fewer than two tests fail is a gap, regardless of coverage.
3. Check whether your level structure is one fact or three — can CI run exactly one level by name?
4. Make dependency resolution its own step, so an outage can never masquerade as a defect.
5. Write down what nothing tests, and put that list in the repository rather than in your head.

None of this needs a new tool or a testing platform. It needs treating the pyramid as a hypothesis about where defects get caught, and then running the experiment.

The whole lab is public — seeded defects, traceability matrix, five workflows, gates: github.com/denisspunk/loan-platform-qa-lab. Clone it, turn on a defect, and see which of your instincts about the pyramid survive.

If your own suite is green and you still cannot say what it would catch, that is usually the first thing I measure in a QA audit engagement — before anyone writes another test.