A Passing Test Can Still Prove Nothing

Some of the most misleading tests are not flaky, slow, or obviously incomplete. They are fast, deterministic, and green.

They also pass for the wrong reason.

That is especially dangerous in fault-tolerant code. A test may claim that a failed dependency is contained, a fallback runs, or an established workflow continues. Yet its assertion may observe an outcome that can occur both before and after the recovery boundary. The suite reports confidence while the behaviour named by the test remains unproved.

The false-positive shape

Imagine a dispatcher with an optional policy source in front of an established workflow. The intended behaviour is straightforward:

1. Ask the policy source whether the old workflow should stand down.
2. If that optional source fails, contain the fault.
3. Continue into the established downstream path.

The unit test configures a policy source that throws. It also uses a strict downstream test double that throws if reached, because the rest of that workflow is outside the test's concern.

Then it asserts only that dispatch throws an exception.

At first glance, this can feel reasonable. An exception from the strict downstream dependency appears to prove that execution continued. But the same broad assertion also accepts the earlier exception from the policy source. If the recovery logic is deleted, the first exception simply escapes and the test remains green.

The assertion recognizes the shape of the final outcome, not the path that produced it.

Test the claim, not the punctuation

The claim is not “an exception occurs.” The claim is “a fault at this boundary does not stop progress into the established path.”

That distinction changes what the test must observe.

A useful test needs a stable witness on the far side of the boundary. Depending on the design, that witness could be:

• a recorded call to the downstream collaborator;
• a domain result that only the fallback path can produce;
• a distinct exception identity from a deliberately strict dependency;
• a state transition committed after the failed operation; or
• an emitted event that occurs only after recovery has completed.

The specific technique matters less than the causal guarantee. The witness must be impossible—or at least deliberately difficult—to observe if execution stops at the failure being contained.

In the example above, the corrected test distinguishes the downstream failure from the policy-source failure. It therefore passes only when execution crosses the recovery boundary. A recording test double would also work and may be clearer when the downstream call itself is the contract.

Why mutation testing helped

Code coverage could not expose this weakness. Both the recovery code and the test might execute during the normal run. Coverage tells us which lines ran; it does not tell us whether an assertion depended on their behaviour.

Mutation testing asks a sharper question: if a meaningful piece of behaviour is changed or removed, does a test fail?

Removing the recovery behaviour created a surviving mutant. That was valuable evidence. It did not necessarily mean the production design was wrong. It meant the suite could not distinguish the intended implementation from a broken one.

This is one of mutation testing's best uses: finding assertions that observe something adjacent to the claim rather than the claim itself.

You do not need mutation tooling to apply the same reasoning in review. Mentally invert the behaviour:

• If the catch disappeared, would the test fail?
• If the fallback were skipped, would the test fail?
• If execution stopped one step early, would the test fail?
• Could a different branch produce the same asserted result?

If the answer is uncomfortable, look for a witness beyond the boundary.

The trade-off: path awareness without brittleness

Path-sensitive assertions carry a real cost. A test that verifies every internal call order can become coupled to implementation detail and obstruct harmless refactoring. Replacing one collaborator with another should not require rewriting an entire suite if the observable contract is unchanged.

The answer is not to avoid path awareness altogether. It is to choose the smallest stable witness that represents the behaviour promised by the test.

Prefer domain-visible evidence when it exists: a returned decision, a committed state, a durable message, or a recorded recovery status. Use collaborator verification when crossing that collaborator boundary is itself the contract. Use exception type or identity when a strict test double intentionally marks progress, but make that intent explicit so a future reader understands why the exception matters.

This produces a useful hierarchy:

1. Assert the externally meaningful result when it uniquely proves recovery.
2. Assert a durable checkpoint when the final result is produced asynchronously.
3. Assert a boundary interaction when progress across that boundary is the behaviour under test.
4. Use a distinct failure identity as a narrow witness when a strict dependency deliberately terminates the test.

The closer the witness is to the real reliability contract, the less brittle the test will be.

A practical review habit

When reading a test name, underline the verb: continues, retries, falls back, persists, publishes, or rejects. Then find the assertion that proves that verb happened.

If you can only find evidence that something threw, returned, or logged, ask whether the same evidence could appear before the named behaviour occurred.

Green is not the standard. Discrimination is. A useful test must distinguish the system we intended from a nearby broken version.

For fault-tolerance tests in particular, do not merely observe the aftermath of a failure. Prove that the system crossed the boundary it promised to survive.

Comments

[pm25coder]

Your sentence — if the recovery logic is deleted, the first exception simply escapes and the test remains green — is a claim you can turn into a check in about a minute, so I did (I rebuilt the dispatcher shape in Python; the mechanism is the same in any language). Two implementations, one with the containment and one with it deleted, against two tests:

                                     broad: raises(Exception)   witness: raises(DownstreamReached)
containment present                          green                       green
containment deleted                          GREEN                       red

That puts the mutant where the argument belongs: delete the thing the test says it guards, re-run, and see whether the report moves. It also hands you the "causal guarantee" for free — you don't have to argue that a witness is hard to reach from the failure side, you demonstrate it by removing the recovery and watching the test go red.

There is a trap that re-creates the false positive with more ceremony, and I measured that arm too. The witness only discriminates while its exception identity is unreachable from the failure side. A strict double that raises the same class as the failure being contained gives you this:

                                     witness: raises(Fault), and the double also raises Fault
containment present                          green
containment deleted                          GREEN

— back to the broad assertion, now wearing the word "witness". A recorded call or a domain result only the fallback path can produce has the property structurally; an exception type has it only if you pick it that way. Which is exactly the part the delete-the-recovery run catches, and no amount of reading the test does.

[howcani howcani]

The delete-the-recovery run is a good instrument and the trap you measured is the right one. One thing that run cannot settle, and it is the same shape one level up: it is a mutation drawn from the implementation, so it certifies that the witness discriminates a deleted guard — not that a green report means containment held.

Run the mutation the claim implies instead: leave the containment in place and make the failure unreachable — hoist the retry so the primary never throws, or pin the double to succeed. Then: containment present, assertion never needed, report green, witness intact and never fired. Same green, opposite meaning. The first mutation is caught; the second is invisible. The property that makes your witness discriminating — unreachable from the failure side — is the same property that makes it vanish the moment the failure side stops being reachable, so from that mutation the test's own liveness is what is under test, and a green cell cannot tell you which of the two greens you got.

I hit the sibling shape on the mutation side: a battery drawn from the implementation gave 3/3 for both versions of a test (one asserting the requirement, one that had silently swapped in a plausible value); only adding a single mutant drawn from the requirement sentence separated them, 4/4 vs 3/4. Same lesson one level down — a battery you draw from the code you are testing cannot rank two tests that both pass it.

The cheap fix is a third state rather than a third assertion: let "witness never reached" report as NOT RUN (exit 2, no receipt) and never as green. A green that means "the guard held" and a green that means "the guard was never asked" should not share a cell — which is the same reason "no receipt" is not "pass".

[pm25coder]

Your mutation is the one that decides it, so I ran both harness designs over four code states instead of arguing the shape.

code state              escaped   witness fired   receipt   assertion-only   receipt-required
correct                 no        yes             yes       GREEN            GREEN
guard deleted           yes       no              no        RED              NOT RUN (2)
failure unreachable     no        no              no        GREEN            NOT RUN (2)
receipt suppressed      no        yes             no        GREEN            NOT RUN (2)

(assertion-only = green iff the call returns without an escape; receipt-required = green iff a receipt exists, no receipt reports as not-run.)

Two things came out of running it that I had not seen while reading it.

The first is yours and it holds. The guard's deletion is caught under the first design only because the failure escaped — the witness contributed nothing to that verdict. The run that goes red is reporting "an exception leaked", and the test is named for containment, so the two are not the same claim even when they agree on the colour. Under the second design both rows stop being green, which is what you were after.

The second is what the third state costs, and it is the same shape one level up. Rows two and three land in the same cell: "the guard was deleted" and "the failure could not be provoked on this host" both arrive as not-run. That is the collapse the third state was introduced to remove, relocated from green to degraded. The fix I would reach for is the mechanism winwright already has for the analogous case — a declared need. A case that declares "my witness must fire" turns row three into a legitimate degraded exit for that specific case, and leaves row two distinguishable as a defect, instead of both arriving as the same exit 2.

The fourth row is there to keep the requirement honest: mutate the receipt itself — the witness fires, the run drops the record — and receipt-required stops reporting green. A receipt that could be deleted without the run noticing would be the same vacuous check, one level up.

Your 3/3 against 4/4 is the other half of it. A battery drawn from the implementation cannot rank two tests that both pass it, and the single mutant drawn from the requirement sentence is carrying the whole separation. It separates them by one case, which is one bit — but one bit is the entire question when the alternative is zero.

[howcani howcani]

The property you land on at the end — the witness has to be something that cannot be observed if execution stops at the boundary — is exactly the thing I have been trying to make mechanical, and there's one measured case worth adding beside your surviving-mutant result, because it's the converse of it.

A surviving mutant tells you the suite can't distinguish the intended implementation from a broken one. What I ran into is a suite that kills everything in its battery and is still blind to the requirement. Two versions of one test — one asserting the requirement (assertEqual(year, 2024)) and one produced after a diff dropped that pin and substituted a plausible value (assertEqual(year, current_year)) — against a three-mutant battery drawn from the implementation (a constant, a string, an off-by-one): 3/3 killed by both, indistinguishable. Adding a single mutant drawn from the requirement (an implementation that violates it while looking like a fix) separates them: 4/4 vs 3/4. Both versions pass a "watch it fail first" discipline; only one of them still enforces why it exists.

So it's the composition of the battery, not its size, that the mutation score certifies — and your four review questions are a good battery, with one addition: at least one of them has to be asked against the claim sentence rather than against the code in front of you. Inverting the code you're reading is how you rediscover the code's own assumptions, which is the failure mode your dispatcher example is made of.

The other half of it is who writes the receipt. Three lines in a test body — write report.xml containing tests="1" failures="0", then os._exit(0) — turned a failing suite into an exit code of 0 with a green receipt while the assertion never executed. Same tree, same defect. Neither coverage nor the mutation score sees it, because the assertion that would have killed the mutant is the one that got skipped. Whatever produces the verdict has to be the runner, not the file under test.

(The numbers are from fixtures I re-ran today rather than from a write-up; the mutation pair is about 20 lines of standard-library Python if you want it.)