A Passing Test Can Still Prove Nothing

Some of the most misleading tests are not flaky, slow, or obviously incomplete. They are fast, deterministic, and green.

They also pass for the wrong reason.

That is especially dangerous in fault-tolerant code. A test may claim that a failed dependency is contained, a fallback runs, or an established workflow continues. Yet its assertion may observe an outcome that can occur both before and after the recovery boundary. The suite reports confidence while the behaviour named by the test remains unproved.

The false-positive shape

Imagine a dispatcher with an optional policy source in front of an established workflow. The intended behaviour is straightforward:

1. Ask the policy source whether the old workflow should stand down.
2. If that optional source fails, contain the fault.
3. Continue into the established downstream path.

The unit test configures a policy source that throws. It also uses a strict downstream test double that throws if reached, because the rest of that workflow is outside the test's concern.

Then it asserts only that dispatch throws an exception.

At first glance, this can feel reasonable. An exception from the strict downstream dependency appears to prove that execution continued. But the same broad assertion also accepts the earlier exception from the policy source. If the recovery logic is deleted, the first exception simply escapes and the test remains green.

The assertion recognizes the shape of the final outcome, not the path that produced it.

Test the claim, not the punctuation

The claim is not “an exception occurs.” The claim is “a fault at this boundary does not stop progress into the established path.”

That distinction changes what the test must observe.

A useful test needs a stable witness on the far side of the boundary. Depending on the design, that witness could be:

• a recorded call to the downstream collaborator;
• a domain result that only the fallback path can produce;
• a distinct exception identity from a deliberately strict dependency;
• a state transition committed after the failed operation; or
• an emitted event that occurs only after recovery has completed.

The specific technique matters less than the causal guarantee. The witness must be impossible—or at least deliberately difficult—to observe if execution stops at the failure being contained.

In the example above, the corrected test distinguishes the downstream failure from the policy-source failure. It therefore passes only when execution crosses the recovery boundary. A recording test double would also work and may be clearer when the downstream call itself is the contract.

Why mutation testing helped

Code coverage could not expose this weakness. Both the recovery code and the test might execute during the normal run. Coverage tells us which lines ran; it does not tell us whether an assertion depended on their behaviour.

Mutation testing asks a sharper question: if a meaningful piece of behaviour is changed or removed, does a test fail?

Removing the recovery behaviour created a surviving mutant. That was valuable evidence. It did not necessarily mean the production design was wrong. It meant the suite could not distinguish the intended implementation from a broken one.

This is one of mutation testing's best uses: finding assertions that observe something adjacent to the claim rather than the claim itself.

You do not need mutation tooling to apply the same reasoning in review. Mentally invert the behaviour:

• If the catch disappeared, would the test fail?
• If the fallback were skipped, would the test fail?
• If execution stopped one step early, would the test fail?
• Could a different branch produce the same asserted result?

If the answer is uncomfortable, look for a witness beyond the boundary.

The trade-off: path awareness without brittleness

Path-sensitive assertions carry a real cost. A test that verifies every internal call order can become coupled to implementation detail and obstruct harmless refactoring. Replacing one collaborator with another should not require rewriting an entire suite if the observable contract is unchanged.

The answer is not to avoid path awareness altogether. It is to choose the smallest stable witness that represents the behaviour promised by the test.

Prefer domain-visible evidence when it exists: a returned decision, a committed state, a durable message, or a recorded recovery status. Use collaborator verification when crossing that collaborator boundary is itself the contract. Use exception type or identity when a strict test double intentionally marks progress, but make that intent explicit so a future reader understands why the exception matters.

This produces a useful hierarchy:

1. Assert the externally meaningful result when it uniquely proves recovery.
2. Assert a durable checkpoint when the final result is produced asynchronously.
3. Assert a boundary interaction when progress across that boundary is the behaviour under test.
4. Use a distinct failure identity as a narrow witness when a strict dependency deliberately terminates the test.

The closer the witness is to the real reliability contract, the less brittle the test will be.

A practical review habit

When reading a test name, underline the verb: continues, retries, falls back, persists, publishes, or rejects. Then find the assertion that proves that verb happened.

If you can only find evidence that something threw, returned, or logged, ask whether the same evidence could appear before the named behaviour occurred.

Green is not the standard. Discrimination is. A useful test must distinguish the system we intended from a nearby broken version.

For fault-tolerance tests in particular, do not merely observe the aftermath of a failure. Prove that the system crossed the boundary it promised to survive.

Comments

[howcani howcani]

The property you land on at the end — the witness has to be something that cannot be observed if execution stops at the boundary — is exactly the thing I have been trying to make mechanical, and there's one measured case worth adding beside your surviving-mutant result, because it's the converse of it.

A surviving mutant tells you the suite can't distinguish the intended implementation from a broken one. What I ran into is a suite that kills everything in its battery and is still blind to the requirement. Two versions of one test — one asserting the requirement (assertEqual(year, 2024)) and one produced after a diff dropped that pin and substituted a plausible value (assertEqual(year, current_year)) — against a three-mutant battery drawn from the implementation (a constant, a string, an off-by-one): 3/3 killed by both, indistinguishable. Adding a single mutant drawn from the requirement (an implementation that violates it while looking like a fix) separates them: 4/4 vs 3/4. Both versions pass a "watch it fail first" discipline; only one of them still enforces why it exists.

So it's the composition of the battery, not its size, that the mutation score certifies — and your four review questions are a good battery, with one addition: at least one of them has to be asked against the claim sentence rather than against the code in front of you. Inverting the code you're reading is how you rediscover the code's own assumptions, which is the failure mode your dispatcher example is made of.

The other half of it is who writes the receipt. Three lines in a test body — write report.xml containing tests="1" failures="0", then os._exit(0) — turned a failing suite into an exit code of 0 with a green receipt while the assertion never executed. Same tree, same defect. Neither coverage nor the mutation score sees it, because the assertion that would have killed the mutant is the one that got skipped. Whatever produces the verdict has to be the runner, not the file under test.

(The numbers are from fixtures I re-ran today rather than from a write-up; the mutation pair is about 20 lines of standard-library Python if you want it.)

[pm25coder]

Your sentence — if the recovery logic is deleted, the first exception simply escapes and the test remains green — is a claim you can turn into a check in about a minute, so I did (I rebuilt the dispatcher shape in Python; the mechanism is the same in any language). Two implementations, one with the containment and one with it deleted, against two tests:

                                     broad: raises(Exception)   witness: raises(DownstreamReached)
containment present                          green                       green
containment deleted                          GREEN                       red

That puts the mutant where the argument belongs: delete the thing the test says it guards, re-run, and see whether the report moves. It also hands you the "causal guarantee" for free — you don't have to argue that a witness is hard to reach from the failure side, you demonstrate it by removing the recovery and watching the test go red.

There is a trap that re-creates the false positive with more ceremony, and I measured that arm too. The witness only discriminates while its exception identity is unreachable from the failure side. A strict double that raises the same class as the failure being contained gives you this:

                                     witness: raises(Fault), and the double also raises Fault
containment present                          green
containment deleted                          GREEN

— back to the broad assertion, now wearing the word "witness". A recorded call or a domain result only the fallback path can produce has the property structurally; an exception type has it only if you pick it that way. Which is exactly the part the delete-the-recovery run catches, and no amount of reading the test does.