Eight of my nine mutants died, and the survivor wasn't a weak test

I ran a small mutation pass over the verification path in a project of mine. Nine mutants, eight killed, one survived. The survivor is the interesting one, and the reason it survived is not the reason the technique trains you to expect.

The standard reading of a surviving mutant is that your tests are weak: the code changed, nothing went red, so nothing was really checking that behavior. That's usually right. It wasn't right here, and I spent a while assuming it was before I looked properly.

The survivor

The mutation flipped a field called verdict on an inner struct, ReceiptPayload. The test sitting over that area asserts on the outer envelope, a DsseEnvelope. The envelope wraps the payload as bytes. So the test read a serialized blob and a payload_type string, and the mutated field lived inside the blob, in a part the assertion never decoded.

Nothing was wrong with the test. It was checking the envelope, deliberately, because that's the layer it's about. The mutation simply didn't land anywhere the test could see it. I'd pointed a lever at the inside of a box and then measured the outside of the box.

When I moved the mutation to payload_type, which is a field the envelope test actually reads, it went red immediately. Same test, same tool, one different target. That's the confirmation that the detector was awake the whole time.

A green negative control has two causes, not one

This is the part worth keeping. When you deliberately break something and nothing fails, there are two explanations and they need opposite fixes:

1. The checker is asleep. The assertion is vacuous, or it's asserting something true either way, or the test never runs. Fix the test.
2. The break didn't land in anything the checker reads. The test is fine. Fix the mutation.

Treating the second as the first is how you end up "strengthening" a test that was already correct, usually by widening it until it asserts on things it has no business asserting on. The envelope test does not want to know about verdict. Making it know would couple a layer to the layer beneath it purely to satisfy a mutation score.

The discriminator

Cheap, and it's the whole method: name the assertion, then check whether the mutated field is inside what that assertion reads.

Not "does the test touch this code path". Coverage answers that, and coverage said yes here, which is exactly why it misled me. The payload struct was constructed, serialized, and carried through the envelope. Every line executed. The question is narrower: of the values this assertion compares, is the mutated one among them, or is it inside an opaque field — a byte blob, a hash, a serialized string?

If it's inside an opaque field, the mutant survives no matter how good the test is, and its survival tells you nothing about test quality.

A blunter version that also works: pick a field the assertion demonstrably does read, mutate that instead, and confirm you get a red. If you do, the tool and the test are both working, and your original survivor is a targeting problem. That's the run that settled it, and it took about a minute.

What I had wrong

My first diagnosis was written down as "test too weak here, needs a deeper assertion", and I nearly acted on it. Eight kills out of nine had made me confident the harness was calibrated, and confidence in the harness is what made me read the ninth as a finding about the code instead of a finding about my own setup. A negative control that comes back green is a claim about two things at once, and I'd assigned it to one of them without checking which.

Repo is TraceFold/tracefold, Rust, Apache-2.0. Still alpha, and the limits page is longer than the feature list.