My self-modifying system's test gate went green on empty output. The statistics noticed before I did.

Part 2 of a series on a 118-day study of a self-modifying multi-agent LLM system. Report and data: doi.org/10.5281/zenodo.23622286.

The system I studied rewrites its own code about a hundred times a day. Every mutation passes a test gate: the change is applied in a git worktree, the targeted tests run, and only a green result is merged.

On one day the gate rejected nothing.

The number that should not exist

In the days before, the gate rejected on average 15.3 mutations per day, about 21.6% of what it saw. On the day in question it saw 67 mutations and rejected 0.

If rejections were independent with p = 0.216, the chance of 67 passes in a row is 0.784⁶⁷ ≈ 8·10⁻⁸. Mutations are not truly independent (the same file and goal template recur), so treat that as an alarm rather than a p-value. Either way, a perfect day is not good news. It is a broken instrument.

What happened

• A mutation removed a WARNING line that a test existed to protect, and it landed.
• The next mutation of the same file entered with that test already red, as "pre-existing red", and also landed.
• Root cause: a hook in the test environment broke test collection inside the worktree. Pytest exited with code 0 and no output. The gate read exit code 0 as "passed".

Zero tests executed, zero tests failed, exit code 0. Every one of those is true, and none of them means "the change is tested".

The fix

The verdict now requires all three:

1. exit code 0,
2. at least one test actually executed,
3. zero failures.

The same pattern, everywhere

This was not a one-off. The same shape kept appearing in the system:

• tests: passed in the mutation registry meant "this stage was not skipped" at 8 of the 9 places that write it.
• Earlier, 10 of 12 landed mutations (83%) had no test that would fail if the mutation were reverted. Targeted tests were picked by whether a test mentioned the module.
• An A/B log had a field called udane ("successful") that was only ever written for runs that survived the gates. Rejected runs vanished from the denominator.

A green check, a status field or an exit code is a declaration of success. The effect is "a test ran, and it would have failed without this change". In a self-modifying system the declarations multiply much faster than anyone can verify them.

What I check now

• Count executed tests, not exit codes.
• Calibrate every gate by mutation: break the code on purpose and make sure the gate goes red. Then restore it and make sure it goes green again.
• Treat a suspiciously perfect day as an incident.

Next in the series: the sixteen measurement rules (R1–R16) I derived from my own instruments failing in exactly this way.