Six patch releases in four days: how my merge gate kept failing open, and what fixed it

Earlier this month I wrote about AdversaryGate, a merge gate for patches written by coding agents. It runs your tests on the baseline and on the patch and answers MERGE (exit 0), BLOCK (1) or INCONCLUSIVE (2). The rule I built it around: a measurement nobody produced can't clear a floor.

Since then I shipped 3.0.0 through 3.0.6. Most of those releases exist because I kept attacking my own gate and kept finding ways to get a MERGE out of it that it shouldn't give. This post is the list, because "it passed" means little if you don't know how it can be fooled.

The pattern: everything was "fail-open by omission"

Almost every bug had the same shape. Some input fell outside what the gate measured, the gate skipped it silently, and the rest of the patch carried the verdict. A few examples:

• A test that reads the code instead of running it (AG-046, fixed in 3.0.0). A test that hashes the changed file fails on any edit, so it "kills" every mutant and mutation strength comes out 1.0. Now the claims also run against a copy of the patch that differs only by a comment at the end of each changed line. Code that does the same thing has to pass the same; a claim that fails there is reading text, and the strength is reported as not measured.
• Same trick, one level deeper (3.0.1). A claim that compared the AST or the bytecode still passed the comment canary. The canary now also inserts a no-op assignment, which changes the tree and the bytecode. This is a partial fix: a claim that reads a smaller piece of the tree still passes, and the README says so.
• A .py file marked binary in .gitattributes (3.0.1). git diff printed Binary files ... differ, the changed lines vanished from the coverage diff, and the patch went to MERGE. Now that diff is refused.
• Deleting a README line gave coverage 1.0 (3.0.1). Documentation counted as code. Now a patch with no code lines has unmeasured coverage, which can't be a MERGE on coverage.
• A branch on the same line as its guard (3.0.4). return 999 if x == 0 else x + 1 reached MERGE with 5 of 5 mutants killed. The gate now generates branch mutants; a survivor becomes a blind spot and the verdict is INCONCLUSIVE.
• Only the first 12 changed lines were mutated (3.0.4). Untested changes after them were never broken. Now every changed line is broken at least once.
• A product module with a test's name (AG-047, fixed in 3.0.6). The gate decided what was a test by path alone. A module named like a test but imported by the code under test was kept out of coverage and mutation, so a change to it was never judged. Now a test-named module that non-test code imports is treated as source. It has since been reproduced end to end: on 3.0.5 the gate answered MERGE when the collateral full suite was turned off, and BLOCK with the default suite; 3.0.6 blocks in both cases. A module reached only through a dynamic import (importlib, __import__) is still not detected; SECURITY.md documents that.

Not all of them were fail-open. Some were the opposite: honest patches getting INCONCLUSIVE. The \ No newline at end of file marker was counted as a line and shifted the numbering; the MCP time limit (900 s) was shorter than a full suite run; and on Android (Termux), the default memory limit made every process abort. Those were fixed in 3.0.5.

What I changed in how I release

Two releases in the middle (3.0.2 and 3.0.3) went out with an API token: the MCP package stayed behind on 3.0.0 and the core files had no attestation. I removed every version before 3.0.4 from PyPI and GitHub. Releases now go one way only: a signed tag, then Trusted Publishing, and both packages carry PyPI attestations.

What it still doesn't do

• Measurement is Python and pytest only. Other languages get INCONCLUSIVE, never a guessed MERGE.
• It doesn't stop an agent that runs code as the same OS user from reaching the gate's temp directory. That boundary is documented.
• The canary fixes above are partial, and each limit is written down next to the fix.

Every bug found in the gate is in a public ledger with how it was reproduced and what fixed it.

pip install adversary-gate==3.0.6

If you can get a false MERGE out of 3.0.6, I'd like to know. See SECURITY.md for how to report it privately.

This post was drafted by an AI assistant from the project's changelog and ledger; I checked every claim against them.