DEV Community: Sanflow

Six patch releases in four days: how my merge gate kept failing open, and what fixed it

Sanflow — Thu, 15 Oct 2026 10:00:28 +0000

Earlier this month I wrote about AdversaryGate, a merge gate for patches written by coding agents. It runs your tests on the baseline and on the patch and answers MERGE (exit 0), BLOCK (1) or INCONCLUSIVE (2). The rule I built it around: a measurement nobody produced can't clear a floor.

Since then I shipped 3.0.0 through 3.0.6. Most of those releases exist because I kept attacking my own gate and kept finding ways to get a MERGE out of it that it shouldn't give. This post is the list, because "it passed" means little if you don't know how it can be fooled.

The pattern: everything was "fail-open by omission"

Almost every bug had the same shape. Some input fell outside what the gate measured, the gate skipped it silently, and the rest of the patch carried the verdict. A few examples:

A test that reads the code instead of running it (AG-046, fixed in 3.0.0). A test that hashes the changed file fails on any edit, so it "kills" every mutant and mutation strength comes out 1.0. Now the claims also run against a copy of the patch that differs only by a comment at the end of each changed line. Code that does the same thing has to pass the same; a claim that fails there is reading text, and the strength is reported as not measured.
Same trick, one level deeper (3.0.1). A claim that compared the AST or the bytecode still passed the comment canary. The canary now also inserts a no-op assignment, which changes the tree and the bytecode. This is a partial fix: a claim that reads a smaller piece of the tree still passes, and the README says so.
A .py file marked binary in .gitattributes (3.0.1). git diff printed Binary files ... differ, the changed lines vanished from the coverage diff, and the patch went to MERGE. Now that diff is refused.
Deleting a README line gave coverage 1.0 (3.0.1). Documentation counted as code. Now a patch with no code lines has unmeasured coverage, which can't be a MERGE on coverage.
A branch on the same line as its guard (3.0.4). return 999 if x == 0 else x + 1 reached MERGE with 5 of 5 mutants killed. The gate now generates branch mutants; a survivor becomes a blind spot and the verdict is INCONCLUSIVE.
Only the first 12 changed lines were mutated (3.0.4). Untested changes after them were never broken. Now every changed line is broken at least once.
A product module with a test's name (AG-047, fixed in 3.0.6). The gate decided what was a test by path alone. A module named like a test but imported by the code under test was kept out of coverage and mutation, so a change to it was never judged. Now a test-named module that non-test code imports is treated as source. It has since been reproduced end to end: on 3.0.5 the gate answered MERGE when the collateral full suite was turned off, and BLOCK with the default suite; 3.0.6 blocks in both cases. A module reached only through a dynamic import (importlib, __import__) is still not detected; SECURITY.md documents that.

Not all of them were fail-open. Some were the opposite: honest patches getting INCONCLUSIVE. The \ No newline at end of file marker was counted as a line and shifted the numbering; the MCP time limit (900 s) was shorter than a full suite run; and on Android (Termux), the default memory limit made every process abort. Those were fixed in 3.0.5.

What I changed in how I release

Two releases in the middle (3.0.2 and 3.0.3) went out with an API token: the MCP package stayed behind on 3.0.0 and the core files had no attestation. I removed every version before 3.0.4 from PyPI and GitHub. Releases now go one way only: a signed tag, then Trusted Publishing, and both packages carry PyPI attestations.

What it still doesn't do

Measurement is Python and pytest only. Other languages get INCONCLUSIVE, never a guessed MERGE.
It doesn't stop an agent that runs code as the same OS user from reaching the gate's temp directory. That boundary is documented.
The canary fixes above are partial, and each limit is written down next to the fix.

Every bug found in the gate is in a public ledger with how it was reproduced and what fixed it.

pip install adversary-gate==3.0.6

If you can get a false MERGE out of 3.0.6, I'd like to know. See SECURITY.md for how to report it privately.

This post was drafted by an AI assistant from the project's changelog and ledger; I checked every claim against them.

My coding agent rewrote the test to agree with its bug. So I built a gate that says INCONCLUSIVE.

Sanflow — Sun, 04 Oct 2026 18:19:43 +0000

Coding agents write fluent code. The expensive failure isn't the code that's obviously broken. It's the patch that looks verified:

the test "passed" because it never actually ran (an import error exits non-zero on both sides, and a naive before/after check reads that as "nothing changed");
coverage cleared the floor because the number was a default nobody measured;
the agent introduced a bug and rewrote the assertion to agree with it. The patch's own tests pass. CI is green.

I built AdversaryGate to catch exactly that. It doesn't ask a model whether the code looks good. It runs your tests and answers one of three things:

Exit	Decision	Meaning
`0`	MERGE	every claim executed and cleared the floors
`1`	BLOCK	something that passed before fails now
`2`	INCONCLUSIVE	something wasn't measured, so it doesn't become a green check

That third answer is the whole point. A measurement nobody produced can't clear a floor.

The case that started it

An agent changes add to return a + b + 1, then edits the test to expect 6 instead of 5. Run the patch's own tests: they pass.

AdversaryGate notices the patch changed a test the baseline already had. It copies the patch's tree, puts back the baseline's copy of the tests (helpers included), and runs them against the new code:

{
  "decision": "block",
  "claims": [{
    "test_id": "test_ops",
    "classification": "regression",
    "oracle": "baseline"
  }],
  "reason": "judged by the baseline's version of test_calc.py and its test files (the patch rewrote test_calc.py): test passes on baseline and fails on patch"
}

The test that runs is the one that existed before the agent touched anything. Bending it doesn't help.

What it measures

1. The same tests on both sides. Name them, or let coverage.py's per-test contexts pick the tests that executed the changed lines. Only a real test failure counts as evidence; import errors, missing tests, timeouts and runs killed by resource limits are unverified, not failed.

2. Diff coverage from artefacts. Computed from your unified diff plus the coverage.py JSON report, test files excluded. Both inputs are recorded by SHA-256, so anyone can recompute the number.

3. Mutation testing on the lines the patch wrote. One mutant per changed line before any line gets a second. The floor reads the lower bound of an 80% Wilson interval, not the raw ratio:

Mutants killed	Ratio	80% lower bound	Clears 0.75?
1 / 1	1.00	0.378	no
4 / 4	1.00	0.709	no
5 / 5	1.00	0.753	yes

One killed mutant is a ratio, not evidence.

4. The full suite on both sides. If it passed on the baseline and fails on the patch, that's a collateral regression: BLOCK.

A passing claim also says how it passed: fixed when the test failed before the patch and passes after (the fix is proven), no_regression when it passed both times.

The patch can't configure its judge

One bug I found in my own gate: a patch could ship a .pytest.ini with addopts = -p plugin, plus a plugin that reported "passed" only while the source had the exact bytes of the bug. Mutants change those bytes, so every mutant died for real. Result: MERGE, exit 0, strength 1.0, with add(2, 3) == 6.

Now every test-harness file in the tree (all five config names pytest 9 reads, conftest.py, sitecustomize.py, *.pth...) must be byte-for-byte the baseline's, compared tree against tree, not from whatever paths the diff reports.

That finding and 31 others are in a public ledger, each with how it was reproduced and whether it's still open. A wrong MERGE is the failure this tool exists to prevent, so a bug in it is treated as a security issue.

Letting an agent call it (MCP)

Since 2.8 it ships an MCP server, so an agent can call the gate before it says "done":

pip install "adversary-gate[mcp]"
adversary-gate-mcp

The design rule: the agent says what to judge; whoever runs the agent says how strictly, and against what. The tool call only takes a repository, claims and test paths. Everything that decides the answer comes from the server's environment:

ADVERSARY_GATE_POLICY: floors, sandbox. An agent that can pass --coverage-floor 0 is grading itself.
ADVERSARY_GATE_BASE_REF: the baseline is the oracle. An agent could commit a rewritten test and name that commit as the base.
ADVERSARY_GATE_PYTHON: the interpreter's site-packages are part of the harness; a plugin installed there runs inside pytest.

There's a ready-made skill for Hermes Agent (adversary-gate-mcp --print-hermes-skill), and it works with any MCP client.

In CI

- uses: actions/checkout@v4
  with:
    fetch-depth: 0
- uses: actions/setup-python@v5
  with:
    python-version: '3.12'
- run: pip install -r requirements.txt pytest coverage
- uses: Sanflow10/adversary-gate@v2
  with:
    base-sha: ${{ github.event.pull_request.base.sha }}

With base-sha, the Action builds the baseline, the diff and a per-test coverage report itself.

What it doesn't do

Measure languages other than Python. A C++ or SQL change gets INCONCLUSIVE, never a false MERGE.
Catch code that knows it's under test. Code that checks "pytest" in sys.modules, or tampers with pytest from inside the process, gets past any test-based gate. That's documented, not hidden.
Decide for you. MERGE means the evidence cleared your floors. It's permission for a human to look, not an instruction to ship.

Site: https://sanflow10.github.io/adversary-gate/
Repo: https://github.com/Sanflow10/adversary-gate
pip install adversary-gate · MIT

I'd like to hear where it gives the wrong answer.