Revert the Fix, Keep the Tests: A Three-Tree Differential Gate for Agent Patches

A green suite after an agent patch tells you the suite is green. It does not tell you the patch is why. To get that, run the same suite three times: on the parent commit, on the patched commit, and on a third tree where the patch's non-test hunks are reverted while its tests stay. Only the last run separates "these tests detect this fix" from "these tests happen to pass today".

I use "agent patch" here for any diff authored mostly by a code assistant, where tests and source arrive in one commit. The method is not agent-specific. The co-authorship is what makes it necessary.

1. Why green-after is the weakest of the three signals

Three common shapes produce a green run that means nothing.

• Tautology. The test asserts a value the patch itself computed, so it can only fail if the patch crashes. I classified these separately in an earlier post; the table in section 4 catches them without extra code.
• Pre-existing green. The test passed before the patch and still passes. The patch changed nothing it observes.
• Fixture-coupled. The test reads a fixture or stub the patch also edited, so both sides move together and the assertion stays true.

Each shape is invisible in a standard CI run and visible in one extra execution. That is the entire trade.

2. Materialize the three trees

Use git worktree so all three trees exist at once and share one object store. No stashing, no branch juggling in the reviewer's shell.

#!/usr/bin/env bash
# diff-gate.sh <patch-commit>
set -euo pipefail

PATCH_REF="${1:?usage: diff-gate.sh <patch-commit>}"
PARENT="$(git rev-parse "${PATCH_REF}^")"
ROOT="$(mktemp -d)"
trap 'rm -rf "$ROOT"' EXIT

git worktree add --detach "$ROOT/parent"  "$PARENT"
git worktree add --detach "$ROOT/patched" "$PATCH_REF"
git worktree add --detach "$ROOT/ablated" "$PATCH_REF"

# Revert only the non-test hunks in the ablated tree.
mapfile -t SRC < <(
  git -C "$ROOT/ablated" diff --name-only "$PARENT" "$PATCH_REF" \
  | grep -Ev '(^|/)(tests?|specs?)/|(_test|\.test|\.spec)\.'
)
if [ "${#SRC[@]}" -gt 0 ]; then
  git -C "$ROOT/ablated" checkout "$PARENT" -- "${SRC[@]}"
fi

The grep -Ev path filter is the fragile part, and the honest part. Any repo with an unusual test layout needs its own pattern.

3. Run all three with a pinned runner

Same interpreter, same lockfile, same seeds, same clock injection. A different Node or Python patch version in one of the three runs converts an environment difference into a verdict.

for tree in parent patched ablated; do
  ( cd "$ROOT/$tree" \
    && npm ci --ignore-scripts \
    && SEED=20261016 TZ=UTC npm test -- --reporter=json > "$ROOT/$tree.json" )
done

Capture machine-readable output per tree. Free-text logs cannot be diffed reliably, and the classifier below needs stable test IDs.

4. Classify with one table

Every test lands in exactly one row. Only the first row is evidence about the patch.

parent	patched	ablated	verdict
fail	pass	fail	sensitive
fail	pass	pass	fix-blind
pass	pass	pass	pre-existing-green
fail	fail	fail	no-fix
pass	fail	fail	regression
fail	fail	pass	inverted
pass	pass	fail	fixture-coupled

The classifier is small enough to keep in the repo and read in a review.

import json, sys
from pathlib import Path

def load(path):
    data = json.loads(Path(path).read_text())
    return {t["id"]: t["status"] for t in data["tests"]}

parent, patched, ablated = (load(p) for p in sys.argv[1:4])

RULES = {
    ("fail", "pass", "fail"): "sensitive",
    ("fail", "pass", "pass"): "fix-blind",
    ("pass", "pass", "pass"): "pre-existing-green",
    ("fail", "fail", "fail"): "no-fix",
    ("pass", "fail", "fail"): "regression",
    ("fail", "fail", "pass"): "inverted",
    ("pass", "pass", "fail"): "fixture-coupled",
}

for test_id in sorted(set(parent) | set(patched) | set(ablated)):
    key = (parent.get(test_id, "pass"),
           patched.get(test_id, "pass"),
           ablated.get(test_id, "pass"))
    print(test_id, RULES.get(key, "unknown"))

Missing tests default to pass, because a test that does not exist cannot fail. If your runner omits collected-but-skipped tests, treat those as unknown instead of assuming a status.

5. Make the gate require one sensitive test

The gate rule is one line: the patch must add or modify at least one test whose verdict is sensitive, and no test may be regression. Everything else is diagnostic, not blocking.

The gate is deliberately asymmetric. It does not require every new test to be sensitive, because some tests legitimately assert invariants the fix does not touch. It does require that the patch's own evidence be causally tied to the patch.

fix-blind is the state worth routing to a human most often. The test fails on the parent and passes with the fix, but also passes when the fix is removed. The usual cause is an import or setup change in the patch that the test depends on for reasons unrelated to the asserted behavior. That is a defect in the test, not in the patch.

6. Quarantine flakes with an expiry and an outsider's signature

Flaky tests break the differential, because they add noise to three runs instead of one. Quarantine them, but under two constraints: the entry expires, and the patch author cannot write it.

# .ci/quarantine.yaml   (numbers below are illustrative, not measurements)
- test_id: checkout.retries_on_5xx
  assertion_hash: 3f9c1a7b44d2
  first_seen: 2026-10-09
  observed_runs: 240
  fail_rate: 0.037
  expires_at: 2026-10-30
  owner: payments-team

Enforce the second constraint in the gate:

AUTHOR="$(git log -1 --format=%ae -- .ci/quarantine.yaml)"
[ "$AUTHOR" = "$AGENT_IDENTITY" ] && { echo "agent cannot self-quarantine"; exit 1; }

A quarantine entry is a claim that a failure was not caused by this patch. Letting the patch author make that claim turns the gate into self-certification. I covered the flake-budget side of this earlier; the differential gate is where that budget gets spent.

7. Where a free runner and free model access fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The expensive step is not the three-tree run. It is observed_runs: 240 above. A fail rate in the low single-digit percent needs a few hundred executions to separate from one percent, per candidate flaky test. Paying for that on the same CI fleet that runs the merge gate is the wrong economics.

This is where I put MonkeyCode's free server option: as a separate, non-blocking runner that executes the repeat-N harness against the parent tree, writes a counter file, and never opens the gate. Its free model access I use one step earlier, on triage only, turning classifier output plus the patch diff into a first-pass label of sensitive, fix-blind, or fixture-coupled for a human to confirm. I have not measured which models are available, what quotas apply, or how long free access lasts, so treat none of those as guarantees and keep the gate functional when the free runner is unreachable.

The rule that keeps this honest: the free tier may measure and may label. It may not decide. The gate reads committed JSON, not a model's opinion.

8. Limitations, and who should not use this

• The ablated tree may not build. If the patch adds a symbol the tests import, reverting the source hunks produces an import error and every test errors in the ablated run. Treat a build failure as inconclusive, never as sensitive.
• Test paths are heuristic. The grep filter will misclassify a repo that keeps tests beside sources with no naming convention. A wrong filter silently moves source into the keep set and inflates fix-blind.
• Three runs cost three times as much. For suites over roughly fifteen minutes, run the differential only on touched packages and accept reduced coverage of the rest.
• Test-only patches are vacuous here. If the patch changes no source files, the ablated tree equals the patched tree and every test lands in pre-existing-green. That is the correct answer: there is no fix to detect.
• Nondeterministic suites break the table. Without seed and clock control, sensitive and fixture-coupled become indistinguishable.

Do not adopt this if your merge gate must stay under a hard wall-clock budget you cannot shard, or if the patch authors are also the only reviewers of quarantine.yaml.

9. Monday checklist

1. Add diff-gate.sh and run it manually against the last five agent patches.
2. Record the verdict distribution before you change any rule.
3. Pick one fix-blind test and read the patch's import and setup diff.
4. Move repeat-N flake measurement off the merge fleet.
5. Set every quarantine entry to expire within 30 days and re-earn its place.

Run the classifier against your own history before you wire it into a required check. The distribution usually says more about test selection than about the agents writing the patches — mine skews to pre-existing-green, which is what you would expect from a suite assembled before agents wrote any of it.

Comments

[Arhan Canli]

I ran the classifier from section 4 against a toy repo, and two things break on the commonest agent-patch shape, a patch that adds new tests and new source files.

1. A test that doesn't exist on the parent defaults to "pass" in parent.get(test_id, "pass"). A new test that fails on the ablated tree then reads as (pass, pass, fail), which your table calls fixture-coupled. It is the best case for the gate, the one that should be sensitive. Treating "absent on parent" as fail for tests that appear in the patch's diff fixes it; "a test that doesn't exist cannot fail" is true but not what you want to count here.

2. git checkout "$PARENT" -- "${SRC[@]}" stops with error: pathspec 'src/new.py' did not match any file(s) known to git when the patch adds a non-test file, and with set -e the script ends before the ablated run. Files added by the patch need git rm -f in the ablated tree, and only the modified ones the checkout.

Do you mark tests as "new in this patch" from the test diff, or from the runner's collected IDs?

[Reid Marlow]

The worktree isolation is solid, but the fragile edge in practice is test-runner discovery. When the ablated tree runs, any new fixture or helper file under tests/ that imports new source modules will trigger an uncaught import error during test collection rather than a test assertion failure. That turns what should be a sensitive verdict into an unhandled collection crash before the matrix can evaluate.