A Single Green Pytest Run Is Not a Merge Signal for Agent Patches

Agent patches routinely survive the pytest invocation that ran while they were being written. That result is a weak merge signal. A patch is only merge-shaped when a second clean run, a reshuffled collection order, and a hashed fixture tree all agree, and when git status --porcelain is empty afterward.

One green command is easy to buy. Agents rewrite goldens, drop examples into on-disk corpora, and leave caches that the next collection silently consumes. The suite then passes for reasons the review diff never shows.

This article is a gate, not a model bake-off. It treats generation as untrusted input. The checks below are local, boring, and cheap enough to put on every agent-authored PR.

The failure mode is disk, not assertion text

Unit assertions can look serious and still be downstream of a polluted tree. Pytest collection order is not a stated contract. CWD is not a stated contract. Neither is the leftover .hypothesis corpus from the run that “proved” the fix.

Three mutation surfaces show up constantly on agent diffs:

1. Fixture bytes. Golden files, CSV snapshots, and protobuf testdata change in the same commit as production code, or they change during the test run itself.
2. Order and CWD. A test passes only because another test warmed a module cache, created a directory, or chdir’d into testdata/.
3. Replay residue. The first run writes __pycache__, .pytest_cache, .hypothesis, or temp files that the second run treats as input.

If any of those move between “before patch” and “after tests,” the green check is describing the workspace, not the program.

What the gate actually decides

Keep the merge rule small. Four predicates, all fail-closed.

Check	Pass means	Fail-closed action
Fixture manifest	testdata/** SHA-256 list matches HEAD unless the PR names every path	Block; require an explicit manifest delta
Shuffled collection	Full suite passes on two distinct collection seeds	Block; quarantine the first failing nodeid
Second clean run	Re-run from a fresh worktree leaves no dirty files	Block on any porcelain line
CWD and env	Tests do not depend on repo-root cwd or inherited env vars the agent exported	Block; fail the wrapper, not the helper

Do not score these. Do not average them. A single miss is a miss.

Protocol: isolate, hash, shuffle, replay

The steps are mechanical. Run them on the PR head, not on the agent’s working directory.

1. Create a throwaway worktree. The agent’s editor junk stays behind. git worktree add is enough.
2. Record fixture hashes before pytest. Only files under an allowlist directory. Ignore caches by construction.
3. Run the suite twice, each time with a different collection seed, from a restored cwd.
4. Re-hash fixtures and inspect porcelain. Any unexplained path is a merge blocker.
5. Delete the worktree. Residue is not evidence. It is contamination.

Illustrative wrapper (unexecuted here; treat as a starting script, not a reported CI metric):

#!/usr/bin/env bash
# scripts/agent_patch_gate.sh — illustrative merge gate
set -euo pipefail

ROOT=$(git rev-parse --show-toplevel)
REV=$(git rev-parse HEAD)
WT=$(mktemp -d "${TMPDIR:-/tmp}/agent-gate.XXXXXX")
MANIFEST_DIR="testdata"
SEED_A=${SEED_A:-20260914}
SEED_B=${SEED_B:-170393}

cleanup() { git worktree remove --force "$WT" 2>/dev/null || rm -rf "$WT"; }
trap cleanup EXIT

git worktree add --detach "$WT" "$REV"
cd "$WT"

hash_tree() {
  local out=$1
  if [[ -d $MANIFEST_DIR ]]; then
    find "$MANIFEST_DIR" -type f -print0 | sort -z | xargs -0 sha256sum > "$out"
  else
    : > "$out"
  fi
}

hash_tree /tmp/fixtures.before

export PYTHONHASHSEED=0
export PYTHONDONTWRITEBYTECODE=1
unset PYTEST_ADDOPTS

run_suite() {
  local seed=$1
  python -m pytest -q \
    --randomly-seed="$seed" \
    --basetemp="$WT/.tmp-$seed" \
    -p no:cacheprovider
}

run_suite "$SEED_A"
run_suite "$SEED_B"

hash_tree /tmp/fixtures.after

if ! diff -u /tmp/fixtures.before /tmp/fixtures.after; then
  echo "gate: fixture bytes changed during tests" >&2
  exit 2
fi

# Caches and corpora are not allowed to become implicit fixtures.
if git -C "$WT" status --porcelain | grep -E '.'; then
  echo "gate: worktree dirty after replay" >&2
  git -C "$WT" status --porcelain >&2
  exit 3
fi

Install the shuffle plugin in the gate image only if the project already accepts it. Pin it. Do not let the agent add it as a “test fix.”

# requirements-gate.txt — illustrative
pytest==8.3.5
pytest-randomly==3.16.0

Collection-seed flags differ by plugin. If you do not use pytest-randomly, shuffle via a small conftest that reorders items from an env seed. The point is two different orders, not a particular package.

Make the manifest a reviewed artifact

Hash lists that live only in CI logs get ignored. Commit a manifest for directories you actually treat as oracles. Review that file like code.

# testdata/MANIFEST.sha256 — illustrative layout
# sha256  path relative to testdata/
9c56cc51b374c3ba95ab8e23fdc4366e2e8cc44d0385cebc5c6d4c56c0b5a3e0  invoices/empty.json
2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3  invoices/sample-001.json

A tiny test then fails when an agent rewrites a golden without updating the manifest. That is the whole point. Silent golden edits are how patches launder behavior changes.

# tests/test_fixture_manifest.py — illustrative
from pathlib import Path
import hashlib

ROOT = Path(__file__).resolve().parents[1] / "testdata"
MANIFEST = ROOT / "MANIFEST.sha256"

def _sha256(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def test_testdata_matches_committed_manifest():
    assert MANIFEST.is_file(), "missing testdata/MANIFEST.sha256"
    listed = {}
    for raw in MANIFEST.read_text().splitlines():
        line = raw.strip()
        if not line or line.startswith("#"):
            continue
        digest, rel = line.split(None, 1)
        listed[rel] = digest

    on_disk = {}
    for path in ROOT.rglob("*"):
        if path.is_file() and path.name != "MANIFEST.sha256":
            rel = path.relative_to(ROOT).as_posix()
            on_disk[rel] = _sha256(path)

    assert on_disk == listed, (on_disk.keys() ^ listed.keys(), "hash mismatch")

When a golden should change, the PR must contain both the bytes and the manifest line. Two-file diffs are reviewable. One-file “updates expected output” diffs are not.

Kill CWD and inherited env on purpose

Agents copy shell state into tests. They export AWS_PROFILE, DEBUG, or a scratch PATH, then the suite only passes in that shell. The gate should start from a sanitized env and a fixed cwd.

# illustrative: run pytest without the agent's shell exports
env -i \
  HOME="$WT/.home" \
  PATH="/usr/bin:/bin" \
  LANG=C.UTF-8 \
  PYTHONHASHSEED=0 \
  python -m pytest -q --randomly-seed="$SEED_A"

Add one explicit test that the process cwd is irrelevant for the modules the agent touched. Label it as a characterization check if you do not yet have a real invariant.

# tests/test_cwd_independence.py — illustrative characterization
from pathlib import Path
import os
import runpy
import subprocess
import sys

def test_cli_entry_does_not_require_repo_root(tmp_path):
    """Proposal: replace 'yourpkg' with the package under review."""
    nested = tmp_path / "nested" / "dir"
    nested.mkdir(parents=True)
    env = os.environ.copy()
    env["PYTHONPATH"] = str(Path(__file__).resolve().parents[1])
    proc = subprocess.run(
        [sys.executable, "-m", "yourpkg", "--help"],
        cwd=nested,
        env=env,
        capture_output=True,
        text=True,
        check=False,
    )
    assert proc.returncode == 0, proc.stderr

If that module name is wrong for your tree, do not fake a pass. Skip the test until the entry point is real. A skipped characterization test is honest. A tautological assert True is not a gate.

Where generation can live without becoming the oracle

Patch authors need a machine that is not the merge machine. If you generate candidate diffs on a remote box so experiments never touch the laptop checkout, MonkeyCode’s free model access and free server option can host that generation step. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not put the gate in the prompt. Prompts are not replayable. Keep scripts/agent_patch_gate.sh and testdata/MANIFEST.sha256 in the repo the PR merges into. The generator may propose a manifest delta. The CI wrapper is the only consumer that counts.

Limitations

Shuffled collection catches order bugs that are deterministic in a given permutation. It does not catch data races that need a thread sanitizer. Two seeds are a sample, not a proof over all n! orders.

Hashing testdata/ fights tests that must rewrite goldens as their purpose (image snapshots with platform font drift, for example). Those trees need a different oracle, or they need to live outside the hashed path.

A second run costs wall clock. On a 40-minute suite the gate is the wrong default. Split a fast hermetic slice for agent patches and keep the long suite on main. Do not pretend the slow suite was shuffled if you only shuffled the slice.

env -i will break tests that honestly need credentials. That is useful information. It is not a reason to disable the gate globally. Pass explicit env keys through a allowlist file reviewed in the same PR.

Worktrees do not isolate Docker sockets, local databases, or listening ports. If the agent’s code talks to a shared service, this protocol will not save you.

Who should not use this

Skip the full protocol when the repository has no hermetic unit slice, when every test hits a shared staging API, or when collection order is a documented feature of a step-wise integration harness. In those trees the honest move is to stop generating patches against the suite, not to wrap pytest twice and call it isolation.

Also skip it for one-off exploratory branches you will throw away. The gate is for merge. It is overhead on a spike.

Merge rule, restated

Green once, in the order the agent left behind, with caches warm and goldens rewritten, is not evidence. Hash the fixtures. Shuffle collection. Run again from a clean worktree. If porcelain is dirty, the patch is not done.

If candidate diffs already come from a free MonkeyCode server, leave them there. Copy only the patch into a worktree that runs this wrapper. The generator can be cheap. The gate should stay strict.