Warm, Ordered, Dirty: Replay an Agent Patch's Test Suite Cold Before You Trust the Green

A green suite is evidence about one process, not about the diff. Agent patches tend to pass in whatever environment the agent last left behind: warmed caches, leftover sqlite files, a module that happened to be imported in a convenient order. The cheap counter-move is to replay the same patch in a clean sandbox, with caches off and test order shuffled, K times, and to read a per-node status matrix instead of the final exit code.

This is not a correctness check. It is an honesty check on the environment. A patch that survives it has earned a weaker but real claim: the green was not produced by residue.

1. Three residue classes that manufacture warm greens

Process and cache residue. Pytest writes .pytest_cache/lastfailed, which --lf and --ff consult; CPython writes __pycache__. An agent that reruns on failure can inherit failure position from the previous attempt. Disable the cache provider and bytecode writing for the replay:

python -m pytest -p no:cacheprovider -q
PYTHONDONTWRITEBYTECODE=1 python -m pytest -q

Filesystem residue. Golden files, *.sqlite, uploaded fixtures, and tmp/ directories from attempt 4 satisfy assertions in attempt 5. The only reliable detector is a fresh directory per run plus a before/after file listing (Section 2, Step 4).

Import and collection residue. An untracked conftest.py, sitecustomize.py, pytest.ini, or *.pth the agent created can make an import resolve or a collection hook fire. git status --porcelain must be part of your patch freeze, or the patch you test is not the patch you merge.

2. Build a cold sandbox from the diff

Four steps, all reproducible.

Step 1 — freeze the patch as a diff, including new files. Untracked files are invisible to git diff; git add -N (intent to add) makes them appear without staging content. It is reversible with git reset.

cd "$REPO"
git add -N .
git diff HEAD > /tmp/agent.patch
sha256sum /tmp/agent.patch | cut -c1-12

Step 2 — export HEAD and apply the patch, with no untracked residue and no build artifacts.

SANDBOX=$(mktemp -d)
git archive HEAD | tar -x -C "$SANDBOX"
patch -d "$SANDBOX" -p1 --batch < /tmp/agent.patch

Step 3 — run K times with a closed environment and a shuffled order. pytest-randomly gives you a seeded shuffle via --randomly-seed; if you do not want another plugin in the loop, collect the node IDs once and shuffle them yourself into an args file.

K=${K:-5}
mkdir -p "$SANDBOX/runs"
for i in $(seq 1 "$K"); do
  HOME="$SANDBOX/home-$i" TMPDIR="$SANDBOX/tmp-$i" \
  PYTHONDONTWRITEBYTECODE=1 \
  python -m pytest -p no:cacheprovider -q -rA \
    --randomly-seed="$i" 2>&1 | tee "$SANDBOX/runs/run-$i.log"
  echo "exit=$?" >> "$SANDBOX/runs/run-$i.log"
done

Plugin-free variant, which also proves that collection itself is order-independent:

python -m pytest --collect-only -q | grep '::' > "$SANDBOX/nodes.txt"
for i in $(seq 1 "$K"); do
  shuf -n "$(wc -l < "$SANDBOX/nodes.txt")" "$SANDBOX/nodes.txt" > "$SANDBOX/nodes-$i.txt"
  xargs -a "$SANDBOX/nodes-$i.txt" python -m pytest -q -rA \
    2>&1 | tee "$SANDBOX/runs/nodes-$i.log"
done

Step 4 — record what the run wrote. A marker file plus -newer gives you the residue list per run.

touch "$SANDBOX/.replay-start"
# ... run the suite ...
find "$SANDBOX" -newer "$SANDBOX/.replay-start" -type f \
  | sed "s|$SANDBOX/||" | grep -v '^runs/' | sort > "$SANDBOX/runs/wrote-$i.txt"

3. Read the matrix, not the exit code

-rA prints one status line per test, which is enough to build a node-by-run matrix. Roughly forty lines of Python:

import collections, glob, json, re

PAT = re.compile(r"^(PASSED|FAILED|ERROR|SKIPPED|XFAIL|XPASS)\s+(\S+)")
matrix = collections.defaultdict(dict)
counts = {}

for path in sorted(glob.glob("runs/*.log")):
    run = re.search(r"run-(\d+)\.log$", path).group(1)
    seen = 0
    for line in open(path):
        m = PAT.match(line)
        if m:
            matrix[m.group(2)][run] = m.group(1)
            seen += 1
    counts[run] = seen

print("collected per run:", json.dumps(counts, sort_keys=True))
for node, runs in sorted(matrix.items()):
    if len(set(runs.values())) > 1:
        print("UNSTABLE", node, json.dumps(runs, sort_keys=True))

Three signatures matter. A node that alternates PASSED/FAILED across seeds is order coupling. A node missing from some runs was never collected, which points at collection-level state. A node that fails in every cold run, when the agent reported green, points at cache residue rather than at your code.

4. From matrix to verdict

Cold replay outcome	Reading	Action
All runs green, identical collected count, no files outside TMPDIR	Patch is order- and residue-independent for this suite	Eligible for review, not proof of correctness
One node flips in exactly one of five runs, not reproducible under a pinned seed	Flake-class	Quarantine with an expiry (Section 5)
One node flips in two or more seeds, or a specific seed reproduces it	Order coupling, not flake	Block; require isolation through fixtures
Collected count differs between runs	Collection side effects in conftest or imports	Block; collection must be pure
Files written outside HOME/TMPDIR	Hidden state inside the worktree	Block; route writes through tmp_path

5. Keep the quarantine honest

The quarantine list is the easiest place to hide a real defect behind the word "flaky". Make it a machine-read file with an expiry and an evidence pointer, and hash the whole file into the verdict record so entries cannot be appended after the replay:

{
  "nodeid": "tests/test_cache.py::test_eviction_order",
  "class": "flake",
  "expires_at": "2026-11-14",
  "evidence": "runs/run-3.log"
}

sha256sum flakefreeze.json >> "$SANDBOX/runs/verdict.txt"

Two rules keep this from rotting. Only a flake-class exit may enter the file; an order-coupling verdict has no path into it. And an entry that reaches expires_at without a new cold replay is treated as a failing test on the next run, because the default outcome of an unreviewed freeze must be failure.

6. Where the free compute fits

The replay loop is K independent suite runs plus one log parser. Every run needs its own sandbox anyway, so the work is trivially parallel and does not want your main CI runner's disk.

That is the gap MonkeyCode's free server option fits into for this workflow: a separate host to run shuffled replays without spending K times your CI minutes, while its free model access covers the tedious part of Section 3 — drafting the status-line parser and the residue classifier. Treat any generated parser as a proposal. It reads build logs and has no way to know whether your -rA output matches the format it assumed, so hand-check one run before you trust the matrix.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The availability claims above (free model access, a free server option) are operator-supplied; I have deliberately not stated quotas, hardware, model names, or durations, because I have no verified numbers for them. Check the current terms before you build a schedule around either.

7. Limitations and who should not use this

Cost scales linearly. K cold replays of a twenty-minute suite is a hundred minutes, and the sandbox export plus patch application adds per-run overhead. Replay the full suite only when the patch touches broad surface; otherwise scope the node list to what the diff can plausibly reach.

Shuffling is a sampler, not a proof. It finds order coupling that a random permutation happens to trigger. Pairwise dependencies between two specific tests can survive every seed you try.

External state is untouched. A shared staging database, a live API, a clock, or a network-dependent fixture is not reset by a fresh sandbox. Cold replay helps only where the state lives in the worktree, the process, or the interpreter cache.

Skip this approach if your suite has legitimate serial ordering with no markers to express it, if you cannot reconstruct the patch as a diff because the agent committed straight to a branch, or if the suite is slow enough that nobody will ever run it K times. A gate that is too expensive to run is a gate that will be switched off within a month.

If you run the loop once, keep the first log where a cold run disagrees with the agent's warm run. That disagreement — not the green — is the artifact worth keeping.