Fixtures Rot Quietly: A Two-Corpus Audit for Agent Patches

Most agent-patch gates watch the source diff. The fixture diff is where the meaning of a test quietly changes.

An agent that edits src/parse.py and tests/fixtures/parse/expected.json in the same patch can produce a fully green run and a broken parser. The test still passes. The assertion still executes. Only the answer key moved. No test-strategy document catches this, because nothing in the test code was touched.

So treat fixtures as inputs, not scenery. Three small artifacts — a manifest, a read trace, and a replay against the pre-patch corpus — turn fixture drift into something CI can name.

1. Hash the corpus before you look at the patch

A manifest is boring on purpose. Path, hash, size, sorted keys, no timestamps, so a re-run on an unchanged tree is byte-identical and the file is diffable.

#!/usr/bin/env python3
# tools/fixture_manifest.py
import hashlib, json
from pathlib import Path

ROOTS = [Path("tests/fixtures").resolve(), Path("tests/data").resolve()]
OUT = Path("artifacts/fixture_manifest.json")

def sha256(p: Path) -> str:
    h = hashlib.sha256()
    with p.open("rb") as fh:
        for chunk in iter(lambda: fh.read(1 << 16), b""):
            h.update(chunk)
    return h.hexdigest()

def build() -> dict:
    files = {}
    for root in ROOTS:
        if not root.is_dir():
            continue
        for p in sorted(root.rglob("*")):
            if p.is_file():
                files[str(p)] = {"sha256": sha256(p), "bytes": p.stat().st_size}
    return {"schema": 1, "roots": [str(r) for r in ROOTS], "files": files}

if __name__ == "__main__":
    OUT.parent.mkdir(parents=True, exist_ok=True)
    manifest = build()
    OUT.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n")
    print(f"{len(manifest['files'])} fixture files -> {OUT}")

Run it once against the base commit (keep that JSON as artifacts/fixture_manifest.base.json) and once against the patched tree. The delta is a set difference, nothing more:

# tools/fixture_delta.py
import json, sys
from pathlib import Path

def load(p):
    f = Path(p)
    return json.loads(f.read_text())["files"] if f.exists() else {}

base, head = load(sys.argv[1]), load(sys.argv[2])
changed = sorted(k for k in base.keys() | head.keys() if base.get(k) != head.get(k))
Path("artifacts/changed_fixtures.txt").write_text("\n".join(changed) + "\n")
print(f"{len(changed)} fixture files changed")

Key paths with absolute resolution on both sides. That is the first bug everyone hits: the manifest stores repo-relative paths, the read trace stores resolved ones, and the set intersection silently returns empty. Empty intersection looks like "no coupling", which is the most dangerous possible false negative here.

2. Record which tests actually read which fixture

Static analysis of conftest.py is a losing game — fixtures arrive via parametrization, helpers, and indirection. Trace the reads instead with a pytest plugin that wraps open and Path.open:

# tools/fixture_trace.py
import builtins, json, os
from pathlib import Path

WATCH = [Path("tests/fixtures").resolve(), Path("tests/data").resolve()]
READS: dict[str, set[str]] = {}
STATE = {"node": None}

def _record(path) -> None:
    try:
        rp = Path(os.fspath(path)).resolve()
    except (TypeError, ValueError):
        return
    if STATE["node"] and any(rp.is_relative_to(w) for w in WATCH):
        READS.setdefault(STATE["node"], set()).add(str(rp))

_open, _path_open = builtins.open, Path.open
builtins.open = lambda f, *a, **k: (_record(f), _open(f, *a, **k))[1]
Path.open = lambda self, *a, **k: (_record(self), _path_open(self, *a, **k))[1]

def pytest_runtest_setup(item):
    STATE["node"] = item.nodeid

def pytest_sessionfinish(session, exitstatus):
    out = Path("artifacts/fixture_reads.json")
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps({k: sorted(v) for k, v in READS.items()}, indent=2, sort_keys=True))

Enable it explicitly so normal developer runs stay untouched:

PYTHONPATH=tools pytest -q -p fixture_trace

Then intersect. These are the tests whose meaning the patch may have moved:

# tools/fixture_coupling.py
import json
from pathlib import Path

reads = json.loads(Path("artifacts/fixture_reads.json").read_text())
changed = {str(Path(l).resolve()) for l in Path("artifacts/changed_fixtures.txt").read_text().split()}
coupled = sorted(t for t, used in reads.items() if set(used) & changed)
Path("artifacts/coupled_nodeids.txt").write_text("\n".join(coupled) + "\n")
print(f"{len(coupled)} coupled tests")

3. Replay the coupled tests against the old bytes

Green on the new corpus proves nothing on its own. Re-run the coupled subset with the base-commit fixtures restored, and compare outcomes. Two outcomes matter: pass→fail means the patch changed what the test demands; pass→pass means the fixture edit was semantically inert and should be justified anyway.

#!/usr/bin/env bash
set -euo pipefail
BASE="${1:?usage: replay.sh <base-sha>}"

mkdir -p artifacts
# 1. snapshot the agent's fixture bytes so the working tree can be restored
find tests/fixtures tests/data -type f -print0 | tar --null -czf artifacts/fixtures.head.tgz -T -

# 2. put the base bytes in place
git archive "$BASE" tests/fixtures tests/data | tar -x -C .

# 3. run only the coupled tests, capture the outcome, restore no matter what
set +e
xargs -a artifacts/coupled_nodeids.txt pytest -q -p no:cacheprovider
RC=$?
set -e
tar -xzf artifacts/fixtures.head.tgz -C .
git status --porcelain tests/fixtures tests/data | grep -q . && { echo "tree dirty after replay"; exit 2; }
exit "$RC"

Adapt the paths; this is a sketch, not a universal script. The git status guard is not decorative — an aborted replay that leaves the base corpus in place poisons every later run on that machine.

4. Decide with a table, not a discussion

Fixtures changed	Coupled tests	Reading	Merge decision
no	—	code-only patch	normal review
yes	fail on base corpus	patch redefined the test	block until the fixture change is argued in the PR
yes	pass on both corpora	cosmetic or reformatting	allow, require a one-line justification
yes	new test only reads new fixture	additive	allow, review the fixture as new code
no	flaky on repeat	nondeterminism, not drift	quarantine ledger entry with an owner and a date

The last row is where teams wrongly reach for a long-lived skip. A flaky test is a scheduling problem with an owner; a fixture-coupled failure is a specification problem. Mixing them produces suites that are green for two unrelated reasons.

Where the free capacity actually helps

This audit is cheap per patch and expensive per volume. Model calls are not the constraint; the constraint is that every candidate patch needs its own manifest, read trace, and replay before a human looks at it. That is a lot of short-lived sandbox runs, and short-lived sandbox runs are exactly the kind of spend that gets stuck in procurement.

MonkeyCode's free model access and free server option are relevant here for a narrow reason: they let you run the full generate → audit → reject loop on candidate patches without a budget conversation first. Verify current terms, limits, and model availability yourself before committing to it — free tiers change, and this article makes no claim about quotas, hardware, or retention. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the free capacity does not buy you is the audit. The manifest, trace, and replay above have to be yours, because they encode your repo's layout and your definition of a fixture.

Limitations and who should skip this

• Invisible reads. The open hook misses fixtures loaded through subprocesses, importlib.resources, memory maps, or a database seeded from SQL. You get a false "no coupling" rather than an error.
• Reformatting noise. A hash changes when a JSON key order changes. If your tooling rewrites fixtures on every run, the manifest delta becomes unreadable; normalize serialization before hashing, or accept the noise consciously.
• Generated fixtures. Anything built at test time never appears in the manifest at rest, so the whole approach is blind to it.
• Replay cost. On a suite where fixtures are gigabytes of media, copying the corpus twice per patch is worse than the bug you are hunting.

Teams with heavy database seeding, binary fixture blobs, or a single shared tests/data directory that every test reads indiscriminately should not start here — the coupling set will be "everything", and the audit degenerates into a full re-run. Start with one fixture root and one module instead.

If your team is already paying for a compute budget, the free tier is not the interesting part; the trace file is. Run the audit for a week, look at how many coupled tests your agent patches actually touch, and let that number decide whether the pipeline is worth building.