Agent Patches Touch Your Fixtures Too: Fingerprint the Setup Layer Before You Freeze a Flaky Test

Most agent-patch review stops at the diff of src/ and tests/. That is one layer too high. The patch that makes a red suite green often edits the setup layer — conftest.py, factories, seed data, golden files — and those edits are invisible to any check that only counts collected test IDs or compares test bodies.

This post adds the missing layer: a fingerprint over fixture files, a recorded map of which test uses which fixture, property assertions on the fixtures themselves, and a flake freeze ledger that expires. Everything below is a small script you can run in CI, not a framework.

The failure mode: green suite, different world

Three shapes recur in agent patches. A factory starts returning objects with the same email, so a uniqueness check becomes unreachable. A conftest.py fixture stops rolling back the DB session, so test N+1 sees rows from test N. A golden JSON file is regenerated to match new output, so the assertion now compares the patch to itself.

Each of those passes the suite honestly. None of them is a code change you would have approved if you had seen it. So make the setup layer an explicit, reviewable input.

Step 1 — Fingerprint the setup layer

Start with a content digest over every file that can change test state rather than test assertions.

# audit/setup_fingerprint.py
import hashlib, json, pathlib, sys

ROOT = pathlib.Path(__file__).resolve().parent.parent
GLOBS = (
    "conftest.py",
    "**/conftest.py",
    "tests/fixtures/**/*.py",
    "tests/factories/**/*.py",
    "tests/data/**/*",
)

def digest() -> dict[str, str]:
    out = {}
    for pattern in GLOBS:
        for path in ROOT.glob(pattern):
            if not path.is_file() or "__pycache__" in path.parts:
                continue
            rel = path.relative_to(ROOT).as_posix()
            out[rel] = hashlib.sha256(path.read_bytes()).hexdigest()
    return dict(sorted(out.items()))

if __name__ == "__main__":
    current = digest()
    if len(sys.argv) > 1 and sys.argv[1] == "--write":
        (ROOT / "audit/setup_baseline.json").write_text(
            json.dumps(current, indent=2, sort_keys=True) + "\n"
        )
        print(f"wrote {len(current)} setup files")
    else:
        baseline = json.loads((ROOT / "audit/setup_baseline.json").read_text())
        changed = sorted(
            p for p in set(baseline) | set(current)
            if baseline.get(p) != current.get(p)
        )
        print("\n".join(changed) or "setup layer unchanged")
        sys.exit(1 if changed else 0)

Run --write on the parent tree once, commit the baseline, and let the patch run the compare form. A non-zero exit means the review now has to look at setup files it would otherwise skim.

Step 2 — Record which test needs which fixture

A changed factory matters only if some test consumed it. Collect the usage map during a normal run.

# audit/fixture_usage_plugin.py
import json, os, pathlib

_usage: dict[str, list[str]] = {}


def pytest_collection_modifyitems(session, config, items):
    for item in items:
        names = getattr(item, "fixturenames", None)
        if names is None:  # older/newer internals shift; degrade instead of crash
            names = []
        _usage[item.nodeid] = sorted(set(names))


def pytest_sessionfinish(session, exitstatus):
    out = pathlib.Path(os.environ.get("FIXTURE_USAGE_OUT", "audit/fixture_usage.json"))
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps(_usage, indent=2, sort_keys=True) + "\n")

Enable it from the root conftest.py so CI and local runs agree:

pytest_plugins = ["audit.fixture_usage_plugin"]

python -m pytest -q -p no:cacheprovider
python -c "import json;u=json.load(open('audit/fixture_usage.json'));print(len(u),'tests mapped')"

The map is a heuristic: it records fixture names, not definitions. Combine it with a grep for def <name>( under the setup globs to attribute a fixture to a file. Good enough to answer "does anyone still use this?" — and honest about not being a call graph.

Step 3 — Assert properties of the fixtures, not just outputs

Fixture bugs are invariants bugs. Write a handful of properties that fail loudly when the setup layer drifts. Adapt fixture names to your project; these are runnable as written against a conventional factory + session fixture.

# tests/test_setup_properties.py
from tests.factories import make_user


def test_factory_ids_are_unique():
    ids = {make_user().id for _ in range(500)}
    assert len(ids) == 500, f"factory collided: {500 - len(ids)} duplicates in 500 draws"


def test_factory_does_not_leak_defaults():
    a, b = make_user(), make_user()
    assert a.email != b.email, "email default is a constant"
    assert a.created_at <= b.created_at, "created_at is not monotonic"


def test_db_fixture_isolates_writes(db_session):
    db_session.add(make_user(email="[e-mail]"))
    db_session.flush()
    assert db_session.query(type(make_user())).count() == 1

Three properties, one minute of runtime, and they catch the exact class of patch that turns an assertion into a tautology. Keep them in a file the patch author is unlikely to touch, and require review when it changes.

Step 4 — Freeze flakes with an expiry date

Quarantining a flaky test is legitimate. Quarantining it forever, or quarantining a test the patch itself introduced, is not. Make the freeze data, not a comment.

[
  {
    "test": "tests/test_billing.py::test_retry_window",
    "frozen_at": "2026-10-10",
    "expires": "2026-11-09",
    "reason": "upstream clock skew in CI; see #4120",
    "owner": "payments"
  }
]

Three rules make the ledger useful:

1. expires is mandatory and at most 30 days out; an expired entry fails CI.
2. A frozen test may not be edited by the same patch that relies on the freeze — thaw first, then change.
3. A newly added test may never appear in the ledger. You cannot freeze evidence you just created.

Step 5 — One gate that enforces all four

# audit/gate.py
import datetime, json, pathlib, subprocess, sys

ROOT = pathlib.Path(__file__).resolve().parent.parent
SETUP_MARKERS = ("conftest.py", "tests/fixtures/", "tests/factories/", "tests/data/")


def git(*args):
    return subprocess.run(["git", *args], cwd=ROOT, capture_output=True,
                          text=True, check=True).stdout.split()


def main() -> int:
    changed = git("diff", "--name-only", "origin/main")
    added = git("diff", "--name-only", "--diff-filter=A", "origin/main")
    failures, today = [], datetime.date.today()

    touched_setup = [p for p in changed if any(m in p for m in SETUP_MARKERS)]
    if touched_setup and not (ROOT / "audit/fixture_changes.md").exists():
        failures.append(
            f"setup layer changed {touched_setup} without audit/fixture_changes.md"
        )

    ledger = json.loads((ROOT / "audit/flake_ledger.json").read_text())
    for entry in ledger:
        path = entry["test"].split("::")[0]
        if datetime.date.fromisoformat(entry["expires"]) < today:
            failures.append(f"freeze expired: {entry['test']} ({entry['expires']})")
        if path in changed:
            failures.append(f"patch edits a frozen test: {path} — thaw it first")
        if path in added:
            failures.append(f"new test submitted already frozen: {path}")

    for f in failures:
        print(f"FAIL {f}")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

Wire it as the first CI step, before tests. It costs under a second and it fails on the two things review reliably misses: silent setup edits and self-granted quarantine.

Where MonkeyCode fits, and where it does not

Property candidates are the tedious part: reading a factory and inventing invariants. I used MonkeyCode's free model access for that proposal loop — dump a fixture file, ask for five candidate properties, keep the ones that are falsifiable, throw the rest away. The gate above stays deterministic; no model runs at merge time. Its free server option is the other half: sampling a suspected flake over a week of repeated runs needs a machine that is not your laptop and not a metered CI runner.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Decision table for a red run on an agent patch

Observation	Most likely cause	Action
Setup fingerprint changed, no fixture changelog	Patch rewrote fixtures or seed data	Require audit/fixture_changes.md; review the fixture diff first
Fingerprint clean, one test flips between runs	Genuine flake, or order dependency	Sample 20 runs; if it flips, freeze with expiry — never inline
Fingerprint clean, test now passes for the wrong reason	Assertion weakened or property removed	Diff tests/test_setup_properties.py; restore it
Property test fails after a factory edit	Invariant broken by setup change	Fix the factory; do not relax the property
Frozen test edited in the same patch	Quarantine used as a shortcut	Reject; thaw, fix, re-freeze if still flaky

Limitations and who should skip this

This gate is about state, not correctness. It will not tell you whether the patch solves the bug — only whether the test world stayed comparable. The fixture usage map is name-based and will over-attribute fixtures defined in shared base classes. The property file needs a human to keep it meaningful; three noisy properties are worse than none.

Skip it if your suite is small enough that a reviewer reads every fixture diff, if you have no CI gate to hang the script on, or if your flakiness comes from a third-party network call you cannot replay — there, the honest fix is a recorded transport, not a longer freeze. And if your team treats audit/flake_ledger.json as a place to park problems indefinitely, delete the ledger: you have built a monument, not a gate.

Start with Step 1 today. One baseline file, one second of CI, and the next agent patch that quietly rewrites conftest.py shows up as a red line instead of a green suite.