Fixture Leases: Catch Agent Patches That Pass Because Another Test Ran First

Most agent-patch gates measure the wrong thing. They rerun the suite and ask "is it green?" — but a green run only tells you the patch didn't break the order in which the suite happened to execute. Shuffle that order and a meaningful fraction of agent patches change verdict.

This post is about the specific failure I keep hitting: a patch that mutates a shared fixture, then passes because some earlier test already put the fixture in the right state. I'll show a self-contained pytest harness that makes fixture ownership explicit (a lease), shuffles execution order across seeds, and sorts every test into one of four buckets you can act on.

The failure mode, precisely

Order-dependent passes are cheap for a model to produce. Give an agent a failing test, and the smallest edit that turns it green may write to a fixture that another test wrote first. Nothing in a single-seed run detects this, because the same sequence runs every time.

Three conditions have to hold at once for it to survive review:

1. A test writes to shared state it does not own (a SQLite file, a JSON config, a golden output).
2. A later test reads that state and does not establish its own preconditions.
3. The suite never runs in a different order.

Remove any one condition and the bug disappears. Condition 3 is the cheapest to remove.

Step 1 — Declare which fixtures exist

Start with a flat manifest. One line per group, name: glob, resolved relative to the repo root.

# fixtures.manifest
db:     fixtures/db/*.sqlite
config: fixtures/config/*.json
golden: fixtures/golden/*.txt

Keep groups coarse. Ten groups are easier to reason about than two hundred files, and a coarse group that flips is still a useful signal.

Step 2 — Instrument ownership with a lease

The plugin below is deliberately dependency-free: no pytest-randomly, no JSON-report plugin. It hashes each group before and after every test, and records a violation when a group the test did not declare changed.

# conftest.py — fixture lease instrumentation (pytest >= 7)
import hashlib, json, os, pathlib, random

ROOT = pathlib.Path(__file__).parent
SEED = int(os.environ.get("LEASE_SEED", "0"))
LEASE_LOG = pathlib.Path(os.environ.get("LEASE_REPORT", "lease-report.jsonl"))
RESULT_LOG = pathlib.Path(os.environ.get("LEASE_RESULTS", "results.jsonl"))


def load_groups() -> dict[str, str]:
    groups = {}
    for line in (ROOT / "fixtures.manifest").read_text().splitlines():
        line = line.strip()
        if line and not line.startswith("#"):
            name, pattern = line.split(":", 1)
            groups[name.strip()] = pattern.strip()
    return groups


GROUPS = load_groups()


def digest(group: str) -> str:
    h = hashlib.sha256()
    for path in sorted(ROOT.glob(GROUPS[group])):
        h.update(path.name.encode())
        if path.is_file():
            h.update(path.read_bytes())
    return h.hexdigest()[:16]


def snapshot() -> dict[str, str]:
    return {g: digest(g) for g in GROUPS}


def declared(item) -> set[str]:
    return {arg for mark in item.iter_markers("writes") for arg in mark.args}


def pytest_runtest_setup(item):
    item._lease_before = snapshot()
    item._lease_owned = declared(item)


def pytest_runtest_teardown(item, nextitem):
    before = getattr(item, "_lease_before", None)
    if before is None:
        return
    after = snapshot()
    dirty = sorted(
        g for g in GROUPS
        if g not in item._lease_owned and before[g] != after[g]
    )
    with LEASE_LOG.open("a") as fh:
        fh.write(json.dumps({"nodeid": item.nodeid, "seed": SEED, "dirty": dirty}) + "\n")


def pytest_runtest_logreport(report):
    if report.when != "call":
        return
    with RESULT_LOG.open("a") as fh:
        fh.write(json.dumps({
            "nodeid": report.nodeid, "seed": SEED, "outcome": report.outcome,
        }) + "\n")


def pytest_collection_modifyitems(session, config, items):
    if os.environ.get("LEASE_KEEP_ORDER") == "1":
        return
    random.Random(SEED).shuffle(items)


def pytest_configure(config):
    config.addinivalue_line("markers", "writes(group): test owns this fixture group")
    config.addinivalue_line("markers", "serial: test requires collection order")

A test that legitimately rewrites the DB fixture declares it:

import pytest

@pytest.mark.writes("db")
def test_migration_rewrites_schema():
    ...

Everything else is implicitly read-only, which is the correct default for most suites.

Step 3 — Run the seed matrix

Eight seeds is enough to separate structural order dependence from noise; more than that mostly buys runtime. The loop is boring and parallelizable:

rm -f lease-report.jsonl results.jsonl
for seed in 1 2 3 4 5 6 7 8; do
  LEASE_SEED=$seed \
  LEASE_REPORT=lease-report.jsonl \
  LEASE_RESULTS=results.jsonl \
  pytest -q -p no:cacheprovider || true
done

|| true matters: a failing seed is data, not an error. Do not let the first red run abort the matrix.

Step 4 — Bucket the results

Triage is thirty lines. The ordering of the branches is the policy: pollution is reported even when the test passes.

# triage.py
import collections, json

leases = [json.loads(l) for l in open("lease-report.jsonl")]
dirty = collections.defaultdict(set)
for rec in leases:
    dirty[rec["nodeid"]].update(rec["dirty"])

outcomes = collections.defaultdict(dict)
for line in open("results.jsonl"):
    rec = json.loads(line)
    outcomes[rec["nodeid"]][rec["seed"]] = rec["outcome"]

for nodeid, seeds in sorted(outcomes.items()):
    seen = set(seeds.values())
    if dirty[nodeid]:
        bucket = "polluter"
    elif seen == {"passed"}:
        bucket = "stable"
    elif "passed" in seen and "failed" in seen:
        bucket = "order-dependent"
    else:
        bucket = "flaky"
    print(f"{bucket:16} {nodeid}  dirty={sorted(dirty[nodeid])}")

Illustrative output from the example layout above (your suite will differ — these are shape samples, not benchmark numbers):

polluter         tests/test_billing.py::test_apply_discount  dirty=['db']
order-dependent  tests/test_billing.py::test_invoice_totals    dirty=[]
stable           tests/test_api.py::test_health               dirty=[]
flaky            tests/test_sync.py::test_retry_backoff       dirty=[]

A node can be both a polluter and order-dependent. The script prints one primary bucket; the dirty list carries the second signal.

What each bucket earns

Bucket	Signal	Action before merge
polluter	Wrote outside its lease	Reject. Either add @pytest.mark.writes(...) with a reviewer's justification, or fix the test to use a per-test copy.
order-dependent	Pass/fail depends on seed	Reject until the test sets up its own preconditions. This is the bucket agent patches land in most often.
flaky	Fails on every seed	Quarantine with an expiry date; do not let it block the patch.
stable	Green on all seeds, no lease breach	Normal review.

The distinction that matters: order-dependent is a code defect and belongs in the patch; flaky is an environment defect and belongs in a queue. Conflating them is how flaky-test freezes end up hiding real bugs.

Where free-tier tooling actually helps

Two parts of this workflow are repetitive enough to be worth offloading, and this is where I've used the free model access and free server option that MonkeyCode provides. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat the availability as operator-supplied and verify current limits yourself before depending on it.

Drafting invariants for order-dependent pairs. For an order-dependent node, the interesting question is what property should hold regardless of order. A prompt built from the patch diff, the polluting group name, and the two nodeids gets you candidate invariants as pytest skeletons. Review them like any other generated code — a property check that cannot fail is worse than no check, so run each candidate against a deliberately broken version of the function before trusting it.

Running the seed matrix. Eight sequential suite runs are CPU-bound and boring, which is exactly the shape of work a free server instance handles well. Generate fixture data on the box rather than shipping a production dump, and keep credentials out of the environment entirely.

Neither use is load-bearing. If the free tier is rate-limited or unavailable, the loop above runs locally; you lose convenience, not correctness.

Limitations, and who should skip this

• Hashing is not free. Large binary fixtures make digest() the slowest part of the run. Shrink the manifest to the files that actually get written.
• Shuffling breaks legitimately ordered tests. Module-scoped fixtures with side effects will fail under shuffle and pass in CI. Mark those @serial and exclude them from the matrix, or fix them — usually the latter.
• Setup errors are invisible. The logreport hook records call-phase outcomes only, so a test that errors during setup appears in no bucket. Extend the hook if your suite has many collection-time failures.
• A lease proves ownership, not correctness. A test can own a group, mutate it correctly, and still be wrong.
• Skip this if your fixtures are all per-test tmp_path copies with no shared files. There is nothing to lease and the manifest is pure overhead.

The short version

Run the suite in a different order before you trust a green run. Record which tests write outside their declared fixtures. Reject order-dependent patches, quarantine genuinely flaky tests, and keep those two decisions separate — that separation is the part most gates get wrong.

If you implement one thing from this post, make it the pytest_runtest_teardown lease check. It catches the patch that passes only because another test ran first, and it costs about forty lines.