Your Agent Patch Edited conftest.py: A Fixture-Contract Gate With a Blast-Radius Map

A patch can weaken forty tests without touching any of them. It edits one shared fixture, the suite stays green, and every downstream test now passes for a different reason. Test-body diffs get reviewed. Setup diffs get skimmed.

This article is about closing that gap cheaply: map which tests can even be affected by a conftest.py change, then check the changed fixtures against a declarative contract instead of trusting that "still green" means "unchanged". The harness below is ordinary pytest — no framework fork, no new service.

The setup layer has no assertions of its own

A fixture is untested code that every test depends on. When a human edits it, review catches the obvious. When an agent edits it while fixing something else, the edit is usually a plausible small widening: scope="function" becomes scope="session", a None fallback appears, a guard assertion inside the fixture disappears because it "caused noise".

None of those show up as a failing test. They show up as a test that no longer proves what its name says.

Step 1 — Map fixture consumers before reading the diff

You cannot judge blast radius by eye. pytest --fixtures-per-test prints consumer relationships, but the text output is awkward to diff. A four-line plugin writes the same information as JSON using item.fixturenames, which is a stable pytest API.

# tools/fixture_map.py
def pytest_collection_finish(session):
    import json, os
    out = os.environ.get("FIXTURE_MAP_OUT")
    if not out:
        return
    data = {item.nodeid: sorted(set(item.fixturenames)) for item in session.items}
    with open(out, "w", encoding="utf-8") as fh:
        json.dump(data, fh, indent=2, sort_keys=True)

Collect only — this is a static pass, so it costs seconds even on a large suite.

FIXTURE_MAP_OUT=artifacts/fixture-map.json \
  pytest --collect-only -q -p tools.fixture_map
python - <<'PY'
import json
m = json.load(open("artifacts/fixture-map.json"))
print(len(m), "tests,", len({f for v in m.values() for f in v}), "distinct fixtures")
PY

Run it on both trees (base and patched) and diff the keys. Tests that appear or vanish are a separate problem. What you want is the intersection: tests whose fixture set changed, plus every test consuming a fixture whose definition changed.

Step 2 — Classify the conftest hunks, not just count them

Not all setup edits are equal. Triage them into four buckets before running anything:

Hunk pattern in conftest.py	Bucket	Why it matters
def fixture_name( added/removed/renamed	Define	New or dead fixture; check the map
scope= changed, autouse=True added	Widen	Lifetime and reach change silently
assert or raise removed inside a fixture	Weaken	Setup no longer validates its own output
default value, fallback, or return payload changed	Drift	Consumers get different data, same signature

A regex over git diff -U0 -- '*/conftest.py' for ^[+-]def (\w+)\( gets you the Define bucket in one line. The other three buckets need a human or a model reading hunks — they are semantic, and pretending a regex catches them is how gates get a reputation for noise.

Step 3 — Give each fixture a contract file

This is the part that replaces missing assertions. Write invariants declaratively so they are reviewable and reviewable diffs stay small.

# tests/fixture_contracts.yaml
fixtures:
  db_session:
    invariants:
      - {check: not_none}
      - {check: min_rows, table: users, n: 3}
  api_client:
    invariants:
      - {check: not_none}
      - {check: has_attr, name: get}
  seeded_orders:
    invariants:
      - {check: len_between, lo: 5, hi: 5}

Enforce it from a normal test module. request.getfixturevalue(name) resolves a fixture by string at runtime, so one parametrized test covers every contract entry.

# tests/test_fixture_contracts.py
import pytest, yaml

CONTRACT = yaml.safe_load(open("tests/fixture_contracts.yaml", encoding="utf-8"))
CASES = sorted(CONTRACT["fixtures"].items())

@pytest.mark.parametrize("name,spec", CASES, ids=[c[0] for c in CASES])
def test_fixture_contract(name, spec, request):
    value = request.getfixturevalue(name)
    for inv in spec.get("invariants", []):
        check = inv["check"]
        if check == "not_none":
            assert value is not None, name
        elif check == "len_between":
            assert inv["lo"] <= len(value) <= inv["hi"], (name, len(value))
        elif check == "has_attr":
            assert hasattr(value, inv["name"]), (name, inv["name"])
        else:
            pytest.skip(f"unimplemented check: {check}")

Skip-on-unknown is deliberate. A contract check that silently passes on a typo is worse than no check — this pytest output makes unimplemented checks visible without failing the build.

Step 4 — Run only the affected subset

Now the gate is cheap: contract tests plus the consumers of changed fixtures. Nothing else.

CHANGED=$(git diff --name-only origin/main -- '*/conftest.py' | xargs -r -n1 dirname)
pytest tests/test_fixture_contracts.py -q
python tools/affected.py \
  --map artifacts/fixture-map.json \
  --changed-fixtures-from $CHANGED > artifacts/affected.txt
pytest @artifacts/affected.txt -q

The subset run answers one question: did behavior change for tests that could not have been affected except through setup? A failure there is a setup regression, not a code regression, and the message should say so.

Two places where a hosted model earns its keep here: drafting the first fixture_contracts.yaml from fixture bodies (mechanical, reviewable), and triaging the semantic buckets from Step 2 when a diff touches several conftests. I use MonkeyCode's free model access for that drafting pass and its free server option when contract checks need a live service the local machine does not run (both availability claims are operator-supplied; verify current terms yourself before depending on them). Generate the YAML, then review every invariant by hand — the model proposes, the reviewer commits.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The decision table

Observation	Reading	Action
Fixture diff exists, zero consumers in map	Dead or dynamically resolved fixture	Warn; check getfixturevalue strings
Contract test fails on patched tree only	Setup weakened	Block merge, exit 2
Contract passes, affected subset changes outcome	Fixture drift with real effect	Investigate, exit 1
Affected subset empty	No reachable setup change	Pass, exit 0
Map changed, no fixture diff	Agent edited test wiring	Ordinary review, not this gate

Exit codes matter more than prose. A gate that only prints warnings gets scrolled past in CI logs.

Limitations, and who should skip this

• Regex attribution is approximate. Fixtures defined in plugins or imported from helper modules will not be attributed to a conftest hunk. Widen the pattern or accept blind spots explicitly.
• Contracts only cover what you write. len_between will not catch a fixture that returns correct-shaped but semantically wrong data.
• Session-scoped fixtures poison the subset run. If a changed fixture is scope="session", ordering effects can hide the difference; re-run the affected subset in isolation.
• Live-service fixtures cost real calls. Contract checks that hit external APIs are not free to run per commit.
• Skip this if your repo has one conftest and two fixtures, or your tests are generated wholesale per patch — you have no stable setup layer to contract against. Fix that first.

A green suite after a setup edit means the setup edit did not break anything the tests check. It says nothing about what the tests still check. Map the consumers, contract the fixtures, run the subset — three files, one CI step.

If you want to prototype the drafting step before wiring a paid model key, the contract file is the right place to start small: one fixture, one invariant, then grow it as review coverage allows.