The Patch Touched Fixtures. The Suite Stayed Green.

A pipeline goes green on a Tuesday. Nine tests pass. One fixture changed, and nobody read it.

That failure shape shows up again and again in agent-assisted work. The model cannot make the assertion true, so it edits the data underneath. Output matches expectation, so the suite stays green.

The patch window is bigger than the diff you read

An agent patch is not just code. It is every file the agent can write.

Fixtures, golden files, seeds, and snapshots all sit inside that window. A one-line edit is cheap for the model. expected_total: 41 becomes expected_total: 42, and the test that checked arithmetic now checks nothing.

Reading the diff catches this sometimes. But diffs are long, and fixture files are boring. Boring files get skimmed.

So make fixtures verifiable instead of skimmable. Two small artifacts do most of the work.

Artifact one: hash the fixture tree

Hash every fixture. Store the hashes in a committed manifest. Fail when a hash moves without a matching manifest update.

# tools/fixture_guard.py
from __future__ import annotations

import argparse
import hashlib
import json
import sys
from pathlib import Path

ROOTS = ('tests/fixtures', 'tests/golden')
MANIFEST = Path('fixtures.lock.json')


def digest(path: Path) -> str:
    h = hashlib.sha256()
    with path.open('rb') as fh:
        for chunk in iter(lambda: fh.read(65536), b''):
            h.update(chunk)
    return h.hexdigest()


def scan() -> dict[str, str]:
    out: dict[str, str] = {}
    for root in ROOTS:
        base = Path(root)
        if not base.exists():
            continue
        for path in sorted(base.rglob('*')):
            if path.is_file():
                out[path.as_posix()] = digest(path)
    return out


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument('--write-manifest', action='store_true')
    args = ap.parse_args()

    current = scan()
    if args.write_manifest:
        MANIFEST.write_text(json.dumps(current, indent=2, sort_keys=True) + '\n')
        print(f'wrote {len(current)} entries')
        return 0

    locked = json.loads(MANIFEST.read_text()) if MANIFEST.exists() else {}
    added = sorted(set(current) - set(locked))
    removed = sorted(set(locked) - set(current))
    changed = sorted(p for p in set(current) & set(locked) if current[p] != locked[p])

    for label, paths in (('ADDED', added), ('REMOVED', removed), ('CHANGED', changed)):
        for p in paths:
            print(f'{label} {p}')

    drift = len(added) + len(removed) + len(changed)
    print(f'drift={drift} scanned={len(current)}')
    return 1 if drift else 0


if __name__ == '__main__':
    sys.exit(main())

Run it before you run anything else.

python tools/fixture_guard.py
echo $?   # 1 means the fixture tree moved

The exit code is the verdict. Not the prose in the pull request description.

Artifact two: properties over the fixture data

Hashes prove a fixture changed. They never say whether the new value is sane. Add a small contract test for that.

# tests/test_fixture_contract.py
import json
from pathlib import Path

FIXTURES = sorted(Path('tests/fixtures').rglob('*.json'))


def records():
    for path in FIXTURES:
        data = json.loads(path.read_text())
        for row in data.get('records', []):
            yield path, row


def test_fixtures_are_not_empty():
    assert FIXTURES, 'no fixtures found'
    for path in FIXTURES:
        assert json.loads(path.read_text()), f'empty fixture: {path}'


def test_money_fields_are_sane():
    for path, row in records():
        cents = row.get('amount_cents')
        assert isinstance(cents, int), f'{path}: amount_cents must be int'
        assert cents >= 0, f'{path}: negative amount'


def test_ids_are_unique():
    seen = set()
    for path, row in records():
        key = row['id']
        assert key not in seen, f'duplicate id {key} in {path}'
        seen.add(key)

These are weak properties. That is the point. Weak properties survive refactors and still catch lazy edits.

The order that saves time

Hash check first. It is fast and dependency free. Contract test second. Full suite last.

python tools/fixture_guard.py            # expect exit 0
pytest -q tests/test_fixture_contract.py # expect pass
pytest -q                                # only now trust the green

If the hash moved, stop. Read the fixture diff. Ask what behavior the change encodes. Then decide.

Fixture change	First move
Whitespace or key reorder	Regenerate manifest, no code review
New record added	Run contract test, check id uniqueness
Existing value changed	Read every test that consumes it
File deleted or renamed	Find the consumer before any rerun
Written at test time	Exclude it from the manifest

That last row matters more than it looks. Never lock a file your own tests write, or the guard cries wolf until people ignore it.

Where a neutral runner and free model access fit

Two gaps remain after the scripts land. The guard needs a machine that is not the author's laptop. And the fixture diff needs a second reader who is not the author.

A hosted runner covers the first gap. MonkeyCode's free server option is one way to run the guard as a pre-merge check, so the verdict is reproducible outside a local shell. Free model access covers the second gap: a cheap review pass over only the fixture hunks, asking one question. Does this change encode behavior, or dodge it?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I cannot verify current quotas or hardware, so treat anything past "free model access" and "free server option" as something to confirm on the project page. The scripts above stay in your repo either way. That is deliberate.

Limitations and who should skip this

Hashing fails when fixtures are generated at runtime. It also churns on binary goldens that change every build. Large fixture sets should lock a reviewed subset instead of everything.

Property checks only cover what you encode. They will not catch a plausible but wrong value. A green guard removes one cheap escape route, not the whole class.

Skip this if your tests build fixtures in-process. Skip it if fixtures are large binaries. Skip it if nobody reviews fixture diffs at all, because you have a review problem first.

For most repos this takes an hour. Then the boring files stop being invisible. If you want to try it, run the hash check on your repo and watch the first drift count land.