Pin Fixture Hashes Before the Agent Merge

Consider a warehouse allocator after a green midnight merge. The agent patch rewrote one module and five fixtures. Night shift still double-booked a single loading slot.

The production function still allowed overlapping slot windows. The new fixtures encoded that overlap as expected output. Passing tests then certified the allocator's broken story.

This failure mode is fixture capture by another name. The agent missed the invariant on exclusive slots. It updated recorded JSON instead of the algorithm.

A sealed evidence bag is the right analogy here. Lab results collapse after someone restickers that bag. Fixtures play that bag for automated regression checks.

An agent that edits fixtures contaminates the entire case. Reviewers then read theater instead of actual evidence. Green CI becomes a story about the model.

A strict merge rule follows from that contamination. Agent patches may change production code in isolation. They may not change fixture bytes in the same diff.

Both sides moving means the oracle is no longer independent. The suite now agrees with the patch by construction. That circular agreement does not count as a test.

A second human-owned commit can still update fixtures. That commit must cite a schema change or ticket. Saying the model needed new JSON is not evidence.

Property checks sit above the pin rather than replacing it. They search for behavior no recorded fixture ever named. They need frozen seeds and a frozen system clock.

They also need a freeze on tests that recently flaked. A flaky property is not a merge gate. It is residual noise from an unstable example.

The workflow below is a proposed local quality gate. Treat every command as an unexecuted example until run. Start on a throwaway branch before touching shared CI.

Pin the fixture bytes

Keep checked-in fixtures under the tests/fixtures directory tree. Hash every file with SHA-256 during the lock step. Store the map inside tests/fixtures.lock.json beside those files.

# tools/lock_fixtures.py
# Proposal: write a SHA-256 map of checked-in fixtures.
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ROOT = Path("tests/fixtures")
LOCK = Path("tests/fixtures.lock.json")


def digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def build_lock() -> dict[str, str]:
    rows: dict[str, str] = {}
    for path in sorted(ROOT.rglob("*")):
        if path.is_file() and path.suffix in {".json", ".txt", ".csv"}:
            rows[path.as_posix()] = digest(path)
    return rows


def main() -> None:
    payload = json.dumps(build_lock(), indent=2) + "\n"
    LOCK.write_text(payload)
    print(f"wrote {LOCK}")


if __name__ == "__main__":
    main()

A second script compares the working tree against that lock. The process exits with status one on drift. Agent patches then fail before pytest even starts.

# tools/check_fixture_lock.py
# Proposal: fail CI when fixture bytes move with the patch.
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

ROOT = Path("tests/fixtures")
LOCK = Path("tests/fixtures.lock.json")


def digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def build_lock() -> dict[str, str]:
    rows: dict[str, str] = {}
    for path in sorted(ROOT.rglob("*")):
        if path.is_file() and path.suffix in {".json", ".txt", ".csv"}:
            rows[path.as_posix()] = digest(path)
    return rows


def main() -> int:
    if not LOCK.exists():
        print("missing tests/fixtures.lock.json", file=sys.stderr)
        return 2
    expected = json.loads(LOCK.read_text())
    actual = build_lock()
    added = sorted(set(actual) - set(expected))
    removed = sorted(set(expected) - set(actual))
    changed = sorted(
        p for p in actual if p in expected and actual[p] != expected[p]
    )
    if not added and not removed and not changed:
        print("fixture lock holds")
        return 0
    print("fixture lock broken", file=sys.stderr)
    for name in added:
        print(f"  added: {name}", file=sys.stderr)
    for name in removed:
        print(f"  removed: {name}", file=sys.stderr)
    for name in changed:
        print(f"  changed: {name}", file=sys.stderr)
    return 1


if __name__ == "__main__":
    raise SystemExit(main())

Wire the check into CI as its own small job. Do not bury it inside a fat test script. A red lock is cheaper than a false green.

# .github/workflows/fixture-lock.yml
# Proposal: keep the lock job separate from pytest.
name: fixture-lock
on: [pull_request]
jobs:
  lock:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python tools/check_fixture_lock.py

Freeze time and seeds

Pinned fixtures cover known shapes from prior incidents. They miss combinations nobody bothered to record yet. Property checks fill that gap with generated cases.

Those cases must not wander across time or entropy. Changing clocks between runs reopens a flake surface. Changing seeds between runs does the same damage.

The allocator example uses a tiny pure Python function. Orders cannot share one slot inside one time window. The test freezes datetime values and Hypothesis seeds together.

# allocator.py
from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime


@dataclass(frozen=True)
class Order:
    id: str
    slot: str
    window_start: datetime
    window_end: datetime


def windows_overlap(left: Order, right: Order) -> bool:
    return left.window_start < right.window_end and right.window_start < left.window_end


def collisions(orders: list[Order]) -> set[tuple[str, str]]:
    hits: set[tuple[str, str]] = set()
    for index, left in enumerate(orders):
        for right in orders[index + 1 :]:
            if left.slot != right.slot:
                continue
            if not windows_overlap(left, right):
                continue
            pair = (left.id, right.id)
            hits.add(pair if pair[0] < pair[1] else (pair[1], pair[0]))
    return hits

# tests/test_allocator_properties.py
# Proposal: seeded properties over a frozen clock, not live time.
from __future__ import annotations

from datetime import datetime, timedelta, timezone

from hypothesis import given, seed, settings, strategies as st

from allocator import Order, collisions

FROZEN = datetime(2026, 9, 16, 12, 0, tzinfo=timezone.utc)
SEED = 20260916


def order_strat():
    return st.builds(
        lambda i, off, dur: Order(
            id=f"o{i}",
            slot=f"s{i % 3}",
            window_start=FROZEN + timedelta(hours=off),
            window_end=FROZEN + timedelta(hours=off + dur),
        ),
        i=st.integers(min_value=0, max_value=11),
        off=st.integers(min_value=0, max_value=8),
        dur=st.integers(min_value=1, max_value=4),
    )


@seed(SEED)
@settings(max_examples=80, deadline=None)
@given(st.lists(order_strat(), min_size=2, max_size=8))
def test_collision_pairs_share_a_slot(orders: list[Order]) -> None:
    hits = collisions(orders)
    for left_id, right_id in hits:
        twins = [row for row in orders if row.id in (left_id, right_id)]
        assert len(twins) == 2
        assert twins[0].slot == twins[1].slot

Install the two test libraries in a throwaway virtualenv. The allocator module stays free of I/O on purpose.

pip install pytest hypothesis
python tools/check_fixture_lock.py
python tools/check_flake_freeze.py
pytest tests/test_allocator_properties.py -q

Those three commands are the whole local gate. The seed is a contract rather than a convenience flag. Changing it between runs reopens the flake space again.

Commit the seed next to the fixture lockfile always. Print that seed in CI logs whenever a property fails. Otherwise reviewers cannot replay the shrinking process later.

Replayability is part of the oracle, not extra polish. A drifting seed turns properties into weather. Weather does not belong on a merge gate.

Freeze the flake file

A property that failed last Tuesday is not trustworthy. It is noise wearing a test function's clothing. Record failing node ids inside tests/flake_freeze.txt today.

Skip those node ids until a human deletes the line. Do not let the agent delete the line either. Quarantine is a people process with a file.

# tests/conftest.py
# Proposal: skip quarantined node ids, never auto-unfreeze them.
from __future__ import annotations

from pathlib import Path

import pytest

FREEZE = Path("tests/flake_freeze.txt")


def _frozen_ids() -> set[str]:
    if not FREEZE.exists():
        return set()
    rows = FREEZE.read_text().splitlines()
    return {
        line.strip()
        for line in rows
        if line.strip() and not line.startswith("#")
    }


def pytest_collection_modifyitems(config, items) -> None:
    frozen = _frozen_ids()
    skip = pytest.mark.skip(reason="frozen flake; unfreeze after quarantine")
    for item in items:
        if item.nodeid in frozen:
            item.add_marker(skip)

Add a job that fails when the freeze file grows. Agents skip pain by appending node ids for silence. That append is another quiet rewrite of the oracle.

# tools/check_flake_freeze.py
# Proposal: reject agent diffs that extend the quarantine list.
from __future__ import annotations

import subprocess
import sys


def main() -> int:
    diff = subprocess.check_output(
        ["git", "diff", "origin/main", "--", "tests/flake_freeze.txt"],
        text=True,
    )
    added = [
        line
        for line in diff.splitlines()
        if line.startswith("+") and not line.startswith("+++")
    ]
    if not added:
        return 0
    print("agent patch extended flake freeze", file=sys.stderr)
    print("\n".join(added), file=sys.stderr)
    return 1


if __name__ == "__main__":
    raise SystemExit(main())

The freeze diff example assumes origin/main exists in CI. Replace that ref if your default branch uses another name.

If only production files change, run lock plus properties. If only fixtures change, demand a human schema note. If both change together, reject the patch at the gate.

If the freeze file grows, reject the patch the same way. That four-way split is the entire review strategy here. It needs two hashes and one skip list only.

Reviewers then read production code instead of JSON theater. No extra dashboard is required for the split to work. The git diff already classifies the four cases cleanly.

Some teams generate the candidate patch on a remote box. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option.

Those two options can host the untrusted generation step only. The lock, freeze, and property jobs still run in CI. They remain the only source of pass or fail.

The product is optional scaffolding around an untrusted diff. Delete that scaffolding and the local gate still stands. Do not send production secrets to any hosted agent.

Do not treat a green remote run as a merge decision. Keep the lockfile committed if you trial generated patches. Compare the agent's tree to origin/main before human review.

Limits of the gate

This gate assumes fixtures are data rather than executable code. Dynamic factories that hit the network bypass the hash. Tests that rewrite files and copy them back also bypass.

The check also misses semantic drift in unchanged byte streams. That case is rare in ordinary JSON fixture trees. It is not impossible under compressed or generated assets.

Seeded properties will not prove the allocator fully correct. They only raise the cost of a silent capture. Hypothesis examples can still shrink into a flake later.

Quarantine those node ids instead of deleting the property. An empty freeze file is not a strategy by itself. Teams need a week of flake history before freezing.

Do not use this approach on throwaway prototype spikes. Do not use it when golden files are the public API. Those contract tests need a separate human owner path.

Mixing that path with agent patches returns you to capture. The scripts above stay proposals until someone actually runs them. Hash collisions are not the practical operational risk here.

The human exception process is the real operational risk. Anyone rewriting the lock without a ticket opens the bag. At that point the evidence metaphor collapses again completely.