Lock the Golden Files. Split Agent Diffs That Touch the Answers.

Treat golden files like a lockfile. If an agent can rewrite expected output in the same pull request as the code under test, the green check is circular.

You do not need a new platform to stop that. You need a committed hash list, a split-diff rule, and a required check that ignores whatever the model claimed about its own patch.

The failure you are blocking

Watch a typical agent PR. It changes src/pricing.py. It also refreshes tests/golden/invoice.json. Both files look related. CI runs. The new function matches the new JSON. Nothing failed, because the spec moved with the implementation.

That is not coverage. That is a tautology. The test did not catch the bug. It absorbed it.

Humans still refresh goldens on purpose. You want that path. You do not want it on the default agent lane, and you do not want CI to rewrite fixtures.lock for anyone.

The contract

Keep three rules. They should fit in a review comment.

1. Every golden file has a SHA-256 in fixtures.lock.
2. An agent-labeled PR may change production code or golden files, not both, unless a human applied fixtures:allow.
3. The merge job runs tests once. No retry loop. A scheduled job can retry. Merge cannot.

If rule 2 fires, the PR stays open. A person inspects the bytes, applies the label, updates the lock in a visible commit, and re-runs the check.

Decision table

Use this at review time. Do not invent extra states until this table holds for a few weeks.

Diff shape	agent label	fixtures:allow	Contract result
Source only	yes	n/a	Pass contract, run tests once
Goldens only	yes	no	Fail split-diff
Source and goldens	yes	no	Fail split-diff
Source and goldens	yes	yes	Pass split; lockfile must update in the same PR
Source and goldens	no (human PR)	n/a	Pass split; CODEOWNERS still reviews goldens
Lockfile drift, goldens unchanged	any	no	Fail hash check

The table is the spec. The scripts below only enforce it.

Step 1 — Inventory the answers

List the files that define expected output. JSON goldens, text fixtures, recorded HTTP payloads, snapshot files. Put the list in the repo so a check cannot quietly drop a path.

mkdir -p tests/golden tests/snapshots scripts

find tests/golden tests/snapshots -type f \
  \( -name '*.json' -o -name '*.snap' -o -name '*.txt' -o -name '*.csv' \) \
  | sort > fixtures.manifest

git add fixtures.manifest

Generate the lock from a clean main. Do this on a tree you already trust, not on the agent branch.

git checkout main
python scripts/write_fixture_lock.py
git add fixtures.lock
git commit -m "Lock golden file hashes"

Example writer. Label this as a starting script, not a benchmarked production service.

# scripts/write_fixture_lock.py
from __future__ import annotations

import hashlib
import pathlib
import sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
MANIFEST = ROOT / "fixtures.manifest"
LOCK = ROOT / "fixtures.lock"


def expand(pattern: str) -> list[pathlib.Path]:
    rel = pattern.strip()
    if not rel or rel.startswith("#"):
        return []
    return sorted(p for p in ROOT.glob(rel) if p.is_file())


def sha256(path: pathlib.Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def main() -> int:
    if not MANIFEST.exists():
        print("missing fixtures.manifest", file=sys.stderr)
        return 2
    rows: list[str] = []
    for line in MANIFEST.read_text().splitlines():
        for path in expand(line):
            rel = path.relative_to(ROOT).as_posix()
            rows.append(f"{sha256(path)}  {rel}")
    LOCK.write_text("\n".join(rows) + ("\n" if rows else ""))
    print(f"wrote {len(rows)} hashes to {LOCK.relative_to(ROOT)}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Boring is the point. If the writer gets clever, the gate gets negotiable.

Step 2 — Fail closed on a drifted hash

The merge job recomputes hashes from the tree it just checked out. If fixtures.lock does not match, and the PR is not allowed to edit goldens, fail. CI must never rewrite the lock. Updating the lock is a human commit you can git blame.

# scripts/check_fixture_lock.py
from __future__ import annotations

import hashlib
import pathlib
import sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
LOCK = ROOT / "fixtures.lock"


def sha256(path: pathlib.Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def main() -> int:
    if not LOCK.exists():
        print("missing fixtures.lock", file=sys.stderr)
        return 2
    failed = 0
    seen: set[str] = set()
    for raw in LOCK.read_text().splitlines():
        if not raw.strip():
            continue
        digest, rel = raw.split("  ", 1)
        seen.add(rel)
        path = ROOT / rel
        if not path.is_file():
            print(f"MISSING  {rel}")
            failed += 1
            continue
        actual = sha256(path)
        if actual != digest:
            print(f"DRIFT    {rel}")
            print(f"  lock   {digest}")
            print(f"  tree   {actual}")
            failed += 1
    if failed:
        print(f"fixture lock failed: {failed} path(s)")
        return 1
    print(f"fixture lock ok: {len(seen)} path(s)")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it the same way CI will:

python scripts/check_fixture_lock.py

A mismatch is a red X. Three common meanings: the agent edited a golden, a human edited a golden and forgot the lock, or someone deleted a file still listed in the lock. All three are merge blockers until a person chooses.

Step 3 — Split the diff for agent PRs

Detect the lane with an agent label, or with a commit trailer Agent-Assisted: true. Then inspect the name-only diff against the base branch. Production paths and golden paths must not both appear unless fixtures:allow is present.

# scripts/split_agent_diff.py
from __future__ import annotations

import os
import subprocess
import sys

GOLDEN_PREFIXES = ("tests/golden/", "tests/snapshots/", "fixtures.lock")
SOURCE_PREFIXES = ("src/", "lib/", "app/")


def names(base: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def main() -> int:
    base = os.environ.get("BASE_SHA", "origin/main")
    labels = set(os.environ.get("PR_LABELS", "").split(","))
    labels = {item.strip() for item in labels if item.strip()}
    is_agent = "agent" in labels or os.environ.get("AGENT_ASSISTED") == "1"
    if not is_agent:
        print("human PR: split-diff skipped")
        return 0
    changed = names(base)
    gold = [p for p in changed if p.startswith(GOLDEN_PREFIXES)]
    src = [p for p in changed if p.startswith(SOURCE_PREFIXES)]
    if src and gold and "fixtures:allow" not in labels:
        print("agent PR changed source and golden files without fixtures:allow")
        print("source:")
        print("\n".join(f"  {p}" for p in src))
        print("goldens:")
        print("\n".join(f"  {p}" for p in gold))
        return 1
    print("agent split-diff ok")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

This is the rule that stops the tautology. The model can still propose a code fix. It can still open a second PR that only updates goldens. It cannot smuggle both through one green check.

Local dry run against main:

export BASE_SHA=origin/main
export PR_LABELS=agent
python scripts/split_agent_diff.py

If you use trailers instead of labels, set AGENT_ASSISTED=1 when git log -1 --pretty=%B contains Agent-Assisted: true.

Step 4 — Wire a required check

Example GitHub Actions workflow. Adapt branch names and prefixes. Do not paste this into a repo whose fixtures live somewhere else and assume it holds.

# .github/workflows/fixture-contract.yml
name: fixture-contract
on:
  pull_request:
    branches: [main]

jobs:
  fixture-contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Split agent diffs
        env:
          BASE_SHA: ${{ github.event.pull_request.base.sha }}
          PR_LABELS: ${{ join(github.event.pull_request.labels.*.name, ',') }}
        run: python scripts/split_agent_diff.py
      - name: Verify fixture lock
        run: python scripts/check_fixture_lock.py
      - name: Tests once, no retry
        run: |
          if [ -f pyproject.toml ] || [ -f pytest.ini ]; then
            python -m pytest -q --maxfail=1
          else
            echo "no pytest config; skip test step in this example"
          fi

Protect main. Require the fixture-contract check. Put CODEOWNERS on the answers so a green job cannot bypass a human on spec files:

# .github/CODEOWNERS
/tests/golden/    @your-org/maintainers
/tests/snapshots/ @your-org/maintainers
/fixtures.lock    @your-org/maintainers
/fixtures.manifest @your-org/maintainers

Write access is not the same as permission to change the spec. Owners review goldens. The check only proves the hashes and the split.

Step 5 — Keep retries off the merge job

Flakes exist. They do not get a vote on merge.

python -m pytest -q --maxfail=1

If a test is unstable, you will see it. That is the point. A retry loop on the PR check is how a weak assertion becomes "green enough." Put soak retries on a schedule after merge if you need them. Do not hide them inside the required job.

When fixtures:allow is present, require the lock update in the same PR:

python scripts/write_fixture_lock.py
git diff --exit-code fixtures.lock

If that diff is non-empty after a golden edit, the author forgot to refresh the lock. Fail. Do not refresh it in CI and push back. The lock is a reviewed artifact, not a build cache.

A local pass is optional

You can run the same three commands on a laptop before git push. A coding assistant that can read the diff will often catch a golden rewrite earlier than you will. That preview is useful. It is not a receipt.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you want that walkthrough on a box you do not have to size yourself, MonkeyCode's free model access and free server option are enough to run check_fixture_lock.py and split_agent_diff.py and list the files that would fail the check. Treat the output as a preview. The required GitHub job is still the only merge authority.

What this does not catch

The hash gate does not stop assert True. It does not stop deleting a test. It does not stop moving a golden file out of the manifest so the lock no longer sees it.

Add a cheap assertion-count diff if that is your next hole. Fail if net assert / self.assert* count drops without tests:allow. That is a different check. Do not overload the lockfile job until you have watched this one hold.

Also watch empty goldens. A file that becomes {} still hashes. The lock will match. The spec died anyway. If that shows up, add a minimum-size rule for selected paths, or review those files by owner every time.

Who should not use this

Skip this if you have no golden files. Skip it if snapshot files are the product, as in some visual-regression suites where every intentional UI tweak must update pixels. Skip it on throwaway spikes that never merge to main.

Also skip it if your agent label is world-writable and you cannot restrict who applies fixtures:allow. A gate that anyone can disable is documentation, not a gate. Restrict label changes, or require the allow label plus a CODEOWNER approval on fixtures.lock.

Close

Commit the lockfile. Split the agent diff. Run tests once. When a human really must refresh expected output, they say so with a label and they own the review.

The model can still write code. It does not get to rewrite the answers in the same move.