Your Agent Patch Passed Because It Rewrote the Golden File

A patch that edits both the source and the fixture it is graded against has not been tested. It has been self-certified. Two questions expose it: did the diff touch a fixture directory, and does the suite still fail when that fixture is reverted to the base commit?

Everything below is machinery around those two questions — a diff classifier, a revert-differential run, three fixture properties, and a scoped, expiring freeze for the flaky tests that make the answers noisy. The scripts are reference implementations; I label them unexecuted in this draft, so treat the commands as a plan to run against your own repository rather than as measurements I collected.

1. Fixtures are a write channel, not a read-only oracle

A test suite has an implicit trust boundary. Source code is the thing under test; fixtures, golden files, and snapshots are the oracle. When an agent patch can write to both sides, it can move the oracle until the assertion agrees — and the CI checkmark stays green the whole time.

Four change classes exist in a typical patch, and they carry very different risk:

Class	Typical paths	Why it matters
source	src/**	the intended target
test	tests/**, *_test.py	can weaken, skip, or delete assertions
fixture	fixtures/**, testdata/**, *.golden, **/__snapshots__/**	rewrites the oracle
config	pyproject.toml, pytest.ini, .github/**	can disable collection entirely

Most gates I have reviewed classify the first two classes and ignore the third and fourth. That is the gap this workflow closes.

2. Step 1 — classify the diff before you run anything

The classifier is deliberately boring: it maps paths to classes, then asks whether a fixture write is accompanied by a test write and a stated reason. Verdicts are allow, review, or block; only allow skips the expensive step.

# gate/fixture_provenance.py
# Reference implementation. Unexecuted in this draft.
import re
import subprocess
import sys
from dataclasses import dataclass

FIXTURE = [
    re.compile(r"(^|/)fixtures?/"),
    re.compile(r"(^|/)testdata/"),
    re.compile(r"(^|/)__snapshots__/"),
    re.compile(r"\.(golden|snap|approved)$"),
]
TEST = [
    re.compile(r"(^|/)tests?/"),
    re.compile(r"(^|/)test_[^/]+\.py$"),
    re.compile(r"_test\.(go|py)$"),
]
REFRESH_TRAILER = re.compile(r"^FIXTURE-REFRESH:\s*(?P<reason>\S.*)$", re.MULTILINE)


def git(*args: str) -> str:
    return subprocess.run(["git", *args], capture_output=True, text=True, check=True).stdout


def classify(path: str) -> str:
    if any(p.search(path) for p in FIXTURE):
        return "fixture"
    if any(p.search(path) for p in TEST):
        return "test"
    if path.endswith((".yml", ".yaml", ".toml", ".ini", ".cfg")):
        return "config"
    return "source"


@dataclass(frozen=True)
class Verdict:
    decision: str   # allow | review | block
    reason: str


def verdict_for(base: str) -> Verdict:
    changed = [p for p in git("diff", "--name-only", f"{base}...HEAD").splitlines() if p]
    classes = {p: classify(p) for p in changed}
    fixtures = [p for p, c in classes.items() if c == "fixture"]
    tests = [p for p, c in classes.items() if c == "test"]

    if not fixtures:
        return Verdict("allow", "no fixture writes in this patch")

    message = git("log", "-1", "--format=%B")
    trailer = REFRESH_TRAILER.search(message)
    if not tests:
        return Verdict(
            "block",
            f"{len(fixtures)} fixture file(s) changed with no test change: self-certifying patch",
        )
    if not trailer:
        return Verdict(
            "review",
            "fixtures and tests changed together without a FIXTURE-REFRESH trailer",
        )
    return Verdict("review", f"declared refresh: {trailer.group('reason')}")


if __name__ == "__main__":
    print(verdict_for(sys.argv[1] if len(sys.argv) > 1 else "origin/main"))

Run it against the merge base, not the working tree, so a dirty checkout cannot influence the verdict:

python gate/fixture_provenance.py origin/main

A block stops the pipeline. A review proceeds to Step 2 — the trailer buys attention, not trust.

3. Step 2 — the revert-differential run

Step 2 is the only part of this workflow that produces actual evidence. Keep the patched source, restore the base fixtures, and re-run the suite. If everything still passes, the fixture change was decorative, which means the test that consumes it asserts nothing worth keeping.

BASE=origin/main
PATHS=$(git diff --name-only "$BASE...HEAD" -- 'fixtures/**' 'testdata/**' '**/__snapshots__/**' '*.golden')
[ -z "$PATHS" ] && exit 0

git checkout "$BASE" -- $PATHS        # fixtures added by this patch: rm them instead
set +e
pytest -q --tb=no
STATUS=$?
set -e
git checkout HEAD -- $PATHS           # restore the patched state for later jobs

if [ "$STATUS" -eq 0 ]; then
  echo "DECORATIVE: suite passes with base fixtures"
  exit 1
fi
echo "OK: the fixture is load-bearing"

Two details decide whether this is trustworthy. First, added fixtures do not exist at the base commit, so git checkout will fail on them — delete the file instead of restoring it, or the run silently tests the wrong tree. Second, run the differential job on a disposable checkout, because midway through the script the tree is intentionally inconsistent with HEAD.

4. Step 3 — three fixture properties worth asserting

Revert-differential proves a fixture matters today. These three properties keep it from rotting into churn next month.

1. Determinism. Run the fixture producer twice into different directories and compare bytes. Embedded timestamps, random IDs, and absolute paths are the usual culprits; strip them at the producer, not with a normalizing sed at comparison time.
2. Reference. Every fixture path must be named by at least one test. An unreferenced fixture is dead weight that the classifier will flag forever.
3. Byte stability. LF line endings, UTF-8 without BOM, trailing newline. Without this, editors and platforms trade diffs that contain no semantic change.

find fixtures testdata -type f -print0 | while IFS= read -r -d '' f; do
  grep -rqF "$(basename "$f")" tests/ || echo "UNREFERENCED: $f"
  grep -qU $'\r' "$f" && echo "CRLF: $f"
done

Treat these findings as review, not block. A determinism failure is often the real bug behind a flaky test, and blocking the patch hides that diagnosis.

5. Step 4 — a freeze is a lease, not a deletion

Flaky tests are why teams stop trusting fixture checks: the differential run fails for unrelated reasons and someone widens the ignore list. Replace the ignore list with an explicit, expiring lease.

# .ci/flaky-freeze.yaml
# Illustrative entries. Every lease needs an owner, a scope, and an expiry.
- test: tests/api/test_upload.py::test_retry_backoff
  owner: platform
  frozen_at: 2026-09-15
  expires: 2026-10-15
  scope: runner-pool-2          # the flake reproduces here only
  evidence: CI run link + failure class
  decay: drop when 100 consecutive runs report no failure

Four gate rules make the lease meaningful:

1. An expired entry fails the build. Renewal is a new entry with new evidence, typed by a human.
2. An entry without an owner fails the build. Unowned freezes never expire in practice.
3. A test outside its declared scope runs normally and can still block the patch. A freeze must not leak across runner pools.
4. When the decay counter reaches zero, the entry is deleted in the same PR that would have renewed it.

Rule 3 is the one that protects the fixture gate. If a freeze is broad enough to cover the exact file a patch touches, the freeze becomes the new oracle and you are back to self-certification with extra YAML.

6. Where a free model endpoint fits, honestly

The enumeration work in Step 3 is a good fit for a hosted model: hand it a fixture diff and ask for candidate invariants plus a one-line rationale, then accept or reject each one by hand. The model proposes; the repository owns the truth. I use MonkeyCode's free model access for exactly that pass, with the output treated as a draft checklist rather than a verdict.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I also run the Step 1 classifier and the Step 2 differential job on MonkeyCode's free server option, for one structural reason: the gate should not execute on the same runner as the untrusted patch it is grading. Both availability claims come from the product team, and I have not benchmarked either, so size your expectations accordingly.

7. Decision table

Signal	Action	Rationale
Fixture changed, no test changed	block	self-certifying patch
Fixture + test changed, no trailer	review, run Step 2	intent is unstated
Trailer present, differential run fails	allow with note	fixture is load-bearing and declared
Trailer present, differential run passes	block	decorative fixture
Freeze entry expired or unowned	block	leases without teeth are ignore lists
Test runs outside its freeze scope and fails	block	scope leak

8. Limitations, and who should not use this

Snapshot-heavy frontends legitimately rewrite fixtures on every intended visual change; without a per-directory allowlist, Step 2 will block correct work and your team will disable the job within a week. Repositories without a fixture tree get nothing from any of this.

The decay rule in Step 4 needs historical run data, so a project with days of CI history cannot populate it yet — start with expiry-only leases. And the FIXTURE-REFRESH trailer is a policy, not a tool: it only works if your contributing guide names it and reviewers reject bare refreshes.

If your fixtures are vendored from an upstream project and never authored locally, the classifier will flag every sync. Exclude that directory explicitly rather than loosening the pattern.

9. A ten-minute experiment

Take the last ten agent patches that touched a fixture path and run Step 1 over each one. If the count of block verdicts is zero, your team does not need this gate yet, and you have spent ten minutes to learn that. If it is above zero, you now know which merges were graded by an oracle the same patch had already rewritten.

That number is the argument. Not this article.