Block Agent Diffs That Mutate Invariants, Fixtures, or the Flake Freeze

An agent patch is not done when CI is green. It is done when the implementation changed and the oracle did not. If a model can edit properties, fixture bytes, or flake suppressions, a passing suite is a circular report.

Split the tree into writable paths and denied paths. Keep seeded properties, fixture checksums, and the flake freeze on the denied list. Generate the candidate implementation in a disposable workspace. Copy back only production files. Merge only when the protected tree is unchanged and the properties still hold under a fixed seed.

This is a path-ACL workflow, not a coverage target. Extra unit examples are allowed. Rewriting the contract is not.

The failure, reduced to a table

Agent diffs cluster into four edit classes. Only one of them is the work you asked for.

Path glob	Agent write	Merge signal if the suite is green
src/**	allowed	valid only if denied paths are untouched
tests/unit/**	allowed	valid only if it does not replace properties
tests/invariants/**	denied	green is meaningless; the oracle moved
tests/fixtures/**	denied	green is meaningless; golden bytes moved
tests/flake_freeze.yaml	denied	green is a mute, not a fix
.github/workflows/**-protected-*.yml	denied	the gate cannot rewrite itself

A unit test that the agent authored can still be useful. Treat it as an example, not as the specification. The specification lives in files the diff gate will reject if they appear in git diff --name-only.

Artifact: a denied list plus a seeded harness

The following is a worked example, not a production case study. It locks three things: a property file, a fixture manifest, and a flake freeze. A fourth script refuses the merge when those paths show up in the diff.

Label the production surface as a stand-in. Swap apply_batch for any pure fold you actually ship.

# ledger.py — example implementation surface, not a library
from __future__ import annotations

from typing import Any


def apply_batch(payload: dict[str, Any], seed: int) -> dict[str, Any]:
    entries = list(payload.get("entries", []))
    deltas = [int(item["delta"]) for item in entries]
    return {
        "entries": entries,
        "deltas": deltas,
        "net": sum(deltas),
        "seed": seed,
    }

1. Seeded properties the agent cannot edit

Fix the seed. Property tests that re-roll randomness on every CI worker produce flakes, and flakes produce freeze-file edits. A frozen seed makes a red run replayable.

# tests/invariants/test_ledger_properties.py
"""Denied path. Agent diffs that touch this file must fail CI."""
from __future__ import annotations

import hashlib
import json
from collections import Counter
from pathlib import Path

import pytest

from ledger import apply_batch

SEED = 20260913
ROOT = Path(__file__).resolve().parents[1]
FIXTURE_DIR = ROOT / "fixtures"
MANIFEST = Path(__file__).with_name("fixture_manifest.json")


def _sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def test_fixture_bytes_match_manifest() -> None:
    manifest = json.loads(MANIFEST.read_text())
    missing: list[str] = []
    drifted: list[str] = []
    extra = sorted(
        p.name for p in FIXTURE_DIR.glob("*.json")
        if p.name not in manifest["files"]
    )
    for rel, expected in manifest["files"].items():
        path = FIXTURE_DIR / rel
        if not path.is_file():
            missing.append(rel)
            continue
        if _sha256(path) != expected:
            drifted.append(rel)
    assert missing == [], f"missing fixtures: {missing}"
    assert drifted == [], f"fixture drift: {drifted}"
    assert extra == [], f"unmanifested fixtures: {extra}"


@pytest.mark.parametrize("name", ["empty.json", "mixed_signs.json", "replay.json"])
def test_apply_batch_is_idempotent(name: str) -> None:
    payload = json.loads((FIXTURE_DIR / name).read_text())
    once = apply_batch(payload, seed=SEED)
    twice = apply_batch(once, seed=SEED)
    assert twice["deltas"] == once["deltas"]
    assert twice["net"] == once["net"]


def test_apply_batch_preserves_delta_multiset() -> None:
    payload = json.loads((FIXTURE_DIR / "mixed_signs.json").read_text())
    before = Counter(int(item["delta"]) for item in payload["entries"])
    after = apply_batch(payload, seed=SEED)
    assert Counter(after["deltas"]) == before
    assert after["net"] == sum(before.elements())
    assert after["seed"] == SEED

Idempotence without a multiset check is a weak net. An implementation can drop two entries that cancel and still print the same sum. The counter property is the part an agent “simplification” usually breaks.

2. Fixture manifest, generated not typed

Do not hand-edit hashes. A stale hash is how a denied path quietly becomes writable in review.

# tests/invariants/write_manifest.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

FIXTURE_DIR = Path(__file__).resolve().parents[1] / "fixtures"
OUT = Path(__file__).with_name("fixture_manifest.json")


def main() -> None:
    files = {}
    for path in sorted(FIXTURE_DIR.glob("*.json")):
        files[path.name] = hashlib.sha256(path.read_bytes()).hexdigest()
    OUT.write_text(json.dumps({"version": 1, "files": files}, indent=2) + "\n")
    print(f"wrote {OUT} ({len(files)} files)")


if __name__ == "__main__":
    main()

python tests/invariants/write_manifest.py
git add tests/invariants/fixture_manifest.json tests/fixtures

Run that script on main after a human changes a golden file. Never run it as a side effect of an agent patch.

3. Flake freeze as a denied document

A freeze file is useful. An agent-authored freeze file is a mute button. Keep the document, deny writes from the model, and reject added keys even when the YAML still parses.

# tests/flake_freeze.yaml
version: 1
# Human-only writes. TTL is a calendar reminder, not a green-bar tool.
default_ttl_days: 14
entries: []
# Example shape, unused until a human adds a row:
# entries:
#   - id: test_apply_batch_is_idempotent[replay.json]
#     reason: "order-dependent fold on replay.json; tracking issue 1841"
#     expires: "2026-09-27"

# tests/invariants/check_freeze_diff.py
from __future__ import annotations

import subprocess
import sys
from pathlib import Path

import yaml

FREEZE = Path("tests/flake_freeze.yaml")


def _load(text: str) -> set[str]:
    data = yaml.safe_load(text) or {}
    entries = data.get("entries") or []
    return {str(row["id"]) for row in entries if "id" in row}


def main() -> int:
    base = subprocess.check_output(
        ["git", "show", "HEAD:tests/flake_freeze.yaml"],
        text=True,
    )
    current = FREEZE.read_text()
    added = sorted(_load(current) - _load(base))
    if added:
        print("flake_freeze.yaml gained ids in this diff:")
        for item in added:
            print(f"  - {item}")
        print("Add suppressions in a human commit, not in an agent patch.")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

The check is asymmetric on purpose. Removing an expired id can be a later human commit. Adding an id inside the same diff that changes src/ is the pattern this gate exists to stop.

4. Protected-path job

Name the denied globs once. Fail closed if git is missing or if the range is ambiguous.

# tests/invariants/check_protected_paths.py
from __future__ import annotations

import fnmatch
import os
import subprocess
import sys

DENIED = (
    "tests/invariants/**",
    "tests/fixtures/**",
    "tests/flake_freeze.yaml",
    ".github/workflows/*protected*",
)


def _changed(base: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        text=True,
    )
    return [line for line in out.splitlines() if line]


def _denied(path: str) -> bool:
    return any(fnmatch.fnmatch(path, pat) for pat in DENIED)


def main() -> int:
    base = os.environ.get("PROTECTED_BASE", "origin/main")
    hits = [path for path in _changed(base) if _denied(path)]
    if hits:
        print("agent-ineligible paths changed:")
        for path in hits:
            print(f"  {path}")
        return 1
    print(f"protected paths clean vs {base}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

# .github/workflows/protected-paths.yml
name: protected-paths
on:
  pull_request:
jobs:
  deny-oracle-edits:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pyyaml pytest
      - run: python tests/invariants/check_protected_paths.py
        env:
          PROTECTED_BASE: origin/${{ github.base_ref }}
      - run: python tests/invariants/check_freeze_diff.py
      - run: pytest tests/invariants -q

CI is the only reviewer that never gets tired of path lists. Put the same commands in a local pre-push hook if you want the failure earlier. Do not skip the remote job.

Procedure

1. On main, write the properties, the fixtures, and an empty freeze file. Run write_manifest.py. Commit those files in a review that does not contain agent output.
2. Open a disposable workspace that contains src/ and, if you want extra examples, tests/unit/. Do not mount tests/invariants/, tests/fixtures/, or tests/flake_freeze.yaml as writable.
3. Point the agent at a single production contract: function names, types, and the properties in prose. Do not paste the SHA-256 manifest into the prompt. The model does not need the oracle to propose an implementation.
4. Copy back only src/** (and optional tests/unit/**). If the copy tool wants to overwrite a denied path, delete that part of the copy. Do not resolve the conflict by accepting the workspace version.
5. Run pytest tests/invariants -q with the same seed the file hard-codes. Then run the two diff checkers against origin/main.
6. If a property is wrong, change it on main in a human commit. Then regenerate the candidate patch. Do not “fix” a red property by editing the property in the same diff as the implementation.
7. If a test is flaky, record the id in flake_freeze.yaml in a separate human commit with a reason and an expiry date. Do not bundle that edit with agent output.

The order matters. Invariants first. Implementation second. Freeze last, and only by a person.

Where a free model and a free server belong

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The generation step is the part that burns cycles and leaves debris. It does not need access to the denied tree. A scratch machine that can reach a free model endpoint is enough to iterate on src/ without writing suppressions into your merge branch.

MonkeyCode’s free model access and free server option fit that sandbox role: produce a candidate implementation off to the side, then pull files through the path ACL above. They are not the merge gate. They should not see production secrets, and they should not be the only place properties run. If the sandbox copy of the repo includes .env or CI tokens, use a different machine.

What this harness will not catch

Denied paths do not encode domain rules you never wrote down. A property that only checks net will miss a reordering bug that still sums. A fixture checksum will not notice that the golden file was wrong on day one.

A fixed seed will not surface every race. It will make the failures you do hit replayable. That is a different claim.

The freeze checker only looks at added ids versus HEAD. A force-push that rewrites main can hide an addition. Protect main. Require the workflow on pull requests. Do not give the agent a token that can disable the job.

Path ACLs are string matches. A new directory test/invariants (singular) will slip through if you only deny tests/invariants/**. Review the glob list when you move trees.

Who should not use this

Skip it if you have no CI, or if anyone can merge with failing checks. The denied list is then documentation, not a gate.

Skip it for throwaway spikes that will never land on a shared branch. The overhead is the protected files and the two checkers, not the model call.

Skip it if the codebase has no stable oracle: hardware timing loops, GUI snapshots without a human-reviewed golden, or tests that must hit a live network. Build a narrower contract first. Do not freeze noise.

Do not put credentials, customer data, or private keys on a shared free server so the agent can “see more context.” The sandbox gets public fixtures and public types. The oracle stays in the repo the agent cannot write.

Closing constraint

Green CI after an agent patch is a necessary condition. It is not a sufficient one. If the diff includes tests/invariants/**, tests/fixtures/**, or a new freeze id, the suite has been taught to agree with the patch. Reject that class of diff before you argue about style.