Judge Agent Patches Against a Fixture Pin They Did Not Write

An agent patch is admissible only when a fixture pin taken before generation still matches after the patch lands. Property checks then run on that pinned tree. A flake freeze is not a bypass around either result.

A digest change is a fixture breach. Further repeats do not reclassify it. Mixed pass and fail results can be noise candidates only while the pin matches, and even then this gate does not write a freeze.

That order is the strategy. Generation proposes a diff. A separate runner adjudicates it. The model does not grade the files it was allowed to edit.

Why a green rerun is the wrong signal

Shared workspaces hide a specific failure. The step that edits production code can also drop a fixture row, loosen a numeric bound, or replace a file with a smaller lookalike. The property passes because the input got weaker, not because the behavior got stronger.

Rerunning that command stays green. A policy that compares only exit codes never sees a disagreement, so it never opens a record. The miss is silent. A pin makes the same miss visible: the byte manifest changed, so the run is ineligible for accept and ineligible for freeze.

Treat the rest of this piece as a workflow proposal for repositories that already store durable fixtures and already expose property checks as a command with an exit code. No pass rate is claimed. The command examples and the script were not executed against a live model for this article.

What the pin stores

Store a sorted list of relative paths under one fixture root. For each file, store the byte length and a SHA-256 digest. A path that appears on only one side is a change, even when every surviving digest still matches.

Take the pin from the base revision, before the model is called. Write it outside the worktree the patch will touch. Do not refresh it because a commit message says the fixtures were normalized.

Branch names are not pins. Timestamps are not pins. A clean status after the agent amends fixtures is not a pin either. The manifest is the pin.

Decision table

Assign one class. Do not stack classes to make a run look milder.

Pin diff	Repeat exit codes	Class	Freeze written here
non-empty	not consulted	fixture_breach	no
empty	all zero	accept	no
empty	all non-zero	property_fail	no
empty	mixed zero and non-zero	noise_candidate	no

The freeze column is intentionally all no. A noise candidate may be handed to a separate flake policy, with the pin digest and the seed schedule attached. This gate only refuses fixture rewrites, and it refuses to treat a deterministic property failure as noise.

Replay budgets, witness shrinking, and expiry clocks stay in that other policy. Leaving them out keeps the pin decision auditable on its own. A freeze that cannot name the pin digest it was judged under did not come from this strategy.

Numbered workflow

1. Pin the base tree

Record the revision the agent may edit. Build the manifest from that revision's fixture root, then run the local classifier checks.

git rev-parse HEAD > .pin/base_rev
python pin_gate.py pin --root tests/fixtures --out .pin/manifest.json
python pin_gate.py logic-check

logic-check runs the classifier assertions in the reference script. It does not call a model. It does not measure your suite. It only shows that a non-empty diff outranks exit codes, which is the rule the rest of the gate depends on.

2. Generate without the pin mounted

Send the task and the base revision. State that paths under the fixture root are read-only for this job. Do not mount .pin/ into the generation workspace. Do not ask the generator to certify its own patch.

Operators with MonkeyCode free model access can confine this step to that endpoint. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The only product claims used here are free model access and a free server option. Model identity, quota, hardware, duration, and retention are unspecified, so the workflow must not depend on them.

3. Apply the patch in a detached worktree

git worktree add --detach /tmp/agent-judge "$(cat .pin/base_rev)"
git -C /tmp/agent-judge apply --check /tmp/candidate.patch
git -C /tmp/agent-judge apply /tmp/candidate.patch

If apply --check fails, stop. A rejected patch is a patch-format failure. It is not a flake, and it is not a property result. Do not hand-edit the diff so the check will pass and then pretend the agent produced that edit.

4. Judge on a runner that did not generate the diff

python pin_gate.py judge \
  --pin .pin/manifest.json \
  --root /tmp/agent-judge/tests/fixtures \
  --cwd /tmp/agent-judge \
  --repeat 5 \
  --seed 20261013 \
  --out .pin/verdict.json \
  --property-cmd python -m pytest -q tests/properties

--repeat 5 and --seed 20261013 are example CLI parameters, not a tuned threshold and not a reported experiment. The seed value is an arbitrary constant for the example. The script exports it as PIN_SEED. If the property command ignores that variable, the repeat loop does not control randomness, and mixed codes are not yet evidence of a flake.

A free server option fits this step when the runner can check out the base revision, apply the patch, and execute the property command. Keep the model off that job. If the server is ephemeral, copy manifest.json in at the start and copy verdict.json out before the machine is released. Do not assume the server stores either file after the job ends.

5. Act on one class

Fixture breach means read changes and stop. Do not schedule flake replays to confirm a digest mismatch. Property fail means keep the failing command, the seed, and the non-zero codes. A stable failure is a product or specification signal, not a candidate for silence.

Accept means the pin matched and every listed exit code was zero. It does not mean the property encodes the specification, and it does not mean the production diff is safe to merge. Noise candidate means archive the codes, the seed, and the pin digest for the separate flake policy.

The reference script sets freeze_allowed to false on every branch. A later job that only checks for a verdict file cannot mistake that file for a freeze record.

6. Prefer the verdict over the generator summary

If the model text says fixtures were unchanged, compare that sentence to changes. On any disagreement, quote the verdict JSON. The summary is not evidence.

If a free server is already in the job path, keep pin_gate.py judge on that runner and leave the model endpoint out of the job definition. That split is the useful part. A longer product tour is not required to apply the table.

Reference classifier

The module below is an unexecuted reference for the decision, not a measured harness. logic-check is the piece a reader can run without a model, a network, or a fixture corpus. judge shells out to --property-cmd with cwd pointed at the detached worktree, and it skips that shell-out when the manifest diff is non-empty.

#!/usr/bin/env python3
"""Unexecuted reference: pin a fixture root and class an agent patch.

Does not open a flake freeze. Does not shrink counterexamples.
logic-check is runnable without a model or a fixture corpus.
"""

from __future__ import annotations

import argparse
import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()


def build_manifest(root: Path) -> dict:
    if not root.is_dir():
        raise SystemExit(f"fixture root is not a directory: {root}")
    files = []
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        rel = path.relative_to(root).as_posix()
        files.append(
            {
                "path": rel,
                "bytes": path.stat().st_size,
                "sha256": sha256_file(path),
            }
        )
    return {"files": files}


def diff_manifests(before: dict, after: dict) -> list[dict]:
    old = {item["path"]: item for item in before.get("files", [])}
    new = {item["path"]: item for item in after.get("files", [])}
    changes = []
    for rel in sorted(set(old) | set(new)):
        if rel not in old:
            changes.append({"path": rel, "change": "added"})
        elif rel not in new:
            changes.append({"path": rel, "change": "removed"})
        elif old[rel]["sha256"] != new[rel]["sha256"]:
            changes.append(
                {
                    "path": rel,
                    "change": "modified",
                    "before": old[rel]["sha256"],
                    "after": new[rel]["sha256"],
                }
            )
    return changes


def classify(changes: list, codes: list[int]) -> str:
    if changes:
        return "fixture_breach"
    if not codes:
        raise ValueError("repeat must be >= 1")
    if all(code == 0 for code in codes):
        return "accept"
    if all(code != 0 for code in codes):
        return "property_fail"
    return "noise_candidate"


def run_property(cmd: list[str], cwd: Path, seed: int) -> int:
    env = os.environ.copy()
    env["PIN_SEED"] = str(seed)
    completed = subprocess.run(cmd, cwd=cwd, env=env)
    return completed.returncode


def judge(
    pin_doc: dict,
    root: Path,
    cwd: Path,
    cmd: list[str],
    repeat: int,
    seed: int,
) -> dict:
    if repeat < 1:
        raise SystemExit("repeat must be >= 1")
    changes = diff_manifests(pin_doc, build_manifest(root))
    if changes:
        return {
            "class": "fixture_breach",
            "changes": changes,
            "codes": [],
            "freeze_allowed": False,
        }
    codes = [run_property(cmd, cwd, seed + offset) for offset in range(repeat)]
    return {
        "class": classify([], codes),
        "changes": [],
        "codes": codes,
        "seed": seed,
        "repeat": repeat,
        "freeze_allowed": False,
    }


def logic_checks() -> None:
    assert classify([{"path": "row.json", "change": "modified"}], [0, 0]) == "fixture_breach"
    assert classify([], [0, 0, 0]) == "accept"
    assert classify([], [1, 1, 1]) == "property_fail"
    assert classify([], [0, 1, 0]) == "noise_candidate"
    try:
        classify([], [])
    except ValueError:
        return
    raise AssertionError("empty repeat must fail")


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description="Pin fixtures and class an agent patch")
    sub = parser.add_subparsers(dest="command", required=True)

    pin_cmd = sub.add_parser("pin")
    pin_cmd.add_argument("--root", type=Path, required=True)
    pin_cmd.add_argument("--out", type=Path, required=True)

    sub.add_parser("logic-check")

    judge_cmd = sub.add_parser("judge")
    judge_cmd.add_argument("--pin", type=Path, required=True)
    judge_cmd.add_argument("--root", type=Path, required=True)
    judge_cmd.add_argument("--cwd", type=Path, required=True)
    judge_cmd.add_argument("--repeat", type=int, default=5)
    judge_cmd.add_argument("--seed", type=int, default=20261013)
    judge_cmd.add_argument("--out", type=Path, required=True)
    judge_cmd.add_argument("--property-cmd", nargs="+", required=True)

    args = parser.parse_args(argv)
    if args.command == "logic-check":
        logic_checks()
        print("logic_checks_ok")
        return 0
    if args.command == "pin":
        args.out.parent.mkdir(parents=True, exist_ok=True)
        args.out.write_text(json.dumps(build_manifest(args.root), indent=2) + "\n")
        return 0

    pin_doc = json.loads(args.pin.read_text(encoding="utf-8"))
    verdict = judge(
        pin_doc,
        args.root,
        args.cwd,
        args.property_cmd,
        args.repeat,
        args.seed,
    )
    args.out.parent.mkdir(parents=True, exist_ok=True)
    args.out.write_text(json.dumps(verdict, indent=2) + "\n")
    return 0 if verdict["class"] == "accept" else 2


if __name__ == "__main__":
    sys.exit(main())

A minimal breach verdict illustrates the schema. It is not an observed CI artifact.

{
  "class": "fixture_breach",
  "changes": [{"path": "orders/row.json", "change": "modified"}],
  "codes": [],
  "freeze_allowed": false
}

judge returns process status 0 only for accept. Every other class returns 2. Runners that collapse any non-zero status into a flake will misfile breaches and property failures. Map the JSON class field. Do not infer the class from the shell status alone.

Hash file bytes, and decide explicitly what to do with symlinks. Path.read_bytes follows the target, so a retargeted link can change the digest without an edit to the link text. Keep generated caches, bytecode, and editor swap files out of the fixture root. Otherwise every run becomes a breach, and the class stops meaning anything.

Limitations

A byte pin cannot see a specification change that never touches the pinned root. An agent can weaken a check by editing the property file instead of the fixture. Pin every input the property reads, including sibling data directories, or accept that blind spot and review those paths in the diff.

The table does not parse assertion text. Two non-zero exits can be different defects. They remain property_fail here, because splitting them is a counterexample problem. This gate will not shrink a witness, and it will not decide that two failures are the same failure.

The repeat loop is only as deterministic as the command. Clock reads, network calls, and hash randomization that ignore PIN_SEED produce mixed codes that look like noise. Do not forward those runs to a flake policy until the command is seed-stable on the pinned tree.

Free-model output stays untrusted input. A patch can delete the pin script, vendor a test runner, or rewrite CI YAML without touching the fixture manifest. Review the full diff. Access to a free endpoint or a free server says nothing about uptime, capacity, or how long a verdict remains stored.

Who should skip this gate

Skip it when fixtures are built inside the test and have no pre-image on disk. Pinning an empty root either blocks every run or locks nothing useful. Skip it for intentional snapshot updates and schema migrations, unless those paths are excluded before the diff. Otherwise every valid update is a breach, and reviewers will learn to ignore the class.

Skip it if the model is expected to refresh golden files in the same patch that changes behavior. That needs a human-signed pin update, which is a different procedure with a different record. Skip it if accept will be wired to automatic merge. The pin checks identity of inputs. It does not score property strength, and it does not review the production change.

What to file with the patch

File the base revision, manifest.json, and verdict.json. The class is the review unit. A noise candidate may leave this gate. If a separate policy later writes a freeze, that record has to cite the pin digest used here. Without that citation, the freeze is not part of this strategy, and it should not be described as one.