Treat Agent-Authored Fixtures as Untrusted Input

A green suite after an agent patch does not validate the new fixtures. It often means the model invented bytes that agree with the code it just wrote. Treat those files as untrusted input. Require a provenance record, keep oracles outside the patch, and freeze flakes by stable test identity so a rename cannot lift the freeze.

Coverage going up is not a counter-argument. An agent can raise line counts by asserting against data it also created. The merge question is narrower: where did the fixture come from, and who is allowed to change it in the same diff as the production code?

What this gate is for

This article proposes a three-record merge gate for Python services that accept agent patches. It is a workflow, not a production study. Commands and modules below are labeled as a harness you can run locally; they are not results from a live fleet.

The three records are independent on purpose.

1. A provenance manifest for every fixture file the patch adds or rewrites.
2. A human-owned property module that the same patch cannot edit.
3. A flake ledger keyed by canonical test identity, not by path.

If any record is missing, the patch does not merge. CI color is ignored until the three files exist and the checks below pass.

Why fixtures are the real merge surface

Production code is visible in review. Fixture JSON is easy to skim and easy to launder. An agent that “fixes” a flake by replacing a recorded payload with a quieter synthetic one will still look like a test improvement. The suite stays green. The bug leaves with the old bytes.

Recorded traces and declared synthetics are both legitimate. The failure mode is mixing them without a label, then letting the same patch change code, fixture, and oracle together. That is one author writing the exam and the answer key.

Record 1: provenance, not snapshots

Keep a manifest next to tests. Each fixture path maps to a source class, a content hash, and an optional trace id. Agent-authored files start as untrusted and cannot become recorded inside the same pull request.

{
  "version": 1,
  "fixtures": {
    "tests/fixtures/checkout/ok.json": {
      "source": "recorded",
      "sha256": "6b1c0e2a9f4d8c71a0b5e3d27c9a11f0e8c4b2a19d7e6f3051c8a4b9e2d0c173",
      "trace_id": "trc_20260911_0041"
    },
    "tests/fixtures/checkout/timeout.json": {
      "source": "synthetic",
      "sha256": "aa19c0d4e8b73301f56a2c9d0e41b87c4d2f90ab11c6e5d8a3b04712fe98c6d1",
      "schema": "checkout_error.v2"
    }
  }
}

Allowed source values are recorded, synthetic, and untrusted. A proposed rule set:

1. recorded requires a trace_id that already exists in the trace store. The patch cannot mint a new id.
2. synthetic requires a schema id. The body must validate against that schema in CI.
3. untrusted is the default for any path the agent created. It may run locally. It cannot merge.
4. A source change recorded -> synthetic or recorded -> untrusted is a hard fail while a flake freeze is active on any test that reads the file.

Hash the file, not the pretty-printed review copy. Agents reformat JSON. Reviewers miss that.

# tools/fixture_prov.py — proposed harness, not a shipped product
from __future__ import annotations

import hashlib, json, sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
MANIFEST = ROOT / "tests" / "fixture_provenance.json"
FIXTURE_ROOT = ROOT / "tests" / "fixtures"
ALLOWED = {"recorded", "synthetic", "untrusted"}

def sha256(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def load_manifest() -> dict:
    return json.loads(MANIFEST.read_text())

def check(changed_paths: list[str]) -> list[str]:
    man = load_manifest()
    errors = []
    for rel in changed_paths:
        path = ROOT / rel
        if not str(path).startswith(str(FIXTURE_ROOT)):
            continue
        key = str(path.relative_to(ROOT))
        rec = man.get("fixtures", {}).get(key)
        if rec is None:
            errors.append(f"missing provenance: {key}")
            continue
        if rec.get("source") not in ALLOWED:
            errors.append(f"bad source for {key}: {rec.get('source')}")
        if rec.get("source") == "recorded" and not rec.get("trace_id"):
            errors.append(f"recorded fixture lacks trace_id: {key}")
        if rec.get("source") == "synthetic" and not rec.get("schema"):
            errors.append(f"synthetic fixture lacks schema: {key}")
        if rec.get("source") == "untrusted":
            errors.append(f"untrusted fixture cannot merge: {key}")
        digest = sha256(path)
        if rec.get("sha256") != digest:
            errors.append(f"hash mismatch {key}: manifest={rec.get('sha256')} disk={digest}")
    return errors

if __name__ == "__main__":
    errs = check(sys.argv[1:])
    if errs:
        print("\n".join(errs))
        sys.exit(1)
    print("fixture provenance: ok")

Wire it to the diff, not to the whole tree. Whole-tree scans hide the files the agent actually introduced.

git diff --name-only origin/main...HEAD -- tests/fixtures \
  | xargs -r python tools/fixture_prov.py

Record 2: oracles the patch cannot edit

Property checks belong in a module that the agent branch cannot modify. The point is not “more asserts.” The point is a second author. If the production change and the oracle change share a diff, the oracle is compromised.

A practical split:

1. Agent may edit src/ and may add tests under tests/agent/.
2. Humans own tests/properties/.
3. CI fails if git diff for the patch includes both src/ and tests/properties/.

# tests/properties/test_checkout_invariants.py — human-owned; label: example
from decimal import Decimal

from checkout import totals

def test_line_sum_matches_subtotal(recorded_order):
    sub = totals.subtotal(recorded_order)
    lines = [Decimal(str(x["cents"])) / 100 for x in recorded_order["lines"]]
    assert sum(lines, Decimal("0")) == sub

def test_tax_never_exceeds_subtotal(recorded_order):
    sub = totals.subtotal(recorded_order)
    tax = totals.tax(recorded_order)
    assert tax >= 0
    assert tax <= sub

def test_rejected_status_has_empty_capture(recorded_order):
    if recorded_order["status"] != "rejected":
        return
    assert recorded_order["capture"] is None

Those three properties do not mention the agent’s new helper names. They mention domain facts. Domain facts survive a rewrite of the implementation. Helper names do not.

Generating a first draft of properties is mechanical. Review is not. A spare machine is enough to run the gate. Free model access is enough to propose invariant stubs from a diff, which a reviewer pastes into tests/properties/ on a follow-up commit the agent does not author.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access and free server option fit this split: draft property stubs on the free models, run fixture_prov.py and the ledger check on the free server, and keep merge authority with the human who owns tests/properties/.

Block mixed diffs with a one-liner.

# tools/forbid_mixed_oracle_diff.sh — proposed
set -euo pipefail
files=$(git diff --name-only origin/main...HEAD)
echo "$files" | grep -q '^src/' || exit 0
if echo "$files" | grep -q '^tests/properties/'; then
  echo "src/ and tests/properties/ changed in the same patch"
  exit 1
fi

Record 3: freeze flakes by identity, not by path

Path-keyed freezes fail. Agents rename the file, split a parametrized case, or move a test into a new class. The freeze record misses. The flake is “fixed” by disappearance.

Canonical identity is the node id after stripping the current path prefix and keeping original names plus parameter ids. Store a hash of that identity. Store the fixture paths the test opened. Store an expiry as a date, not as “until green.”

{
  "version": 1,
  "freezes": {
    "c7e10b9a21d44f0e": {
      "node_id_canon": "TestCapture::test_retries[timeout-2]",
      "fixtures": ["tests/fixtures/checkout/timeout.json"],
      "reason": "timeout flake; do not swap recorded payload",
      "expires": "2026-09-25",
      "allow_source_change": false
    }
  }
}

# tools/flake_ledger.py — proposed harness
from __future__ import annotations

import hashlib, json, re, sys
from datetime import date
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
LEDGER = ROOT / "tests" / "flake_ledger.json"
PATH_NOISE = re.compile(r"^(tests/[^:]+::)")

def canon(node_id: str) -> str:
    return PATH_NOISE.sub("", node_id)

def ident(node_id: str) -> str:
    return hashlib.sha256(canon(node_id).encode()).hexdigest()[:16]

def blocked(node_id: str, fixture_path: str, new_source: str) -> str | None:
    rec = json.loads(LEDGER.read_text())["freezes"].get(ident(node_id))
    if rec is None:
        return None
    if date.fromisoformat(rec["expires"]) < date.today():
        return None
    if fixture_path in rec["fixtures"] and new_source != "recorded" and not rec["allow_source_change"]:
        return f"freeze {ident(node_id)} blocks source change on {fixture_path}"
    return None

The ledger does not skip the test. Skipping is how flakes vanish. The ledger only forbids provenance edits and identity churn while the freeze is live. When the date passes, a human either deletes the freeze or files a real fix. Agents do not extend expires.

Detect identity churn by comparing canonical names in the patch against frozen names, ignoring file moves.

python - <<'PY'
import subprocess, re, json, hashlib
from pathlib import Path
ledger = json.loads(Path("tests/flake_ledger.json").read_text())["freezes"]
frozen = {v["node_id_canon"] for v in ledger.values()}
diff = subprocess.check_output(["git","diff","-U0","origin/main...HEAD","--","tests"], text=True)
removed = re.findall(r"^-\s*def (test_\w+)", diff, re.M)
for name in removed:
    hits = [c for c in frozen if name in c]
    if hits:
        raise SystemExit(f"frozen test identity removed: {name} -> {hits}")
print("flake identity: ok")
PY

Merge order

Run the checks in this order. Cheap, then binding.

1. List fixture paths in the patch. Fail if any path lacks a manifest row.
2. Verify hashes and source labels. Fail untrusted on merge branches.
3. Fail mixed src/ plus tests/properties/ diffs.
4. Load the flake ledger. Fail source changes and identity removals under a live freeze.
5. Run the human-owned properties against recorded fixtures only.
6. Run the rest of the suite. Treat that result as necessary, not sufficient.

# tools/agent_fixture_gate.sh — proposed
set -euo pipefail
base=${1:-origin/main}
mapfile -t fx < <(git diff --name-only "$base"...HEAD -- tests/fixtures)
python tools/fixture_prov.py "${fx[@]}"
bash tools/forbid_mixed_oracle_diff.sh
python tools/flake_identity_guard.py
pytest -q tests/properties -k recorded
pytest -q

Decision table

Patch shape	Provenance	Oracle file	Flake ledger	Merge
Code only, recorded fixtures unchanged	n/a	unchanged	unchanged	allowed after properties pass
New synthetic fixture, schema present, properties unchanged	synthetic	unchanged	no freeze on those tests	allowed
New fixture with no manifest row	missing	any	any	reject
Agent marks fixture recorded without an existing trace id	forged	any	any	reject
Same PR edits src/ and tests/properties/	any	compromised	any	reject
Flake freeze live, fixture source becomes synthetic	swap	any	hit	reject
Test file renamed, canonical identity still frozen	any	any	identity match	freeze still applies
Freeze expired, human removed the row	any	any	absent	normal rules

Limitations

This gate does not prove the recorded trace was correct on the day it was captured. Bad production data becomes a pinned lie. It also does not replace contract tests against a real dependency. Schema-valid synthetic fixtures can still encode a wrong business rule.

Hash pins break on intentional fixture upgrades. That is the cost. If you rotate a recorded payload, you rotate the manifest in a PR that does not also rewrite the implementation. Do not combine those edits to save a round trip.

Canonical identity is only as stable as your test names. If the suite already uses generated names, hash the source of the parameters, not the printed node id. Otherwise the ledger will false-positive on every parametrization change.

Who should not use this: solo prototypes with no recorded traffic, teams that already forbid agent-written tests entirely, and codebases whose fixtures are regenerated every run from a live sandbox. Those groups do not have a provenance problem. They have a different one. Adding a ledger there only produces noise.

The harness is also a poor fit if reviewers cannot own tests/properties/. An unreviewed property file is another answer key. In that case, stop generating stubs until a person is named as the oracle owner.

What to keep when you strip the tools

Three rules survive without any script. Do not merge a fixture the patch also invented. Do not let the same diff change production code and the properties that bless it. Do not let a flake disappear by rename or by quieter data.

If you already have a machine that can run pytest, the gate is those three checks in order. Free model access is optional and only for drafting invariants a human still has to accept. The merge decision stays with the provenance file, the oracle split, and the ledger — not with a green job name.