Demote Equality Tests When an Agent Mutates Their Fixtures

A merge gate that still treats assertEqual(actual, golden) as evidence is already compromised when the same patch rewrites the golden. Score three things independently: schema validity of the fixture, properties over a derived view that never reads the new bytes, and a quarantine budget for failures that do not replay. None of those scores should be produced only on the author's laptop.

Agent patches fail this pattern in a boring way. The code under test moves. The snapshot file moves with it. CI stays green. The equality test did not check a behavior. It checked that two writes in one diff agree with each other.

This workflow is a proposed pre-merge filter, not a production study. It uses git path classification, a JSON Schema check, and a small Python property file that must live outside the fixture directory. Label every example below as unexecuted until you run it on a real pull request.

Why equality collapses

Golden-file tests are characterization tests. They are useful when a human froze the bytes and a later patch is forbidden from touching them. They stop being tests when the author of the code is also the author of the expected output.

Agents do that rewrite because it is the cheapest way to clear a red job. Deleting an assertion is louder. Editing testdata/order.json is quiet. Quiet is the failure mode.

A fixture lock that forbids every golden edit is too blunt. Some patches should update fixtures: a negotiated wire format, a versioned catalog, a migrated column. The gate needs a third state between "bytes must match" and "bytes may change." That state is: the file may change, equality on that file is demoted, and a schema plus derived properties still have to pass.

Three independent scores

Do not fold these into one boolean.

1. Schema score. The mutated fixture must parse and validate. Invalid JSON that happens to match a rewritten expected string is not a pass.
2. Derived-property score. Cardinality, key sets, sort stability, round-trip type, and range checks run on a view computed from the fixture. The property module must not import the golden path.
3. Quarantine budget. A failure that does not replay on a pinned remote command spends budget. It does not freeze the test, and it does not count as green.

Equality on a mutated golden becomes an informational annotation. It can show up in the review comment. It cannot vote for merge.

Step 1: Classify paths in the patch

Run this on the merge ref, not on a dirty worktree. The classifier only needs git and the standard library.

git fetch origin main
git diff --name-status origin/main...HEAD > /tmp/patch.names
python3 fixture_gate.py /tmp/patch.names

# fixture_gate.py — proposed classifier, unexecuted here
from __future__ import annotations

import pathlib
import sys

FIXTURE_MARKERS = (
    "/testdata/",
    "/fixtures/",
    ".snap",
    ".golden",
    ".expected.",
)
PROPERTY_ROOT = pathlib.Path("properties")
SCHEMA_ROOT = pathlib.Path("schemas")


def is_fixture(path: str) -> bool:
    lower = path.replace("\\", "/").lower()
    return any(m in lower for m in FIXTURE_MARKERS)


def load_name_status(path: str) -> list[tuple[str, str]]:
    rows = []
    for line in pathlib.Path(path).read_text().splitlines():
        if not line.strip():
            continue
        status, rel = line.split("\t", 1)
        rows.append((status[0], rel))
    return rows


def sibling_property(rel: str) -> pathlib.Path:
    stem = pathlib.Path(rel).name.split(".")[0]
    return PROPERTY_ROOT / f"{stem}_view_test.py"


def sibling_schema(rel: str) -> pathlib.Path:
    stem = pathlib.Path(rel).name.split(".")[0]
    return SCHEMA_ROOT / f"{stem}.schema.json"


def main() -> int:
    rows = load_name_status(sys.argv[1])
    mutated = [rel for status, rel in rows if is_fixture(rel) and status in {"A", "M", "R"}]
    if not mutated:
        print("fixture_gate: no fixture mutation; equality tests may remain gating")
        return 0
    missing = []
    for rel in mutated:
        prop = sibling_property(rel)
        schema = sibling_schema(rel)
        if not prop.is_file():
            missing.append(f"missing derived property: {prop} (for {rel})")
        if not schema.is_file():
            missing.append(f"missing schema: {schema} (for {rel})")
    if missing:
        print("fixture_gate: demote equality; blocking gaps:")
        print("\n".join(missing))
        return 2
    print("fixture_gate: fixture mutation detected; equality demoted")
    print("\n".join(mutated))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

A non-zero exit is a missing artifact, not a failed equality. That distinction matters. Reviewers keep asking why CI is red. The answer should be "no derived view," not "bytes differ."

Step 2: Demote equality, do not delete it

Keep the golden assertion. Change how the runner reports it. Informational failures belong in a log file that merge is allowed to ignore. Gating failures belong in the process exit code.

# run_lanes.py — proposed two-lane runner
import subprocess
import sys

GATING = ["tests/properties", "tests/schema"]
INFO = ["tests/goldens"]


def run(paths: list[str]) -> int:
    cmd = [sys.executable, "-m", "pytest", "-q", *paths]
    return subprocess.call(cmd)


def main() -> int:
    info_rc = run(INFO)
    pathlib_note = "informational golden lane rc=%s" % info_rc
    print(pathlib_note)
    return run(GATING)


if __name__ == "__main__":
    import pathlib  # noqa: F401 — printed note only
    raise SystemExit(main())

If your suite cannot split those directories yet, start with a pytest marker. Mark equality tests that import a mutated path as golden. Then invoke -m "not golden" on the gating lane. The marker is mechanical. The policy is the split.

Step 3: Put properties on a derived view

A property that opens the golden file is just equality with extra functions. Compute a view first. Persist the view next to the test, not next to the fixture, so a later patch cannot satisfy the gate by rewriting both.

# properties/order_view_test.py — proposed, unexecuted
from __future__ import annotations

import json
import pathlib
from decimal import Decimal
from typing import Any

FIXTURE = pathlib.Path("testdata/order.json")  # read once, never asserted byte-for-byte


def view(doc: dict[str, Any]) -> dict[str, Any]:
    items = doc["items"]
    totals = [Decimal(str(i["cents"])) for i in items]
    return {
        "n_items": len(items),
        "skus": sorted({i["sku"] for i in items}),
        "cents_sum": int(sum(totals)),
        "all_cents_nonneg": all(t >= 0 for t in totals),
        "currency": doc["currency"],
    }


def test_derived_invariants() -> None:
    doc = json.loads(FIXTURE.read_text())
    v = view(doc)
    assert v["n_items"] >= 1
    assert v["currency"] in {"USD", "EUR", "JPY"}
    assert v["all_cents_nonneg"]
    assert v["cents_sum"] == int(doc["cents_total"])
    assert len(v["skus"]) == len(set(v["skus"]))

Add one more check that does not care about the stored total: permutation stability. Shuffle is a poor local test. Sort-then-sum is enough and deterministic.

def test_sum_is_order_independent() -> None:
    doc = json.loads(FIXTURE.read_text())
    forward = sum(int(i["cents"]) for i in doc["items"])
    reverse = sum(int(i["cents"]) for i in reversed(doc["items"]))
    assert forward == reverse == int(doc["cents_total"])

If the agent updates cents_total and the item list together, both tests still constrain the pair. If it only rewrites the golden total, the derived view fails. That is the point of demotion: equality would have passed.

Step 4: Schema-check the new fixture

Do not hand-roll a second parser if the fixture is JSON. A draft schema is enough to reject empty objects and type flips.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["currency", "cents_total", "items"],
  "properties": {
    "currency": {"type": "string", "minLength": 3, "maxLength": 3},
    "cents_total": {"type": "integer", "minimum": 0},
    "items": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "required": ["sku", "cents"],
        "properties": {
          "sku": {"type": "string", "minLength": 1},
          "cents": {"type": "integer"}
        },
        "additionalProperties": false
      }
    }
  },
  "additionalProperties": false
}

python3 -m pip install check-jsonschema
check-jsonschema --schemafile schemas/order.schema.json testdata/order.json

YAML and CSV follow the same split: validate structure, then derive a view. Binary goldens (images, protobuf wire dumps) need a different derived view, such as dimensions or field presence. Byte equality stays informational there too, once the patch touched the file.

Step 5: Quarantine with a budget, never a freeze

A freeze flag is how flakes become permanent skips. Use a counter instead. A test may enter quarantine only after a remote replay fails to reproduce the local failure. Each quarantine entry spends one unit. Merge is blocked when spend exceeds the budget for that suite.

# quarantine.py — proposed state machine
from __future__ import annotations

import json
import pathlib
from dataclasses import dataclass

BUDGET = 3
STATE = pathlib.Path(".quarantine.json")


@dataclass
class Entry:
    nodeid: str
    spends: int
    last_remote_rc: int
    input_hash: str


def load() -> dict[str, Entry]:
    if not STATE.is_file():
        return {}
    raw = json.loads(STATE.read_text())
    return {k: Entry(**v) for k, v in raw.items()}


def save(items: dict[str, Entry]) -> None:
    STATE.write_text(json.dumps({k: vars(v) for k, v in items.items()}, indent=2) + "\n")


def record(nodeid: str, remote_rc: int, input_hash: str) -> int:
    items = load()
    cur = items.get(nodeid, Entry(nodeid, 0, remote_rc, input_hash))
    if remote_rc == 0:
        # Local fail, remote pass: spend, do not skip forever.
        cur.spends += 1
        cur.last_remote_rc = 0
        cur.input_hash = input_hash
        items[nodeid] = cur
        save(items)
        spent = sum(e.spends for e in items.values())
        return 2 if spent > BUDGET else 0
    # Remote also failed: this is a regression, not a flake.
    return 1

Replay has to include the input hash. A quarantine record without the input is just a skip list. Compute the hash from the fixture bytes plus the command that loaded them.

python3 - <<'PY'
import hashlib, pathlib, sys
cmd = sys.argv[1:]
body = pathlib.Path("testdata/order.json").read_bytes()
print(hashlib.sha256(b"\0".join([body] + [a.encode() for a in cmd])).hexdigest())
PY
pytest -q tests/properties/order_view_test.py

Classification belongs off the laptop. Clock skew, extra pytest plugins, and a dirty pip cache all manufacture one-off failures. A free remote runner is the right place to spend the budget. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft candidate derived-view properties from a fixture schema, and the free server option is enough to replay the same command away from the author's machine. The gate itself does not depend on that product. Any pinned remote runner that accepts a command and returns an exit code works.

Decision table

Fixture in diff?	Schema	Derived properties	Remote replay of a local fail	Merge
No	n/a	existing suite	n/a	Equality may still gate
Yes	fail	n/a	n/a	Block
Yes	pass	missing file	n/a	Block
Yes	pass	fail	n/a	Block
Yes	pass	pass	remote fail	Block (regression)
Yes	pass	pass	remote pass, budget left	Allow, spend 1
Yes	pass	pass	remote pass, budget exceeded	Block
Yes	pass	pass	no local fail	Allow; equality is info only

The table has no row that says "freeze the test." Skipping is not a score.

Limitations

The classifier is filename heuristics. A golden stored as src/testdata.py with a giant string constant will slip through until you add that path. Schema drafts reject structure, not domain meaning. A property that asserts True or re-reads the golden will satisfy the file-existence check and still be useless.

Remote replay does not pin CPU, libc, or locale unless you do that yourself. A free server is not a reproducible-build system. If your flakes are data races, a second machine will only sample them differently. This filter also does not replace review of legal or contractual snapshots. When the fixture is the deliverable, a human owns the byte change.

Derived views have a coverage hole: they ignore fields you forgot to project. Pair this gate with a schema additionalProperties: false so extra keys cannot hide in the unprojected tail. That pairing is still weaker than a full model of the domain.

Who should not use this

Skip the demotion rule if no agent (or intern, or formatter) is allowed to edit testdata. A strict fixture lock is simpler there. Skip the quarantine budget if you cannot replay with an input hash. Skip the whole split if the repository is a one-file script with no git history. Do not use this as an auto-approve policy for agent pull requests. Three green scores mean the cheapest self-pass was blocked. They do not mean the patch is correct.

If you already isolate agent jobs on a remote runner, drop fixture_gate.py on the PR diff there before treating a local green as merge evidence. The informational golden lane will still be noisy. That noise is now a review comment, not a vote.