Make the Test Diff Boring Before Merging an Agent Patch

Green CI is the wrong merge signal for an agent-authored diff. The useful signal is a classified parent-versus-head test diff: fixture digests that did not move, properties that still fail a known-bad seed, and a freeze ledger whose flake statuses are unchanged.

Agent patches fail in a narrow way. They add tests that describe the new code, skip a noisy case, or rewrite an assertion until the suite is quiet. A reviewer who only reads production files will miss that. A reviewer who diffs the oracle will not.

This article proposes a local merge gate. It is a workflow template, not a production study. Snippets below are unlabeled as executed results; treat them as a reproducible checklist against any two git trees.

What "boring" means

A boring test diff has three properties.

1. Every human fixture file has the same content digest as on the parent commit, or a human-owned changelog explains the digest change.
2. Every property check still falsifies a checked-in bad seed. If the property goes green on that seed, the property died.
3. No frozen flake changed result, skip mark, or file path. Agents do not thaw tests.

If any lane moves without a human note, the patch is not mergeable. The production diff can wait.

Why parent-versus-head beats "all tests passed"

A test runner returning 0 on HEAD does not tell you whether HEAD invented its own oracle. Compare the two trees.

Parent is the last human-trusted commit. Head is the agent patch. The gate classifies every test-related path into fixture, property, freeze, agent_test, or other. Then it scores the delta.

Untrusted paths are not automatically rejected. They are barred from being the only reason the suite is green.

Artifact 1: content-addressed fixtures

Store fixtures as bytes, not as "the test that imports the new helper." Hash the input and the expected output separately. A patch that "fixes" a fixture by editing the expected blob is a test-diff event, not a silent pass.

# fixture_digest.py — template, unexecuted against your tree
from __future__ import annotations

import hashlib
import json
from pathlib import Path

FIXTURE_ROOT = Path("tests/fixtures")


def digest_file(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def catalog(root: Path = FIXTURE_ROOT) -> dict[str, str]:
    out: dict[str, str] = {}
    for path in sorted(root.rglob("*")):
        if path.is_file() and path.suffix in {".json", ".txt", ".bin", ".csv"}:
            rel = path.relative_to(root).as_posix()
            out[rel] = digest_file(path)
    return out


def write_lock(path: Path = Path("tests/fixtures.lock.json")) -> None:
    path.write_text(json.dumps(catalog(), indent=2, sort_keys=True) + "\n")

Compare lockfiles across commits with a structural check, not a raw git diff on whitespace.

# check_fixture_lock.py — template
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

LOCK = "tests/fixtures.lock.json"
NOTE = Path("tests/FIXTURE_CHANGELOG.md")


def show(ref: str, path: str) -> dict:
    proc = subprocess.run(
        ["git", "show", f"{ref}:{path}"],
        capture_output=True,
        text=True,
    )
    if proc.returncode != 0 or not proc.stdout.strip():
        return {}
    return json.loads(proc.stdout)


def main(parent: str) -> None:
    before = show(parent, LOCK)
    after = json.loads(Path(LOCK).read_text()) if Path(LOCK).exists() else {}
    moved = sorted(k for k in set(before) | set(after) if before.get(k) != after.get(k))
    if not moved:
        return
    if not NOTE.exists() or parent not in NOTE.read_text():
        sys.stderr.write("fixture digests moved without changelog for parent " + parent + "\n")
        sys.stderr.write("\n".join(moved) + "\n")
        raise SystemExit(2)


if __name__ == "__main__":
    main(sys.argv[1])

The lockfile is human-owned. An agent patch that changes tests/fixtures.lock.json without a matching note in tests/FIXTURE_CHANGELOG.md fails the gate. That is the whole fixture rule.

Artifact 2: properties that must still fail

A property check is only a merge oracle if it can fail. Keep a seed that is known to violate the invariant. Run that seed on every agent patch. If the seed no longer fails, the property was weakened, deleted, or made self-describing by the same change that claims to implement it.

# properties/invariants.py — template
from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Seed:
    name: str
    payload: bytes
    must_fail: bool


# Human-owned. Do not accept agent edits to this list on the same PR.
KNOWN_BAD = [
    Seed("empty_batch", b"", must_fail=True),
    Seed("duplicate_keys", b'{"a":1,"a":1}', must_fail=True),
]


def parse_batch(payload: bytes) -> list[tuple[str, int]]:
    """Replace with the unit under test. Shown as a stub."""
    raise NotImplementedError


def check_unique_keys(rows: list[tuple[str, int]]) -> None:
    keys = [k for k, _ in rows]
    if len(keys) != len(set(keys)):
        raise AssertionError("duplicate keys")


def run_known_bad() -> None:
    expected = [s for s in KNOWN_BAD if s.must_fail]
    failures = 0
    for seed in expected:
        try:
            rows = parse_batch(seed.payload)
            check_unique_keys(rows)
        except Exception:
            failures += 1
            continue
        raise AssertionError(f"property no longer fails on seed {seed.name}")
    if failures != len(expected):
        raise AssertionError("known-bad corpus did not all fail")

The important line is not the parser. It is the assertion that the bad seed still fails. Agent patches that "fix" the seed, skip it, or catch the exception inside the property are test-diff events.

Classify properties in a sidecar so the gate can refuse self-referential ones:

# tests/properties.index
# kind          path                         owner
invariant       properties/invariants.py     human
metamorphic     properties/roundtrip.py      human
# agent-authored properties are recorded, never used as the green signal
generated       properties/agent_guess.py    agent

A metamorphic row is useful when the exact bytes are unknown but a relation is stable: parse then serialize, sort then unique, encode then decode. Keep those relations on the human index. Do not let the same patch both introduce the relation and supply its only example.

Artifact 3: a freeze ledger, not a skip comment

A flake that is skipped in source is invisible to reviewers watching production files. Put flake state in a ledger keyed by test node id. The agent may not change the row.

{
  "version": 1,
  "entries": [
    {
      "nodeid": "tests/test_replay.py::test_timeout_window",
      "status": "frozen",
      "reason": "timing depends on wall clock",
      "expires": null,
      "owner": "human"
    }
  ]
}

expires: null means a human must thaw it. A date field is allowed. An agent patch that sets status to active, deletes the row, or renames the test without a ledger update is rejected.

# freeze_gate.py — template
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

LEDGER = "tests/freeze_ledger.json"


def git_show(ref: str, path: str) -> str:
    proc = subprocess.run(
        ["git", "show", f"{ref}:{path}"],
        capture_output=True,
        text=True,
    )
    if proc.returncode != 0:
        return ""
    return proc.stdout


def ledger_map(raw: str) -> dict[str, dict]:
    if not raw.strip():
        return {}
    data = json.loads(raw)
    rows = {}
    for row in data.get("entries", []):
        key = row["nodeid"]
        rows[key] = {
            "status": row.get("status"),
            "reason": row.get("reason"),
            "expires": row.get("expires"),
            "owner": row.get("owner"),
        }
    return rows


def assert_freeze_unchanged(parent: str) -> None:
    before = ledger_map(git_show(parent, LEDGER))
    after = ledger_map(Path(LEDGER).read_text() if Path(LEDGER).exists() else "")
    if before != after:
        sys.stderr.write("freeze ledger moved; human thaw required\n")
        raise SystemExit(2)


if __name__ == "__main__":
    assert_freeze_unchanged(sys.argv[1])

The comparison is structural, not textual. Key order may change. Status, node id, owner, and expiry may not.

Numbered gate

Run this on every agent patch. Parent is HEAD before applying the patch, or the default-branch tip.

1. Record PARENT=$(git rev-parse HEAD) before the agent edits anything.
2. Apply the patch on a throwaway branch.
3. Build the fixture catalog on parent and head. Fail if digests moved without tests/FIXTURE_CHANGELOG.md naming that parent SHA.
4. Run run_known_bad() against head. Fail if any must-fail seed passes.
5. Diff tests/properties.index. Fail if a row with owner=human changed owner or path.
6. Run assert_freeze_unchanged "$PARENT". Fail on any ledger delta.
7. Collect test node ids added under tests/ that are not listed as human fixtures or human properties. Label them agent_test. They may run. They cannot be the sole green signal.
8. Print a three-column report: path, lane, verdict. Merge only if every oracle lane is unchanged or human_noted.

Minimal report shape:

path                         lane        verdict
tests/fixtures/batch.json    fixture     unchanged
properties/invariants.py     property    unchanged
tests/freeze_ledger.json     freeze      unchanged
tests/test_agent_new.py      agent_test  ignored_as_oracle
src/batch.py                 other       n/a

Wire the commands as a single script so the report is the review artifact, not a screenshot of a local test run.

# oracle_diff.sh — template
set -euo pipefail
PARENT="${1:?parent sha}"
python check_fixture_lock.py "$PARENT"
python freeze_gate.py "$PARENT"
python -c "from properties.invariants import run_known_bad; run_known_bad()"
python classify_test_paths.py --parent "$PARENT" --head HEAD

classify_test_paths.py can be a short git diff --name-status filter. Map tests/fixtures/ to fixture, properties/ plus tests/properties.index to property, tests/freeze_ledger.json to freeze, other tests/** additions to agent_test, and everything else to other.

Decision table

Test-diff event	Merge?	Why
Production code changes, all three oracle lanes unchanged	Yes	Oracle did not move
Fixture digest changes with changelog + reviewer note	Yes	Human resized the oracle
Fixture digest changes, no changelog	No	Expected output was edited to match the patch
Known-bad seed now passes	No	Property lost falsifiability
New tests added only by the agent	Not as oracle	May run; cannot green the patch
Frozen node id skipped, renamed, or deleted	No	Flake was thawed by the patch
Freeze ledger expiry edited by the agent	No	Thaw is a human action
Property index owner flipped to agent	No	Oracle ownership moved with the code
Metamorphic relation added with a human index row and a failing seed	Yes	New oracle is falsifiable and owned
Metamorphic relation added only inside agent tests	No	Relation is not an oracle yet

Where a free coding environment fits

Generating candidate patches is cheap enough to do off the critical path if the follow-up work is this gate, not another round of prompt retries. MonkeyCode's free model access and free server option are relevant here only as a place to produce the candidate diff without shipping the fixture corpus to a paid endpoint.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The gate itself does not depend on that environment. Copy the lockfile, the known-bad seeds, and the freeze ledger into any CI image. If you already generate patches some other way, skip the generator and keep the oracle diff. The scripts above are the part to copy.

Limitations

This workflow does not measure mutation score, production incident rate, or reviewer time. It does not replace a fuzzer. It will not catch a wrong invariant that a human wrote and an agent faithfully implemented.

Content-addressed fixtures assume the expected bytes are stable. Snapshot tests that embed timestamps, hostnames, or unordered maps will thrash the lockfile. Freeze the snapshot format first, or exclude those files from the catalog.

Known-bad seeds only protect the properties you remembered to seed. An invariant with no failing seed is not an oracle under this scheme. Add the seed before the agent runs, not after.

The freeze ledger cannot see flakes that never failed in CI. If a test is noisy only on one runner, record that node id by hand. Do not wait for the agent to skip it.

Parent-versus-head classification also fails closed when git history is shallow. Fetch enough depth for git show $PARENT:tests/fixtures.lock.json to resolve. A missing parent lockfile should be a hard error on a repo that already committed one, and a no-op only on first introduction.

Who should not use this

Do not install this gate on a repository that has no human-owned tests yet. You would freeze an empty oracle and then reject every useful patch.

Do not use it as a reason to stop reading production diffs. A boring test diff plus a logic error in src/ is still a logic error.

Do not apply it to exploratory branches where the fixtures are the work. Prototype first. Turn the gate on when the oracle should stop moving.

Teams that already have contract tests, Pact files, or recorded HTTP cassettes can map those artifacts onto the fixture lane. They should not duplicate them as a second lockfile without a reason.

A small CI stub

# .github/workflows/oracle-diff.yml — template
name: oracle-diff
on:
  pull_request:
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: parent-vs-head oracle diff
        run: |
          PARENT="${{ github.event.pull_request.base.sha }}"
          bash oracle_diff.sh "$PARENT"

Wire the Python entrypoints so a non-zero exit fails the job. Keep the scripts boring. The value is the classification, not the framework.

The merge question is not "did tests pass." It is "did the oracle move." If the test diff is boring, the production diff is finally worth reading.