Four Test Hunk Classes. Merge Only One With Production Code.

An agent patch that changes production code and the tests that police it in the same commit is not a controlled experiment. The control moved with the treatment. Split the test tree. Classify every test hunk. Merge production code only against tests the agent was not allowed to edit.

That is the policy. The rest of this article is a path ACL, a two-phase merge, a flake-freeze schema that captures seed and clock instead of deleting the test, and a small Python gate you can run before review. It is a proposed workflow, not a report of a production incident.

Why path ACLs beat commit messages

Commit messages lie under automation. An agent can write fix tests for new behavior while deleting a boundary case, rewriting a fixture, or silencing a property. Reviewers then argue about intent. The filesystem does not.

A test permission model treats paths as capabilities. Production code may change. Characterization tests may be added, later, against already-merged code. Property modules and existing fixture bytes may not change in the agent’s diff at all. Flaky tests may be frozen with replay inputs. They may not vanish.

This is adjacent to assertion scoring and oracle isolation, but it is not the same control. You are not measuring how weak an assert became. You are deciding which files the authoring agent is allowed to write.

Four hunk classes

Every test-side hunk in an agent diff falls into one class. The merge gate reads the class. It does not read the PR title.

Class	Typical path	Allowed with src/ in the same commit?	Gate action
SRC_ONLY	src/**	n/a	Allow phase 1 if properties and fixtures are untouched
PROP_TOUCH	tests/properties/**	Never	Deny. Properties are human-owned
FIXTURE_REWRITE	existing file under tests/fixtures/**	Never	Deny. Bytes are content-addressed
FIXTURE_ADD	new tests/fixtures/<sha256>.*	No. Phase 2 only	Allow if the filename is the digest of the contents
UNIT_EDIT_WITH_SRC	tests/unit/** plus src/**	No	Deny. Split the commit
UNIT_EDIT_ALONE	tests/unit/** only	n/a	Allow phase 2 against merged code
TEST_DELETE	any test file or node	Never	Deny unless the nodeid is in the freeze file
FLAKE_FREEZE	flake_freeze.yml plus no test deletion	No	Allow freeze entries that capture seed and clock

Only SRC_ONLY merges with production code. Everything else waits, or it is rejected. The table is the review. Commentary is optional.

Two-phase merge

Number the phases so CI can fail closed.

1. Phase 1, code vs frozen tests. The agent may patch src/**. The runner executes tests/properties/** and the existing fixture set. git diff --exit-code -- tests/properties tests/fixtures must be clean. Unit tests may run, but the agent may not modify them yet.
2. Phase 2, tests vs merged code. After phase 1 lands, a separate change may add characterization tests or hash-named fixtures. That change must not edit src/**. If production code needs another edit, it goes back to phase 1.
3. Refuse mixed commits. If src/ and tests/unit/ both appear in one commit, classify it as UNIT_EDIT_WITH_SRC and reject. Do not accept a squash that hides the mix.
4. Keep properties readable, not writable. You may put invariant names in the agent’s context. CI still treats tests/properties as append-only by humans.

Phase 1 answers a single question: does the new code preserve independently specified behavior. Phase 2 answers a different question: do we want more examples. Mixing them produces a tautology. The new tests pass because they were written for the new code.

Freeze flakes by seed and clock, not by deletion

Deleting a flaky test raises the pass rate and lowers the information rate. A freeze list is cheaper than that loss if the test has a determinism seam.

Capture the inputs that made the failure non-reproducible. Seed. Clock. Timezone. Locale. Working directory. An environment allowlist. Then keep the test file. Mark the nodeid frozen. Replay on a runner that injects those values.

Proposed freeze schema (YAML, versioned):

version: 1
entries:
  - id: parser_emoji_tz
    path: tests/unit/test_parser.py
    nodeid: tests/unit/test_parser.py::test_handles_emoji
    reason: timezone-dependent formatting
    frozen_at: "2026-09-09"
    seed: 174221
    clock: "2026-01-15T00:00:00Z"
    cwd: "."
    env_allowlist: ["TZ", "LANG"]
    owner: platform
    replay_command: >-
      TZ=UTC LANG=C.UTF-8 pytest -q
      tests/unit/test_parser.py::test_handles_emoji
      --seed=174221

Rules for the freeze file:

1. A TEST_DELETE hunk is allowed only when the deleted nodeid is present in entries and replay_command is non-empty. Prefer not deleting. Skip in default CI, replay on a scheduled job.
2. Missing seed or clock is a schema error if reason mentions time, random, or order.
3. env_allowlist is closed. Unlisted variables must not leak into replay.
4. owner is a human team, not the agent. Frozen tests without an owner are invalid.
5. Do not encode an expiry date as a substitute for a seed. Expiry without replay inputs is just delayed deletion.

If the test has no seam for seed or clock, it does not belong in this freeze file. It belongs in a human review of whether the behavior is specified at all.

A proposed gate you can run on a worktree

The script below is a proposed classifier. It is not a benchmark. It reads git diff --name-status against a base ref, loads a freeze file, and prints a machine-readable decision. Run it on a clean worktree that already contains the agent patch. It requires Python 3.11+ and git on PATH.

#!/usr/bin/env python3
"""Classify test hunks for an agent patch. Proposed gate, not a verdict from prod."""
from __future__ import annotations

import argparse
import hashlib
import subprocess
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    yaml = None  # type: ignore

SRC_PREFIXES = ("src/", "lib/")
PROP_PREFIX = "tests/properties/"
FIXTURE_PREFIX = "tests/fixtures/"
UNIT_PREFIX = "tests/unit/"
FREEZE_PATH = "flake_freeze.yml"


def git_name_status(base: str) -> list[tuple[str, str]]:
    raw = subprocess.check_output(
        ["git", "diff", "--name-status", base, "--"],
        text=True,
    )
    rows: list[tuple[str, str]] = []
    for line in raw.splitlines():
        if not line.strip():
            continue
        parts = line.split("\t")
        status, path = parts[0], parts[-1]
        rows.append((status[0], path))
    return rows


def load_frozen_nodeids(root: Path) -> set[str]:
    path = root / FREEZE_PATH
    if not path.exists():
        return set()
    if yaml is None:
        raise SystemExit("PyYAML required to validate freeze entries")
    data = yaml.safe_load(path.read_text()) or {}
    if data.get("version") != 1:
        raise SystemExit("flake_freeze.yml version must be 1")
    nodeids = set()
    for entry in data.get("entries", []):
        for key in ("nodeid", "seed", "clock", "owner", "replay_command"):
            if not entry.get(key):
                raise SystemExit(f"freeze entry {entry.get('id')!r} missing {key}")
        nodeids.add(entry["nodeid"])
    return nodeids


def fixture_name_matches_bytes(root: Path, rel: str) -> bool:
    p = root / rel
    if not p.is_file():
        return False
    digest = hashlib.sha256(p.read_bytes()).hexdigest()
    return digest in p.name


def classify(base: str, root: Path) -> tuple[str, list[str]]:
    rows = git_name_status(base)
    frozen = load_frozen_nodeids(root)
    src = any(p.startswith(SRC_PREFIXES) for _, p in rows)
    notes: list[str] = []
    classes: set[str] = set()

    for status, path in rows:
        if path.startswith(PROP_PREFIX):
            classes.add("PROP_TOUCH")
            notes.append(f"property path touched: {path}")
        elif path.startswith(FIXTURE_PREFIX) and status in {"M", "D"}:
            classes.add("FIXTURE_REWRITE")
            notes.append(f"fixture rewrite: {status} {path}")
        elif path.startswith(FIXTURE_PREFIX) and status == "A":
            if fixture_name_matches_bytes(root, path):
                classes.add("FIXTURE_ADD")
            else:
                classes.add("FIXTURE_REWRITE")
                notes.append(f"new fixture name is not sha256 of bytes: {path}")
        elif path.startswith(UNIT_PREFIX) and status == "D":
            classes.add("TEST_DELETE")
            notes.append(f"unit test deleted: {path}")
        elif path.startswith(UNIT_PREFIX):
            classes.add("UNIT_EDIT_WITH_SRC" if src else "UNIT_EDIT_ALONE")
        elif path == FREEZE_PATH:
            classes.add("FLAKE_FREEZE")
        elif path.startswith(SRC_PREFIXES):
            classes.add("SRC_ONLY")

    if "PROP_TOUCH" in classes or "FIXTURE_REWRITE" in classes:
        return "DENY", notes
    if "TEST_DELETE" in classes:
        return "DENY", notes + [f"freeze nodeids currently listed: {len(frozen)}"]
    if "UNIT_EDIT_WITH_SRC" in classes:
        return "DENY", notes + ["split phase 1 (src) from phase 2 (unit tests)"]
    if classes <= {"SRC_ONLY", "FLAKE_FREEZE"} or classes == {"SRC_ONLY"}:
        return "ALLOW_PHASE_1", notes
    if classes <= {"UNIT_EDIT_ALONE", "FIXTURE_ADD", "FLAKE_FREEZE"}:
        return "ALLOW_PHASE_2", notes
    if not classes:
        return "ALLOW_NOOP", notes
    return "DENY", notes + [f"unresolved classes: {sorted(classes)}"]


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--base", default="origin/main")
    parser.add_argument("--root", default=".")
    args = parser.parse_args()
    decision, notes = classify(args.base, Path(args.root))
    print(decision)
    for n in notes:
        print(f"# {n}")
    return 0 if decision.startswith("ALLOW") else 1


if __name__ == "__main__":
    sys.exit(main())

Wire it as a required check:

python classify_test_diff.py --base origin/main --root .
pytest -q tests/properties tests/unit --seed=0

Phase 1 CI should fail if the classifier exits non-zero, even when pytest is green. A green suite on a contaminated test tree is not evidence.

Where an isolated runner fits

The classifier is only as strong as its filesystem. If it runs in the same workspace the agent just mutated, a prompt can rewrite classify_test_diff.py, the freeze file, or the property tree after the model is done “helping.” Run phase 1 on a runner that checks out the patch read-only and keeps tests/properties and tests/fixtures from the base ref.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you do not already operate an isolated runner, MonkeyCode’s free model access and free server option can host that phase-1 pass. The merge classes above do not depend on that host. A free model may propose a hunk class in review comments. It must not be the oracle that returns ALLOW_PHASE_1.

Keep the model on the comment path. Keep the gate on git diff and pytest.

What this does not catch

Path ACLs do not detect a property that was never written. They do not detect a fixture that is validly hash-named and still tautological. They do not detect a unit test added in phase 2 that only exercises the happy path.

They also do not replace mutation testing, contract tests against an external API, or a human reading of security-sensitive diffs. A classifier that only sees file names will allow a one-line logic bug that existing properties do not mention.

Clock injection fails when the code under test reads time through a channel you did not wrap: HTTP Date headers, file mtimes, database now(), or a native extension. Seed injection fails when randomness comes from the network or from unordered hash iteration that you never pinned. In those cases, freeze-by-seed is the wrong tool. Specify the behavior or quarantine the subsystem.

Who should not use this

Do not install this policy on a repo with no independent tests. You will only freeze an empty tree and rubber-stamp SRC_ONLY diffs.

Do not use it as a reason to skip human review on authentication, payments, or privacy code. A path ACL is not an authorization model for product risk.

Do not let the agent own flake_freeze.yml. An agent that can freeze a test can hide a race. Freeze entries need a human owner field that CI verifies against a CODEOWNERS-style list.

Do not apply two-phase merge to generated snapshots that are the product. If the repository’s artifact is the snapshot, you need a different oracle, not a ban on fixture writes.

Closing rule

Agent patches are hypotheses about production code. Tests are the control group. If the same author writes both in one commit, you no longer have a control group. Classify the hunk. Split the commit. Freeze flakes with seed and clock. Leave properties to humans.