A Frozen Test Is Not Coverage: A Merge Gate for Agent Patches

A green CI job after an agent patch is not a result. It is a result only when three facts are true at the same time: every production hunk in the diff is covered by at least one live property check, every fixture those checks read still matches a pinned digest, and none of those covering tests sit on the flake freeze list.

Freeze files exist for a narrow reason. Non-deterministic tests should not block a pipeline. They also punch holes in the only signal you have when a model emits diffs faster than a reviewer can read them. This article specifies a merge rule that treats a frozen covering test as missing coverage, not as a pass.

The workflow is mechanical. You keep three files in the repo. A small gate reads the agent diff, those files, and git. Then it fails closed.

Why freeze lists go stale under agent patches

A flake freeze is a temporary exemption. In a human-paced repo it is usually short-lived. Agent patches change the rate. Cheap diffs raise the number of commits that touch a module without anyone noticing that the only precise test for that module was quarantined last week.

Two failure modes show up immediately. They are boring, and they are common.

First, the agent rewrites a fixture so the new, incorrect behavior matches the file on disk. The suite stays green. A digest of the fixture bytes would have caught it if you hashed the file before the run.

Second, the agent changes a parser, a serializer, or a retry loop whose only strict test is frozen. CI reports success because the frozen test never ran. That is not coverage. It is silence.

A third mode is quieter. The agent edits the freeze file, the hunk map, or the property test in the same commit as production code. If that is allowed, the other two files are theater.

The three artifacts

Keep these paths stable. The gate should not search the tree for whatever looks like a lockfile.

1. test/contract/fixtures.lock.json — SHA-256 digests of every fixture the property suite is allowed to read.
2. test/contract/flake-freeze.json — quarantined node ids, each with a reason and an owner. Frozen tests contribute zero coverage.
3. test/contract/hunk-map.json — production path prefixes mapped to the property tests that must remain live if those paths change.

Properties live in tests/properties/. They are the only tests that count toward the merge rule. Example-based tests can stay in tests/examples/ for local debugging. They do not lift the gate.

Step 1 — Pin fixture bytes, not fixture names

Name stability is not content stability. Hash the bytes.

# tools/write_fixture_lock.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

FIXTURE_ROOT = Path("tests/fixtures")
LOCK_PATH = Path("test/contract/fixtures.lock.json")


def digest_tree(root: Path) -> dict[str, str]:
    out: dict[str, str] = {}
    for path in sorted(root.rglob("*")):
        if path.is_file():
            rel = path.relative_to(root).as_posix()
            out[rel] = hashlib.sha256(path.read_bytes()).hexdigest()
    return out


if __name__ == "__main__":
    lock = {
        "fixtures_root": FIXTURE_ROOT.as_posix(),
        "files": digest_tree(FIXTURE_ROOT),
    }
    LOCK_PATH.parent.mkdir(parents=True, exist_ok=True)
    payload = json.dumps(lock, indent=2) + "\n"
    LOCK_PATH.write_text(payload, encoding="utf-8")
    print(f"wrote {len(lock['files'])} digests to {LOCK_PATH}")

Run this only from a human-owned change that intentionally updates golden input. Label agent-authored fixture edits as untrusted until a reviewer regenerates the lock in a follow-up commit. If the gate sees a digest mismatch, it fails before pytest starts. There is nothing to retry.

Command to regenerate after a reviewed fixture change:

python tools/write_fixture_lock.py
git add test/contract/fixtures.lock.json tests/fixtures

Step 2 — Write properties that name the invariant

A property check is a relation that must hold for a class of inputs, not a single recorded example. The examples below are executable pytest. They are not production measurements.

# tests/properties/test_parse_roundtrip.py
import json
from pathlib import Path

import pytest

from mypkg.parse import parse_record, serialize_record

FIXTURES = Path("tests/fixtures/records")


def _load(name: str) -> bytes:
    return (FIXTURES / name).read_bytes()


@pytest.mark.property
@pytest.mark.parametrize("name", ["ok-minimal.json", "ok-unicode.json", "ok-nested.json"])
def test_parse_serialize_roundtrip_preserves_object(name: str) -> None:
    raw = _load(name)
    obj = parse_record(raw)
    again = parse_record(serialize_record(obj))
    assert again == obj


@pytest.mark.property
@pytest.mark.parametrize("name", ["ok-minimal.json", "ok-unicode.json"])
def test_unknown_fields_survive_roundtrip(name: str) -> None:
    raw = json.loads(_load(name))
    raw["x-experimental"] = {"k": 1}
    obj = parse_record(json.dumps(raw).encode("utf-8"))
    dumped = json.loads(serialize_record(obj))
    assert dumped["x-experimental"] == {"k": 1}

Two properties are enough to start. Round-trip equality catches silent field drops. Explicit extra-field survival catches tidy schema stripping, which agent patches introduce often because tidy objects are common in training data.

If you later add a generative library, keep the same markers. The gate keys off pytest.mark.property and the hunk map, not off the generator. Weak properties still pass weak patches. The gate does not fix that. It only stops you from counting a skipped test as a check.

Register the marker once so unknown-mark warnings do not hide real failures:

# pytest.ini
[pytest]
markers =
    property: live invariant used by the agent merge gate

Step 3 — Quarantine flakes with a coverage penalty

The freeze file is not a skip list with comments. Each entry must declare that it no longer counts as coverage for any production path.

{
  "version": 1,
  "entries": [
    {
      "nodeid": "tests/examples/test_retry.py::test_backoff_jitter",
      "reason": "clock jitter; not a property",
      "owner": "platform",
      "covers": [],
      "frozen": true
    },
    {
      "nodeid": "tests/properties/test_parse_roundtrip.py::test_parse_serialize_roundtrip_preserves_object[ok-nested.json]",
      "reason": "nested fixture currently non-deterministic under CI locale",
      "owner": "parse-owners",
      "covers": ["src/mypkg/parse.py"],
      "frozen": true
    }
  ]
}

Read the second entry carefully. It freezes one parametrized node, and it names the production file that node used to cover. After the freeze, src/mypkg/parse.py has a hole unless another live property still maps to it.

That remaining property may be narrower. test_unknown_fields_survive_roundtrip does not load ok-nested.json. Nested regressions can slip while CI stays green. Shrinking the live set shrinks the invariant. Record that shrinkage in review comments, not in a skip.

Do not put expiry dates in this file unless a job also fails when a date passes. A date nobody enforces is documentation. This gate does not depend on calendar behavior.

Wire pytest to skip frozen node ids without deleting them:

# tests/conftest.py
import json
from pathlib import Path

import pytest

FREEZE = Path("test/contract/flake-freeze.json")


def pytest_collection_modifyitems(items: list[pytest.Item]) -> None:
    data = json.loads(FREEZE.read_text(encoding="utf-8"))
    frozen = {e["nodeid"] for e in data["entries"] if e.get("frozen")}
    skip = pytest.mark.skip(reason="frozen flake; zero coverage credit")
    for item in items:
        if item.nodeid in frozen:
            item.add_marker(skip)

Skipping is not passing. The merge gate below is what converts the skip into a coverage hole.

Step 4 — Map hunks before the agent runs

Convention (src/foo.py to tests/properties/test_foo.py) is a start. It is not sufficient when one parse.py is asserted by three property modules. Declare the map.

{
  "version": 1,
  "rules": [
    {
      "path_prefix": "src/mypkg/parse.py",
      "live_properties": [
        "tests/properties/test_parse_roundtrip.py::test_parse_serialize_roundtrip_preserves_object",
        "tests/properties/test_parse_roundtrip.py::test_unknown_fields_survive_roundtrip"
      ]
    },
    {
      "path_prefix": "src/mypkg/serialize.py",
      "live_properties": [
        "tests/properties/test_parse_roundtrip.py::test_parse_serialize_roundtrip_preserves_object"
      ]
    }
  ]
}

Unmapped production paths are a hard fail when they appear in the diff. Missing map entries are cheaper to add than false greens. Put the map in review for any new package path before the first agent patch on that path.

Step 5 — Fail closed on the diff

The gate below is a proposal you can run as python tools/agent_merge_gate.py. It uses only the standard library.

# tools/agent_merge_gate.py
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
LOCK = ROOT / "test/contract/fixtures.lock.json"
FREEZE = ROOT / "test/contract/flake-freeze.json"
HUNK_MAP = ROOT / "test/contract/hunk-map.json"
FIXTURE_ROOT = ROOT / "tests/fixtures"
FORBIDDEN_CONTRACT = {
    "test/contract/fixtures.lock.json",
    "test/contract/flake-freeze.json",
    "test/contract/hunk-map.json",
}


def git_changed_files() -> list[str]:
    cmd = ["git", "diff", "--name-only", "HEAD~1", "HEAD"]
    out = subprocess.check_output(cmd, cwd=ROOT, text=True)
    return [line.strip() for line in out.splitlines() if line.strip()]


def verify_fixtures() -> list[str]:
    lock = json.loads(LOCK.read_text(encoding="utf-8"))
    errors: list[str] = []
    files = lock["files"]
    for rel, expected in files.items():
        path = FIXTURE_ROOT / rel
        if not path.is_file():
            errors.append(f"missing fixture: {rel}")
            continue
        actual = hashlib.sha256(path.read_bytes()).hexdigest()
        if actual != expected:
            errors.append(f"digest mismatch: {rel}")
    for path in FIXTURE_ROOT.rglob("*"):
        if path.is_file():
            rel = path.relative_to(FIXTURE_ROOT).as_posix()
            if rel not in files:
                errors.append(f"unlocked fixture: {rel}")
    return errors


def frozen_nodeids() -> set[str]:
    data = json.loads(FREEZE.read_text(encoding="utf-8"))
    return {e["nodeid"] for e in data["entries"] if e.get("frozen")}


def mapped_properties(changed: list[str]) -> dict[str, list[str]]:
    rules = json.loads(HUNK_MAP.read_text(encoding="utf-8"))["rules"]
    required: dict[str, list[str]] = {}
    for path in changed:
        if not path.startswith("src/"):
            continue
        matches = [r for r in rules if path.startswith(r["path_prefix"])]
        if not matches:
            required[path] = []
            continue
        props: list[str] = []
        for rule in matches:
            props.extend(rule["live_properties"])
        required[path] = sorted(set(props))
    return required


def is_blocked(prop: str, frozen: set[str]) -> bool:
    for nodeid in frozen:
        if nodeid == prop or nodeid.startswith(prop + "["):
            return True
    return False


def main() -> int:
    changed = git_changed_files()
    errors = verify_fixtures()
    src_hits = [p for p in changed if p.startswith("src/")]
    contract_hits = sorted(FORBIDDEN_CONTRACT.intersection(changed))

    if contract_hits and src_hits:
        errors.append(
            "contract files and production sources changed together: "
            + ", ".join(contract_hits)
        )

    frozen = frozen_nodeids()
    required = mapped_properties(changed)
    for path, props in required.items():
        if not props:
            errors.append(f"no hunk map for {path}")
            continue
        live = [p for p in props if not is_blocked(p, frozen)]
        if not live:
            errors.append(
                f"coverage hole: {path} has no live property"
            )

    extra = [
        p for p in changed
        if p.startswith("tests/properties/") or p.startswith("tests/fixtures/")
    ]
    if extra and src_hits:
        errors.append(
            "properties or fixtures changed with production code: "
            + ", ".join(extra)
        )

    if errors:
        print("MERGE REFUSED")
        for item in errors:
            print(f"- {item}")
        return 1
    print("MERGE GATE PASS")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The script is conservative on purpose. If a freeze entry prefixes a mapped property, that property does not count. If production sources and contract files move together, the commit is refused. If fixtures or property tests move with src/, the commit is refused.

An agent can still propose those edits. They land in a second, human-owned commit after the lock is rewritten. is_blocked fails closed when any parametrized node under a mapped property is frozen. That is stricter than a large parametrize list needs. Relax it only after the hunk map lists each live node id explicitly.

CI order:

python tools/agent_merge_gate.py
pytest -m property -q

Do not invert it. Pytest on a mutated fixture can still pass. The digest check has to go first.

Decision table

Diff contents	Fixture digests	Live property for each src/ hunk	Gate
src/ only	match	yes	pass
src/ only	mismatch	yes	refuse
src/ only	match	covering test frozen	refuse
src/ only	match	path missing from hunk map	refuse
src/ + tests/fixtures/	any	any	refuse
src/ + contract files	any	any	refuse
contract files only	lock rewritten without src/	n/a	human review; no auto-merge
freeze file only	match	n/a	allow, then re-check the map on the default branch

The last row is easy to get wrong. A freeze-only commit can create holes for the next agent patch. Re-run the gate against a synthetic "every src/ file changed" mode if you want to catch that immediately. Add that mode later. Do not skip the rows above while you wait for it.

Where a free model and a free server fit

The gate does not need a model. The model needs the gate.

If you generate candidate patches with free model access and execute CI on a free server option, keep the roles split. The model proposes a diff. The server runs the lock in verify mode, then agent_merge_gate.py, then pytest -m property. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option; those are relevant here only as a place to emit diffs and run this gate, not as a replacement for the three contract files.

Do not send the freeze file to the model as context to update. Models will often unfreeze tests or rewrite digests so the job goes green. Pass the gate output, not the contract, if you ask a model to propose a follow-up.

Limitations

This is a merge rule, not a proof.

It will not catch a wrong invariant. If both properties encode the same mistake, the agent can implement that mistake and pass. It will not catch integration failures that live only in UI or network tests. Those tests are often the ones you freeze, which is why they must not count as coverage.

The hunk map is declared, not measured. A test can import parse.py and never assert on it. Line coverage would tighten the map. It also adds a toolchain dependency this article avoids. If your map and your imports drift, you get a false sense of mapping, not a false green from a freeze.

The git comparison uses HEAD~1. Rebase-heavy workflows and squash merges need git merge-base against the target branch. That change is local. The contract files stay the same. First commits on an empty repo will fail the diff command; seed the three files in a human commit before any agent run.

Do not use this approach if any of the following is true:

• The repo has no property or unit layer, only flaky end-to-end jobs.
• Agents are required to update golden fixtures in the same commit as code, such as protobuf codegen or UI snapshots. Split those pipelines. Do not weaken this gate to make them fit.
• You need statistical performance claims. Nothing here measures latency or model quality.
• A frozen test is your only regression net for a safety-critical path. Unfreeze it or replace it before you accept machine-authored diffs.

Cheap patches make tests the scarce resource. Freeze lists make some of those tests silent. Hash the fixtures, keep properties live, and treat a frozen covering test as a hole. Then read the patch.

If you already generate diffs with a free model, run the gate on the free server before you open the diff. The three files are the review.