Subtract the Freeze Surface Before You Score an Agent Patch

A green run that skipped frozen tests is not a score. It is a missing observer. If an agent patch only edits lines that frozen tests uniquely cover, the result is unscored, not pass.

Skip lists convert flake noise into false confidence. An agent that rewrites a retry loop, a clock helper, or a cache key can live entirely under tests you already froze. The suite stays green. The behavior is unobserved.

This article treats a flaky freeze as a coverage hole. Property checks stay outside the writable tree. Fixture hashes stay fail-closed. Frozen tests do not disappear. Their unique line set is subtracted from the scored surface before any patch verdict is emitted.

The observation problem

CI typically encodes flakes as skips. That is convenient for humans. It is the wrong primitive for agent scoring.

A skip removes an assertion. It also removes the only coverage some lines ever receive. The scorer then attributes “all relevant tests passed” to a diff that no remaining test executed. That is a classification error, not a flake.

Three surfaces have to stay distinct:

1. Harness properties. Immutable invariants. The patch cannot rewrite them.
2. Fixture-locked tree tests. Runnable, but fixture bytes are hashed and fail closed if the patch mutates them.
3. Freeze surface. Tests listed in a freeze manifest. Unique coverage from those tests is subtracted. Overlap with the diff yields unscored.

Do not merge those lists. A property that can be edited by the agent is not a harness. A fixture the agent can rewrite is not locked. A freeze that only skips is not a coverage hole; it is a lie.

Artifact: a coverage-subtraction gate

The following workflow is a proposal you can run locally. It does not require a particular vendor. It does require three inputs at a parent SHA: a freeze manifest, a coverage map keyed by test node id, and the agent unified diff.

freeze.json (example)

{
  "schema": 1,
  "frozen": [
    {
      "nodeid": "tests/test_retry.py::test_backoff_under_jitter",
      "reason": "clock-jitter race",
      "bound_sha": "a1b2c3d4"
    }
  ]
}

Bind the freeze to a SHA. A freeze that floats across history will subtract the wrong lines after a rename. The scorer should refuse an unbound freeze. It should also refuse a freeze file that the incoming patch itself modifies.

coverage_by_test.json (example, produced on the parent tree)

{
  "tests/test_retry.py::test_backoff_under_jitter": {
    "src/retry.py": [40, 41, 42, 43, 88]
  },
  "tests/test_retry.py::test_backoff_deterministic": {
    "src/retry.py": [40, 41, 42, 60, 61]
  },
  "harness/test_invariants.py::test_retry_never_negative": {
    "src/retry.py": [12, 13, 14]
  }
}

Collect this map before the patch is applied. Coverage collected on the patched tree answers a different question: “what ran after the edit?” The freeze hole is defined on the parent. Mixing the two maps hides deletions.

Proposed scorer (subtract_freeze.py)

#!/usr/bin/env python3
"""Subtract unique freeze coverage from an agent diff. Proposal / example."""
from __future__ import annotations

import json
import re
import sys
from collections import defaultdict
from pathlib import Path
from typing import Dict, Iterable, Set, Tuple

LineSet = Set[Tuple[str, int]]
HUNK_RE = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")


def load_json(path: Path):
    return json.loads(path.read_text(encoding="utf-8"))


def as_lines(file_map: Dict[str, Iterable[int]]) -> LineSet:
    return {(path, int(n)) for path, nums in file_map.items() for n in nums}


def union_coverage(cov: dict, nodeids: Iterable[str]) -> LineSet:
    acc: LineSet = set()
    for nodeid in nodeids:
        acc |= as_lines(cov.get(nodeid, {}))
    return acc


def parse_diff_changed_lines(diff_text: str) -> LineSet:
    """Map added/replaced lines in a unified diff. File deletes count as (path, 0)."""
    changed: LineSet = set()
    current = None
    new_line = 0
    for raw in diff_text.splitlines():
        if raw.startswith("+++ "):
            rhs = raw[4:].strip()
            current = None if rhs == "/dev/null" else rhs[2:] if rhs.startswith("b/") else rhs
            continue
        m = HUNK_RE.match(raw)
        if m:
            new_line = int(m.group(1))
            continue
        if current is None:
            continue
        if raw.startswith("+") and not raw.startswith("+++"):
            changed.add((current, new_line))
            new_line += 1
        elif raw.startswith("-") and not raw.startswith("---"):
            changed.add((current, 0 if new_line == 0 else new_line))
        else:
            new_line += 1
    return changed


def fixture_hash_violations(patch_files: Set[str], lock: dict) -> list[str]:
    locked = set(lock.get("paths", []))
    return sorted(path for path in patch_files if path in locked)


def main(argv: list[str]) -> int:
    freeze = load_json(Path(argv[1]))
    cov = load_json(Path(argv[2]))
    lock = load_json(Path(argv[3]))
    diff_text = Path(argv[4]).read_text(encoding="utf-8")
    harness_prefix = argv[5] if len(argv) > 5 else "harness/"

    frozen_ids = [row["nodeid"] for row in freeze.get("frozen", [])]
    if any("bound_sha" not in row for row in freeze.get("frozen", [])):
        print("UNSCORED\tunbound freeze entries")
        return 2

    all_ids = set(cov)
    live_ids = all_ids - set(frozen_ids)
    frozen_lines = union_coverage(cov, frozen_ids)
    live_lines = union_coverage(cov, live_ids)
    unique_freeze = frozen_lines - live_lines

    changed = parse_diff_changed_lines(diff_text)
    patch_files = {path for path, _ in changed}

    if any(path == "freeze.json" or path.endswith("/freeze.json") for path in patch_files):
        print("FAIL\tpatch edits freeze.json")
        return 1
    if any(path.startswith(harness_prefix) for path in patch_files):
        print("FAIL\tpatch edits harness properties")
        return 1

    locked_hits = fixture_hash_violations(patch_files, lock)
    if locked_hits:
        print("FAIL\tfixture lock: " + ",".join(locked_hits))
        return 1

    # Lines the diff cannot be observed on, even if later tests are green.
    overlap = {(p, n) for (p, n) in changed if n != 0 and (p, n) in unique_freeze}
    changed_code = {(p, n) for (p, n) in changed if n != 0}
    if not changed_code:
        print("UNSCORED\tdiff produced no added lines")
        return 2
    if overlap and overlap == changed_code:
        print("UNSCORED\tdiff lives entirely under unique freeze coverage")
        return 2
    if overlap:
        ratio = len(overlap) / len(changed_code)
        print(f"UNSCORED\tfreeze overlap ratio={ratio:.2f} lines={len(overlap)}")
        return 2

    print("CONTINUE\tlive surface covers the diff")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))

Exit 0 means the live surface still observes the diff. It does not mean the patch is correct. It only means you are allowed to spend a test run on it. Exit 1 is a hard fail. Exit 2 is unscored: do not promote, do not retry-as-green, do not fold into a pass rate.

Fixture lock file (fixture_lock.json)

{
  "paths": [
    "tests/fixtures/retry_corpus.json",
    "tests/fixtures/clock_vectors.bin"
  ]
}

Hash those paths in a separate step if you need byte equality. Path membership is the cheap gate. Bytes are the real lock. A patch that “fixes flakiness” by widening a fixture is a fixture rewrite, not a product fix.

Numbered procedure

Run the gate on the parent SHA, then apply the patch only if the subtraction result is CONTINUE.

1. Check out the parent commit. Record HEAD as bound_sha for every freeze entry you still accept.
2. Collect per-test coverage on the parent tree. Keep harness tests and tree tests in the same map. Keep node ids stable.
3. Confirm freeze.json is not in the agent’s writable path, or confirm the scorer rejects edits to it (the script above does).
4. Run subtract_freeze.py freeze.json coverage_by_test.json fixture_lock.json agent.patch harness/.
5. On exit 2, stop. Publish unscored and the overlap set. Do not run the expensive suite.
6. On exit 1, stop. That is a policy fail: harness edit or fixture lock break.
7. On exit 0, run harness properties first. Then run non-frozen tree tests. Do not reintroduce frozen node ids in this phase.
8. Emit a three-state verdict: pass_clean, fail, or unscored. Never coerce unscored into pass because “the remaining tests were green.”

Harness properties belong in a directory the patch cannot touch. A minimal example, labeled as a stand-in rather than a production invariant:

# harness/test_invariants.py  — not writable by the agent patch
from retry import delay_for_attempt

def test_retry_never_negative():
    for attempt in range(0, 32):
        assert delay_for_attempt(attempt) >= 0

def test_retry_monotone_non_decreasing():
    prev = delay_for_attempt(0)
    for attempt in range(1, 32):
        current = delay_for_attempt(attempt)
        assert current >= prev
        prev = current

Those two tests are cheap. They do not replace coverage subtraction. They catch the class of patches that “fix” retries by clamping everything to zero or by deleting the backoff. Run them even when the freeze surface is large. Especially then.

Decision table

Condition	Verdict	Why
Patch edits harness/	fail	Observer was rewritten
Patch edits freeze.json	fail	Freeze is not a self-service skip
Patch edits a locked fixture path	fail	Oracle moved
Diff lines ⊆ unique freeze coverage	unscored	No live observer
Diff lines ∩ unique freeze coverage ≠ ∅	unscored	Partial observer; do not guess
Diff has no added lines	unscored	Deletion-only needs a separate gate
Live coverage includes every added line, harness pass, fixtures intact	continue to tree tests	Observation still exists

Partial overlap is unscored on purpose. Splitting a patch into “the observed hunk” and “the frozen hunk” is a human task. An automated scorer that accepts 70% observed diffs will systematically bless the unobserved 30%.

Isolated scoring, not a shared workspace

The subtraction gate is only as honest as the filesystem it reads. If the same process that proposed the patch can rewrite coverage_by_test.json, freeze.json, or harness/, the verdict is circular.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A practical split is: a free model proposes the diff; a free server that does not share the proposer’s workspace runs subtract_freeze.py and the harness. MonkeyCode’s free model access and free server option fit that split. They do not change the rule. The scorer still has to refuse freeze edits and still has to treat unique frozen coverage as a hole.

Keep the coverage map and the freeze manifest on the scoring host. Mount the agent worktree as an input, not as the authority for those files. If you cannot enforce that mount, the rest of this workflow is theater.

What this does not claim

Coverage subtraction does not fix flakes. It prices them. A large freeze surface shrinks the set of patches you are allowed to score. That is the point. Teams that freeze half the suite will see unscored often. That is a freeze budget problem, not a scorer bug.

The parser above is a proposal. It treats added lines as the unit of observation. Comment-only edits, generated files, and binary blobs will mis-fire. If your agent regularly patches .proto generated sources, build a file-class filter before you compute overlap. If coverage is non-deterministic across runs, stabilize the map first; subtracting jitter produces jittery verdicts.

This workflow also does not detect tautological tests, seed drift, or oracle-host leakage. Those are separate gates. Stacking them without keeping freeze coverage subtracted still leaves a hole under the skip list.

Who should not use this

Do not use coverage subtraction as a substitute for deleting a freeze. If a test has been frozen longer than you can name a bound SHA for, the test is not flaky. It is unowned.

Do not use it on repositories without per-test coverage. File-level coverage is too coarse: a frozen test and a live test that share a file will hide the hole.

Do not use it when the agent is allowed to edit the harness, the freeze manifest, or the fixture lock. Policy has to precede scoring. A model with those write permissions can always manufacture a pass.

Do not use it as a merge criterion by itself. CONTINUE only means the live surface still sees the diff. Correctness still requires the property run and the remaining tree tests.

The useful output is the three-state verdict. Count unscored next to fail. If unscored dominates, your freeze surface is the product defect. Shrink it, or stop scoring agent patches on that tree until live tests observe the code you let models touch.

If you already isolate scoring from the proposer, run the subtraction gate on that host before any suite that costs wall time. A free server is enough hardware for this check. The constraint is the mount, not the slogan.