Subtract Frozen Tests From Coverage Before You Merge an Agent Patch

A green pipeline is not a proof. If the suite still contains frozen flaky tests, coverage numbers include lines that no reliable test currently exercises. Subtract those tests first. Then merge an agent patch only if proven coverage, fixture digests, and the property suite all stay within policy.

Frozen tests inflate the merge bar

Agent patches optimize for the signals a gate publishes. Skip entries, xfail marks, and coverage badges are signals. When a flaky test is frozen, CI usually stops failing on it. It does not automatically remove that test from the proof set.

A freeze is a lease on unreliability. It is not a pass. Treat the leased nodeid as missing evidence.

The failure mode is quiet. The agent leaves the freeze file alone, or it appends a nodeid. The JUnit report stays green. The coverage badge ticks up. Reviewers see a smaller diff than the residual risk.

This is a structural gap, not a one-week anecdote. Any merge queue that treats "not failing" as "proven" will accept patches that only look safe because flakes are parked.

Three numbers, not one badge

Replace a single coverage number with three computed values.

1. Proven coverage. Line and branch coverage collected only from tests that are not frozen and whose leases have not expired.
2. Fixture digest delta. The count of content-addressed fixture files whose SHA-256 changed in the patch.
3. Property hold. Pass or fail of a property suite whose seed corpus is not writable by the agent.

A patch may raise raw coverage and still fail this bar. That outcome is intended. Raw coverage still counts frozen tests. Proven coverage does not.

Keep the three numbers in CI artifacts. Do not recompute them by hand in a review comment.

Maintain a freeze ledger with a TTL

Store freezes in the repo, next to the tests. Do not keep them in a chat transcript or in an agent's scratch directory.

The ledger below is a policy example. It is not telemetry from a production fleet.

# freeze.toml
version = 1
max_active_leases = 8
reject_if_expired = true
reject_if_agent_adds_lease = true

[[leases]]
nodeid = "tests/test_billing.py::test_invoice_total_currency_round"
reason = "intermittent tz boundary on CI runners"
owner = "platform-billing"
expires = "2026-09-26"

[[leases]]
nodeid = "tests/test_cache.py::test_stampede_window"
reason = "timing-dependent under load"
owner = "platform-cache"
expires = "2026-09-19"

Four rules keep the ledger honest.

1. Every lease needs an owner, a reason, and an ISO-8601 date.
2. Expired leases fail the gate. They do not become silent skips.
3. Agent patches cannot add leases. Growing the freeze set requires a human-authored trailer on the commit.
4. max_active_leases is a budget. Crossing it is a merge failure, not a warning.

Pick the budget from suite size, not from hope. A 200-test library that already carries eight freezes is telling you the suite is the problem. Do not raise the cap to land a model-generated diff.

Accounting workflow

The workflow is a proposed local gate. Paths and commands are examples. Label the output as computed policy, not as a measured incident.

1. Collect the test universe

set -euo pipefail
pytest --collect-only -q | tee collected.txt

Parse nodeids from that list. The universe is the denominator for every later ratio.

2. Load leases and expire them against the clock

Read freeze.toml on the merge worker. Compare expires to the worker's UTC date. If reject_if_expired is true, leftover expired rows are a hard fail.

Do not let the agent pass a clock of its own. Use the CI system's time.

3. Subtract frozen nodeids from the coverage run

python freeze_account.py emit-deselect \
  --freeze freeze.toml \
  --collected collected.txt \
  > deselect.args

pytest @deselect.args \
  --cov=src \
  --cov-report=json:proven_coverage.json \
  --cov-fail-under=0

emit-deselect writes pytest --deselect flags. Coverage then records only proven tests. Keep a second job that runs the full suite, including frozen nodeids, if you still want flake telemetry. Do not use that job as the merge bar.

4. Lock fixtures by digest

find tests/fixtures -type f -print0 \
  | sort -z \
  | xargs -0 sha256sum \
  > fixtures.lock

git diff --exit-code fixtures.lock

If the lockfile changes, require a human trailer:

git log -1 --format=%B | grep -qx 'Fixture-Lock: human'

An agent-only commit that rewrites golden files fails. Humans may update fixtures. The gate's job is to make that update explicit.

5. Run properties from a read-only seed path

chmod a-w tests/properties/seeds
pytest tests/properties -q

The agent may add examples under a scratch directory. It may not shrink or rewrite the seed corpus used for the merge bar. A property that only encodes the patch's own output is not a hold. It is a tautology. Keep seeds that predate the patch.

Example property file. This is illustrative, not a harvested production test.

# tests/properties/test_invoice_properties.py
import json
from decimal import Decimal
from pathlib import Path

from src.billing import total_with_tax

SEEDS = Path(__file__).parent / "seeds"


def test_tax_never_negative_on_seeds():
    paths = sorted(SEEDS.glob("*.json"))
    assert paths, "property seed corpus is empty"
    for path in paths:
        payload = json.loads(path.read_text())
        result = total_with_tax(payload)
        assert result >= Decimal("0")
        assert result >= Decimal(str(payload["net"]))

6. Emit a machine-readable decision

Print JSON the merge queue can parse. Reviewers still read the table. The queue refuses the merge when decision is not MERGE.

Artifact: freeze_account.py

Stdlib only on Python 3.11+. The script accounts for tests. It does not execute the suite by itself.

#!/usr/bin/env python3
"""Account for frozen tests before an agent patch merge.

Example helper. Adapt paths. Not a measured benchmark.
"""
from __future__ import annotations

import argparse
import json
import re
import sys
import tomllib
from datetime import date, datetime, timezone
from pathlib import Path

NODE_RE = re.compile(r"^(tests/\S+::\S+)")


def load_freeze(path: Path) -> dict:
    data = tomllib.loads(path.read_text())
    if "leases" not in data:
        raise SystemExit("freeze file missing [[leases]]")
    return data


def parse_collected(path: Path) -> list[str]:
    nodeids = []
    for line in path.read_text().splitlines():
        m = NODE_RE.match(line.strip())
        if m:
            nodeids.append(m.group(1))
    return nodeids


def classify(freeze: dict, today: date) -> tuple[list[dict], list[dict]]:
    active, expired = [], []
    for lease in freeze["leases"]:
        exp = date.fromisoformat(str(lease["expires"]))
        if exp < today:
            expired.append(lease)
        else:
            active.append(lease)
    return active, expired


def cmd_emit_deselect(args: argparse.Namespace) -> int:
    freeze = load_freeze(args.freeze)
    collected = set(parse_collected(args.collected))
    today = datetime.now(timezone.utc).date()
    active, expired = classify(freeze, today)
    if freeze.get("reject_if_expired", True) and expired:
        print("expired leases present:", file=sys.stderr)
        for row in expired:
            print(f"  {row['nodeid']} expired {row['expires']}", file=sys.stderr)
        return 2
    unknown = [row for row in active if row["nodeid"] not in collected]
    if unknown:
        print("leases for missing nodeids:", file=sys.stderr)
        for row in unknown:
            print(f"  {row['nodeid']}", file=sys.stderr)
        return 2
    budget = int(freeze.get("max_active_leases", 0) or 0)
    if len(active) > budget:
        print(f"freeze budget exceeded: {len(active)} > {budget}", file=sys.stderr)
        return 2
    for row in active:
        print(f"--deselect={row['nodeid']}")
    return 0


def cmd_decide(args: argparse.Namespace) -> int:
    freeze = load_freeze(args.freeze)
    today = datetime.now(timezone.utc).date()
    active, expired = classify(freeze, today)
    collected = parse_collected(args.collected)
    coverage = json.loads(args.coverage.read_text())
    totals = coverage.get("totals", {})
    proven = float(totals.get("percent_covered", 0.0))
    reasons = []
    budget = int(freeze.get("max_active_leases", 0) or 0)
    if expired and freeze.get("reject_if_expired", True):
        reasons.append("expired_leases")
    if len(active) > budget:
        reasons.append("freeze_budget")
    if freeze.get("reject_if_agent_adds_lease", True) and args.agent_added_leases:
        reasons.append("agent_added_lease")
    if proven < args.min_proven:
        reasons.append("proven_coverage")
    if args.fixture_delta and not args.human_fixture_lock:
        reasons.append("fixture_digest")
    if args.property_failures:
        reasons.append("property_hold")
    decision = "MERGE" if not reasons else "REJECT"
    report = {
        "today_utc": today.isoformat(),
        "collected_tests": len(collected),
        "active_leases": len(active),
        "expired_leases": len(expired),
        "proven_coverage_percent": proven,
        "min_proven_coverage_percent": args.min_proven,
        "fixture_digest_delta": args.fixture_delta,
        "property_failures": args.property_failures,
        "agent_added_leases": args.agent_added_leases,
        "reasons": reasons,
        "decision": decision,
    }
    json.dump(report, sys.stdout, indent=2)
    print()
    return 0 if decision == "MERGE" else 1


def main() -> int:
    parser = argparse.ArgumentParser(
        description="Frozen-test accounting for agent patches"
    )
    sub = parser.add_subparsers(dest="cmd", required=True)

    p1 = sub.add_parser("emit-deselect")
    p1.add_argument("--freeze", type=Path, required=True)
    p1.add_argument("--collected", type=Path, required=True)
    p1.set_defaults(fn=cmd_emit_deselect)

    p2 = sub.add_parser("decide")
    p2.add_argument("--freeze", type=Path, required=True)
    p2.add_argument("--collected", type=Path, required=True)
    p2.add_argument("--coverage", type=Path, required=True)
    p2.add_argument("--min-proven", type=float, default=70.0)
    p2.add_argument("--fixture-delta", type=int, default=0)
    p2.add_argument("--property-failures", type=int, default=0)
    p2.add_argument("--agent-added-leases", type=int, default=0)
    p2.add_argument("--human-fixture-lock", action="store_true")
    p2.set_defaults(fn=cmd_decide)

    args = parser.parse_args()
    return args.fn(args)


if __name__ == "__main__":
    raise SystemExit(main())

If max_active_leases is omitted, the script treats the budget as zero. That is fail-closed. Set the budget in freeze.toml before the first merge.

Wire the decide step after pytest:

PROPERTY_FAILURES=0
pytest tests/properties -q || PROPERTY_FAILURES=1

python freeze_account.py decide \
  --freeze freeze.toml \
  --collected collected.txt \
  --coverage proven_coverage.json \
  --min-proven 70 \
  --fixture-delta "$(git diff --numstat -- fixtures.lock | wc -l)" \
  --property-failures "$PROPERTY_FAILURES" \
  --agent-added-leases "$(git diff HEAD~1 -- freeze.toml | grep -c '^+\[\[leases\]\]' || true)"

Example output. Synthetic. Not a measured run.

{
  "today_utc": "2026-09-12",
  "collected_tests": 214,
  "active_leases": 2,
  "expired_leases": 0,
  "proven_coverage_percent": 71.6,
  "min_proven_coverage_percent": 70.0,
  "fixture_digest_delta": 0,
  "property_failures": 0,
  "agent_added_leases": 0,
  "reasons": [],
  "decision": "MERGE"
}

If reasons is non-empty, stop the merge queue. Do not negotiate the JSON in a review thread after the fact.

Decision table

Signal	Pass condition	Fail condition
Proven coverage	percent_covered from the deselected run ≥ policy floor	Floor missed after frozen nodeids are removed
Freeze budget	active_leases ≤ max_active_leases	Budget exceeded or expired rows remain
Lease authorship	Agent diff does not add [[leases]] rows	Agent grows the freeze set
Fixture digests	fixtures.lock unchanged, or Fixture-Lock: human present	Golden files rewritten without a human trailer
Property hold	Seeded properties pass; seed directory stays read-only	Failures, empty seeds, or seed rewrites

Read the table top to bottom. The first fail is enough. Do not average a fail against a strong coverage number.

Generating the diff is optional. Accounting is not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Candidate patches can come from a free model endpoint. MonkeyCode's free model access and free server option can produce those diffs without standing up a private inference box. They should not own freeze.toml, fixtures.lock, or the property seeds. Run freeze_account.py on a worker you control. The three-number bar stays on your side of the network.

Limitations

Nodeid subtraction is only as stable as pytest nodeids. Parametrized tests that change brackets will leak out of the lease or will freeze the wrong row. Pin nodeids after collection, not from memory.

Proven coverage still misses untested branches. A high proven percentage on a small run set is a weak bar. Pair it with the property suite.

The ledger can be rubber-stamped. If lease owners approve every extension, max_active_leases becomes decoration. Audit owners on a cadence. Expire first. Explain second.

This workflow does not classify tautological tests. A property that asserts f(x) == f(x) will pass and will not save you. Hold seeds that predate the patch. Add a separate assertion-density check if that is in scope.

It also does not replace review for authorization, crypto, or privacy-sensitive changes. Coverage accounting is a merge filter. It is not a threat model.

Who should not use this

Skip this gate if the repo has no freeze file and no flake history. The machinery is overhead.

Skip it if agents are allowed to edit CI YAML or the freeze ledger. The gate assumes a permission boundary.

Skip it if tests cannot be collected by stable nodeids, or if coverage cannot be produced as JSON. The artifact depends on both.

Skip it for suites where fixtures are regenerated on every run by design, such as some snapshot-heavy UI packs, unless you first split human-owned goldens from agent-owned scratch.

Close the loophole, then merge

The merge token is the three-number tuple. It is not the green check. Frozen tests remain a liability until their leases expire and they pass again. Until then, they do not count as proof.