A Frozen Test Can't Justify a Patch: Lease-Bound Freezes and Witnessed Properties for Agent Patches

The most expensive test in an agent patch is the one nobody runs. It sits in the suite, green, on a quarantine list, and the patch quietly points at it as proof.

Here is the conclusion first: three kinds of test evidence should never share one bucket — properties, examples, and frozen tests. Only the first two can justify a diff. A freeze is a short-term loan, so give it a lease, an owner, and an expiry date, then make the patch declare which tests it stands on so a small script can refuse the claim.

The method below assumes you already review agent patches and that flaky tests exist in your repo. It does not assume any particular agent vendor.

The failure mode: a freeze becomes a receipt

Quarantines are supposed to remove noise. In practice they remove evidence. A test gets skipped because it fails once in forty runs, and six weeks later an agent patch adjusts the exact behavior that test described. The reviewer sees a green suite, sees the skipped test in the same file, and reads it as "known flaky, unrelated."

Three things go wrong at once. The freeze has no expiry, so nobody revisits it. The patch never states which tests it relies on, so nothing can be checked against the ledger. And the replacement evidence, usually a property test written the same afternoon, is often narrow enough that it cannot fail.

Classify evidence before you read the diff

Evidence class	What it asserts	Can justify a patch?	Default failure response
Property (invariant)	A statement over an input space	Yes, with a witness (Step 5)	Fix the code, not the property
Example (fixture-backed)	One concrete scenario	Yes, if not frozen and its fixture is unchanged	Check fixture write-set first
Frozen (quarantined)	Nothing, currently	No	Re-freeze with a lease, or delete

The table is the whole policy. Everything else is plumbing to enforce it.

Step 1 — Name the three classes in the repo

File naming beats markers, because agents edit markers less reliably than they edit files. Use a prefix convention and wire it into pytest in one place.

# tests/conftest.py (proposal - verify against your pytest version)
def pytest_collection_modifyitems(config, items):
    for item in items:
        path = str(item.fspath).replace("\\", "/")
        if "/test_prop_" in path:
            item.add_marker("property")
        elif "/test_example_" in path:
            item.add_marker("example")

Then a targeted run is boring and explicit:

pytest -q -m property
pytest -q -m example

A file without a prefix is legacy and gets treated as example evidence with no property rights. That asymmetry is deliberate.

Step 2 — Give every freeze a lease

A freeze without an expiry is a deletion with extra steps. Store freezes in one YAML file, reviewed like code, and require four fields: owner, since, expires, and an issue link.

# testing/freeze-ledger.yaml
freezes:
  - test: tests/test_example_billing.py::test_proration_rounds_down
    file: tests/test_example_billing.py
    since: 2026-10-02
    expires: 2026-10-23
    owner: payments
    issue: ENG-4412
    reason: "fails roughly 1 in 40 runs on CI clock skew"

Twenty-one days is a reasonable default ceiling. Long enough to fix a clock dependency, short enough that the person who froze it is still around to answer for it.

Step 3 — Make the patch declare its justifying tests

This is the part most review processes skip. Require a small machine-readable claim next to the diff, produced by whoever or whatever wrote the patch.

{
  "patch_id": "agent-8841",
  "changed_files": ["src/billing/proration.py"],
  "justifying_tests": [
    "tests/test_example_billing.py::test_proration_rounds_down",
    "tests/test_prop_refund.py::test_refund_never_increases_balance"
  ]
}

An empty justifying_tests list is a valid answer. It means "no test justifies this change," which is exactly what a reviewer needs to know before reading a single hunk.

Step 4 — Audit the claim, not the test suite

Sixty lines of Python catch the whole failure class. The script never runs the tests; it checks the argument.

#!/usr/bin/env python3
# tools/freeze_audit.py (proposal - review before adding to CI)
import datetime as dt
import json
import sys
from pathlib import Path

import yaml

LEDGER = Path("testing/freeze-ledger.yaml")
MAX_FREEZE_DAYS = 21


def main(claim_path: str) -> int:
    today = dt.date.today()
    raw = yaml.safe_load(LEDGER.read_text()) or {}
    ledger = {e["test"]: e for e in raw.get("freezes", [])}
    claim = json.loads(Path(claim_path).read_text())

    problems = []
    for test in claim.get("justifying_tests", []):
        if test in ledger:
            e = ledger[test]
            problems.append(f"{test}: cited as justification but frozen since {e.get('since')}")

    for test, e in ledger.items():
        if not e.get("owner"):
            problems.append(f"{test}: freeze has no owner")
        if not e.get("issue"):
            problems.append(f"{test}: freeze has no tracking issue")
        expires = e.get("expires")
        if not expires:
            problems.append(f"{test}: freeze has no expiry")
            continue
        if dt.date.fromisoformat(str(expires)) < today:
            problems.append(f"{test}: freeze expired on {expires}")
        since = dt.date.fromisoformat(str(e.get("since", today)))
        if (today - since).days > MAX_FREEZE_DAYS:
            problems.append(f"{test}: frozen {(today - since).days} days (>{MAX_FREEZE_DAYS})")

    if problems:
        print("freeze-audit: BLOCKED")
        for p in problems:
            print(f"  - {p}")
        return 1

    print(f"freeze-audit: OK ({len(ledger)} active freezes)")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "claim.json"))

Run it locally first, because the exit code is the product:

python tools/freeze_audit.py claim.json; echo "exit=$?"
# freeze-audit: BLOCKED
#   - tests/test_example_billing.py::test_proration_rounds_down: cited as justification but frozen since 2026-10-02
# exit=1

That output is the review comment you wanted: the patch is not wrong, its argument is unpaid.

Step 5 — Require a witness for every property

Freezes push people toward properties, and properties have their own vacuity problem. A property test that passes against a broken implementation is decoration. The cheap defense is a witness: mutate the function under test to something obviously wrong and require the property to fail.

# tests/test_witness_refund.py (proposal - not executed here)
import pytest
import tests.test_prop_refund as prop


def test_property_catches_inverted_refund(monkeypatch):
    monkeypatch.setattr(prop, "refund", lambda balance, amount: balance + amount)
    with pytest.raises(AssertionError):
        prop.test_refund_never_increases_balance()

If the witness passes, the property is not evidence. Send it back. One witness per property file is enough to keep the habit honest, and it costs a few lines.

Step 6 — Run the parts that are cheap on the cheap tier

Nothing in Steps 1 to 5 requires a paid model call or a large runner. The audit script is I/O and date math. The witness tests are ordinary pytest. The flake estimate is repetition:

pip install pytest-repeat
pytest -q --count=20 tests/test_example_billing.py::test_proration_rounds_down

This is where MonkeyCode's free model access and free server option fit, and only here. The operator currently lists a 10M-token allowance and a free server; treat both as availability you can plan around, not as a guarantee of capacity or permanence.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A workable split: use free model access to draft property candidates and witness mutations from a diff, which is high-volume, low-stakes generation; run the n-times repetition job on the free server, since it is long and dumb; keep the merge decision, the ledger edits, and the freeze approvals on human review. The audit stays the authority. A model can propose a witness; only a failing run confirms one.

Limitations and who should skip this

This workflow is heavier than most teams need. Skip it if you have fewer than a dozen agent-authored patches a week, if your suite finishes in under a minute, or if flakiness is genuinely rare — a plain quarantine comment and a calendar reminder will do. Skip it too if your tests cannot be run selectively, because the claim file assumes you can name and execute individual tests.

The honest gaps: the audit checks that a cited test is not frozen, not that it covers the changed lines; a patch can cite an unrelated passing test and still pass. The witness check catches vacuous properties, not properties that encode the wrong requirement. And lease expiry creates pressure to re-freeze rather than fix, which is why the expiry needs an owner with authority to say no.

What to take away

Separate properties from examples from freezes. Give every freeze a lease and an owner. Require each patch to name the tests it stands on. Then let a sixty-line script read the argument instead of the diff.

The ledger is the artifact worth copying, even if you throw away every line of code above. If you want to try the repetition job without paying for a runner, the free server option is the low-risk place to start — bring your own ledger, because the discipline is the point and the compute is not.