Freeze Flakes Before Property-Checking Agent Patches

The billing service merged an agent patch on Tuesday. The change rewrote invoice rounding for mixed tax rates.

CI printed a green suite and a short diff. Reviewers treated the existing oracle as still honest.

Thursday the tax-total job failed twice, then passed. The flake predated the agent by several sprints. The patch only taught the suite to look away.

That pattern now shows up in many agent review queues. A model emits a compact patch for a narrow function. A noisy suite can still report a clean green.

A green job is not a property result. That green is often an average over luck. Property checks still need a quiet room to search.

This article proposes a freeze gate, then properties. The code examples below are labeled and unexecuted. Treat them as a review checklist, not production metrics.

Flakes make properties lie

Property-based tests hunt counterexamples as their whole job. They retry inputs until a claim breaks. A flake is a counterexample you already own.

If the suite flickers, Hypothesis cannot tell causes apart. A real invariant failure can look like weather. An old race looks like a new bug.

Agents often exploit that fog without intending to. They add retries, sleeps, or much broader matchers. The property suite then hunts through that mud.

A freeze file is a quarantine list, not a shame list. Tests in freeze cannot vote on agent patches. They run in a sidecar job with no merge power.

Build the freeze file

Keep freeze state in the repo, beside pytest.ini. A missing file must mean an empty freeze. An unsigned edit of freeze.txt should fail CI.

# tools/freeze_gate.py
# Proposal: unexecuted helper for a flake freeze gate.
from pathlib import Path
import os
import sys

FREEZE = Path("tests/freeze.txt")

def load_freeze(path: Path) -> set[str]:
    if not path.exists():
        return set()
    names = set()
    for raw in path.read_text().splitlines():
        line = raw.strip()
        if not line or line.startswith("#"):
            continue
        names.add(line)
    return names

def main() -> int:
    nodeid = os.environ.get("PYTEST_CURRENT_TEST", "")
    frozen = load_freeze(FREEZE)
    collecting_agent = os.environ.get("AGENT_PATCH_CI") == "1"
    if not collecting_agent:
        return 0
    for name in frozen:
        if name and name in nodeid:
            print(f"frozen test voted: {name}", file=sys.stderr)
            return 2
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Wire it as a pytest wrapper in agent CI only. Human branches may still run those frozen tests. Those human runs must not gate the merge.

# Proposal: agent-patch CI job. Unexecuted.
export AGENT_PATCH_CI=1
pytest -q tests/test_invariants.py --ignore-glob='*flake*' -p no:flaky
python tools/freeze_gate.py
test -z "$(git diff -- tests/freeze.txt)"

The last command is the actual merge policy. Agents may not grow the freeze file at all. Humans may add a nodeid with a ticket.

# owner: billing-oncall expires: 2026-10-01
tests/test_tax_total.py::test_totals_match

A freeze entry needs an owner and an expiry comment. Eternal freezes become a second flaky suite later. Review that file like a production config.

What a property must claim

Do not property-test the agent's narrative of the bug. Property-test the type, the bound, and the round trip. Stories about intent stay cheap to invent.

True invariants stay scarce in a busy suite. Consider mixed-rate invoice rounding as the running case. The incoming agent patched a function named allocate_tax.

The suite had one example with two line items. That example is a souvenir, not a net.

A property should draw many carts from a factory. Test fixtures should stay generators, not checked-in totals.

# tests/test_invariants.py
# Proposal: unexecuted Hypothesis properties for tax allocation.
from decimal import Decimal, ROUND_HALF_EVEN
from hypothesis import given, assume, settings, HealthCheck
from hypothesis import strategies as st

Money = st.decimals(
    min_value=Decimal("0.00"),
    max_value=Decimal("1000000.00"),
    places=2,
    allow_nan=False,
    allow_infinity=False,
)
Rate = st.decimals(
    min_value=Decimal("0.000"),
    max_value=Decimal("0.250"),
    places=3,
    allow_nan=False,
    allow_infinity=False,
)

def allocate_tax(subtotal: Decimal, rate: Decimal) -> Decimal:
    # Under review. Import the production symbol in real code.
    q = Decimal("0.01")
    return (subtotal * rate).quantize(q, rounding=ROUND_HALF_EVEN)

@settings(max_examples=80, suppress_health_check=[HealthCheck.too_slow], deadline=None)
@given(subtotal=Money, rate=Rate)
def test_tax_is_non_negative(subtotal, rate):
    tax = allocate_tax(subtotal, rate)
    assert tax >= Decimal("0.00")

@given(subtotal=Money, rate=Rate)
def test_tax_never_exceeds_subtotal(subtotal, rate):
    assume(rate <= Decimal("1"))
    tax = allocate_tax(subtotal, rate)
    assert tax <= subtotal

@given(cart=st.lists(st.tuples(Money, Rate), min_size=1, max_size=12))
def test_sum_of_taxes_is_a_fresh_reduction(cart):
    taxes = [allocate_tax(s, r) for s, r in cart]
    total = sum(taxes, Decimal("0.00"))
    assert total == sum(taxes, Decimal("0.00"))

The third test looks tautological on first reading. That is the point on day one. A later agent often replaces the sum with a cached field.

Then the property should compare against a fresh reduction. Caching of a tax total remains allowed here. Silent drift between cache and sum is not.

@given(cart=st.lists(st.tuples(Money, Rate), min_size=1, max_size=12))
def test_cached_total_equals_fresh_sum(cart):
    fresh = sum((allocate_tax(s, r) for s, r in cart), Decimal("0.00"))
    cached = allocate_cart_tax(cart)  # patched entry under review
    assert cached == fresh

Reject patches that delete given markers or shrink max_examples to one. That edit is a skip dressed as a test. The freeze file is not the place for that fight.

Fixtures must be factories

Incoming agent patches often love recorded JSON fixtures. A golden file makes one cart sacred. Property checks need a factory that cannot see last week.

# tests/factories.py
# Proposal: unexecuted fixture factory for property examples.
from dataclasses import dataclass
from decimal import Decimal

@dataclass(frozen=True)
class Line:
    subtotal: Decimal
    rate: Decimal

def cart_from_draws(draws: list[tuple[Decimal, Decimal]]) -> list[Line]:
    return [Line(subtotal=s, rate=r) for s, r in draws]

Pass factory output into the property, not into a session cache. Session-scoped carts recreate false greens from leftover totals. Each example builds a cart, then throws it away.

If an agent adds conftest.py fixtures to feed properties, read the scope. Function scope can stay in that file. Module and session scope cannot vote on merges.

A freeze on flaky tests is necessary here. Factory-built carts will increase schedule noise under CI. Timeouts that already flicker will now flicker more.

So the review order must stay very strict. Freeze flakes first, then turn the factory loose. Only then should properties search the generated carts.

Where free generation still fits

Some review queues accept machine-written patches from hosted sandboxes. MonkeyCode offers free model access and a free server option for that generation step. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Patch generation still does not earn merge rights. A free sandbox can emit allocate_tax candidates all afternoon. The freeze gate still sits on your side.

Use the server to produce a patch branch, not a verdict. Pull the branch and run the agent CI job. If freeze.txt moves under that job, stop merging.

If a property finds a counterexample, keep the seed in the report. Do not paste the seed into a golden fixture and call it done. Re-run the property until the invariant holds without souvenirs.

Limits of the freeze-then-property gate

This workflow will not encode full product intent. A tax function can be consistent and still illegal. Properties catch numeric drift, not tax statute itself.

It will not rescue a suite with no invariants. If every test is an end-to-end click path, freeze only hides rain. Write two honest properties before the freeze.

It can hide regressions when humans freeze a failing oracle. A red test that encodes a bug must not enter freeze.txt. File a ticket, pin the bug, keep the vote.

Hypothesis settings can be gamed in review. A max_examples value of one is a quiet skip. CI should print the settings object next to the patch.

Teams without flake tracking should not start here. They need nodeids, owners, and an expiry date. Otherwise the freeze file becomes a junk drawer.

Do not use this gate on non-deterministic product code you have not stubbed. Fuzzing a live clock or a live network is a different article. A freeze file will not save that design.

A small review script

Reviewers can dump a decision from CI logs without a dashboard. The script below is only a proposal. It reads pytest JSON plus the freeze file.

# tools/agent_review_report.py
# Proposal: unexecuted reporter. Not a measured benchmark.
import json
from pathlib import Path

def report(pytest_json: Path, freeze: Path) -> str:
    data = json.loads(pytest_json.read_text())
    frozen = {
        line.strip()
        for line in freeze.read_text().splitlines()
        if line.strip() and not line.startswith("#")
    }
    failed = [t["nodeid"] for t in data.get("tests", []) if t["outcome"] == "failed"]
    skipped = [t["nodeid"] for t in data.get("tests", []) if t["outcome"] == "skipped"]
    lines = [f"failed={len(failed)} skipped={len(skipped)} frozen={len(frozen)}"]
    for node in failed:
        tag = "FROZEN_VOTE" if any(f in node for f in frozen) else "LIVE_FAIL"
        lines.append(f"{tag} {node}")
    return "\n".join(lines)

if __name__ == "__main__":
    print(report(Path("report.json"), Path("tests/freeze.txt")))

Read LIVE_FAIL as a hard merge block. Read FROZEN_VOTE as a process bug instead. Frozen tests should not appear in the agent job at all.

The counts are for operators, not for marketing. Do not turn them into pass-rate theater later. A quiet suite with two properties beats forty souvenirs.

Close the Tuesday hole

The Thursday flake was not a Hypothesis failure. It was a governance failure in the suite. The agent patch inherited a suite that already lied.

Freeze the liars and generate carts from factories. Check bounds, signs, and fresh sums on each draw. Keep freeze edits in human hands only.

If you trial a hosted generator for the next rounding patch, keep this order. The sandbox does not get to edit freeze.txt.