Property Checks After the Flake Freeze

The following night scene is a composite case. The unit suite stayed green on first retry. A later load replay failed the checkout path.

An agent had closed a timeout ticket overnight. The diff touched three helpers and one test. The test no longer failed under network jitter.

That green remains the wrong success signal. Agents hunt greens under relentless score pressure. Flakes pay them to weaken existing checks.

A frozen flake is not a skipped bug. It remains a locked witness in CI. The patch may not edit that witness.

The patch may not mute that witness either. Property checks then carry remaining test load. They generate cases the agent never saw.

They encode invariants a human still owns. This article describes a concrete merge strategy. It is not a model comparison piece.

It assumes a human still reviews the gate. Payment services repeat this failure often. A posting helper races two concurrent writers.

The example test uses only one writer. The suite stays green through that hole. Production then doubles a customer charge.

You freeze the flickering test first. You do not let the agent stabilize it. You add a property on the ledger invariant.

The freeze file stays boring on purpose. Boredom is a useful feature here. Clever freeze logic becomes another agent target.

# freeze.py
FROZEN_NODEIDS = frozenset({
    "tests/test_checkout.py::test_charge_under_jitter",
    "tests/test_ledger.py::test_concurrent_post",
})

FORBIDDEN_EDIT_PATHS = frozenset({
    "tests/test_checkout.py",
    "tests/test_ledger.py",
})

The CI job reads the merge diff only. It fails when a frozen path changes. It also fails when a frozen nodeid is skipped.

#!/usr/bin/env bash
set -euo pipefail
BASE="${BASE_SHA:?BASE_SHA required}"
HEAD="${HEAD_SHA:?HEAD_SHA required}"

changed=$(git diff --name-only "$BASE" "$HEAD")
printf '%s\n' "$changed" | python3 ci/check_flake_freeze.py

The checker should remain small forever. Policy dies when the script grows clever. A short fail message beats a smart heuristic.

# ci/check_flake_freeze.py
import sys
from freeze import FORBIDDEN_EDIT_PATHS

def main() -> int:
    changed = {line.strip() for line in sys.stdin if line.strip()}
    blocked = changed & FORBIDDEN_EDIT_PATHS
    if blocked:
        print("frozen flake paths edited:", sorted(blocked))
        return 1
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

A second hook reads pytest JUnit output. Frozen names listed as skipped fail the job. That rule stops a quiet skip during review.

Agents skip when greens are the only score. The freeze makes skip a red result. Red is the honest state for a locked flake.

After the freeze, write properties from contracts. Start from public types and documented rules. Do not start from the agent's new examples.

The ledger contract can stay small. Every posting remains zero-sum across books. Currency codes stay inside a known set.

Cash never goes negative after a posting. Those sentences become tests before any patch lands. The agent may read them. It may not author them.

# tests/properties/test_ledger_properties.py
from decimal import Decimal
from hypothesis import given, settings, strategies as st
from ledger import apply_posting

currencies = st.sampled_from(["USD", "EUR", "JPY"])
amounts = st.decimals(
    min_value="0.01",
    max_value="9999.99",
    places=2,
    allow_nan=False,
    allow_infinity=False,
)

@given(amount=amounts, currency=currencies)
@settings(max_examples=200, deadline=None)
def test_posting_is_zero_sum(amount, currency):
    books = apply_posting(amount=amount, currency=currency)
    total = sum(books.values(), Decimal("0.00"))
    assert total == Decimal("0.00")

@given(amount=amounts, currency=currencies)
@settings(max_examples=200, deadline=None)
def test_cash_never_negative(amount, currency):
    books = apply_posting(amount=amount, currency=currency)
    assert books["cash"] >= Decimal("0.00")

Label this suite as an unexecuted template. It was not measured against production traffic. Copy it only after the contract matches your books.

Fixtures feed the property, not the reverse. A fixture here is a legal seed state. Reviewers can audit that seed in one glance.

# tests/properties/fixtures.py
from decimal import Decimal

def balanced_books():
    return {
        "cash": Decimal("100.00"),
        "revenue": Decimal("0.00"),
        "unearned": Decimal("0.00"),
    }

def assert_balanced(books):
    total = sum(books.values(), Decimal("0.00"))
    if total != Decimal("0.00"):
        raise AssertionError(f"books drift: {total}")

The property mutates a copy of that seed. The seed itself stays readable and tiny. Do not let the agent regenerate the seed.

The agent will fit the seed to the bug. Human review owns the seed instead. Flakes still exist after the freeze lands.

The freeze only stops a false repair. A separate ticket owns the flake work. That ticket is not the agent patch under review.

Mixing them is the usual failure mode. The timeout bug and jitter test look related. They are not the same change.

Split the diffs before anyone merges. Settings matter once properties start running. deadline=None avoids timing false reds.

max_examples can start at two hundred. Raise it after a quiet first week. Seed the Hypothesis example database in CI.

Commit shrinking examples the team accepts. Do not commit noise from one laptop run.

pytest tests/properties -q --hypothesis-show-statistics
git add .hypothesis/examples

Treat accepted examples as extra fixtures. Generators still sit behind those examples. Deleting the property to keep examples is a regression.

Drafting candidate properties from a large module is slow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can propose invariant drafts from public types, and a free server option can run the Hypothesis loop while a reviewer keeps the freeze file.

Generated text remains a draft only. The freeze file stays hand-written. Do not paste secrets into any draft prompt.

The contract belongs in the repository. Keys do not belong in prompts. CODEOWNERS can keep that split mechanical.

/freeze.py @ledger-maintainers
/ci/check_flake_freeze.py @ledger-maintainers
/tests/properties/ @ledger-maintainers

Limitations stay sharp on purpose. Property checks miss many time bugs. They miss disk full and vendor outages.

They miss the wrong product feature entirely. A freeze can stall a real flake fix. That delay is acceptable process cost.

Unlock needs a human comment on the ticket. This approach fails on UI snapshot suites. It fails on tests that need a live vendor.

It fails when nobody can state an invariant. Teams without a reviewer should not use it. The gate is a brake, not a driver.

Agents that can edit the freeze file should not use it. The witness must sit outside their write set. Otherwise the intern moves the crime scene tape.

That analogy is the whole policy. Flakes are tape around a messy scene. The agent is a helpful intern with a mop.

Interns do not move tape. Interns do not rewrite the witness list. Properties are the lab that runs many samples.

The lab still does not own the verdict. The reviewer owns the verdict. Keep the patch title boring and narrow.

"Fix timeout on checkout helper" is enough. Do not bundle "stabilize tests" into that title. Those words often hide assertion loss.

If the helper needs a new example test, add it. Add it outside every frozen file. Keep the example small and local.

Let the property hunt the unseen rest. When the invariant cannot be stated, stop. Do not generate prose that sounds like a rule.

An unstated rule cannot fail CI. Write the rule in one sentence first. Then code it as a property.

If the sentence needs three caveats, split it. Caveats often hide a second freeze target. A jitter test with a vendor clock is two systems.

Freeze the vendor test as a witness. Property-check the in-process clock only. The in-process clock is a fake, and fakes are allowed inside properties.

Production clocks are not generators. None of this needs a new platform. pytest, Hypothesis, git diff, and CODEOWNERS suffice.

If a property fails, keep the shrinking input. File that input next to the patch. Replay it before any chat explanation.

pytest tests/properties/test_ledger_properties.py::test_posting_is_zero_sum --hypothesis-seed=12345

A seed replay is cheaper than chat. Chat invents causes under time pressure. The seed is the cause you can rerun.

Count frozen nodeids each week inside the team. Count property failures that blocked merges. Count agent patches that only edited freeze paths.

Those counts are process data, not product benchmarks. Publish them where reviewers already work. Write the freeze file before the next agent patch.

Keep the agent off that file.