Property Checks and a Flake Freeze for Agent Patches

The ticket asked for a tax rounding fix. An agent produced a small and confident patch. Existing unit tests stayed green on every fixture.

A three cent refund still broke the money invariant. Generated patches chase the examples already in git. They rarely state the rules those examples imply.

Flaky tests make the merge picture much worse. A flickering assertion is not a stable signal. It is weather on the test runner.

This article treats agent patches as untrusted diffs. The gate uses fixture pins, properties, and a flake freeze. Remove every product mention and the method still holds.

Think of example tests as streetlights on one block. They only show the curb you already know. Property checks walk the rest of the street.

A flake freeze unscrews the bulbs that flicker. You do not merge by a flickering bulb.

The worked example is a checkout money helper. Discount applies first and tax applies second. Refunds must invert the same rounding story.

# money.py — worked example, not a production incident
from decimal import Decimal, ROUND_HALF_EVEN

CENT = Decimal("0.01")

def apply_discount(amount: Decimal, rate: Decimal) -> Decimal:
    if amount < 0 or not (Decimal("0") <= rate <= Decimal("1")):
        raise ValueError("invalid discount")
    raw = amount * (Decimal("1") - rate)
    return raw.quantize(CENT, rounding=ROUND_HALF_EVEN)

def apply_tax(amount: Decimal, rate: Decimal) -> Decimal:
    if amount < 0 or rate < 0:
        raise ValueError("invalid tax")
    raw = amount * (Decimal("1") + rate)
    return raw.quantize(CENT, rounding=ROUND_HALF_EVEN)

def checkout_total(amount: Decimal, discount: Decimal, tax_rate: Decimal) -> Decimal:
    return apply_tax(apply_discount(amount, discount), tax_rate)

Treat the module as a fixture under test. Do not treat it as a measured outage. No latency or traffic numbers are claimed here.

Pin the fixtures before any agent touches money.py. A pin is a frozen input table plus one hash. The agent may add rows after human review.

It may not edit expected cents without a human. That single rule stops silent golden edits. Agents love to tidy JSON and “fix” totals.

[
  {"amount": "10.00", "discount": "0.10", "tax": "0.07", "total": "9.63"},
  {"amount": "0.03", "discount": "0.00", "tax": "0.00", "total": "0.03"},
  {"amount": "19.99", "discount": "0.15", "tax": "0.08875", "total": "18.48"}
]

Save that table as fixtures/checkout_cases.json. Keep the JSON boring and the hash mandatory. An agent that tidies JSON fails the pin first.

# test_fixtures.py
import hashlib
import json
from decimal import Decimal
from pathlib import Path

from money import checkout_total

PIN = Path("fixtures/checkout_cases.json")
# Placeholder only. Replace after hashing the file on disk.
PIN_SHA = "REPLACE_WITH_SHA256_OF_CHECKOUT_CASES"

def test_fixture_file_is_pinned():
    digest = hashlib.sha256(PIN.read_bytes()).hexdigest()
    assert digest == PIN_SHA

def test_pinned_cases_match_totals():
    cases = json.loads(PIN.read_text())
    for row in cases:
        got = checkout_total(
            Decimal(row["amount"]),
            Decimal(row["discount"]),
            Decimal(row["tax"]),
        )
        assert got == Decimal(row["total"])

Compute PIN_SHA during setup, not from memory. Label the digest in this draft as a placeholder. The assertion is the contract either way.

Pinned fixtures stop silent expectation drift on agent diffs. They still do not prove general correctness alone. That remaining job belongs to the property checks.

One useful property here is monotonic money. Larger amounts must not yield smaller totals. Rates stay fixed when that comparison runs.

Another property rejects negative cash outright and early. A third property demands quantization to two places. Refund logic needs a narrower fourth property.

A zero discount and zero tax must preserve cents. Mixed rates are harder, so state only defensible rules here. A property you cannot defend will reject good patches.

# test_properties.py — proposed suite, not executed in this draft
from decimal import Decimal

from hypothesis import assume, given, strategies as st

from money import checkout_total

Money = st.decimals(
    min_value=Decimal("0.00"),
    max_value=Decimal("10000.00"),
    places=2,
    allow_nan=False,
    allow_infinity=False,
)
Rate = st.decimals(
    min_value=Decimal("0.00"),
    max_value=Decimal("0.50"),
    places=4,
    allow_nan=False,
    allow_infinity=False,
)

@given(amount=Money, discount=Rate, tax=Rate)
def test_total_is_two_places(amount, discount, tax):
    total = checkout_total(amount, discount, tax)
    assert total == total.quantize(Decimal("0.01"))

@given(amount=Money, discount=Rate, tax=Rate)
def test_total_not_below_zero(amount, discount, tax):
    assert checkout_total(amount, discount, tax) >= 0

@given(a=Money, b=Money, discount=Rate, tax=Rate)
def test_total_grows_with_amount(a, b, discount, tax):
    assume(a < b)
    left = checkout_total(a, discount, tax)
    right = checkout_total(b, discount, tax)
    assert left <= right

@given(amount=Money)
def test_zero_rates_preserve_amount(amount):
    total = checkout_total(amount, Decimal("0"), Decimal("0"))
    assert total == amount

Install pytest, hypothesis, and pytest-randomly in the project. Run the properties on every agent diff. Do not run them only on recited examples.

Hypothesis will search corners the prompt never named. An agent patch that switches Decimal to float dies here. So does a patch that clamps totals with int.

Unit fixtures might still glow green after that. The properties will not glow with them. Green examples are a curb, not a proof.

Float rounding looks harmless on 10.00 carts. It collapses on 0.03 refunds and repeating rates. Decimal with banker's rounding is the invariant, not a style choice.

Flakes remain the last leak in the gate. Network stubs, clocks, and unordered sets all flicker. Agents also emit tests that depend on dict order.

Those tests pass twice then fail on another shard. A flake freeze treats instability as a merge blocker. The freeze rule is blunt and local.

If a test disagrees with itself across two runs, freeze it. It stays skipped until a human rewrites it. The patch cannot ride a flickering name into main.

# freeze_flakes.py — proposed harness, not executed in this draft
import json
import subprocess
import sys
from pathlib import Path

LEDGER = Path("flake_freeze.json")

def failed_names(seed: str) -> set[str]:
    proc = subprocess.run(
        ["pytest", "-q", f"--randomly-seed={seed}"],
        check=False,
        capture_output=True,
        text=True,
    )
    names: set[str] = set()
    for line in proc.stdout.splitlines():
        if "FAILED" in line:
            names.add(line.split()[0])
    return names

def main() -> int:
    first = failed_names("1")
    second = failed_names("2")
    flakes = sorted(first.symmetric_difference(second))
    LEDGER.write_text(json.dumps({"frozen": flakes}, indent=2))
    if flakes:
        print("frozen flaky tests:")
        print("\n".join(flakes))
        return 2
    print("flake freeze clean")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Wire the freeze into CI after pins and properties. Order matters more than tool brand here. Pins fail fast on edited golden fixtures.

Properties fail on broken invariants next in line. The freeze fails on non-determinism last of all. That sequence keeps cheap failures cheap.

python -m pytest test_fixtures.py test_properties.py
python freeze_flakes.py

Skip frozen names on the next pytest run. Do not delete them from the tree. Deletion hides the evidence of the instability.

# conftest.py
import json
from pathlib import Path

import pytest

ledger_path = Path("flake_freeze.json")
FROZEN: set[str] = set()
if ledger_path.exists():
    FROZEN = set(json.loads(ledger_path.read_text()).get("frozen", []))

def pytest_collection_modifyitems(items):
    for item in items:
        if item.nodeid in FROZEN or item.name in FROZEN:
            item.add_marker(pytest.mark.skip(reason="frozen flake"))

That skip is not a pardon for the patch. It is a quarantine tag in the ledger. The patch still cannot rely on the skipped test.

Humans own the rewrite of frozen tests. An agent that deletes a skip to go green fails the review. The ledger is evidence, not a todo the model may clear.

Where does a coding agent fit this loop. The agent proposes the diff and nothing else. The three gates stay outside the model.

Free model access is enough to draft the patch. A free server is enough to run the suite off-laptop. The oracle still lives in Decimal, hashes, and pytest.

MonkeyCode provides free model access and a free server option for that draft-and-run loop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The product is optional infrastructure, not the oracle.

Keep the oracle local to your repository. Keep Decimal, hashes, and pytest on your side. The server can compile and run tests.

It should not get a vote on invariants. A model that updates PIN_SHA for you has already escaped the gate. Treat hash files as keys you do not hand over.

A short decision path keeps the merge honest. If the pin digest moved, reject the patch. If a property failed, reject the patch.

If the freeze ledger grew, reject the patch. Only a human may expand fixtures or unfreeze a test. That is the entire merge policy.

Limitations are sharp and easy to miss. Property tests need true invariants to help. Wrong properties reject good patches without mercy.

Weak properties accept bad patches in silence. Hypothesis also burns CPU on wide strategies. Narrow the money bounds before you widen the rates.

The flake freeze fails closed on purpose. Chronic infrastructure noise will stall otherwise clean merges. Do not use it as a substitute for fixing the runner.

Seed control needs pytest-randomly or an equivalent plugin. Without a seed, two runs do not isolate order bugs. The ledger then freezes the wrong names.

Some teams should skip this approach entirely. Teams without a deterministic core should skip it. Model output as the unit under test will never freeze cleanly.

Pure UI animation work will not freeze cleanly either. Security patches still need review beyond money properties. This gate does not replace threat modeling.

Do not ask the agent to write the freeze ledger. Do not ask it to update the pin digest. Those files are the lock on the gate.

Giving the model the lock returns empty meaning. Green tests then mean nothing again overnight. The three-cent refund comes back with a different stack trace.

The refund of three cents is the whole lesson. Examples are necessary and still not sufficient. Properties describe the street the examples miss.

Frozen flakes keep the remaining lights from lying. Use that order on the next agent diff.