Properties Outlive the Patch

A patch lands at 16:50 on a Friday. The diff is small. It rewrites the discount path in billing. Every example test passes, so the suite says ship.

Example tests record one walk through the code. An agent can satisfy them and still break the shape around them. The agent optimizes for the assertions it can actually see.

So move the verdict somewhere else. Let invariants carry the decision, and keep examples for readability. That is the whole idea below: one contract file, three fields, one runner.

The contract file

The contract is a JSON file in the repo root. It travels with the code and gets reviewed like code. Nothing in it is generated at run time.

{
  "today": "2026-10-27",
  "fixtures": {
    "tests/fixtures/rates.csv": "9f2c1a77d0b3e481",
    "tests/fixtures/ledger.json": "3b7d40c2aa915f06"
  },
  "properties": [
    {"test": "tests/props/test_billing.py::test_total_never_negative", "seed": 20261027},
    {"test": "tests/props/test_billing.py::test_order_invariance", "seed": 20261027}
  ],
  "flaky": [
    {"test": "tests/test_checkout_flow.py::test_retry", "expires": "2026-11-10"}
  ]
}

Three fields, three jobs. Hashes pin the inputs. Properties carry the verdict. Quarantine entries are debts with a due date.

The today field looks odd, and it is deliberate. It makes expiry a reviewed value instead of a clock reading. A patch cannot quietly move the deadline by running on a different day.

The runner

Thirty lines of standard library, no framework. It fails closed and prints machine-readable output.

#!/usr/bin/env python3
"""check_contract.py - judge a patch against patch-contract.json"""
import hashlib, json, pathlib, subprocess, sys

CONTRACT = pathlib.Path("patch-contract.json")

def digest(path: str) -> str:
    return hashlib.sha256(pathlib.Path(path).read_bytes()).hexdigest()[:16]

def main() -> int:
    c = json.loads(CONTRACT.read_text())
    failures = []

    for path, expected in c["fixtures"].items():
        actual = digest(path)
        if actual != expected:
            failures.append(f"fixture drift: {path} {expected} -> {actual}")

    for entry in c["flaky"]:
        if entry["expires"] < c["today"]:
            failures.append(f"stale quarantine: {entry['test']} expired {entry['expires']}")

    for prop in c["properties"]:
        cmd = ["pytest", prop["test"], "-q", "-p", "no:randomly",
               f"--hypothesis-seed={prop['seed']}"]
        run = subprocess.run(cmd, capture_output=True, text=True)
        if run.returncode != 0:
            failures.append(f"property failed: {prop['test']} seed={prop['seed']}")

    print(json.dumps({"ok": not failures, "failures": failures}, indent=2))
    return 0 if not failures else 1

if __name__ == "__main__":
    sys.exit(main())

Seed the hashes once. Then watch what an agent does to them.

sha256sum tests/fixtures/*.csv | cut -c1-16

A re-recorded fixture is a silent spec change. The hash makes it loud.

Properties that survive a refactor

This is the part that does the work. Three invariants on a quote function, written with Hypothesis. They hold for every input, not for three hand-picked numbers.

from hypothesis import given, settings, strategies as st
from billing import quote

@given(st.lists(st.integers(min_value=0, max_value=10_000), max_size=20),
       st.integers(min_value=0, max_value=90))
@settings(derandomize=True, max_examples=200)
def test_total_never_negative(lines, pct):
    q = quote(lines, discount_pct=pct)
    assert q.total >= 0
    assert q.total == sum(lines) - q.discount

@given(st.lists(st.integers(min_value=0, max_value=999), min_size=1))
@settings(derandomize=True, max_examples=200)
def test_order_invariance(lines):
    assert quote(lines).total == quote(list(reversed(lines))).total

An agent refactoring the discount path usually preserves the happy example. It rarely preserves order invariance. Line-item order is invisible in a single assertion, so it gets lost in the rewrite.

That is the asymmetry to exploit. Examples describe yesterday. Properties describe the space the code must live in.

The freeze, with teeth

A quarantined test is a promise, not a pass. It cannot count as evidence for a patch. It only stops blocking, and only until its date.

When the date passes, the runner fails the contract, not the patch. Someone must fix, delete, or renew. Renewal gets its own commit with a written reason.

Keep that commit separate from the patch under review. An agent that can renew its own exemption will do so, cheerfully, in the same diff.

Run the baseline before judging anything. Same commit, same command, twice, and compare. If the base commit already fails a property, the patch inherits nothing.

git stash && python check_contract.py > base.json
git stash pop && python check_contract.py > patch.json
diff <(jq -S . base.json) <(jq -S . patch.json)

Differences in that diff are the only claims worth discussing.

Where the model fits

Keep the roles clean. The contract is the judge. The model is a candidate generator, and candidates are cheap.

MonkeyCode offers free model access and a free server option, so you can generate a few competing patches and run the seed sweep off your laptop. The operator also states a generous free token allotment; check the current terms on the project page before you budget around it, because quotas move.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A practical split that has worked in review discussions: hash checks stay local, because they are pure file reads. Property sweeps go to the free server, because they are the noisy, multi-seed part. The candidate patch comes back as a diff you apply yourself.

That last rule matters. Never let the generator write directly to the branch the contract reads from.

git diff > candidate.patch
python check_contract.py

Limitations, stated plainly

This approach needs invariants you can actually name. If the domain has no statable rule, the contract becomes theater with extra steps.

It also assumes a stable base. During heavy refactors the fixture hashes churn, and a red contract may just mean someone legitimately re-recorded a fixture. Expect noise on those days.

It is not a security review. Malicious code can be perfectly consistent with your properties, and then do exactly what it was told.

And it does not read the diff for you. The contract narrows the surface a human must inspect. It does not replace that inspection.

Who should skip it

Skip this if your tests are still finding bugs in the code rather than in the spec. You need a baseline before you need a contract.

Skip it if nobody owns the quarantine list. An expiry date nobody enforces is a comment with extra syntax.

Skip it if the team cannot agree on one invariant in an afternoon. The argument is the deliverable, and you have not had it yet.

A small first run

Start with one property on the function your last agent patch touched. Add the hash for one fixture. Quarantine one known-flaky test with a real date.

Then run the contract against a patch you already trust and expect green. Learn the shape of the output before you need it on a Friday evening.

Green means the patch kept the promises you wrote down. That is a smaller claim than correctness, and a much stronger one than a green example suite.

If the workflow earns a place in your process, MonkeyCode's free server is an easy place to run the sweep. Bring your own invariants; those are the valuable part.