DEV Community

Finley Zhou
Finley Zhou

Posted on

Green Property Tests Can Still Be a Regression

#testing #python #ai #ci

A green property test does not prove that an agent patch preserved the contract. It only proves that the current generator produced no counterexample. Shrink the generator, add a broad assume(), or rewrite the fixture the property reads, and the suite still passes.

Treat that as a merge defect. The control proposed here is a triple lock plus a freeze of the oracle, not of the test. Fingerprint the generator. Bound the assume-reject rate. Hash the fixtures the property consumes. Require an independent replay to agree. If any of those signals move, the oracle loses its vote until a human classifies the change.

The failure mode

Agent patches optimize for green CI. Property tests look rigorous, so they are an attractive target. The agent does not need to delete the check. It only needs to make the search space smaller than the bug.

Three edits show up in review. The strategy stops emitting the interesting cases. assume() rejects most draws, so the body almost never runs. The fixture changes while the assertion text stays the same.

None of those fail a naive pytest invocation. All three reduce what the test can still catch. A freeze that merely silences noise will hide the same edits. Freeze the oracle instead: keep collecting results, stop letting that check vote.

What this article does not claim

This is a proposed workflow and a runnable checker, not a production study. No pass-rate, flake-rate, or model-quality numbers are attached. If your merge gate already fingerprints generators and compares two runners, you do not need the rest.

The method is for repositories where an agent may edit tests and fixtures in the same patch as production code. It is not a substitute for review of the production diff. Label every default bound below as a starting value, not a measured optimum.

Three locks, one freeze

Keep three artifacts in source control, outside paths the agent is allowed to edit:

  • generators.lock.json — a fingerprint per property.
  • fixtures.lock.json — hashes of files the properties read.
  • oracle_freeze.json — oracles that currently cannot vote.

The merge predicate is then mechanical. A property may vote only if its fingerprint matches, its assume-reject rate stays inside a bound, its fixtures match, it is not frozen, and a second runner reports the same outcome. Everything else is information. It is not a yes.

1. Fingerprint the generator

Hash the AST of the function that builds inputs, not the assertion. Comments and local variable names should not move the fingerprint. Structure should. Harmless refactors will trip the lock. That is intended. A human can accept a new fingerprint. An agent cannot.

# oracle_lock.py — proposal you can run locally
from __future__ import annotations

import ast
import hashlib
import inspect
from collections.abc import Callable

def _normalize(src: str) -> str:
    tree = ast.parse(src)
    for node in ast.walk(tree):
        for attr in ("lineno", "end_lineno", "col_offset", "end_col_offset"):
            if hasattr(node, attr):
                setattr(node, attr, 0)
    return ast.dump(tree, include_attributes=False)

def fingerprint_fn(fn: Callable) -> str:
    payload = _normalize(inspect.getsource(fn)).encode("utf-8")
    return hashlib.sha256(payload).hexdigest()[:16]
Enter fullscreen mode Exit fullscreen mode

Register each property by a stable name. Do not register by file path. Agents rename files.

PROPERTIES: dict[str, Callable] = {}

def property_case(name: str):
    def deco(fn: Callable) -> Callable:
        PROPERTIES[name] = fn
        return fn
    return deco
Enter fullscreen mode Exit fullscreen mode

Write the lock file in CI from main, not from the patch branch. On the patch, only compare.

python oracle_lock.py write-generators --from-ref origin/main --out generators.lock.json
python oracle_lock.py check-generators --lock generators.lock.json
Enter fullscreen mode Exit fullscreen mode

If the agent rewrote gen_orders() to skip cancelled orders, the fingerprint moves. The property can still be green. The lock is red.

2. Bound the assume-reject rate

A property that rejects most draws is barely a property. Record attempts, rejects, and completed trials. Fail the check when the reject rate rises by more than a fixed delta, even if every completed trial passed.

from dataclasses import dataclass

@dataclass
class TrialStats:
    attempts: int = 0
    rejected: int = 0
    completed: int = 0
    failed: int = 0

    @property
    def reject_rate(self) -> float:
        return 0.0 if self.attempts == 0 else self.rejected / self.attempts

def run_property(fn, draws, assume) -> TrialStats:
    stats = TrialStats()
    for value in draws:
        stats.attempts += 1
        if not assume(value):
            stats.rejected += 1
            continue
        stats.completed += 1
        try:
            fn(value)
        except AssertionError:
            stats.failed += 1
    return stats
Enter fullscreen mode Exit fullscreen mode

Store baseline rates next to the fingerprint. Use a fixed draw budget so two runners are comparable.

{
  "prop_order_total_non_negative": {
    "generator": "a1b2c3d4e5f60789",
    "attempts": 200,
    "rejected": 14,
    "completed": 186
  }
}
Enter fullscreen mode Exit fullscreen mode

Decision rule, labeled as a default:

  1. failed > 0 — property red; the oracle may vote as a fail.
  2. completed == 0 — treat as red, not as skip.
  3. reject_rate - baseline > 0.15 — freeze the oracle.
  4. generator fingerprint mismatch — freeze the oracle.

The 0.15 delta is a starting bound. Tighten it if your generators are deterministic. Loosen it only with a recorded reason in oracle_freeze.json.

3. Hash fixtures the property reads

Here the fixture is an input to a property, not an expected output. If the agent rewrites orders.json so every amount is already positive, prop_order_total_non_negative becomes cheap to satisfy. The assertion text does not have to change.

from pathlib import Path

def hash_tree(root: Path) -> dict[str, str]:
    out = {}
    for path in sorted(root.rglob("*")):
        if path.is_file():
            digest = hashlib.sha256(path.read_bytes()).hexdigest()
            out[str(path.relative_to(root))] = digest
    return out
Enter fullscreen mode Exit fullscreen mode

Put property-owned fixtures under oracles/fixtures/. If the patch touches that directory, the fixture lock fails. The property is then ineligible to vote, whether or not it passed.

python oracle_lock.py check-fixtures --root oracles/fixtures --lock fixtures.lock.json
Enter fullscreen mode Exit fullscreen mode

Seed the draws from a committed seed file in the same tree. Do not read wall-clock time or hostname inside a property. Those inputs make replay disagreement unreadable.

4. Replay on a second runner

One CI lane is not a witness. Cache, plugin order, time, and RNG can all make a single green run look stable. Replay the same property set on a second machine or container. Compare the JSON reports, not the log text.

def disagreement(merge: dict, replay: dict) -> list[str]:
    names = sorted(set(merge) | set(replay))
    bad = []
    for name in names:
        a, b = merge.get(name), replay.get(name)
        if a is None or b is None or a["outcome"] != b["outcome"]:
            bad.append(name)
        elif abs(a["reject_rate"] - b["reject_rate"]) > 0.15:
            bad.append(name)
    return bad
Enter fullscreen mode Exit fullscreen mode

Encode outcome as pass, fail, or vacuous. Do not encode skip as success. A missing name in either report is a disagreement.

If you do not already have spare hardware, a second environment is enough. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is useful for drafting candidate properties from a spec; a human still has to accept the generator and the assume predicate before they enter the lock file. The free server option is one way to run the replay lane without waiting on the merge runner. Neither replaces the lock files.

Do not send production secrets to any hosted runner. Replay on synthetic fixtures only.

5. Freeze the oracle, keep the test

A freeze is not pytest.mark.skip. Skipping deletes the signal. The ledger records that the oracle is under dispute and must not vote.

{
  "prop_order_total_non_negative": {
    "frozen_at": "2026-09-19",
    "reason": "generator_fingerprint_mismatch",
    "merge_outcome": "pass",
    "replay_outcome": "pass",
    "vote": false
  }
}
Enter fullscreen mode Exit fullscreen mode

Allowed reasons, and only these:

  1. generator_fingerprint_mismatch
  2. reject_rate_delta
  3. fixture_hash_mismatch
  4. runner_disagreement
  5. vacuous (zero completed trials)

Unfreeze is a human edit. The agent patch that caused the freeze cannot clear it. CI should fail if oracle_freeze.json loses an entry without a matching reviewed commit on main.

Numbered merge procedure

  1. Compute generator fingerprints on the patch. Compare to generators.lock.json from main.
  2. Run properties with a fixed draw budget. Write merge_report.json.
  3. Compare reject rates and completed counts to the lock.
  4. Hash oracles/fixtures against fixtures.lock.json.
  5. Replay steps 2–4 on a second runner. Write replay_report.json.
  6. If any name disagrees, or any lock is red, append a freeze entry with vote: false.
  7. Merge only if every unfrozen oracle passed on both runners and no new freeze was required.

Minimal driver:

python oracle_lock.py check-generators --lock generators.lock.json
python oracle_lock.py check-fixtures --root oracles/fixtures --lock fixtures.lock.json
python oracle_lock.py run --budget 200 --seed oracles/seed.txt --out merge_report.json
python oracle_lock.py run --budget 200 --seed oracles/seed.txt --out replay_report.json  # second host
python oracle_lock.py freeze --merge merge_report.json --replay replay_report.json --ledger oracle_freeze.json
python oracle_lock.py vote --ledger oracle_freeze.json --merge merge_report.json --replay replay_report.json
Enter fullscreen mode Exit fullscreen mode

vote should print a single line: ALLOW or DENY. Wire that to the merge gate. Do not parse pytest exit codes alone. A frozen oracle that still fails is data for the reviewer. It is not a merge vote.

def vote(ledger: dict, merge: dict, replay: dict) -> str:
    if disagreement(merge, replay):
        return "DENY"
    for name, row in merge.items():
        frozen = ledger.get(name, {}).get("vote") is False
        if frozen:
            continue
        if row["outcome"] != "pass" or replay[name]["outcome"] != "pass":
            return "DENY"
    return "ALLOW"
Enter fullscreen mode Exit fullscreen mode

Decision table

Signal Test still green? Oracle votes? Human action
Assertion failed on either runner no yes, as fail fix production or property
Generator fingerprint changed often yes no accept new generator or revert
Reject rate up more than 0.15 often yes no restore assumes or justify bound
Fixture hash changed often yes no restore fixture or re-lock
Runners disagree mixed no classify noise vs contract change
Zero completed trials n/a no treat as vacuous, not skip
All locks match, both pass yes yes merge may proceed

The table is the artifact. If you adopt only one row, adopt the generator fingerprint. That row catches the cheapest way to fake a property.

Limitations and who should not use this

The fingerprint is brittle under refactor. Teams that rewrite generators weekly will freeze constantly. That is a process mismatch, not a checker bug.

The 0.15 reject-rate delta is arbitrary. It will miss a slow squeeze of the input domain. It will also false-freeze noisy filters. Record the bound next to the lock so it cannot drift in a comment.

Two runners do not create independence if they share a cache volume, a seeded RNG hidden in a plugin, or the same mutable temp directory. Copy the lock files. Do not copy .pytest_cache.

Do not use this workflow when:

  • Tests are not allowed to block merge at all.
  • The agent is forbidden from touching tests, and review already covers generator edits.
  • Properties are not deterministic given a seed and a fixture tree.
  • You need a statistical flake model. This ledger is a gate, not a reliability estimator.

Hosted replay is the wrong tool for data that cannot leave your network. Run the second lane on an internal runner in that case. Free-model drafts are the wrong source of truth for oracles. They are a source of candidates.

What to keep in the repository

Commit the three lock files, the checker, and a one-page oracles/README.md that lists who may unfreeze. Keep production tests where they are. The oracle tree is a contract the agent does not own.

If you already have a second runner, point it at this checker. The lock files are the product. The host that replays them is not.

Top comments (0)