Green Suite, Three Unchecked Claims: Domain Fingerprints, Fixture Re-derivation, and a Freeze Budget

A passing test run on an agent patch proves one thing: the tests that executed, on the inputs they happened to use, with the fixtures as they currently exist, returned green. That is a narrow claim wearing a broad costume.

I stopped treating the green run as the verdict and started extracting three numbers from every agent patch before merge: whether the property domains moved, whether the fixtures can still be rebuilt from their recipes, and how much of the suite is frozen. Each number answers a question the pass/fail bit cannot.

This is not a general theory of correctness. It is a gate I built after watching an agent make a property test green by shrinking its input range from 512 characters to 8, and watching another hand-edit a fixture until a parser test agreed with it.

1. Pin the property domain as data, then fingerprint it

Property-based tests have a weakness that example-based tests do not: their input domain lives in code, in the same file as the assertion, and is therefore editable by the same patch that is supposed to be under test.

An agent asked to make a failing property test pass has three cheap moves. Tighten the generator, add a filter that rejects the awkward cases, or lower max_examples. All three keep the test green. None of them show up in a test-count diff, because the test is still collected and still passes.

The fix is to move the domain out of the test and into a declared spec that gets hashed.

# tests/domains.py
DOMAINS = {
    "parse_header": {
        "alphabet": {"min_codepoint": 32, "max_codepoint": 126},
        "min_size": 1,
        "max_size": 512,
        "edge_cases": ["empty_after_bom", "crlf_only", "duplicate_keys"],
    },
    "parse_header_unicode": {
        "alphabet": {"min_codepoint": 0x20, "max_codepoint": 0x2FFF},
        "min_size": 0,
        "max_size": 256,
        "edge_cases": [],
    },
}

# tools/domain_fingerprint.py
import hashlib, json
from tests.domains import DOMAINS

def fingerprint(domains: dict) -> str:
    blob = json.dumps(domains, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()

if __name__ == "__main__":
    print(fingerprint(DOMAINS))

The test then reads the spec instead of hardcoding it:

from hypothesis import given, settings, strategies as st
from tests.domains import DOMAINS

D = DOMAINS["parse_header"]
HEADER_TEXT = st.text(
    alphabet=st.characters(**D["alphabet"]),
    min_size=D["min_size"],
    max_size=D["max_size"],
)

@given(s=HEADER_TEXT)
@settings(max_examples=200, deadline=None)
def test_parse_header_preserves_raw(s):
    assert parse_header(s).raw == s

Commit the fingerprint into a file the gate reads, e.g. tests/domains.sha256. Any patch that changes DOMAINS changes the hash. That is not a failure by default; it is a stop sign that requires a human sentence in the PR: which domain moved and why.

What this buys you: a tightened generator becomes a visible event instead of an invisible edit. What it does not buy you: proof that the remaining domain is interesting. If the original domain was already trivial, fingerprinting just freezes your mistake. That is why the edge_cases list exists as a reviewable artifact rather than living in someone's head.

2. Split fixture drift into two failure modes

A fixture is evidence. Evidence that has been edited to match the code under test is not evidence anymore; it is a transcript of the bug.

Most fixture checks I have seen hash the file and compare. That catches hand-edits and misses the case I care about more: a fixture that still matches its hash but can no longer be regenerated, because the recipe rotted when a dependency moved. The file is now a relic nobody can re-derive, and the test depending on it is testing a museum piece.

So each fixture carries a recipe, and the check runs the recipe.

// tests/fixtures/invoices/2024_q3.provenance.json
{
  "target": "2024_q3.xml",
  "sha256": "<hex digest of the committed fixture>",
  "recipe": ["python", "tools/mk_invoice.py", "--seed", "7", "--out", "{out}"],
  "generated_at": "2026-10-09"
}

# tools/check_fixtures.py
import hashlib, json, pathlib, subprocess, tempfile

def sha(p: pathlib.Path) -> str:
    return hashlib.sha256(p.read_bytes()).hexdigest()

def check(root="tests/fixtures"):
    drift = []
    for prov in pathlib.Path(root).rglob("*.provenance.json"):
        meta = json.loads(prov.read_text())
        target = prov.parent / meta["target"]
        if not target.exists():
            drift.append((str(target), "missing-target")); continue
        if sha(target) != meta["sha256"]:
            drift.append((str(target), "edited-without-recipe")); continue
        with tempfile.TemporaryDirectory() as td:
            out = pathlib.Path(td) / meta["target"]
            cmd = [a.replace("{out}", str(out)) for a in meta["recipe"]]
            try:
                subprocess.run(cmd, check=True, capture_output=True, timeout=120)
            except subprocess.SubprocessError:
                drift.append((str(target), "recipe-failed")); continue
            if sha(out) != meta["sha256"]:
                drift.append((str(target), "recipe-no-longer-reproduces"))
    return drift

if __name__ == "__main__":
    for path, kind in check():
        print(f"{kind}\t{path}")

Three outcomes, three different responses. edited-without-recipe blocks: someone changed the evidence. recipe-failed blocks: the recipe itself is broken, which usually means the fixture generator depends on something that moved. recipe-no-longer-reproduces should warn loudly and open a ticket, because it frequently means a library version bump changed output formatting, which is exactly the kind of change you want to know about before it shows up as a mystery failure in staging.

Run this inside a network-disabled job. A fixture recipe that reaches out at generation time is not a fixture recipe.

3. Give the freeze a number, not a policy

Freezing a flaky test is a legitimate move and an easy place to hide. The difference between the two is accounting.

Measure flakes first, with tooling pytest already ships:

mkdir -p out
for i in 1 2 3 4 5; do
  pytest -q -p no:cacheprovider --junitxml="out/run$i.xml"
done

Parse the JUnit XML, map each test id to the set of statuses seen across runs, and any test with more than one status is a flake candidate for this patch:

# tools/flake_diff.py
import sys, xml.etree.ElementTree as ET
from collections import defaultdict

seen = defaultdict(set)
for path in sys.argv[1:]:
    for case in ET.parse(path).iter("testcase"):
        tid = f"{case.get('classname')}::{case.get('name')}"
        bad = case.find("failure") is not None or case.find("error") is not None
        seen[tid].add("fail" if bad else "pass")

for tid, statuses in sorted(seen.items()):
    if len(statuses) > 1:
        print(tid)

Five runs is a starting point, not a detector. It finds the flakes that fire often enough to matter for triage and misses the ones that fire at one in fifty. Treat the output as a candidate list, not a certificate.

The freeze ledger is a file with a budget:

{
  "budget_ratio": 0.02,
  "entries": [
    {
      "test": "tests/test_webhooks.py::test_retry_timing",
      "reason": "upstream timer jitter under parallel load",
      "owner": "@finley",
      "added": "2026-10-09",
      "review_by": "2026-10-23"
    }
  ]
}

Two derived numbers go into the gate: frozen_ratio = frozen entries divided by collected tests, and overdue = entries past review_by. The ratio blocks when it exceeds budget. Being past review does not block a merge by itself, but it does add a line to the PR body listing the owner, which is enough pressure to make the entry move.

Frozen tests should still execute. Keep them out of the blocking job with a marker and a deselect pattern, and run them in a second non-blocking job that reports without gating:

pytest -q -m "not frozen" --junitxml=out/blocking.xml
pytest -q -m "frozen"     --junitxml=out/observed.xml || true

If a frozen test passes a hundred times in the observed job, that is your signal to promote it back. Freezes that stop running are just deletions with extra steps.

4. Combine the three numbers into one verdict

Signal	Threshold	Effect
Domain fingerprint changed	any change	Block until a reviewer note names the moved domain
Fixture drift edited-without-recipe	>= 1	Block
Fixture drift recipe-no-longer-reproduces or recipe-failed	>= 1	Block, open a ticket in the same PR
New flake candidate from k runs	>= 1	Block until fixed or entered in the ledger
frozen_ratio	> budget_ratio	Block
overdue entries	>= 1	Non-blocking, listed in PR body

# tools/gate.py (illustrative)
def verdict(domains_moved, fixture_drift, new_flakes, frozen_ratio, budget):
    blocking = []
    if domains_moved: blocking.append("domain-fingerprint")
    if any(k == "edited-without-recipe" for _, k in fixture_drift):
        blocking.append("fixture-edited")
    if any(k in ("recipe-failed", "recipe-no-longer-reproduces") for _, k in fixture_drift):
        blocking.append("fixture-recipe")
    if new_flakes: blocking.append("new-flake")
    if frozen_ratio > budget: blocking.append("freeze-budget")
    return "block" if blocking else "pass", blocking

These thresholds — five runs, a 2% freeze budget, a two-week review window — are the numbers I picked as starting points. They are policy, not findings. The gate's value is that the three signals get compared against the same values on every patch, so drift is visible as a trend rather than as an argument.

Where the agent work fits

The slow part of this workflow is authoring the DOMAINS spec and the fixture recipes for coverage that never had them. That drafting is a reasonable thing to hand to a model, and it is also the part I now route through MonkeyCode's free model access for first drafts, then review line by line before anything is committed to tests/domains.py. The repeated suite runs go to MonkeyCode's free server option so the five-run sweep does not occupy my machine while I work on something else.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The gate itself has no dependency on any model or host. If you run it entirely locally with hand-written specs, every section above still applies.

What this gate does not do

It does not verify that a property is worth asserting. A tautological property with a perfectly fingerprinted domain is still a tautology; that check belongs earlier in review.
It does not detect flakes that need more than a handful of runs to surface, and five runs will miss most of them.
It does not distinguish a legitimate domain narrowing from a dishonest one — it only forces the question to be asked in writing.
It does not help if fixtures are generated inline in test bodies, because there is nothing stable to hash. That code has to be extracted first, and that is real work.

Skip this approach if your suite runs in under ten seconds, if you have no property-based tests, or if your team has no shared ownership of the frozen ledger — a budget nobody reconciles becomes a licence to skip.

The habit worth keeping

The specific files matter less than the sequence: fingerprint what the test is allowed to explore, prove the fixtures can be rebuilt, count what is frozen. Three numbers, one verdict, written down before merge.

A second reviewer catches what the agent did not understand. A gate catches what the agent understood perfectly well.

If you want to try the drafting half of this, start with one property test's domain spec and one fixture recipe — the free model access is enough for that, and the gate will tell you honestly whether the draft moved anything it should not have.