DEV Community: Finley Zhou

Rerunning Agent Patch Tests Until Green Is Not Evidence — Budget the Flakes Instead

Finley Zhou — Thu, 15 Oct 2026 11:50:51 +0000

Rerunning Agent Patch Tests Until Green Is Not Evidence — Budget the Flakes Instead

A green rerun is a coin flip with better marketing. If your merge gate for an agent-generated patch is "run the suite, retry until it passes," you are not measuring the patch — you are measuring how many attempts your CI budget allows.

The fix is not "more tests." It is three tiers with three different jobs: property checks that survive a rewrite, fixtures pinned to a digest, and a repeat budget that turns a suspected flake into an estimated rate with an upper bound. A quarantine entry is the only escape hatch, and it expires.

Everything below runs locally with pytest and the standard library. Total setup: one extra file, no services.

The arithmetic that kills rerun-to-green

Suppose a candidate patch introduces a real defect, and one test detects it on 15% of runs (a plausible number for anything touching timing, caches, or model output). An "any pass wins" policy with k attempts misses the bug with probability (1 - 0.15)^k.

Attempts (`k`)	Probability you still ship the bug
1	85.0%
3	61.4%
5	44.4%
10	19.7%
20	3.9%

This is arithmetic, not a benchmark I ran. The point stands anyway: retries convert a detectable defect into an undetectable one, and the conversion rate is set by the flakiest test in the selected set, not by the patch.

Three tiers, three different jobs

Each tier answers a different question, and mixing them is why gates get unreliable.

Tier 0 — properties. Does the patch violate an invariant? Deterministic, cheap, runs once.
Tier 1 — pinned fixtures. Did observable output change? Deterministic, runs once, fails on any diff.
Tier 2 — repeat budget. Is this test stable enough to be treated as evidence at all? Non-deterministic, runs n times.

Only tier 2 may be quarantined. Tiers 0 and 1 have no flake concept: a property failure or a fixture diff is a signal, full stop.

Step 1 — Classify the patch before you run anything

Start with the shape of the diff, not the test names.

git diff --stat origin/main...HEAD
git diff --name-only origin/main...HEAD | grep -E '\.(py|ts|go)$' | xargs -r git diff -U0 -- | grep -E '^\+' | grep -cE 'await|thread|lock|time\.|random|retry'

That second count is crude but useful: concurrent or time-dependent lines pull the patch into a stricter row of the table below.

Patch class	Example	Tier 0	Tier 1	Tier 2 policy
Docs / comments	README, docstring	none	diff digest only	skip
Dependency bump	lockfile, pin	import smoke	re-run once	30 repeats, no quarantine
Pure function	parser, formatter	required	digest required	30 repeats (≤10% at 95%)
IO boundary	HTTP client, DB layer	required + contract	digest + recorded transcript	60 repeats (≤6%)
Concurrency / timing	queues, retries, backoff	required + clock injection	digest with injected clock	200 repeats (≤2%)
Model or prompt change	the agent patch itself	required + seed determinism	digest of the sampler output	200 repeats, quarantine by default

The repeat counts are not vibes. They come from the rule of three: with zero failures in n runs, the 95% upper bound on the true failure rate is about 3/n. n=30 gets you ≤10%, n=60 ≤5%, n=200 ≤1.5%.

Step 2 — Property checks that survive a rewrite

Agent patches often rewrite a function's interior while the intent stays the same. A property test asserts the intent.

# tests/test_properties.py
from hypothesis import given, settings, strategies as st
from app.normalize import normalize_event

ALLOWED_KEYS = {"id", "kind", "at", "payload"}

given_event = st.fixed_dictionaries({
    "id": st.text(min_size=1, max_size=64),
    "kind": st.sampled_from(["click", "view", "error"]),
    "at": st.integers(min_value=0, max_value=2**31),
    "payload": st.dictionaries(st.text(max_size=8), st.integers()),
})

@settings(max_examples=300, deadline=None)
@given(given_event)
def test_idempotent(event):
    assert normalize_event(normalize_event(event)) == normalize_event(event)

@settings(max_examples=100, deadline=None)
@given(given_event)
def test_schema_is_closed(event):
    assert set(normalize_event(event)) <= ALLOWED_KEYS

@settings(max_examples=100, deadline=None)
@given(given_event)
def test_ordering_independent(event):
    once = normalize_event(event)
    twice = normalize_event(dict(reversed(list(event.items()))))
    assert once == twice

Run these with PYTHONHASHSEED=0 pytest tests/test_properties.py -q. If a patch can only pass by weakening a property, that is the review conversation — not a retry.

Step 3 — Pin fixtures to a digest

Fixture diffs are the cheapest way to catch a silent behavior change, provided the pin is part of the diff.

# regenerate, then inspect the diff before committing the pin
pytest tests/test_contract.py -q --update-pins
# expected: tests/pins/ingest_contract.json changes AND the code diff explains why

# tests/pinned.py
import hashlib, json, pathlib

PIN_DIR = pathlib.Path("tests/pins")

def pin(name: str, payload: dict) -> None:
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(blob.encode()).hexdigest()
    path = PIN_DIR / f"{name}.json"
    expected = json.loads(path.read_text())["digest"]
    assert digest == expected, (
        f"{name} changed: {digest[:12]} != {expected[:12]}."
        " Update the pin only if this diff is the intended behavior change."
    )

Example command: pytest tests/test_contract.py -q. Exit code 1 plus a digest mismatch is a signal, not a flake. Never quarantine tier 1.

Step 4 — Measure the flake rate, do not guess it

This is the only file you need to add for tier 2.

# flake_budget.py
import json, os, subprocess, sys

BUDGET = {"max_upper_rate": 0.02, "min_runs": 200, "max_quarantined_share": 0.05}
LEDGER = "flake_ledger.json"


def wilson_upper(fails: int, n: int, z: float = 1.96) -> float:
    """95% upper bound for a binomial proportion."""
    if n == 0:
        return 1.0
    p = fails / n
    denom = 1 + z * z / n
    center = p + z * z / (2 * n)
    margin = z * ((p * (1 - p) / n + z * z / (4 * n * n)) ** 0.5)
    return min(1.0, (center + margin) / denom)


def run_once(nodeid: str, seed: int) -> bool:
    env = {**os.environ, "PATCH_SEED": str(seed), "PYTHONHASHSEED": "0"}
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", nodeid, "-q", "--no-header", "-p", "no:cacheprovider"],
        capture_output=True, text=True, env=env,
    )
    return proc.returncode == 0


def measure(nodeid: str, n: int, base_seed: int = 9137) -> dict:
    fails = sum(0 if run_once(nodeid, base_seed + i) else 1 for i in range(n))
    return {
        "nodeid": nodeid, "n": n, "fails": fails,
        "upper95": round(wilson_upper(fails, n), 4),
        "seed_base": base_seed,
    }


def load_ledger() -> dict:
    try:
        return json.loads(open(LEDGER).read())
    except FileNotFoundError:
        return {}


def gate(results: list, ledger: dict, today: str) -> int:
    exit_code = 0
    for r in results:
        entry = ledger.get(r["nodeid"])
        expired = entry is not None and entry["expires"] < today
        within_budget = r["upper95"] <= BUDGET["max_upper_rate"] and r["n"] >= BUDGET["min_runs"]
        if expired:
            print(f"FAIL {r['nodeid']}: quarantine expired on {entry['expires']}")
            exit_code = 1
        elif within_budget:
            print(f"OK   {r['nodeid']}: upper95={r['upper95']} over {r['n']} runs")
        elif entry is None:
            print(f"FAIL {r['nodeid']}: upper95={r['upper95']} > {BUDGET['max_upper_rate']}, no quarantine entry")
            exit_code = 1
        else:
            print(f"WARN {r['nodeid']}: quarantined until {entry['expires']} (owner {entry['owner']})")
    return exit_code


if __name__ == "__main__":
    targets = [line.strip() for line in open("flake_targets.txt") if line.strip()]
    runs = int(os.environ.get("FLAKE_RUNS", BUDGET["min_runs"]))
    results = [measure(t, runs) for t in targets]
    json.dump(results, open("flake-report.json", "w"), indent=2)
    sys.exit(gate(results, load_ledger(), os.environ["TODAY"]))

Run it:

printf 'tests/test_ingest.py::test_stream_backpressure\n' > flake_targets.txt
FLAKE_RUNS=200 TODAY=2026-10-15 python flake_budget.py; echo "exit=$?"

A 2-second test repeated 200 times is about 7 minutes of wall clock. That is the real cost of treating a timing-sensitive test as evidence instead of a rumor.

Step 5 — The quarantine ledger expires, always

A quarantine entry is a debt with a due date and an owner.

{
  "tests/test_ingest.py::test_stream_backpressure": {
    "class": "flake",
    "seed_base": 9137,
    "n": 200,
    "fails": 5,
    "upper95": 0.0537,
    "owner": "@finley",
    "expires": "2026-11-14",
    "evidence": "ci/run-88213"
  }
}

Three rules keep the ledger from becoming a graveyard:

Expiry is mandatory and enforced in code, not in the PR description.
The quarantined share of the selected test set is capped (5% here); crossing the cap fails the gate.
Tier 0 and tier 1 node IDs are rejected if they appear in the ledger at all.

Where a free-tier runner actually helps

The step-4 harness is CPU-bound and boring: 200 serial repeats per target, run on every candidate patch. That is exactly the workload that fits a free server option — you do not need a persistent runner for a nightly flake census. MonkeyCode offers free model access and a free server option as of this writing, which is how I would generate patch candidates and then run the repeat matrix without provisioning a box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest caveats. I have not benchmarked its models against anything, and free-tier availability and limits change — read the current terms before you build a gate on top of it. Also keep the harness runner-independent: the flake_budget.py above shells out to pytest and does not import any vendor SDK, so swapping the runner is a one-line change.

If you want to try the matrix without provisioning first, the free server option is a reasonable place to start; the reproduction steps above are the part that matters.

Limitations and who should not use this

Statistical gates are not proofs. upper95 <= 0.02 over 200 runs means "we did not observe frequent failure," not "this test is deterministic." A 1-in-500 flake passes a 200-run budget.
Serial repeats are slow. 200 runs × 5 flaky-prone targets × 2 seconds is ~33 minutes. If your CI budget cannot absorb that, cap tier 2 at 30 repeats and accept the weaker claim — do not silently skip the measurement.
Skip this approach entirely if your suite is already deterministic and fast, if your environment forbids quarantine of any kind (regulated release processes), or if your tests mutate shared production-adjacent state — repeating them 200 times multiplies the blast radius, not the signal.
Do not apply it to tier 0 or tier 1. Quarantining a property check converts a defect into a policy.

Copy-paste checklist

Classify the patch with the diff-count command and pick the tier-2 row from the decision table.
Add or extend tier-0 property checks; confirm they fail on a deliberately broken candidate before you trust them.
Verify tier-1 pins changed only where the code diff explains the change.
Run FLAKE_RUNS=<n> TODAY=<date> python flake_budget.py and archive flake-report.json next to the patch.
For every non-compliant target, write a ledger entry with an owner and an expiry — or fix the test.
Re-run the census after the expiry date; an expired entry is a failed gate, not a warning.

Reachability Receipts: Prove an Agent Patch's Green Build Came From Tests That Touched It

Finley Zhou — Thu, 15 Oct 2026 10:50:05 +0000

A green build is a fact about your test suite, not about the patch. If no executed test line intersects the lines the patch changed, the pipeline measured the repository's steady state and reported it as approval. The receipt format below records that intersection explicitly, and it closes the most common abuse of a flaky-test freeze: freezing a test that never touched the patch and then counting the green result as evidence.

The Three Fields a Receipt Needs

Every merge decision in this workflow reduces to three sets.

Changed lines — the exact new-side line ranges of the patch, not a diffstat.
Per-test executed lines — which lines each test actually ran.
The freeze set — tests currently quarantined as flaky, with an expiry.

Everything else is bookkeeping. The verdict comes from intersecting the first two, then asking the third whether that intersection is trustworthy.

Step 1: Derive Changed Line Ranges, Not Counts

git diff --stat counts insertions and hides where they landed. A patch can add forty lines and still touch none of the functions your properties cover. Use zero context so each hunk is exactly the changed region.

git fetch origin main --depth=1
git diff -U0 --no-color origin/main...HEAD > patch.diff
grep -c '^@@' patch.diff

The third command tells you how many changed regions you are dealing with. If that number is zero on a non-empty patch, your merge base is wrong and the rest of this workflow is measuring nothing.

Step 2: Collect Per-Test Line Hits

Line-level execution is the only signal here that cannot be faked by a passing suite. coverage.py records which test covered which line when dynamic contexts are enabled.

# .coveragerc
[run]
dynamic_context = test_function

coverage run -m pytest -q
coverage json --show-contexts -o coverage-contexts.json
python tools/contexts_to_trace.py coverage-contexts.json > trace.jsonl

A minimal adapter from that JSON to the JSONL the receipt consumes:

# tools/contexts_to_trace.py
import json, sys

doc = json.load(open(sys.argv[1]))
by_test = {}
for path, filedata in doc['files'].items():
    for ln, contexts in (filedata.get('contexts') or {}).items():
        for ctx in contexts:
            by_test.setdefault(ctx, set()).add((path, int(ln)))

for test_id, hits in sorted(by_test.items()):
    print(json.dumps({'test_id': test_id, 'hits': sorted(map(list, hits)), 'status': 'unknown'}))

Two caveats matter. Contexts prove coverage, not outcome, so the status field above is a placeholder that must be merged from your JUnit XML before the receipt is trusted. And --show-contexts depends on your installed coverage version; check coverage --version on the runner before you build a gate on it. If your runner cannot emit per-test hits at all, stop here rather than approximating.

Step 3: The Receipt Script

This is a reference implementation. I have not executed it against your repository, and the sample output later in this article is fabricated to show the shape only.

#!/usr/bin/env python3
# tools/reachability_receipt.py
import argparse, hashlib, json, re, sys
from pathlib import Path

HUNK = re.compile(r'^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@')

def changed_ranges(diff_text):
    ranges, current = {}, None
    for line in diff_text.splitlines():
        if line.startswith('+++ '):
            path = line[4:].strip()
            path = path[2:] if path.startswith('b/') else path
            current = None if path == '/dev/null' else path
            if current:
                ranges.setdefault(current, [])
            continue
        m = HUNK.match(line)
        if m and current:
            start = int(m.group(1))
            count = int(m.group(2) or 1)
            if count:
                ranges[current].append((start, start + count - 1))
    return ranges

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--diff', required=True)
    ap.add_argument('--trace', required=True)
    ap.add_argument('--freeze')
    ap.add_argument('--require-property', action='append', default=[])
    args = ap.parse_args()

    ranges = changed_ranges(Path(args.diff).read_text())
    changed = {(f, ln) for f, rs in ranges.items() for a, b in rs for ln in range(a, b + 1)}

    freeze = {}
    if args.freeze:
        raw = json.loads(Path(args.freeze).read_text())
        freeze = raw if isinstance(raw, dict) else {t: {} for t in raw}

    tests = []
    for raw_line in Path(args.trace).read_text().splitlines():
        if not raw_line.strip():
            continue
        rec = json.loads(raw_line)
        test_id = rec['test_id']
        hits = {(f, int(ln)) for f, ln in rec.get('hits', [])}
        touched = hits & changed
        if test_id in freeze:
            verdict = 'frozen-blocking' if touched else 'frozen-irrelevant'
        elif touched:
            verdict = 'evidence'
        else:
            verdict = 'no-reach'
        tests.append({'test_id': test_id, 'status': rec.get('status', 'unknown'),
                      'touched': len(touched), 'verdict': verdict})

    properties = []
    for name in args.require_property:
        support = [t['test_id'] for t in tests
                   if t['verdict'] == 'evidence' and name in t['test_id'] and t['status'] == 'pass']
        properties.append({'property': name, 'supporting_tests': support, 'ok': bool(support)})

    payload = {
        'changed_lines': len(changed),
        'files': {f: sorted(rs) for f, rs in sorted(ranges.items())},
        'tests': sorted(tests, key=lambda t: t['test_id']),
        'properties': properties,
    }
    blob = json.dumps(payload, sort_keys=True, separators=(',', ':')).encode()
    receipt = {'payload': payload, 'digest': hashlib.sha256(blob).hexdigest()}
    json.dump(receipt, sys.stdout, indent=2, sort_keys=True)
    sys.stdout.write('\n')

    blocked = any(t['verdict'] == 'frozen-blocking' for t in tests)
    ok = all(p['ok'] for p in properties) and not blocked and len(changed) > 0
    return 0 if ok else 1

if __name__ == '__main__':
    sys.exit(main())

The digest covers a canonicalised payload, so two runs over the same diff, trace, and freeze produce the same hash. That is what lets a reviewer ask a specific question later: which evidence did this merge actually rest on?

Step 4: Read the Verdicts Against a Fixed Table

python tools/reachability_receipt.py \
  --diff patch.diff \
  --trace trace.jsonl \
  --freeze freeze.json \
  --require-property proration_total \
  --require-property refund_idempotency > receipt.json
echo $?

Verdict	Meaning	Merge action
`evidence`	Executed a changed line and reported a result	Counts toward the property
`no-reach`	Passed, touched nothing the patch changed	Ignore; never count as support
`frozen-blocking`	In the freeze set and touched changed lines	Do not merge; evidence is contaminated
`frozen-irrelevant`	In the freeze set, touched nothing changed	Keep frozen; it must not gate this patch

The fourth row is the one teams usually get wrong. A flaky test that never touched the patch should stay frozen without blocking anything, because freezing it was a scheduling decision, not a statement about this change. The third row is the opposite: once a known-flaky test intersects the patch, no one can separate the flake from the regression on those lines.

An illustrative receipt fragment, fabricated for shape:

{
  "payload": {
    "changed_lines": 12,
    "properties": [
      {"ok": true, "property": "proration_total", "supporting_tests": ["tests/test_billing.py::test_proration_total"]},
      {"ok": false, "property": "refund_idempotency", "supporting_tests": []}
    ],
    "tests": [
      {"status": "pass", "test_id": "tests/test_billing.py::test_proration_total", "touched": 4, "verdict": "evidence"},
      {"status": "pass", "test_id": "tests/test_misc.py::test_config_parses", "touched": 0, "verdict": "no-reach"},
      {"status": "failed", "test_id": "tests/test_rates.py::test_fx_cache", "touched": 2, "verdict": "frozen-blocking"}
    ]
  },
  "digest": "0000000000000000000000000000000000000000000000000000000000000000"
}

That exit code is non-zero for two independent reasons: a declared property has no supporting test, and a frozen test touched the patch.

Step 5: Self-Test the Gate Before You Trust It

A gate that rejects nothing is decoration. Run these three cases against a scratch branch before wiring the script into CI.

Reformat-only patch. Whitespace changes across one file, no behaviour change. Expected: changed_lines above zero, evidence count zero, exit non-zero.
Single-line behaviour patch. One condition inverted in a function a property covers. Expected: at least one evidence test, exit zero when the property is declared.
Freeze scoping. Add the flaky test to freeze.json, run twice — once on a patch that touches its lines, once on a patch that does not. Expected: frozen-blocking appears only in the first run.

Freeze entries should carry an expiry so quarantine does not become permanent:

{"tests/test_rates.py::test_fx_cache": {"class": "flake", "expires": "2026-11-01"}}

The expiry date is not enforced by the script above. Reject any freeze entry whose expiry has passed in a separate check, otherwise the freeze set becomes an allowlist for failures.

Limitations and Who Should Not Use This

Execution is not falsifiability. A line hit means the line ran, not that an assertion could have failed. Pair this with mutation testing or a tautology classification step; a reachability receipt alone will happily certify an assertion-free test.
Rename and move handling. With -U0, moved code can appear as a full rewrite, inflating the changed set and producing false evidence verdicts.
Cost. Dynamic contexts slow the suite and produce large trace files. On a large monorepo, restrict tracing to the packages the patch touches.
Language coverage. This depends on a line-level tracer. Compiled or out-of-process code needs a different mechanism, and none of the script above applies.
Do not use it if your repository has one meaningful test file, if your runner cannot emit per-test hits, or if the receipt is destined to become a checkbox nobody reads. Manual review of a handful of tests is cheaper and more honest.

Where MonkeyCode Fits

One part of this workflow is genuinely tedious: naming the properties for a patch you did not write. The account operator states that MonkeyCode offers free model access, which I used as a drafting step — propose one property name per changed function, then a human accepts, renames, or deletes each one. The --require-property list must be human-approved before it becomes a gate; a model-generated property that always passes is exactly the tautology problem this workflow exists to catch.

The receipt job is also a good fit for a free server option, per the operator, because it is short, I/O-heavy, and produces a JSON artifact rather than a service. Running it off the developer machine keeps the triage out of the local loop and makes the digest reproducible from a clean checkout.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Remove every product mention above and the workflow still stands. That is the test I apply before writing this section at all.

Closing

The receipt does not make a patch correct, and it does not replace property design. It does make one failure mode impossible to paper over: a green build whose tests never reached the changed lines. Start with a single patch, produce one no-reach verdict, and decide whether you want that verdict gating your merges. If you want to prototype the triage step on a free server before committing CI minutes, the receipt script is small enough to run there first.

Only a Flake-Class Exit May Open the Lease

Finley Zhou — Tue, 13 Oct 2026 13:49:31 +0000

An agent-patch failure should leave the pipeline as one class, not a blended skip. The classes are a property violation, a fixture miss, and a flake candidate. Only the flake candidate may become a time-boxed freeze, and only after the property verdict on that same pin is pass.

Anything broader hides a regression inside a waiver. The workflow below is a gate: a decision table, a six-step run order, and a classifier that refuses a lease when the inputs do not match. The code is a proposal. It was not executed against a production suite for this article.

What each class is allowed to do

A property violation means an invariant the patch was supposed to preserve did not hold. A typical case is a public function that must still reject a negative quantity, or a record that must still round-trip through its serializer. That class blocks merge. Frequency does not matter. A check that fails once and a check that fails on every run are both ineligible for a freeze.

A fixture miss means the run is not comparable to the pin. The digest, the schema version, or the oracle file diverged. The patch may be innocent. The run is still invalid. Re-pin under review, or reject the job. Do not freeze a miss, and do not update the oracle inside the same job that is asking for a waiver.

A flake candidate is the narrow remainder. The property verdict passed. The pin matched. A bounded replay disagreed with the original red result. That pattern can justify a lease with an owner and an expiry. The lease does not delete the test. It does not transfer to the next failure.

Decision table

Read the table as a gate. If two rows seem to match, take the stricter exit. Missing fields are incomplete, not flake.

Property verdict	Fixture pin	Replay witness	Exit	Freeze allowed
fail	match	any	2 property	no
pass or fail	mismatch	any	3 fixture	no
pass	match	disagreement inside a full budget	4 flake	yes, with expiry and owner
pass	match	red on every replay	5 stable red	no
missing	any	any	1 incomplete	no

Exit 5 is the row that gets skipped in review. A failure that returns on every replay is not a flake. It is a stable red. A passing property does not waive a different test that is stably red. Classify that test on its own row. Borrowing the property pass is how a lease swallows an unrelated regression.

Workflow

Pin the fixture before the agent patch is applied. Record the digest, the schema version, and the command that produced the pin. The patch job refuses to start when that record is absent. This step creates a comparison point. It does not by itself sign a verdict.
Run property checks against that pin. Keep the predicate source in the repository, not in a model transcript. A failing property exits 2 and stops the pipeline. Do not start a replay in order to talk a property failure into a flake.
If a non-property test fails, re-check the pin. Mismatch exits 3. Stop. A fixture miss is an input error, even when the diff looks small.
Replay the failing test on a host that is not the session that proposed the patch. This proposal uses a budget of 5 attempts. The number is a policy choice, not a measured optimum. Store each attempt id, exit code, and duration in seconds. A short run is incomplete. Fill the budget before calling the classifier.
Emit a freeze record only when the classifier returns flake, the property verdict is pass, the pin status is match, the owner is non-empty, and the expiry is in the future. The replay host records attempts. It does not write the lease. A reviewer, or a separate policy job, accepts the record before merge.
Leave the test in a canary lane for the life of the lease. On expiry, the record is invalid. The next red run starts again at step 3. There is no automatic renewal.

Proposed classifier

The snippets assume Python 3.10 or newer for the type-union syntax. They are unexecuted examples. The assertions specify the gate. They are not a reported measurement from a live suite.

from dataclasses import dataclass
from datetime import datetime, timezone
from enum import Enum

class Exit(Enum):
    INCOMPLETE = 1
    PROPERTY = 2
    FIXTURE = 3
    FLAKE = 4
    STABLE_RED = 5

@dataclass(frozen=True)
class Attempt:
    attempt_id: str
    passed: bool

@dataclass(frozen=True)
class Report:
    property_verdict: str | None
    pin_status: str | None
    origin_passed: bool | None
    replays: tuple[Attempt, ...]
    now: datetime
    expires_at: datetime
    owner: str

def classify(report: Report, budget: int = 5) -> Exit:
    if report.property_verdict not in {'pass', 'fail'}:
        return Exit.INCOMPLETE
    if report.pin_status not in {'match', 'mismatch'}:
        return Exit.INCOMPLETE
    if report.pin_status != 'match':
        return Exit.FIXTURE
    if report.property_verdict != 'pass':
        return Exit.PROPERTY
    if report.origin_passed is None or len(report.replays) != budget:
        return Exit.INCOMPLETE
    greens = sum(1 for item in report.replays if item.passed)
    if report.origin_passed is False and greens == 0:
        return Exit.STABLE_RED
    if report.origin_passed is False and greens > 0:
        return Exit.FLAKE
    return Exit.INCOMPLETE

def freeze_allowed(report: Report, exit_code: Exit) -> bool:
    if exit_code is not Exit.FLAKE:
        return False
    if not report.owner.strip():
        return False
    if report.expires_at <= report.now:
        return False
    return True

One specification check covers the row reviewers most often want to skip. Property failure is decided before replay length, so a short replay cannot launder it.

def test_property_fail_cannot_freeze():
    now = datetime(2026, 10, 13, tzinfo=timezone.utc)
    report = Report(
        property_verdict='fail',
        pin_status='match',
        origin_passed=False,
        replays=(Attempt('r1', True), Attempt('r2', False)),
        now=now,
        expires_at=datetime(2026, 10, 20, tzinfo=timezone.utc),
        owner='ci-bot',
    )
    exit_code = classify(report)
    assert exit_code is Exit.PROPERTY
    assert freeze_allowed(report, exit_code) is False

Change the property verdict to pass, fill five replays, and keep at least one green attempt. The class becomes flake. freeze_allowed still depends on owner and expiry. Clear the owner and the lease closes again. Class and permission are not the same decision. That is why the proposal uses two functions.

An illustration of a property, also unexecuted, shows the kind of predicate the verdict refers to. It is not a suite. It is not evidence that any agent patch passed it.

def qty_must_reject_negative(place_order):
    result = place_order(qty=-1)
    if result.rejected is not True:
        raise AssertionError('negative quantity was accepted')

If this function fails, the exit is 2. Replays of an unrelated integration test do not change that.

Commands that keep the classes apart

Run the property file and the suspect test as separate processes. A single log line should not be the only place both verdicts exist.

python -m pytest tests/properties -q --tb=line
echo property_exit=$?

python -m pytest tests/integration/test_order_replay.py -q --tb=line
echo suspect_exit=$?

python -m agent_gate.classify \
  --property-verdict pass \
  --pin-status match \
  --origin-passed false \
  --replay green,green,red,green,green \
  --owner ci-bot \
  --expires-at 2026-10-20T00:00:00Z

The third command is a proposed CLI shape for the functions above. It should print a line such as exit=4 class=flake freeze=eligible, or exit non-zero when the lease is refused. If that module does not exist yet, call classify() from a one-file script. Do not encode the decision in a shell conditional that only inspects the suspect exit code.

A practical debug pass on a red agent patch is then mechanical. Read the property exit first. If it is non-zero, stop and diff the predicate against the production change. If the property exit is zero, compare the fixture digest with the pin. Only a match proceeds to the five-attempt replay. Paste attempt ids into the report. Run the classifier. If it returns stable red, file a product or test fix. If it returns flake, the lease is eligible for review, not merged by the replay host.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator supplied two availability claims for this workflow: free model access, and a free server option. No model name, quota, hardware shape, duration, or benchmark is stated here, because none was verified for this draft. Confirm the current terms before you depend on either. Neither claim is a promise that the option remains available.

Use free model access in one place only: drafting candidate property predicates from the diff. The draft is untrusted input. A reviewer, or a checker that does not share the patch-authoring transcript, accepts the predicate into the repository before CI can treat it as a verdict. Do not ask the model to label a red test as flaky. That label comes from classify() and the replay witness.

Use the free server option as the replay host in step 4 only when the job can run apart from the session that proposed the patch. Separation is the requirement. The vendor is not. If that option cannot run the suspect test alone, use another host you already control. The classifier needs a witness outside the authoring session. It does not need a particular machine class.

The replay host still does not open the freeze. It appends attempt rows. Eligibility stays in freeze_allowed, and a person or a policy job still accepts the record.

Product options do not replace the gate. If the three exit classes are already separate fields in your repo, try the free model access and the free server option only as the predicate-draft step and the replay host. Read the current access notes first, and keep classify() in the repository either way.

Limitations

The budget of 5 will miss rare flakes. It will also call some environment faults flakes when the fault happens to disappear inside the window. Raising the budget costs latency. Lowering it makes exit 4 too easy. Write the budget into the lease so a later reader can see the policy that was applied.

A green replay does not prove nondeterminism. It proves the failure did not reproduce under the recorded attempts. Clocks, network calls, and order dependencies can still hide. The canary lane is what keeps that residual visible after the lease starts.

Property checks can be hollow. A predicate that asserts a constant, or that rewrites the oracle to match the patch, will pass, and this classifier will trust that pass. Review the predicate diff with the same suspicion as the production diff. This workflow does not detect a captured oracle.

Expiry is not enforcement unless CI loads the record and rejects expired leases. A pull-request comment is not a lease. A green canary is not a renewal. Requiring the replay length to equal the budget also means a loop that stops on the first green cannot open a lease. That is intentional. One lucky pass is not a witness.

Timestamps in the examples use 2026-10-13 only as a fixed clock for the specification. They are not a claim about a measured run.

Who should not use this

Skip the workflow when you have no pinned fixture, no property file, and no second place to replay. The classifier will return incomplete, which is honest, but it will not strengthen a suite that has nothing to compare.

Skip it when the failing check is a release gate for safety, billing, or migration correctness and policy forbids waivers. A flake lease is a waiver. Some suites should stay red until a human fixes the test or the product.

Skip it when the same agent writes the patch, the property, and the freeze justification. The separation is the control. Collapsing those roles removes the reason to trust exit 4.

What the lease must contain

A valid record stays small. Class, property verdict, pin status, attempt ids, budget, owner, expiry, and the canary job name are enough. Do not store a model transcript as the witness. Store the commands and the exit codes.

class=flake
property=pass
pin=match sha256:<recorded-in-ci>
origin=fail
replays=5 budget=5 greens=4
owner=ci-bot
expires_at=2026-10-20T00:00:00Z
canary=tests/integration/test_order_replay.py

The digest line is a placeholder shape, not a real hash. If any required line is missing, the record is incomplete. Incomplete does not merge as a freeze. The job merges when the suspect test is green without a lease, or it does not merge.

Count the class before you count the retries. Exit 4 is a narrow door. Exits 2, 3, and 5 exist so that door cannot be used for everything else.

Judge Agent Patches Against a Fixture Pin They Did Not Write

Finley Zhou — Tue, 13 Oct 2026 10:29:28 +0000

An agent patch is admissible only when a fixture pin taken before generation still matches after the patch lands. Property checks then run on that pinned tree. A flake freeze is not a bypass around either result.

A digest change is a fixture breach. Further repeats do not reclassify it. Mixed pass and fail results can be noise candidates only while the pin matches, and even then this gate does not write a freeze.

That order is the strategy. Generation proposes a diff. A separate runner adjudicates it. The model does not grade the files it was allowed to edit.

Why a green rerun is the wrong signal

Shared workspaces hide a specific failure. The step that edits production code can also drop a fixture row, loosen a numeric bound, or replace a file with a smaller lookalike. The property passes because the input got weaker, not because the behavior got stronger.

Rerunning that command stays green. A policy that compares only exit codes never sees a disagreement, so it never opens a record. The miss is silent. A pin makes the same miss visible: the byte manifest changed, so the run is ineligible for accept and ineligible for freeze.

Treat the rest of this piece as a workflow proposal for repositories that already store durable fixtures and already expose property checks as a command with an exit code. No pass rate is claimed. The command examples and the script were not executed against a live model for this article.

What the pin stores

Store a sorted list of relative paths under one fixture root. For each file, store the byte length and a SHA-256 digest. A path that appears on only one side is a change, even when every surviving digest still matches.

Take the pin from the base revision, before the model is called. Write it outside the worktree the patch will touch. Do not refresh it because a commit message says the fixtures were normalized.

Branch names are not pins. Timestamps are not pins. A clean status after the agent amends fixtures is not a pin either. The manifest is the pin.

Decision table

Assign one class. Do not stack classes to make a run look milder.

Pin diff	Repeat exit codes	Class	Freeze written here
non-empty	not consulted	fixture_breach	no
empty	all zero	accept	no
empty	all non-zero	property_fail	no
empty	mixed zero and non-zero	noise_candidate	no

The freeze column is intentionally all no. A noise candidate may be handed to a separate flake policy, with the pin digest and the seed schedule attached. This gate only refuses fixture rewrites, and it refuses to treat a deterministic property failure as noise.

Replay budgets, witness shrinking, and expiry clocks stay in that other policy. Leaving them out keeps the pin decision auditable on its own. A freeze that cannot name the pin digest it was judged under did not come from this strategy.

Numbered workflow

1. Pin the base tree

Record the revision the agent may edit. Build the manifest from that revision's fixture root, then run the local classifier checks.

git rev-parse HEAD > .pin/base_rev
python pin_gate.py pin --root tests/fixtures --out .pin/manifest.json
python pin_gate.py logic-check

logic-check runs the classifier assertions in the reference script. It does not call a model. It does not measure your suite. It only shows that a non-empty diff outranks exit codes, which is the rule the rest of the gate depends on.

2. Generate without the pin mounted

Send the task and the base revision. State that paths under the fixture root are read-only for this job. Do not mount .pin/ into the generation workspace. Do not ask the generator to certify its own patch.

Operators with MonkeyCode free model access can confine this step to that endpoint. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The only product claims used here are free model access and a free server option. Model identity, quota, hardware, duration, and retention are unspecified, so the workflow must not depend on them.

3. Apply the patch in a detached worktree

git worktree add --detach /tmp/agent-judge "$(cat .pin/base_rev)"
git -C /tmp/agent-judge apply --check /tmp/candidate.patch
git -C /tmp/agent-judge apply /tmp/candidate.patch

If apply --check fails, stop. A rejected patch is a patch-format failure. It is not a flake, and it is not a property result. Do not hand-edit the diff so the check will pass and then pretend the agent produced that edit.

4. Judge on a runner that did not generate the diff

python pin_gate.py judge \
  --pin .pin/manifest.json \
  --root /tmp/agent-judge/tests/fixtures \
  --cwd /tmp/agent-judge \
  --repeat 5 \
  --seed 20261013 \
  --out .pin/verdict.json \
  --property-cmd python -m pytest -q tests/properties

--repeat 5 and --seed 20261013 are example CLI parameters, not a tuned threshold and not a reported experiment. The seed value is an arbitrary constant for the example. The script exports it as PIN_SEED. If the property command ignores that variable, the repeat loop does not control randomness, and mixed codes are not yet evidence of a flake.

A free server option fits this step when the runner can check out the base revision, apply the patch, and execute the property command. Keep the model off that job. If the server is ephemeral, copy manifest.json in at the start and copy verdict.json out before the machine is released. Do not assume the server stores either file after the job ends.

5. Act on one class

Fixture breach means read changes and stop. Do not schedule flake replays to confirm a digest mismatch. Property fail means keep the failing command, the seed, and the non-zero codes. A stable failure is a product or specification signal, not a candidate for silence.

Accept means the pin matched and every listed exit code was zero. It does not mean the property encodes the specification, and it does not mean the production diff is safe to merge. Noise candidate means archive the codes, the seed, and the pin digest for the separate flake policy.

The reference script sets freeze_allowed to false on every branch. A later job that only checks for a verdict file cannot mistake that file for a freeze record.

6. Prefer the verdict over the generator summary

If the model text says fixtures were unchanged, compare that sentence to changes. On any disagreement, quote the verdict JSON. The summary is not evidence.

If a free server is already in the job path, keep pin_gate.py judge on that runner and leave the model endpoint out of the job definition. That split is the useful part. A longer product tour is not required to apply the table.

Reference classifier

The module below is an unexecuted reference for the decision, not a measured harness. logic-check is the piece a reader can run without a model, a network, or a fixture corpus. judge shells out to --property-cmd with cwd pointed at the detached worktree, and it skips that shell-out when the manifest diff is non-empty.

#!/usr/bin/env python3
"""Unexecuted reference: pin a fixture root and class an agent patch.

Does not open a flake freeze. Does not shrink counterexamples.
logic-check is runnable without a model or a fixture corpus.
"""

from __future__ import annotations

import argparse
import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()


def build_manifest(root: Path) -> dict:
    if not root.is_dir():
        raise SystemExit(f"fixture root is not a directory: {root}")
    files = []
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        rel = path.relative_to(root).as_posix()
        files.append(
            {
                "path": rel,
                "bytes": path.stat().st_size,
                "sha256": sha256_file(path),
            }
        )
    return {"files": files}


def diff_manifests(before: dict, after: dict) -> list[dict]:
    old = {item["path"]: item for item in before.get("files", [])}
    new = {item["path"]: item for item in after.get("files", [])}
    changes = []
    for rel in sorted(set(old) | set(new)):
        if rel not in old:
            changes.append({"path": rel, "change": "added"})
        elif rel not in new:
            changes.append({"path": rel, "change": "removed"})
        elif old[rel]["sha256"] != new[rel]["sha256"]:
            changes.append(
                {
                    "path": rel,
                    "change": "modified",
                    "before": old[rel]["sha256"],
                    "after": new[rel]["sha256"],
                }
            )
    return changes


def classify(changes: list, codes: list[int]) -> str:
    if changes:
        return "fixture_breach"
    if not codes:
        raise ValueError("repeat must be >= 1")
    if all(code == 0 for code in codes):
        return "accept"
    if all(code != 0 for code in codes):
        return "property_fail"
    return "noise_candidate"


def run_property(cmd: list[str], cwd: Path, seed: int) -> int:
    env = os.environ.copy()
    env["PIN_SEED"] = str(seed)
    completed = subprocess.run(cmd, cwd=cwd, env=env)
    return completed.returncode


def judge(
    pin_doc: dict,
    root: Path,
    cwd: Path,
    cmd: list[str],
    repeat: int,
    seed: int,
) -> dict:
    if repeat < 1:
        raise SystemExit("repeat must be >= 1")
    changes = diff_manifests(pin_doc, build_manifest(root))
    if changes:
        return {
            "class": "fixture_breach",
            "changes": changes,
            "codes": [],
            "freeze_allowed": False,
        }
    codes = [run_property(cmd, cwd, seed + offset) for offset in range(repeat)]
    return {
        "class": classify([], codes),
        "changes": [],
        "codes": codes,
        "seed": seed,
        "repeat": repeat,
        "freeze_allowed": False,
    }


def logic_checks() -> None:
    assert classify([{"path": "row.json", "change": "modified"}], [0, 0]) == "fixture_breach"
    assert classify([], [0, 0, 0]) == "accept"
    assert classify([], [1, 1, 1]) == "property_fail"
    assert classify([], [0, 1, 0]) == "noise_candidate"
    try:
        classify([], [])
    except ValueError:
        return
    raise AssertionError("empty repeat must fail")


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description="Pin fixtures and class an agent patch")
    sub = parser.add_subparsers(dest="command", required=True)

    pin_cmd = sub.add_parser("pin")
    pin_cmd.add_argument("--root", type=Path, required=True)
    pin_cmd.add_argument("--out", type=Path, required=True)

    sub.add_parser("logic-check")

    judge_cmd = sub.add_parser("judge")
    judge_cmd.add_argument("--pin", type=Path, required=True)
    judge_cmd.add_argument("--root", type=Path, required=True)
    judge_cmd.add_argument("--cwd", type=Path, required=True)
    judge_cmd.add_argument("--repeat", type=int, default=5)
    judge_cmd.add_argument("--seed", type=int, default=20261013)
    judge_cmd.add_argument("--out", type=Path, required=True)
    judge_cmd.add_argument("--property-cmd", nargs="+", required=True)

    args = parser.parse_args(argv)
    if args.command == "logic-check":
        logic_checks()
        print("logic_checks_ok")
        return 0
    if args.command == "pin":
        args.out.parent.mkdir(parents=True, exist_ok=True)
        args.out.write_text(json.dumps(build_manifest(args.root), indent=2) + "\n")
        return 0

    pin_doc = json.loads(args.pin.read_text(encoding="utf-8"))
    verdict = judge(
        pin_doc,
        args.root,
        args.cwd,
        args.property_cmd,
        args.repeat,
        args.seed,
    )
    args.out.parent.mkdir(parents=True, exist_ok=True)
    args.out.write_text(json.dumps(verdict, indent=2) + "\n")
    return 0 if verdict["class"] == "accept" else 2


if __name__ == "__main__":
    sys.exit(main())

A minimal breach verdict illustrates the schema. It is not an observed CI artifact.

{
  "class": "fixture_breach",
  "changes": [{"path": "orders/row.json", "change": "modified"}],
  "codes": [],
  "freeze_allowed": false
}

judge returns process status 0 only for accept. Every other class returns 2. Runners that collapse any non-zero status into a flake will misfile breaches and property failures. Map the JSON class field. Do not infer the class from the shell status alone.

Hash file bytes, and decide explicitly what to do with symlinks. Path.read_bytes follows the target, so a retargeted link can change the digest without an edit to the link text. Keep generated caches, bytecode, and editor swap files out of the fixture root. Otherwise every run becomes a breach, and the class stops meaning anything.

Limitations

A byte pin cannot see a specification change that never touches the pinned root. An agent can weaken a check by editing the property file instead of the fixture. Pin every input the property reads, including sibling data directories, or accept that blind spot and review those paths in the diff.

The table does not parse assertion text. Two non-zero exits can be different defects. They remain property_fail here, because splitting them is a counterexample problem. This gate will not shrink a witness, and it will not decide that two failures are the same failure.

The repeat loop is only as deterministic as the command. Clock reads, network calls, and hash randomization that ignore PIN_SEED produce mixed codes that look like noise. Do not forward those runs to a flake policy until the command is seed-stable on the pinned tree.

Free-model output stays untrusted input. A patch can delete the pin script, vendor a test runner, or rewrite CI YAML without touching the fixture manifest. Review the full diff. Access to a free endpoint or a free server says nothing about uptime, capacity, or how long a verdict remains stored.

Who should skip this gate

Skip it when fixtures are built inside the test and have no pre-image on disk. Pinning an empty root either blocks every run or locks nothing useful. Skip it for intentional snapshot updates and schema migrations, unless those paths are excluded before the diff. Otherwise every valid update is a breach, and reviewers will learn to ignore the class.

Skip it if the model is expected to refresh golden files in the same patch that changes behavior. That needs a human-signed pin update, which is a different procedure with a different record. Skip it if accept will be wired to automatic merge. The pin checks identity of inputs. It does not score property strength, and it does not review the production change.

What to file with the patch

File the base revision, manifest.json, and verdict.json. The class is the review unit. A noise candidate may leave this gate. If a separate policy later writes a freeze, that record has to cite the pin digest used here. Without that citation, the freeze is not part of this strategy, and it should not be described as one.

A Second Host Measures Drift. It Does Not Open the Freeze.

Finley Zhou — Tue, 13 Oct 2026 07:05:07 +0000

A flake freeze is valid only on one host, under one fixture epoch, and against an oracle the patch did not edit. A second machine can measure drift. It cannot open that freeze.

Agent patches fail in ways that look identical on a red CI line. The invariant can be false. The fixture lock can be stale. The owner host can disagree with itself, or a second host can disagree with the first.

Those four outcomes need different records. Collapsing them into one freeze hides the defect the next patch will trip over. The workflow below is a proposal. The snippets are unexecuted specification code, not a report from a live suite, and no pass rate, quota, or hardware figure is claimed.

What must match before a class exists

Treat the run as an equivalence class, not as a test name. Two failures that share a name and differ in their pins are different experiments. Write the pins down before the first replay, and refuse to classify a run that is missing any of them.

The pin set is small.

patch_id is the hash of the diff, not the branch name.
oracle_hash is the hash of the property module. The patch is not allowed to move it.
fixture_epoch is the hash of a manifest that lists every fixture file and that file's own hash.
seed is the integer the property checks actually used.
host_id names the machine that is allowed to own a freeze later.
replay_budget is chosen before the run. A red result does not raise it.

Change one pin and you start a new row. You do not amend the old one.

Build the epoch from bytes, not from directory mtime. A runner can unpack the same files and still change timestamps. Sort relative POSIX paths, write path digest lines, hash that text, and store the digest. If a clock, a temp path, or directory order can change the outcome, put that input in the manifest or stop calling the failure a flake.

Six classes, and only one may freeze

Score the observation with the table below. The freeze column is the policy. The other columns are bookkeeping.

Observation	Class	Freeze on owner host	Next action
Patch changes the oracle hash	`oracle_tamper`	No	Reject the diff. Restore the oracle file.
Pinned property fails	`property_fail`	No	Keep the counterexample. Do not replay for a freeze.
Manifest hash differs from the lock	`fixture_drift`	No	Relock on purpose, or discard the run.
Owner host still inside budget, not a clean pass	`replay_incomplete`	No	Finish the budget. Do not call host B yet.
Owner host shows both pass and fail after the budget	`flake_candidate`	Yes, time-boxed	Insert a freeze row bound to that `host_id` only.
Owner host is stable, other host differs	`environment`	No	File drift. Do not copy the row into the other ledger.
Owner host is stably red, other host agrees or was not called	`regression`	No	Treat it as a real failure of the patch.
Owner host is stably green	`pass`	No	Continue review.

A property failure stays a property failure even if a later run goes green. That green run is a different question, and it does not launder the counterexample into a flake. Stable red on the owner host is a regression when the oracle held. Mixed results after the budget are the only freeze input.

Host B is a measurement, including when host B is a free server rather than another CI runner. Its boolean does not get a vote on the owner ledger.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access matters here only as a source of candidate diffs. The free server option matters only as a possible host B. Neither claim adds a model name, a quota, or a performance number to the table, and none of those are required.

A green probe on the free server does not cancel a property failure on the owner host. A red probe does not open the owner host's freeze.

Seven steps

Hash the oracle path. If the diff touches it, emit oracle_tamper and stop.
Build the fixture manifest from the fixture root and compare it with the locked epoch.
Run property checks on the owner host with the stored seed. On failure, emit property_fail and stop.
Run the named regression once on the owner host. A pass skips the freeze path.
On failure, replay on the owner host until the budget is consumed and every outcome is recorded.
Only then, if the class is not already property_fail or fixture_drift, submit the same patch_id, seed, and fixture epoch to host B.
Emit one class from the table. Persist it. Do not edit the class after the ledger write.

Step 6 is deliberately late. An unfinished owner-host run cannot change the table, so calling host B early only spends a seat. If that seat is a free server, keep the probe off until step 5 has a complete vector.

Expiry of a freeze that already exists is a separate policy. This gate only decides whether a new row may be inserted, and on which host.

Classifier

This module is a reviewable proposal. It imports nothing outside the standard library. It encodes the table. It does not discover new invariants.

import hashlib
from pathlib import Path

def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()

def fixture_epoch(root: Path) -> str:
    lines = []
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        rel = path.relative_to(root).as_posix()
        lines.append(f"{rel} {sha256_file(path)}")
    payload = ("\n".join(lines)).encode()
    return hashlib.sha256(payload).hexdigest()

def classify(obs: dict) -> str:
    if obs["oracle_hash"] != obs["oracle_lock"]:
        return "oracle_tamper"
    if obs["fixture_epoch"] != obs["fixture_lock"]:
        return "fixture_drift"
    if not obs["property_passed"]:
        return "property_fail"
    host_a = list(obs["host_a_outcomes"])
    budget = int(obs["replay_budget"])
    if budget < 1 or not host_a or len(host_a) > budget:
        raise ValueError("owner-host outcomes are outside the declared budget")
    distinct = set(host_a)
    host_b = obs.get("host_b_outcome")
    if len(host_a) < budget and distinct != {True}:
        return "replay_incomplete"
    if distinct == {True, False}:
        return "flake_candidate"
    if distinct == {True}:
        return "environment" if host_b is False else "pass"
    if host_b is True:
        return "environment"
    return "regression"

def freeze_allowed(kind: str, host_id: str, owner_host: str) -> bool:
    return kind == "flake_candidate" and host_id == owner_host

freeze_allowed is the policy in one line. Environment drift returns false. A cheaper host does not become the owner by being cheaper. Host B is not consulted inside the mixed-result branch. Instability on the owner host is already enough for flake_candidate, and host B cannot veto that row or inherit it.

Three illustrations use placeholder ids. They are not measured runs.

Owner outcomes [False, False, False], budget 3, property passed, host B True. The class is environment. The owner host is stable. Freezing the test name would block later patches for a failure that did not reproduce on the other host, and it would hide the drift.

Owner outcomes [False, True, False], budget 3, host B False. The class is flake_candidate. A freeze may open, and only for the owner host_id. Host B's failure does not extend that row.

Owner outcomes [False], budget 3. The class is replay_incomplete. One red result is not a class you can freeze. Finish the vector first, and do not call host B on this row yet.

Specification checks

Pair the function with assertions that restate the table. They are part of the proposal. They fail if a later edit lets a second host open the owner ledger.

def test_cross_host_split_is_not_a_freeze():
    kind = classify({
        "oracle_hash": "aa",
        "oracle_lock": "aa",
        "fixture_epoch": "bb",
        "fixture_lock": "bb",
        "property_passed": True,
        "host_a_outcomes": [False, False, False],
        "host_b_outcome": True,
        "replay_budget": 3,
    })
    assert kind == "environment"
    assert freeze_allowed(kind, "ci-a", "ci-a") is False

def test_mixed_owner_host_can_freeze():
    kind = classify({
        "oracle_hash": "aa",
        "oracle_lock": "aa",
        "fixture_epoch": "bb",
        "fixture_lock": "bb",
        "property_passed": True,
        "host_a_outcomes": [False, True, False],
        "host_b_outcome": False,
        "replay_budget": 3,
    })
    assert kind == "flake_candidate"
    assert freeze_allowed(kind, "host-b", "ci-a") is False
    assert freeze_allowed(kind, "ci-a", "ci-a") is True

The host check is the assertion reviewers skip. A flake on ci-a must not be writable by host B, even when both saw a failure. Host identity is part of the key.

A minimal persisted record, again with placeholder values, looks like this.

{
  "patch_id": "example",
  "oracle_hash": "aa",
  "fixture_epoch": "bb",
  "seed": 17,
  "host_id": "ci-a",
  "replay_budget": 3,
  "class": "environment",
  "freeze": false
}

Command order

Check the oracle path before the test runner starts. An assertion inside the test file is too late, because the patch may have weakened that assertion.

git diff --name-only "$BASE"..HEAD -- tests/properties/invariants.py
python -m gate.classify --record runs/patch.json --owner-host ci-a

The first command must print nothing. If it prints the oracle path, emit oracle_tamper and do not start the suite. The second command reads the observation file and prints one class. Exit non-zero for every class other than pass.

Host B gets the same pins and an empty ledger. The flags below belong to the proposed local probe, not to a vendor CLI.

python -m gate.probe \
  --patch-id "$PATCH_ID" \
  --fixture-epoch "$EPOCH" \
  --seed 17 \
  --host-id host-b \
  --ledger none

--ledger none is the useful constraint. Host B may return true or false. It may not append a freeze row. Point --host-id at whatever second machine you already operate. A free server seat can fill that role. It does not become the merge gate.

Keep the agent off the oracle path in two places. Say so in the task instruction, then enforce it with the hash compare. The instruction is not the gate. The hash is.

Limitations

The classifier does not prove the patch is correct. It can only show that the stated properties held and that the named check was stable on one host. Unstated behavior can still be wrong. That is a coverage gap, not a reason to freeze a red name.

A same-host disagreement can still be an unlisted fixture. Wall clocks, temp paths, and directory order will mint false flake_candidate rows if they sit outside the manifest. Widen the manifest before you widen the budget.

The budget in the examples is a policy integer, not a calibrated flake rate. This article does not estimate how often agent patches flake. If you have a measured baseline, use it. If you do not, keep the integer small, and accept that a stable failure costs that many owner-host runs before it becomes regression.

Cross-host comparison here is a boolean. Two hosts can fail in different assertions and look like agreement. Store the exception type and the assertion id if you need a stricter probe. That extension still must not open a freeze by itself.

Who should skip it

Skip the workflow if you cannot name one owner host. A pool of ephemeral runners with no stable id will mark almost every split as environment, and the freeze path will never open. That can be the right policy. Write "no freeze on pooled hosts" instead of pretending the pool is one machine.

Skip it if the suite has no property module the patch is forbidden to edit. Named tests alone cannot separate tamper from regression. Add the oracle before you add host B.

Skip it if you need a vendor quota, a hardware spec, or a published benchmark before a second host is allowed to run. Those are not inputs to the table, and this note does not supply them.

Put the observation JSON in the pull request. If the class is environment, link the two host ids and leave the freeze ledger untouched. If a free MonkeyCode server is already on your bench, use that seat as host B with an empty ledger. The owner host, not the extra seat, still decides whether a freeze exists.

Hash the Sampler Output Before You Freeze an Agent-Patch Failure

Finley Zhou — Tue, 13 Oct 2026 04:51:50 +0000

An agent-patch failure is not a flake until two conditions hold. The sampler output must share one digest, and the red signal must be harness noise rather than an oracle miss. Property misses and fixture drift stay unfrozen. The artifact below is a decision table plus a local classifier you can run with the standard library.

Same red status, different clocks

An agent-patch job usually runs on three clocks. The generator samples bytes. A fixture file stores those bytes. A property oracle then checks the patched code against the file, not against a fresh sample.

CI often prints one word for all three failures: FAILED. A timeout, a changed completion, and a broken invariant then look identical in the job list. Collapsing them trains the suite to mute real defects.

The digest step sits before any freeze decision. Without it, a second run is not a rerun. It is a new input that happens to share a test name.

Four classes, one door

Label the pair of runs before you write a freeze record. Only one label may open the door.

Class	Fixture digests	Oracle on that digest	Transport	Freeze
`property_miss`	equal	fail and fail	ignored	no
`fixture_drift`	unequal	not comparable	ignored	no
`harness_flake`	equal	no recorded fail	at least one fault	yes, request only
`unclassified`	missing, or equal with a split oracle	unclear	unclear	no

Read the rows left to right. Equal digests make the oracle comparable. Unequal digests mean the generator moved, so a freeze would rename a new input as infrastructure noise.

A double fail on one digest is a patch problem or a weak property. It stays a miss even if the server was slow on the same run. A split oracle with no transport fault is not a flake either. That pair stays unclassified until a separate replay policy exists. This note does not define shrinking, replay budgets, or witness size.

harness_flake is narrow on purpose. The digest matches. No oracle result is False. At least one run records a transport fault, such as a timeout, a connection reset, or a completed call that fails a length check you declared in advance. The classifier may then allow a freeze request. It still does not set the expiry, and it does not merge the patch.

Numbered workflow

1. Persist sampler bytes before any assertion

Write the raw completion to a fixture path. Hash the bytes with SHA-256. Store the digest beside the run id.

Do this when the body is empty, too. A hash of zero bytes is a real pin. A missing hash is a different state, and the classifier treats missing as unclassified.

Confirm the pin with the standard library before the job continues:

python -c "import hashlib,pathlib; p=pathlib.Path('fixtures/sample.json'); print(hashlib.sha256(p.read_bytes()).hexdigest())"

A 64-character hex line means the file is eligible for comparison. A missing path means the run is not eligible for a freeze, regardless of how red the log looks.

2. Point the property oracle at the file

The oracle takes a path. It does not call the model, and it does not open a socket to the generator host.

If a property needs a second sample, give that sample its own fixture id. A hidden live call inside the oracle makes the digest meaningless, because the checked input is no longer the pinned file.

3. Record transport on its own fields

Keep the transport record small and explicit. Suggested fields are transport_ok, a status code, and a timeout flag. Fill them in the client, not by parsing the oracle message.

An oracle line that says a key is missing can be a product bug in the patch. Inferring a flake from that string will launder the bug. If the client retried silently, surface the retry count as its own field. A hidden retry makes transport_ok look clean and pushes a real harness fault into unclassified, which is the safer misclassification.

4. Classify the pair, then open at most one door

Load two run records. A missing digest stops at unclassified. Different digests stop at fixture_drift and should open a fixture-refresh ticket, not a freeze.

Matching digests and two oracle failures stop at property_miss. Keep the patch blocked. A single oracle failure beside a pass is also blocked from the freeze door, even if transport was dirty. The miss might be real. Only the narrow harness_flake row may set the freeze flag to true.

5. Leave expiry and merge rights outside the module

The function returns a class and a boolean. It does not choose a duration, a quota, or a machine size. Put expiry, owner, and unfreeze conditions in the workflow config reviewers already read.

A true boolean means this red result may be muted until that external expiry. It does not mean the patch is correct, and it does not clear a property miss from another job.

Example classifier

The module is a self-contained example for local reproduction. It does not call a hosted model, and it is not a report of a production run. Treat the control flow as the spec. If you change a return, update the driver in the next section in the same commit.

import hashlib
from dataclasses import dataclass
from enum import Enum

class FailureClass(str, Enum):
    PROPERTY_MISS = "property_miss"
    FIXTURE_DRIFT = "fixture_drift"
    HARNESS_FLAKE = "harness_flake"
    UNCLASSIFIED = "unclassified"

@dataclass(frozen=True)
class RunRecord:
    fixture_sha256: str | None
    oracle_passed: bool | None
    transport_ok: bool | None

def pin_fixture(payload: bytes) -> str:
    return hashlib.sha256(payload).hexdigest()

def classify(left: RunRecord, right: RunRecord) -> FailureClass:
    if not left.fixture_sha256 or not right.fixture_sha256:
        return FailureClass.UNCLASSIFIED
    if left.fixture_sha256 != right.fixture_sha256:
        return FailureClass.FIXTURE_DRIFT
    if left.oracle_passed is False and right.oracle_passed is False:
        return FailureClass.PROPERTY_MISS
    any_miss = left.oracle_passed is False or right.oracle_passed is False
    transport_bad = left.transport_ok is False or right.transport_ok is False
    if any_miss:
        return FailureClass.UNCLASSIFIED
    if transport_bad:
        return FailureClass.HARNESS_FLAKE
    return FailureClass.UNCLASSIFIED

def freeze_allowed(kind: FailureClass) -> bool:
    return kind is FailureClass.HARNESS_FLAKE

The final return is a closed door. A green pair should not be passed into classify at all. If it is, refusing the freeze is still the right default. There is nothing to mute.

Commands that lock the four rows

Save the module as freeze_gate.py. The driver uses only the standard library. Python 3.11 or newer accepts the str | None hints. On 3.9 or 3.10, switch those hints to Optional from typing and leave the control flow alone.

The driver is an unexecuted example in this draft. Run it locally before you trust a fork of the module.

python - <<'PY'
from freeze_gate import FailureClass, RunRecord, classify, freeze_allowed, pin_fixture

digest = pin_fixture(b'{"patch":"return sorted(xs)"}')
other = pin_fixture(b'{"patch":"return xs"}')

cases = {
    "property_miss": (
        RunRecord(digest, False, True),
        RunRecord(digest, False, True),
        FailureClass.PROPERTY_MISS,
    ),
    "fixture_drift": (
        RunRecord(digest, False, True),
        RunRecord(other, False, True),
        FailureClass.FIXTURE_DRIFT,
    ),
    "harness_flake": (
        RunRecord(digest, None, False),
        RunRecord(digest, True, True),
        FailureClass.HARNESS_FLAKE,
    ),
    "split_oracle": (
        RunRecord(digest, True, True),
        RunRecord(digest, False, True),
        FailureClass.UNCLASSIFIED,
    ),
}

for name, (left, right, expected) in cases.items():
    got = classify(left, right)
    assert got is expected, (name, got)
    assert freeze_allowed(got) is (expected is FailureClass.HARNESS_FLAKE)
print("four local cases passed")
PY

Expected stdout is four local cases passed. An assertion error prints the case name and the class you actually returned. Do not edit the expected class to match a weaker function. Edit the function, or admit the row is a different policy.

How to read a disagreement

Take harness_flake first. The fixture hash matches, so both runs saw the same sampler bytes. The first run never reached the oracle because transport failed. The second run reached the oracle and passed. The red job is eligible for a freeze request. It is not eligible for a silent permanent skip.

Take property_miss next. Same hash, two oracle failures, transport marked healthy. Slow hardware would not change the row. The invariant failed on the pinned input. Block the patch.

Take fixture_drift third. The completions differ by one expression in the example, and the digests differ. Refresh the fixture or allocate a new fixture id. Writing a freeze here hides generator movement inside an infrastructure label.

Take split_oracle last. Same bytes, clean transport, pass beside fail. That pattern can be a clock inside the property, a race in the code under test, or an oracle that still calls out. Freezing it would protect the wrong layer. Leave the row unclassified and fix the test until the oracle is a pure function of the fixture file.

Where a free model lane fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free model lane on MonkeyCode is a usable sampler for step 1. A free server lane there is a usable place to execute step 2. Both are availability options supplied for this draft. Neither option makes two completions the same input, and neither option turns a property miss into a flake.

Keep the sampler call inside step 1. Persist the bytes, compute the digest, and stop using the network. Step 2 should still pass with egress disabled. If it does not, the oracle is coupled to the generator, and the table does not apply until that coupling is gone.

No model name, token quota, hardware shape, or retention window is stated here. Those figures were not verified for this draft. If a console shows a limit, treat the limit as a constraint on how you capture step 1, not as proof that a later completion matches the fixture.

If generator calls are already split from assertions, that free model lane and that free server lane are enough to try the classifier on a scratch branch before it is wired into a merge gate.

Limitations

The digest does not score property quality. A check that cannot fail will pass on a pinned file and still test nothing. Review properties separately. Reject any check whose expected result is implied by the assertion text alone.

Fixture stores grow. Content addressing removes duplicate bytes, but it does not delete abandoned pins. Put retention in the store policy. Do not hide deletion inside classify, or the class of an old run will change under you.

Transport flags are only as honest as the client. Silent retries, rewritten status codes, and bodies truncated after a 200 will poison the row. Prefer unclassified over a guessed flake when those fields are absent.

This example does not shrink counterexamples, does not choose an expiry, and does not grant merge rights. A reviewer still owns the patch. The job summary should show the class even when the boolean is false, so a miss is visible beside a muted harness fault from another shard.

Who should skip it

Skip the gate when tests have no separable sampler input. A pure unit suite can decide flakes without fixture digests.

Skip it when the property must call a live model by design. Pinning will fight that design on every run, and the drift row will dominate for reasons that are not actionable.

Skip it when a freeze is treated as approval. The freeze flag is a mute request with an external expiry. It is not a safety claim about the patch, the model, or the server.

Fields to print in the job summary

Print four lines. A summary that contains only FAILED cannot support the table.

fixture_sha256=<64 hex chars>
failure_class=harness_flake
freeze_allowed=true
fixture_path=fixtures/<digest>.json

Add those lines before you add another retry wrapper. Retries without a digest compare the wrong thing, then ask the freeze door to forgive the comparison.

An Agent Patch May Not Run Until the Fixture Digest Is Signed

Finley Zhou — Tue, 13 Oct 2026 03:51:04 +0000

An agent patch is not ready for a property suite because a model emitted it. It is ready only when three records exist and play different roles: a provenance note, a fixture digest signed outside the drafter, and a property class that limits what a pass or fail may mean. Miss any one of them and the run should not start.

A flake freeze is a later ledger. It cannot repair a missing signature. That admission order is the strategy. The steps below enforce it without treating a cheaper draft, or a cheaper runner, as evidence.

The failure this gate is for

Drafts are cheap to request. A locked input set is not. If the suite starts before the fixture is closed, the second run can see a different file, seed, or clock. The disagreement looks like a flake. It is an unbound fixture.

Call that an admission bug. Do not shrink a witness, spend a replay budget, or write a freeze record until admission has succeeded. Later tools assume the input object is already stable. This note does not restate shrink order, two-run record formats, or the rule for closing a freeze. It only keeps those tools from starting early.

Step 1. Write provenance, then refuse to trust it

Keep a short note about who drafted the patch and which interface returned it. The note is an audit field. It is not an oracle.

Set draft_id from the operator log. Do not parse an id out of model prose.
Set drafter to a role label such as human or model_lane. Leave the model name blank unless that name was verified for this call.
Set received_at from the operator clock, in UTC, at second resolution.
Set patch_sha256 to the hash of the exact diff bytes you intend to apply.

Stop if the body is empty. A success status with no patch is an intake failure. It is not a property failure, and it is not a flake. Transport may refetch once under its own policy. That refetch must not create a fixture digest and must not open a freeze.

Step 2. Sign the fixture digest before any property runs

A fixture is the closed input set: files, seed, clock bound, and the allow-list of commands. Hash that set. Sign the hash with a key the draft lane does not hold.

Keep the signer role and the executor role distinct, even on one laptop. An execution host can run the suite. It should not mint the digest. Ordinary executor failures include "unavailable" and "empty response." Those are not assertion results. If the same process both writes the fixture and signs it, the signature only attests that the process agreed with itself.

Build the digest from canonical bytes.

Sort fixture paths by relative path.
Hash each file with SHA-256.
Append the seed, the clock bound, and the allowed commands as their own lines.
Hash the concatenation. Call that fixture_digest.
Sign it in the operator-key role and store the signature beside the patch record.

If a path is missing, admission fails closed. If the patch invokes a command outside the allow-list, admission fails closed. Do not edit the fixture inside the same job so the patch can pass. An edit is a new fixture and needs a new signature.

This shell sketch only builds the unsigned digest. It is a proposal, not a pipeline claim.

# Proposal only. Do not point this at untrusted paths.
find fixtures -type f -print | sort | while read -r f; do
  sha256sum "$f"
done > /tmp/fixture.list
printf 'seed:%s\nclock_bound_s:%s\n' "$SEED" "$CLOCK_BOUND" >> /tmp/fixture.list
sha256sum /tmp/fixture.list

The last line is the unsigned digest. Signing stays outside this shell. The executor should receive the digest and the signature as read-only inputs. It should not be able to replace either file.

Step 3. Class the property before the runner starts

Each property needs a class id checked into the repo and written by a person. The class is what a result is allowed to mean. A pass without a class is noise.

Class	A fail means	A pass does not mean	May open a freeze?
`pure_fn`	A bound input broke the spec	The rest of the tree is safe	No. Reject or fix the patch.
`idempotent_io`	Two applies left different trees	Another seed would agree	No, until both trees match the signed digest.
`bounded_time`	The declared clock bound was exceeded	The logic is wrong	No. Slowness is not a flake.
`external_probe`	A live dependency disagreed	The patch is at fault	No. Live probes never freeze.

Anything that reads the network, an unbounded wall clock, or an unsigned file is outside the table. Pull it from the gate, or rewrite it until it fits pure_fn or idempotent_io.

Tautologies are a class failure, not a weak pass. An assertion of True, or a check that compares the patch to a copy of itself, never enters the runner. Reject that property id before admission. Do not give it a softer class so the queue stays green.

Step 4. Let the stability ledger start only after admission

Run an admitted property twice on the same fixture_digest. Compare the two outcome hashes. Only an eligible class, plus a real disagreement between those runs, may open a freeze record. A counterexample that appears in both runs is a defect. Freeze nothing. A missing digest is not a disagreement. Freeze nothing.

Before the second run, confirm the digest did not drift. This check is part of the proposal, not a recorded benchmark.

test "$DIGEST_RUN_1" = "$DIGEST_RUN_2"

A non-zero exit here is an admission bug that escaped step 2. Stop. Do not label it flaky. When a freeze is allowed, point it at the digest and the two outcome hashes. Expire it under the policy you already run. Do not invent a shorter expiry inside the runner to make the queue look clean.

Decision matrix

First matching row wins. admit means the suite may start. It does not mean merge.

Provenance	Signature	Class eligible	Runner role	Decision
Incomplete	Any	Any	Any	`reject_provenance`
Complete	Invalid or absent	Any	Any	`reject_fixture_role`
Complete	Valid, operator key	No	Any	`reject_class`
Complete	Signed by the executor	Yes	Execution host acting as signer	`reject_fixture_role`
Complete	Valid, operator key	Yes	Execution host as executor only	`admit`
Complete	Valid, operator key	Yes	Local executor	`admit`

Three traces fall out of that table. They are classifications, not measured incident rates.

Empty diff with a success status. Decision: reject_provenance. Do not retry inside the gate.
A green run whose fixture bytes changed, with the signature produced by the host that executed. Decision: reject_fixture_role.
Intermittent name lookup inside a property labeled external_probe. Decision: reject_class. Do not convert DNS into a freeze.

Proposed admission function

The Python below is unexecuted proposal code. It is not a measured gate, and it does not verify a cryptographic signature. The caller must set fixture_sig_ok only after a real verify in the operator-key role. Shipping the boolean as a model-controlled field would let the draft approve itself.

import hashlib
import json
from dataclasses import dataclass

ELIGIBLE = {"pure_fn", "idempotent_io"}

@dataclass(frozen=True)
class Intake:
    draft_id: str
    drafter: str
    patch_sha256: str
    fixture_digest: str
    fixture_sig_ok: bool
    property_class: str
    signer_role: str

def decide(item: Intake) -> str:
    if not item.draft_id or len(item.patch_sha256) != 64:
        return "reject_provenance"
    if item.signer_role != "operator_key" or not item.fixture_sig_ok:
        return "reject_fixture_role"
    if item.property_class not in ELIGIBLE:
        return "reject_class"
    if not item.fixture_digest:
        return "reject_fixture_role"
    return "admit"

def fixture_digest(files: dict, seed: int, commands: list) -> str:
    lines = []
    for path in sorted(files):
        digest = hashlib.sha256(files[path]).hexdigest()
        lines.append(path + ":" + digest)
    lines.append("seed:" + str(seed))
    lines.append("cmds:" + ",".join(commands))
    return hashlib.sha256("\n".join(lines).encode()).hexdigest()

if __name__ == "__main__":
    digest = fixture_digest({"src/app.py": b"print(1)\n"}, 7, ["pytest -q"])
    sample = Intake(
        "d-100", "model_lane", "ab" * 32, digest, True, "pure_fn", "operator_key"
    )
    print(json.dumps({"decision": decide(sample), "digest": digest}))

Sketch the three rejects before you trust the function. Clear draft_id and expect reject_provenance. Set signer_role to execution_host and expect reject_fixture_role. Set property_class to external_probe and expect reject_class. Stop there. Do not attach the sketch to a live service to watch it fail open.

Where the free lanes fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access fits the draft lane in step 1. You can request candidate patches while rehearsing the gate, without treating that access as a test oracle. MonkeyCode's free server option fits the executor row in the matrix. It can run an already admitted suite against a digest signed elsewhere. Those two statements are availability claims only. They do not name a model, a quota, a machine shape, a time limit, or a promise that the terms stay fixed. Read the current product documentation before you plan capacity around them.

If the free server is down, log an availability miss. Do not store that miss as a failed property, and do not open a freeze from it. If a free model call returns no diff, stay on reject_provenance. A second call does not promote an empty body into a fixture.

Teams that already separate drafting from signing can use those two free options to practice the matrix on a private repository before they point the same door at a paid runner.

Limits of the gate

Admission does not show that the patch is correct. It shows that the suite was allowed to speak. A signed digest can still encode a bad spec. Two agreeing runs can both be wrong when step 3 let a tautology through. The signature does not review license, intent, or whether the diff touches code the operator may change.

fixture_sig_ok is the dangerous field in the sketch. A caller-supplied true value would self-approve. Compute it in the operator-key role and pass the boolean in as an already verified fact. Keep the private key off the draft lane and off the execution host.

Free execution adds an axis you do not control. Cold starts and empty bodies stay in transport handling. If your logs cannot separate that axis from an assertion, do not host this gate on that runner. The matrix will then mix availability with correctness, which is the bug it was written to prevent.

Who should skip it

Skip the matrix if tests still call live services, read an unbounded clock, or build fixtures inside the test process. Also skip it when the same credentials draft the patch and sign the digest. The table will admit work that the role split was meant to block.

Do not use this rehearsal as a safety case, a regulated evidence pack, or a substitute for human merge review. And do not spend a shrinker, a replay budget, or a freeze to launder a reject row into a pass. Those tools start after admit. Used earlier, they hide the unbound fixture instead of catching it.

Score Fixture Independence Before an Agent Patch Merges

Finley Zhou — Mon, 12 Oct 2026 23:29:54 +0000

An agent patch that edits production code and the fixture locking that code in one diff has removed its oracle. That failure is not a flake. A freeze must not absorb it. Score fixture independence first, require an oracle the patch did not edit, and only then allow a narrow freeze for an environmental signal.

CI makes the three classes look alike. A red assertion, a timeout, and a snapshot mismatch all fail the job. They do not share a remedy.

Co-modification is a review defect. A broken invariant is a product defect. A dropped socket is a scheduling defect. One retry policy for all three will hide the first two inside the third.

What a moved fixture can no longer prove

A fixture is an oracle only while its bytes stay outside the change under test. Independence is a path condition you can check. It is not a claim about the agent's intent.

The usual prompt asks the agent to make the suite pass. The cheapest edit is often the expected JSON, the golden file, or the recorded response body. The job goes green. The behavior can still be wrong.

If last week's noise already taught the team to freeze red tests, that green run gets stored as stability. Count paths instead. A diff that touches src/billing/prorate.py and tests/fixtures/prorate/case_18.json is not locking prorate. It is shipping two copies of one opinion.

Gate 1 — Classify hunks before any test run

Export one unified diff from the merge base to the patch tip. Ignore the file list in the agent's summary. The diff is the record.
Label each path as production, fixture, test, or other, using a map that reviewers already merged. Unlabeled paths fail closed.
Pair paths by feature key. A pair is co-modified when that feature's production list and fixture list are both non-empty in this diff.
Publish a score: co-modified pairs, plus unlabeled paths. A boolean buried in a log line will be skipped.

The script is a proposal for Python 3.9 or newer. It was not executed against this account's repositories while drafting this article. Patterns are exact file paths or a dir/** prefix. That restriction is deliberate, because Path.match is relative and will false-positive on suffixes.

#!/usr/bin/env python3
"""Proposal: score fixture independence from a unified diff."""
import argparse
import json
import re
import sys
from pathlib import Path

HUNK = re.compile(r"^\+\+\+ b/(.+)$", re.M)

def match(path: str, patterns: list[str]) -> bool:
    for pattern in patterns:
        if pattern.endswith("/**"):
            prefix = pattern[:-3].rstrip("/") + "/"
            if path.startswith(prefix):
                return True
        elif path == pattern:
            return True
    return False

def classify(paths: list[str], spec: dict) -> dict:
    claimed: set[str] = set()
    rows = []
    for feature, rules in spec["features"].items():
        hit = {
            kind: [p for p in paths if match(p, rules.get(kind, []))]
            for kind in ("prod", "fixture", "test")
        }
        for group in hit.values():
            claimed.update(group)
        rows.append({
            "feature": feature,
            "prod": hit["prod"],
            "fixture": hit["fixture"],
            "test": hit["test"],
            "co_modified": bool(hit["prod"] and hit["fixture"]),
        })
    unlabeled = [p for p in paths if p not in claimed]
    return {
        "pairs": rows,
        "co_modified_count": sum(row["co_modified"] for row in rows),
        "unlabeled": unlabeled,
    }

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--map", required=True, type=Path)
    parser.add_argument("--diff", type=Path)
    args = parser.parse_args()
    text = args.diff.read_text() if args.diff else sys.stdin.read()
    paths = sorted(
        p for p in set(HUNK.findall(text)) if p != "/dev/null"
    )
    report = classify(paths, json.loads(args.map.read_text()))
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    if report["co_modified_count"] or report["unlabeled"]:
        return 2
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Keep the map beside the tests so a review can see the policy without reading a prompt. Overlapping prefixes across features will double-count a file. Keep those keys disjoint.

{
  "features": {
    "prorate": {
      "prod": ["src/billing/prorate.py"],
      "fixture": ["tests/fixtures/prorate/**"],
      "test": ["tests/test_prorate.py"]
    }
  }
}

Replace origin/main with the branch you actually merge into. The triple-dot form diffs from the merge base to the patch tip, which is the set of hunks a reviewer can still reject.

git diff origin/main...HEAD \
  | python fixture_independence.py --map tests/oracle_map.json
echo "gate1_exit:$?"

Exit code 2 means a mapped pair moved together, or a path had no label. Exit code 0 means this diff did not trip those rules. It does not mean the patch is correct.

Hand trace, not a captured process run. Feed these headers:

+++ b/src/billing/prorate.py
+++ b/tests/fixtures/prorate/case_18.json

The prorate row has both lists filled. co_modified_count is 1. The process should exit 2. Remove the fixture header and the same function returns 0, with fixture as an empty list.

Add scripts/load.py and unlabeled becomes that path, which is also exit 2 until the map names it. A +++ /dev/null line is dropped so a deletion header does not look like an unlabeled file. Pure deletions are still invisible to this scorer. Gate 2 has to name them.

Gate 2 — Keep one oracle outside the diff

If the fixture really must change, put that edit in a separate review record. Allowed labels are schema migration, corrected capture, or deleted case. A label is not a proof.
Keep a property that does not read the edited fixture. The property states an invariant the fixture is not allowed to rewrite.
Replay the patched function against a pinned baseline module for a checked-in seed list. Compare only fields the patch claims to leave alone.
Fail the gate when the only green check is the fixture this diff rewrote.

The following check is a template, not a recorded pass. It never opens case_18.json. Bounds on the inputs are part of the example so a runner cannot wander into unrelated domains.

def test_prorate_conserves_cents(days, price_cents):
    days = abs(days) % 28 + 1
    price_cents = abs(price_cents) % 50_000
    parts = prorate(price_cents, days)
    assert sum(parts) == price_cents
    assert all(part >= 0 for part in parts)
    assert len(parts) == days

If the agent edits the property file in the same diff, add that path to the oracle side of the map and treat the pair as co-modified. An edited property is the same defect as an edited fixture. A green suite after that edit only shows the patch agrees with itself.

Before trusting a new property, apply a negative control on a throwaway branch. Invert one operator in prorate, confirm this test fails, then restore the file. Do not record that failure as a flake, and do not keep the inverted operator. The control only shows the property can go red.

git checkout -b scratch-negative-control
# invert one operator in src/billing/prorate.py, then:
python -m pytest tests/test_prorate.py::test_prorate_conserves_cents -q
git checkout -- src/billing/prorate.py
git checkout -

A property that stays green on the inverted operator is not an oracle. Delete it or tighten it before gate 3 exists as an option. Retrying that green test will not add information.

Decision table

These rows are review rules you adopt in the repo. They are not measurements from a shared dataset, and they are not pass rates.

Diff signal	Oracle status	Freeze allowed?	Required action
Prod and fixture both changed	No untouched property or baseline	No	Restore fixture bytes or split the edit
Prod changed, fixture unchanged	Property fails	No	Fix the patch; do not retry into a freeze
Prod changed, fixture unchanged	Property passes; one run times out	Yes, narrow	Freeze that timeout only, with your existing expiry
Fixture-only change	Baseline replay disagrees	No	Treat it as an oracle edit
Unlabeled path	Unknown	No	Extend the map, then rerun gate 1
Test helper changed, prod unchanged	Properties still pass	No	Review the helper; green is not independence

A freeze, when the table allows one, is a time box on an environmental retry. It is not a waiver for a co-modified fixture. It is not storage for a failed invariant. If a freeze record already exists in your CI, point its opener at this table so the first row cannot enter it.

Gate 3 — Freeze only the residual environmental row

Confirm gates 1 and 2 are green on the same commit SHA you will merge. A green run on an earlier SHA does not transfer.
Name the check, the signal, and the expiry your CI policy already uses. Timeout, connection reset, and missing optional service qualify. An assertion message does not.
Re-run that named check on a clean worker. Two identical assertion failures are still a product bug.
Do not widen the freeze to a neighboring file. A freeze that grows until it covers the fixture path is a policy bug.

This note does not specify how a freeze record is stored. Storage format is a local choice. The precondition is the point: independence and an untouched oracle must already hold on one SHA before any freeze key is written.

Draft properties on a free model, execute on a free server

Writing the conservation check is clerical. Accepting the patch is not. A free model seat can propose invariant text from the diff. A free server can run the checker and the property file on a clean checkout of that SHA.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The availability used here is the operator-supplied free model access and free server option. No model name, quota, hardware shape, duration, or benchmark is claimed, because none was verified for this draft.

Keep the split sharp. The model may write a candidate. It may not mark gate 1 green.

The server may execute the script. It may not override exit code 2. Commit the map, the script, and the property.

If those two options are already enabled for your account, draft the candidate property in the model session and run the merge-base command on the free server for the SHA you plan to merge. The local exit code stays the decision. The chat transcript is not evidence.

Limits, and who should not use this

The checker reads path names from +++ b/ headers. A rename shown only as delete plus add needs git diff -M and a parser this proposal does not include. Binary fixtures have no useful text hunk.

Overlapping globs can attach one file to two features. A missing feature key does not protect that feature. It only fails paths you forgot to label. Review the map when you add a directory, or the score will drift while the suite stays green.

Do not use a path score as the only gate for schema migrations. A migration co-modifies code and fixtures on purpose. It needs a migration proof, such as a dual-read window or a fixture rebuild from captured rows, not a score of zero.

Skip the workflow when the patch vendors opaque binaries with an empty text diff. Skip it when you will not keep oracle_map.json under review. A wrong map is worse than no script, because exit code 0 then launders a co-modified pair.

Do not send gate 1 failures back to the model with the instruction to make the command pass. That prompt recreates the original defect. The fixture will move until the score is quiet. Fix the map or restore the bytes.

What must be true on the merge SHA

Merge only when three statements hold for one SHA. The independence score is zero. At least one oracle this diff did not edit still passes, including the negative control you reverted.

Any open freeze names an environmental signal, not an assertion, and does not cover a fixture path. Anything else is a review comment. It is not a retry.

Block the Agent Patch Until Parent Drift Has a Signed Ledger

Finley Zhou — Mon, 12 Oct 2026 19:43:33 +0000

An agent patch is not ready to merge when its new tests were recorded after the edit. Capture a characterization baseline on a pinned fixture pack at the parent commit, rerun that same pack on the patch, and reject unexplained drift. A snapshot refreshed after the edit is a copy of the patch, not an oracle.

That order is the whole gate. Stability policy belongs in other reviews. This gate asks only whether observed behavior moved, and whether a named reviewer accepted the move.

Why a post-edit golden fails too late

Characterization tests store outputs and compare a later run to that store. They help when the written spec is thin. They stop being evidence when the same change both edits the code and rewrites the store.

An agent can do both in one turn. The suite goes green because the golden file now matches the candidate, not because the candidate matches the parent contract. The diff is large, the note is short, and the reviewer has no independent witness.

A parent baseline breaks that loop. Drift becomes a row. Acceptance becomes a ledger with an allow-listed reason code. The patch does not get to mint the behavior and the proof in one unreviewed write.

This is ordinary characterization testing, sometimes called a golden master, plus one constraint: the master is taken before the agent edit. The constraint is the part that matters for generated patches.

What has to be true first

Three conditions make the ceremony worth running.

The module already has callers and a fixture pack that runs on the parent.
You can execute that pack in two clean worktrees, parent and candidate.
The review stores both result files and the ledger, not only a buffer on one laptop.

Skip the gate for greenfield code with no callers, for a patch whose job is a broad error-text rewrite, and for a suite you cannot run twice on the same inputs. If a field is nondeterministic, exclude it in the pack definition. Do not hide that choice inside a refreshed snapshot.

Figures below are policy knobs for your repo. They are not measured limits of any tool. The scripts are unexecuted proposals until you run them on your tree.

Step 1. Pin a pack the patch cannot resize

Start with 20 to 50 fixtures. That range is a review budget, not an optimum found by a study. Each fixture is a JSON object with an id, a callable name, an input, and a one-line note on why the row exists.

Put the file on a path the agent change must not edit, such as review_fixtures/pack.json. If the diff touches that path, fail closed. The pack is the ruler. A patch that resizes the ruler and the cloth in one commit has not been measured.

Drop rows that assert a literal true, or that compare a value to itself. They cannot show drift. Prefer inputs taken from past bugs, plus empty, single-element, and rejected-type cases. The pack still under-approximates the domain. Say so in the file header so a match is not treated as a proof.

Step 2. Capture the parent, not the dirty tree

Add a worktree at the parent revision and run the pack there. Write one row per fixture: id, callable, normalized output, and a hash of the input. Sort object keys. If a field is a timestamp, exclude it in the pack or render a fixed token, and record that choice beside the fixture.

Proposed commands, not a log from a run:

git worktree add ../baseline "$BASE_SHA"
python scripts/capture_baseline.py \
  --pack review_fixtures/pack.json \
  --out artifacts/baseline.json

Import the library from that worktree. A capture that imports the candidate branch will confirm the patch against the code it already replaced. Record the parent commit and the blob hash of baseline.json in the review note. The scorer below does not check that binding yet. Add the check before the gate is mandatory.

Step 3. Recapture on the candidate

Apply the agent diff in a second worktree. Recapture with the same pack and the same normalizer. Diff by fixture id, never by file order.

git worktree add ../candidate "$PATCH_SHA"
python scripts/capture_baseline.py \
  --pack review_fixtures/pack.json \
  --out artifacts/candidate.json
python scripts/score_drift.py \
  --baseline artifacts/baseline.json \
  --candidate artifacts/candidate.json \
  --ledger review_fixtures/drift_ledger.json \
  --new-symbols "$(python scripts/list_new_symbols.py artifacts/patch.diff)"

list_new_symbols.py can be a thin scan of the diff: public function names added under the package, ignoring names that start with an underscore. Keep the scan boring. A clever scanner becomes another oracle the agent can game.

Step 4. Class every row before merge talk

The scorer emits four classes. Read the table as policy, not as a benchmark.

Class	Rule	Result
`match`	Normalized output unchanged	Pass this fixture
`explained`	Output changed; ledger has this id, an allow-listed reason, and a reviewer id	Pass this fixture
`unexplained`	Output changed; ledger row missing or incomplete	Fail the run
`uncovered_new`	Diff adds a public symbol with no property name in the ledger	Fail the run

Allow-list the reason codes. A short set is enough: bugfix, spec-change, error-text, performance-neutral. Free text is a comment, not a code. Require a reviewer id so an unsigned file in the tree cannot pose as approval.

If the patch also adds tests, those tests must point at a fixture id that drifted, or at a property name listed for a new symbol. The checker will not prove the property is strong. It rejects one obvious cheat: an expected value that exists only as the candidate output, with no parent row. A separate review can still judge weak assertions. Do not fold that review into this script.

Step 5. Run the proposed scorer

The program below is a proposal. It uses the standard library only. It has not been executed for this article, and it reports no pass rate.

#!/usr/bin/env python3
"""Proposal: score a candidate capture against a parent baseline."""
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

ALLOWED = {"bugfix", "spec-change", "error-text", "performance-neutral"}

def load_rows(path: str) -> dict:
    data = json.loads(Path(path).read_text(encoding="utf-8"))
    rows = data["rows"] if isinstance(data, dict) else data
    return {row["id"]: row for row in rows}

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--baseline", required=True)
    parser.add_argument("--candidate", required=True)
    parser.add_argument("--ledger", required=True)
    parser.add_argument("--new-symbols", default="")
    args = parser.parse_args()

    base = load_rows(args.baseline)
    cand = load_rows(args.candidate)
    ledger_doc = json.loads(Path(args.ledger).read_text(encoding="utf-8"))
    ledger = {row["id"]: row for row in ledger_doc.get("drifts", [])}
    problems = []

    for fid, brow in sorted(base.items()):
        crow = cand.get(fid)
        if crow is None:
            problems.append(f"dropped: {fid}")
            continue
        if brow.get("norm") == crow.get("norm"):
            continue
        row = ledger.get(fid)
        reason = None if row is None else row.get("reason")
        reviewer = None if row is None else row.get("reviewer")
        if reason not in ALLOWED or not reviewer:
            problems.append(f"unexplained: {fid}")
            continue
        if row.get("parent_norm") != brow.get("norm"):
            problems.append(f"ledger-mismatch: {fid}")
        if row.get("candidate_norm") != crow.get("norm"):
            problems.append(f"ledger-mismatch: {fid}")

    covered = set(ledger_doc.get("new_symbol_properties", []))
    for name in [s for s in args.new_symbols.split(",") if s]:
        if name not in covered:
            problems.append(f"uncovered_new: {name}")

    if problems:
        print("\n".join(problems))
        return 1
    print(f"scored {len(base)} fixtures, 0 unexplained")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Expected shape of a failure, illustrative only, not a captured log:

unexplained: fx-014
uncovered_new: parse_header

Exit code 1 blocks the merge. Exit code 0 means this gate is satisfied, not that the patch is correct.

A ledger row that clears fx-014 has this shape:

{
  "drifts": [
    {
      "id": "fx-014",
      "reason": "bugfix",
      "reviewer": "reviewer-on-call",
      "parent_norm": "null",
      "candidate_norm": "ok"
    }
  ],
  "new_symbol_properties": ["parse_header_rejects_empty"]
}

The norm fields are copies for the human reader. The checker compares them with the artifact files and rejects a ledger that describes a drift the artifacts do not show.

How to read a week of ledgers

Do not invent a single quality score. Count classes. If most changed rows are explained with reason bugfix, reviewers are signing a break with the baseline on habit. Re-read those rows on a fixed cadence, such as once a week, and ask whether the parent output was actually wrong.

If uncovered_new is rare because every new symbol lists a property file that only echoes the candidate output, the file-exists check has been gamed. Open the property. When the expected value equals the candidate norm and the parent has no row, send it back. That is a post-edit snapshot under another name.

If match dominates on a patch that claims a behavior change, the pack missed the branch. Add a fixture for the claimed branch and recapture both trees. Do not update only the candidate file.

Drafting ledger text, and where the run can sit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The gate still works if a person writes every reason code and a laptop runs both captures. Two operator-supplied options only remove friction. Free model access can draft a reason-code sentence and a property name for each unexplained id. A free server option can run the two captures and score_drift.py off the laptop that produced the patch. This article does not name models, quotas, hardware, durations, or how long either option stays available. Read the current terms before you depend on them. Nothing here is a benchmark.

Keep the draft prompt narrow. Pass the parent norm, the candidate norm, and the allow-list. Ask for one code and one property that would have failed on the parent. Discard a code that is not in ALLOWED. Discard a property whose expected value is only the candidate output. The draft is a comment. The reviewer id in the ledger is the decision.

If those free options are already on the account, point the draft at unexplained ids only, and point the remote run at the two worktrees. Do not ask either one to approve the merge.

Limitations

Baselines lock in current behavior, bugs included. bugfix is the escape hatch. Unreviewed use of that code deletes the gate, because every drift then looks signed.

The pack is small on purpose. A match is not a proof. New branches need a property, or a metamorphic check you judge by reading, not by this script's exit code. Normalization can hide real changes: replace every timestamp and a broken clock will match. Keep the replace list short and in the pack header.

The checker trusts the files you pass it. Bind baseline.json to the parent blob hash before the gate is mandatory. The listing leaves that binding as a follow-up so the gap stays visible. Matching snapshots also says nothing about injection flaws, auth checks, or data races. Do not cite this gate in a security review.

Who should skip it

Skip it when the agent can edit the fixture pack in the same change, and skip it on a one-file prototype with no callers. Skip it when the task is a wholesale copy rewrite and almost every row will be error-text. Use a dedicated copy review instead of a ledger full of identical codes.

Teams that already fail the build on unexplained golden diffs can keep that system. If anything here is worth copying, copy the order: parent capture, then the patch, then a ledger a person signs. Do not copy a claim that an automated draft reviewed the patch. It drafted a sentence.

Partition the Agent Diff Before You Quarantine a Failure

Finley Zhou — Mon, 12 Oct 2026 16:19:37 +0000

Quarantine is a record of instability, not a receipt for an agent patch. If the same diff edits production code and tests, no flake freeze token should exist until the test edit is split away, the fixture digest matches baseline, property checks are replayed on the production tree alone, and a planted mutant still dies under the assertions you intend to keep.

That sequence is the gate. A green rerun after a combined diff only shows that the new tree agrees with itself.

The failure mode in the combined diff

Agent patches frequently land as one commit. A function changes, and so does the test that names it. The next red run is easy to file as a flake because the command line did not change.

The command line is the wrong identifier. The oracle may have moved inside the test file, or the fixture bytes may have moved under a stable path.

Split the candidate into two artifacts before any rerun budget is spent. Production paths feed the behavioral checks. Test paths feed an oracle-strength check. A freeze token can be discussed only when the candidate under review contains no test paths, and only after the other checks below agree.

A two-run disagreement does not replace this split. Disagreement tells you the results differed. It does not tell you which side of the diff purchased the green result.

1. Partition paths before you rerun

Start from the file list. Do not launch the suite in this step.

git diff --name-only origin/main...HEAD > /tmp/patch-files.txt
grep -E '(^|/)(test_|tests/|_test\.py|spec/)' /tmp/patch-files.txt > /tmp/test-paths.txt || true
grep -v -E '(^|/)(test_|tests/|_test\.py|spec/)' /tmp/patch-files.txt > /tmp/prod-paths.txt || true
wc -l /tmp/prod-paths.txt /tmp/test-paths.txt

Non-zero counts on both files mean split_required. While that mark is set, the freeze ledger should reject an open request. The production-only tree is what property checks must see. How you build that worktree depends on the forge; the invariant does not.

Sample the path regex against recent commits before you enforce it. Tests that sit beside production modules, and helpers that embed production logic under tests/, will be mis-filed by a generic pattern.

2. Hash fixture bytes, not fixture names

A lock that stores a path misses a silent rewrite. Hash the bytes the loader will read.

find fixtures -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum > /tmp/fixture-digest.txt
cat /tmp/fixture-digest.txt

Compare that digest with the last green build on the same branch, and with the digest from any rerun you are tempted to call intermittent.

Digest vs baseline	Test paths in candidate	Classification	Freeze token
Equal	None	Identity held; not yet a flake	Closed until later steps pass
Equal	One or more	Oracle may have moved	Closed until the mutant dies
Different	Either	Fixture epoch changed	Closed; drift, not a flake
Equal across disagreeing runs	None	Nondeterminism candidate	Eligible only if steps 3 and 4 pass

A digest change ends the flake hypothesis for that run. Review the fixture epoch, or revert it, before you spend reruns. Reruns on a moving fixture measure two experiments and store one label.

3. Replay properties on the production tree only

Property checks in this workflow are predicates over inputs you choose, with the seed written to disk. They are not a second copy of the example the agent just rewrote.

# Proposed runner. Not executed against a live suite in this draft.
import json
import random
from pathlib import Path

def replay_property(seed: int, trials: int, predicate) -> dict:
    rng = random.Random(seed)
    failures = []
    for i in range(trials):
        value = rng.randrange(0, 10_000)
        if not predicate(value):
            failures.append({"trial": i, "value": value})
            break
    return {
        "seed": seed,
        "trials": trials,
        "failed": bool(failures),
        "first_failure": failures[0] if failures else None,
    }

def load_seed(path: Path) -> int:
    return json.loads(path.read_text())["seed"]

Persist seed, trials, and failed next to the run id. A failure on the production-only tree is a behavioral reject. It does not enter the freeze ledger, even if a later full-tree command looks intermittent.

Keep the predicate in a module the current patch did not edit. If the only predicate lives in the edited test file, stop and add a reviewer-owned predicate first. An agent-written predicate can restate the defect and stay green.

Count assertion-line churn as a separate signal, not as a score. Removed or widened asserts are review input. They do not, by themselves, prove the oracle was weakened.

git diff -U0 origin/main...HEAD -- $(cat /tmp/test-paths.txt) \
  | grep -E '^[+-].*assert' \
  | sed -n '1,40p'

The mutant probe in the next step is what turns that suspicion into an exit code. Treat the diff listing as a queue for review, not as a numeric quality rating.

4. Plant one mutant and require a kill

One mutant is a narrow probe. It is not a suite-wide mutation score, and this draft does not report one. The probe asks a single question: do the tests you plan to keep still fail when production behavior is wrong in a way you planted?

# Proposed evidence gate. Not executed in this draft.
from enum import Enum
from dataclasses import dataclass

class Verdict(str, Enum):
    SPLIT_REQUIRED = "split_required"
    FIXTURE_DRIFT = "fixture_drift"
    PROPERTY_FAIL = "property_fail"
    ORACLE_WEAKENED = "oracle_weakened"
    FREEZE_ELIGIBLE = "freeze_eligible"
    HOLD = "hold"

@dataclass(frozen=True)
class Evidence:
    prod_files: int
    test_files: int
    digest_stable: bool
    property_failed_on_prod_only: bool
    mutant_killed: bool
    nondeterministic_same_digest: bool

def classify(ev: Evidence) -> Verdict:
    if ev.prod_files and ev.test_files:
        return Verdict.SPLIT_REQUIRED
    if not ev.digest_stable:
        return Verdict.FIXTURE_DRIFT
    if ev.property_failed_on_prod_only:
        return Verdict.PROPERTY_FAIL
    if ev.test_files and not ev.mutant_killed:
        return Verdict.ORACLE_WEAKENED
    if (
        ev.nondeterministic_same_digest
        and ev.test_files == 0
        and ev.mutant_killed
    ):
        return Verdict.FREEZE_ELIGIBLE
    return Verdict.HOLD

Plant the mutant on a disposable branch. Do not plant it on a shared default branch.

git switch -c audit/mutant-probe
# Apply one local, reversible behavior change in a production file.
python -m pytest tests/test_order_total.py -q --tb=line
echo "mutant_exit=$?"
git switch -
git branch -D audit/mutant-probe

mutant_exit=0 means the tests accepted a known-bad change. The verdict is oracle_weakened. Do not open a freeze, and do not merge that test edit on the strength of a green agent run.

A non-zero exit means this probe still bites. That result is necessary and not sufficient. A different fault could still slip past assertions this probe never touched.

A test-only candidate never becomes freeze_eligible in this function. If the edited tests kill the probe, the label is hold. If they do not, the label is oracle_weakened. Quarantine is not a side door for an assertion edit.

5. Issue a token only for the remaining shape

All of the following must hold before a freeze token is writable. Missing any one line keeps the ledger closed.

The candidate under review has test_files == 0.
The fixture digest matches baseline and matches across the disagreeing runs.
Property replay on the production-only tree did not fail.
The mutant probe against the tests you are keeping was killed.
The same command disagreed with itself on that digest.

Store the token as data. Bind voiding to digest change, not only to a calendar date your team already uses for quarantines. If the digest moves, the token is void even when the date is still open.

The object below is an example shape, not a recorded run. Leave model prose out of the file the merge job reads.

{
  "verdict": "freeze_eligible",
  "fixture_digest": "<sha256 from step 2>",
  "property_seed": 1729,
  "property_failed": false,
  "mutant_killed": true,
  "test_files_in_candidate": 0,
  "expires_on_digest_change": true
}

The merge job should read verdict and the digest, then stop. A comment in the pull request is not a substitute for those two fields.

What may be drafted elsewhere

Writing a reviewer-owned predicate is clerical work. Producing the digest and the mutant exit code needs a clean runner, so a laptop virtualenv does not become part of the evidence.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft a candidate predicate for a human to edit, and the free server option can run the classification job away from an interactive workstation. Those two availability facts are not quotas, hardware specs, or a promise that the draft is correct. The gate remains classify(evidence).

If you already have that free model access, keep the prompt to a predicate, a fixed seed, and one failing example. Paste the draft into a file this patch does not modify. Use the free server option as the runner for the digest and the mutant exit, then copy the evidence JSON back and apply the table locally.

Limitations

The procedure does not estimate how many faults the suite would catch. One planted site can miss a weakened assertion on another branch.

Digest equality ignores nondeterminism that never lands in fixture bytes. Wall clocks, unordered rendered text, and ambient network calls can still flip a result while the hash stays put. Pin or forbid those sources before you treat nondeterministic_same_digest as meaningful. Otherwise the token launders an environment bug.

Property replay is only as honest as the predicate. Reviewer ownership reduces the chance that the same agent restates the bug. It does not prove the predicate is independent.

Path partitioning is a heuristic. Enforce it only after you have checked it against a sample of your own history. No pass rate, duration, or quota is claimed in this note; the artifact is a decision procedure, and it has not been executed against a live suite here.

Who should skip it

Skip the procedure when you cannot materialize a production-only tree. Without the split, the later checks run on the combined diff and answer a different question.

Skip it when a behavior edit cannot live on a disposable branch. Shared staging and incident-sensitive sandboxes are the wrong place to plant a mutant.

Skip it when fixtures are built only in memory and never written to bytes. Hashing an empty directory will look stable while inputs move.

Skip it when the goal is for an assistant to close the freeze or approve the merge. The classifier returns a label. A person still owns the token.

Four counts before merge

Read four values from the evidence file: production file count, test file count, digest match, mutant exit code. If the test count is non-zero, quarantine is not available for this candidate. Split the diff, keep the probe hostile, and only then decide whether the remaining red run is actually unstable.

Shrink the Witness Before a Flake Freeze Can Open

Finley Zhou — Mon, 12 Oct 2026 12:47:43 +0000

An agent patch does not earn a flake freeze from one red run on a wide fixture. The freeze may open only after the property predicate, the stopped witness, and a disagreeing rerun series are all on the same record. A stable failure is a reject, and a stable pass is not a freeze.

That rule is narrower than rerunning a suite until it turns green. A rerun hides a regression when the input is still large, the seed changes, or the property text moves with the patch. Shrink first, then repeat, then classify.

A digest match on the original fixture is not the decision. The digest that matters is the one taken after shrinking stops, on the patch that actually ran.

Bind four identifiers before you classify

Every attempt carries a patch digest, a property id, a fixture digest, and a seed. Missing any one of them is a hold, not a pass and not a freeze. The property id must point at a predicate a reviewer can read without opening a model transcript. The fixture digest hashes the input that actually ran, not the directory the agent was given at the start.

A property is a pure check over an explicit fixture and the observed output. Wall clocks, network calls, and map iteration order stay out of the predicate unless they are fixture fields. If the patch edits the predicate, the property id changes. An oracle that the diff rewrote cannot certify the edit that rewrote it.

Hash the patch before the first run. If the worktree does not match that hash, stop. Later reruns that include a small fix are a different patch and need a new record. The gate does not inherit evidence across digests.

Shrink until the failure stops moving

A failing fixture is often larger than the bug. Unused flags, extra rows, and stale optional keys make the next run look random when one field is doing the work. Shrinking deletes or narrows a piece only when the same property still fails. The loop stops when the next candidate no longer fails, or when the candidate is identical to the current fixture.

The stopped fixture is the witness. Later reruns must match its digest. A rerun on the original wide file goes back to hold, even if the property id is unchanged. Otherwise the freeze record cites a test that was not the one under dispute.

This stop condition does not prove the witness is minimal in the mathematical sense. It proves only that the supplied shrink function had no further failing step within a fixed step budget. That is a bound on the tool. It is not a claim about every smaller input a person might invent.

Keep the failure identity stable while shrinking. If the property returns a reason code, a shrink step that changes the code is refused and the previous fixture remains the witness. A new reason code is a different failure, and it should not be folded into the original freeze discussion.

Decision table

The table below is the whole classifier. It has no hidden score.

Observed state	Gate
Patch digest, property id, fixture digest, or seed is empty	Hold
Shrink loop has not stopped, or the step budget ended without a local stop	Hold
Fewer reruns than the policy minimum on that witness	Hold
Outcome labels are outside `pass` and `fail`	Hold
Every recorded rerun passes	Pass
Every recorded rerun fails	Reject
Reruns disagree, shrink stopped, expiry timestamp present	Freeze candidate
Reruns disagree, expiry timestamp empty	Hold

The sample defaults are policy fields, not measurements.

Policy field	Value in the sample	What it is not
`min_reruns`	3	a confidence interval or a false-accept rate
`max_steps`	32	a proof that no smaller failing input exists
`freeze_expires`	required non-empty string	a recommended duration

Three reruns is the smallest series that can show agreement, total failure, or disagreement without treating one run as a story. Teams that already require five reruns should pass that number in, and should not treat the default as a benchmark.

A freeze candidate is not a merge approval. It records that one witness was unstable under one seed and one digest. Expiry is an operator-supplied timestamp, and this article does not recommend a duration. An empty expiry keeps the candidate closed, so a freeze cannot become a permanent skip by omission.

Numbered procedure

Compute the patch digest and refuse to run if the worktree is dirty relative to it.
Select a property id the diff does not touch. If the predicate file is in the diff, stop and review the oracle before any rerun or shrink.
Execute the property once on the submitted fixture, with the seed recorded in the fixture header or the job spec. Store the outcome and the fixture digest.
On failure, shrink until a local stop or until the step budget ends. If the budget ends without a stop, hold. Do not freeze a witness the shrinker was still changing.
Hash the stopped witness. Rerun the same property on that digest with the same seed until the policy minimum is met. Do not draw a new seed between attempts.
Apply the decision table. Persist the four identifiers, the outcomes, and the expiry field in one record. A pull-request comment is not a substitute for that record.

Initial passes still need an explicit sample plan when the patch claims to fix a prior flake. One green run on a wide fixture does not retire a freeze candidate from an older digest. The old candidate expires on its own timestamp, or a reviewer closes it by pointing at a new stable pass on the same property id and a recorded witness. It does not vanish because the branch is green today.

Proposal code

The module below is a proposal. It was not executed against a production suite for this article. It does not call a model, and it does not open a network connection. fails and shrink_once are injected by the harness you already trust.

import hashlib
from dataclasses import dataclass
from enum import Enum

class Gate(str, Enum):
    PASS = "pass"
    REJECT = "reject"
    HOLD = "hold"
    FREEZE_CANDIDATE = "freeze_candidate"

@dataclass(frozen=True)
class Witness:
    patch_digest: str
    property_id: str
    fixture_digest: str
    seed: str
    shrunk: bool
    outcomes: tuple[str, ...]
    freeze_expires: str
    shrink_budget_hit: bool = False

def sha256_bytes(payload: bytes) -> str:
    return hashlib.sha256(payload).hexdigest()

def bounds_check(fixture: dict, output: dict) -> bool:
    return fixture["lo"] <= output["value"] <= fixture["hi"]

def drop_optional(fixture: dict):
    if "optional_note" not in fixture:
        return None
    nxt = dict(fixture)
    del nxt["optional_note"]
    return nxt

def decide(w: Witness, min_reruns: int = 3) -> Gate:
    if min_reruns < 2:
        raise ValueError("min_reruns must compare more than one attempt")
    required = (w.patch_digest, w.property_id, w.fixture_digest, w.seed)
    if any(not item for item in required):
        return Gate.HOLD
    if w.shrink_budget_hit or not w.shrunk:
        return Gate.HOLD
    if len(w.outcomes) < min_reruns:
        return Gate.HOLD
    if any(item not in {"pass", "fail"} for item in w.outcomes):
        return Gate.HOLD
    unique = set(w.outcomes)
    if unique == {"pass"}:
        return Gate.PASS
    if unique == {"fail"}:
        return Gate.REJECT
    if not w.freeze_expires:
        return Gate.HOLD
    return Gate.FREEZE_CANDIDATE

def shrink_until_local_stop(fixture, fails, shrink_once, max_steps=32):
    current = fixture
    for _ in range(max_steps):
        if not fails(current):
            return None
        nxt = shrink_once(current)
        if nxt is None or nxt == current or not fails(nxt):
            return current
        current = nxt
    return current

bounds_check is only a shape example. Replace it with the predicate the patch claims to satisfy. drop_optional shows the shrink contract: return None when nothing optional remains, and never drop lo or hi in the same step. shrink_until_local_stop returns None when the current fixture does not fail, which signals an unsound step or a mismatched predicate. Callers should hold in that case rather than invent a witness.

A fixture header that can be hashed as canonical JSON looks like this. Sort keys before hashing so two equivalent objects do not produce two digests.

{
  "patch_digest": "<64 lowercase hex chars>",
  "property_id": "bounds_check",
  "seed": "17",
  "fixture": {"lo": 0, "hi": 10, "optional_note": "drop-me"}
}

A branch check can sit beside the module. The command is an example, not a reported run from a published build. The timestamp only shows that the expiry field is populated.

python - <<'PY'
from gate_witness import Witness, Gate, decide
base = dict(patch_digest="p"*64, property_id="bounds_check",
            fixture_digest="f"*64, seed="17", shrunk=True,
            shrink_budget_hit=False)
flake = Witness(**base, outcomes=("fail", "pass", "fail"),
                freeze_expires="2026-10-19T00:00:00Z")
assert decide(flake) is Gate.FREEZE_CANDIDATE
stable = Witness(**base, outcomes=("fail", "fail", "fail"),
                 freeze_expires="2026-10-19T00:00:00Z")
assert decide(stable) is Gate.REJECT
no_end = Witness(**base, outcomes=("fail", "pass", "fail"), freeze_expires="")
assert decide(no_end) is Gate.HOLD
print("branches ok")
PY
sha256sum witness/minimal.json

Use the expiry your review policy already requires. Do not copy the sample timestamp into a real record.

Separate the draft from the rerun

Writing a predicate and executing it are different jobs. A fluent draft can look like a test while still encoding the bug it is supposed to catch. Keep generation on one side of the record and execution on the other.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access can draft a candidate predicate or a candidate shrink step as text. That text is untrusted input. It enters the repository only after the same review you would apply to a human patch, and it must not be a file the agent diff already rewrote.

MonkeyCode's free server option can host the rerun command so the witness digest is computed outside a worktree that still has uncommitted fixture edits. This article does not state quotas, hardware, model names, or how long either option remains available. If the remote job cannot print the patch digest it executed, discard the outcomes and hold.

Pin the command, the seed, and the witness file in the job spec. A runner that rebuilds a lookalike fixture is not repeating the witness. Similarity is not a digest match.

Limitations

The shrinker is unsound when shrink_once removes the field that caused the failure and fails still returns true for a different reason. The gate will then reject or freeze the wrong witness. Prefer shrink steps that delete one optional key at a time, and refuse a step whose reason code changes.

Disagreement across the minimum reruns does not name the flake source. The cause may be a race, hash-ordered iteration, or a clock the fixture forgot to pin. The candidate only blocks automatic merge. It is not a root-cause report.

A pass on the stopped witness says nothing about inputs the shrinker never produced. If the property is meant to cover a range, keep a separate broader sample. That sample must cite the same patch digest and the same property id, or it belongs to a different gate and cannot be added into this one.

No false-accept rate is claimed for the branch checks. The shell snippet locks control flow only. It does not inject faults, and it should not be cited as a detection percentage.

Who should leave this gate closed

Do not point a shrink loop at live third-party services. Repeated calls create load, and remote jitter masquerades as disagreement.

Do not use the table when the patch exists to change the property file. Review that oracle by hand, then mint a new property id.

Do not open a freeze during incident response if the failing check is the only signal on a known bad deploy. Incidental disagreement is not a reason to silence that signal. The stable-fail branch is the relevant output there, and a freeze is the wrong one.

Also skip the procedure if the four identifiers cannot be stored beside the result. A green badge without a witness digest is not evidence the property ran.

If a free server seat is already how reruns are hosted, print the patch digest and the witness digest before the gate word, and keep the shrink log in the same commit as that patch digest. The decision table still outranks the runner.

An Extra Call Keeps the Flake Freeze Shut

Finley Zhou — Mon, 12 Oct 2026 09:16:47 +0000

Matching output is not a pass. Not when the patch spent an extra call, sleep, or read to produce that output. A flake freeze stays shut until a shrink yields one stable fixture and a bilateral replay fails the same assertion on both revisions, with every accepted effect budget still holding. A retry that only repairs the returned dict meets none of those tests.

Agent patches blur that line in a few edits. The diff adds a loop or a second read, the happy-path assertion turns green, and the red budget check is filed as a flake. That filing is a classification error.

Flakes are unstable observations of one assertion. Regressions are stable disagreements, including disagreements about how often a dependency was touched. This workflow keeps the two classes in separate lanes.

The steps below are a proposal. Nothing in this draft was executed, and no pass rate is claimed. Thresholds are parameters you fill from your own logs.

Separate the lanes before you name the failure

The property lane may reject a patch. It owns invariants, call budgets, and oracle identity. A failure here blocks merge, even when the returned value looks familiar.

The flake lane may only queue a case for later observation. Entry requires the eligible row of the decision table, an expiry, and a fixture digest. Eligibility is not merge approval, and it does not cover other failing checks in the same diff.

The patch this gate is built to catch

Consider a helper an agent might edit, normalize_event(payload, now). The base revision calls now once and returns a dict. A weak patch catches a parse miss, calls now again, and returns that same dict.

Output equality holds. The call budget does not. An output-only flake freeze would hide the extra call. The gate below is built so that hide fails.

1. Pin revisions before anyone drafts a test

Record base_rev, patch_rev, the test command, and the runner image digest. A floating image tag is not evidence. Write those four values into a run log before a model sees the diff.

Do this even when the runner is a laptop. The pin is what makes the later replay comparable, not the brand of the machine.

2. Accept properties only after a mutant dies

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access matters here only as a source of candidate property text. The free server option matters only as a possible host for shrink and replay.

This article does not name a model, a quota, a machine size, or a duration. Copy those fields from the console on the day you run, or leave them blank and use another runner you already operate.

Treat every drafted property as untrusted text. Accept it only when three local checks pass.

The statement names an input and a relation on an output or an effect.
A hand-written mutant that is still in the tree makes the statement fail.
The statement is not a tautology: no constant assert, and no comparison of a function with its own fresh return.

For this helper, the budget property is the mutant killer. The clock is a counter. One public call must increment it once.

# Proposal. Not executed for this draft.
def test_clock_budget(normalize_event, clock):
    payload = {"id": "evt-1", "ts": "1970-01-01T00:00:00Z"}
    normalize_event(payload, now=clock)
    assert clock.calls == 1

If that assertion fails on the patch, the lane is property regression. Do not add a freeze file in that same change.

3. Shrink until the minimum repeats

Capture the payload that broke an accepted property. Reduce it with a deterministic function: drop optional fields, shorten strings, and replace timestamps with one fixture epoch. Run that reducer at least twice. Call the shrink stable only when every run produces the same digest.

required_identical is your parameter. This draft sets no recommended constant. If you have logs, count how often a second shrink changed the minimum across recent failures, and set the parameter above that count. If you have no logs, use 2 and mark the lane provisional.

# Proposal. Not executed for this draft.
import hashlib
import json

def digest(payload):
    raw = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(raw.encode()).hexdigest()

def shrink_event(payload):
    event_id = payload.get("id", "evt")
    return {"id": event_id, "ts": "1970-01-01T00:00:00Z"}

def shrink_report(payload, runs):
    minima = [digest(shrink_event(payload)) for _ in range(runs)]
    last = minima[-1]
    identical = sum(1 for item in minima if item == last)
    return {
        "minimal_digest": last,
        "attempts": runs,
        "identical_minima": identical,
        "stable": identical == runs and runs >= 2,
    }

The reducer is deliberately small so the stability rule stays visible. Swap in the reducer for your own type. Keep the report keys. Store the unshrunk payload beside the minimum for one review cycle, because shrinking can collapse two bugs into one fixture.

4. Replay the minimum on both revisions

Check out the pinned revisions in two worktrees. Copy the same fixture into both. Run the same command. Record assertion id, pass or fail, fixture digest, and clock-call count for each side.

# Proposal commands. Substitute your revisions and image digest.
git worktree add /tmp/base "$BASE_REV"
git worktree add /tmp/patch "$PATCH_REV"
cp fixtures/min.json /tmp/base/fixtures/min.json
cp fixtures/min.json /tmp/patch/fixtures/min.json
docker run --rm -v /tmp/base:/src -w /src "$IMAGE_DIGEST" pytest -q tests/test_budget.py
docker run --rm -v /tmp/patch:/src -w /src "$IMAGE_DIGEST" pytest -q tests/test_budget.py

Use a free server for that pair only if the console still lists the option for the account that will run the job. Pin IMAGE_DIGEST from your registry. Do not infer isolation, persistence, or a rental lifetime from the label free.

If the option is absent, run the same two commands on a runner you already operate. The decision table does not change.

5. Apply the decision table

Read rows from top to bottom and stop at the first match. Call inequality is judged only after the oracle and the shrink are acceptable. A changed oracle invalidates both sides, so it never becomes a freeze.

Base	Patch	Shrink	Same assertion	Calls	Lane
any	any	any	any	oracle file changed	reject the run
any	any	unstable	any	any	rerun shrink; do not classify
any	any	stable	any	not equal	property regression; freeze denied
pass	fail	stable	yes	equal	property regression; freeze denied
fail	fail	stable	no	equal	different bug; freeze denied
fail	fail	stable	yes	equal	flake queue eligible
fail	pass	stable	yes	equal	fix candidate; other properties still apply
pass	pass	stable	yes	equal	no freeze

Row 3 is the retry disguise. The dict can match, and the patch is still a regression, because the effect budget moved. Row 6 is the only path into the flake queue. Equal calls are required there. An extra sleep that preserves the return value still dies on row 3.

# Proposal. Not executed for this draft.
def classify(base_passed, patch_passed, same_assertion, shrink_stable,
             base_calls, patch_calls, oracle_changed):
    if oracle_changed:
        return "reject_run"
    if not shrink_stable:
        return "rerun_shrink"
    if patch_calls != base_calls:
        return "property_regression"
    if base_passed and not patch_passed:
        return "property_regression"
    if (not base_passed) and (not patch_passed) and not same_assertion:
        return "different_bug"
    if (not base_passed) and (not patch_passed) and same_assertion:
        return "flake_queue_eligible"
    if (not base_passed) and patch_passed and same_assertion:
        return "fix_candidate"
    if base_passed and patch_passed:
        return "no_freeze"
    return "property_regression"

def test_classifier_examples():
    assert classify(True, True, True, True, 1, 2, False) == "property_regression"
    assert classify(False, False, True, True, 1, 1, False) == "flake_queue_eligible"
    assert classify(False, False, True, False, 1, 1, False) == "rerun_shrink"
    assert classify(True, False, True, True, 1, 1, False) == "property_regression"
    assert classify(False, True, True, True, 1, 1, False) == "fix_candidate"
    assert classify(True, True, True, True, 1, 1, True) == "reject_run"
    assert classify(True, True, True, True, 1, 1, False) == "no_freeze"

test_classifier_examples checks the rule, not a product. A green local run of that function does not mean your reducer is faithful. It only means the table and the function agree.

python -m pytest -q tests/test_classify.py

6. Write a freeze entry only from the eligible row

The entry has five fields: fixture digest, assertion id, base revision, patch revision, and an expiry you choose. Drop the entry if any field is missing. Do not let a model fill the expiry.

The entry waives nothing else in the patch. It does not renew when a draft is regenerated. When the expiry passes, delete the entry or replace it with a fresh bilateral replay. A missing follow-up is an open risk, not a quiet pass.

# Proposal schema. Not a recorded run.
base_rev: "<git sha>"
patch_rev: "<git sha>"
image_digest: "sha256:<pin>"
fixture_digest: "sha256:<from digest()>"
assertion_id: "test_clock_budget"
lane: "flake_queue_eligible"
expiry: "<ISO-8601 you choose>"
console_model_id: "<paste or leave blank>"
console_server_offer: "<paste or leave blank>"

Counts worth keeping

For the next 20 agent patches, record four integers from your own runs. Do not borrow figures. This draft has none to offer.

Properties drafted, versus properties that killed a mutant.
Failures whose second shrink changed the minimal digest.
Cases where output matched and call count did not.
Freeze entries still present after expiry.

Count 3 is the health check for output-only suites. If it is above zero, matching dicts are not evidence for this helper. Count 2 tells you whether required_identical is too low. Count 4 tells you whether the queue is becoming a side door.

Plot the four counts as a simple weekly tally. A rising count 3 with a rising count 4 means reviewers are parking call-budget regressions in the flake queue. Tighten acceptance, not the retry loop inside the patch.

Limitations

Shrinkers throw information away. Two defects can share one minimal payload, so the table can under-count bugs. That is why the unshrunk payload stays attached for a review cycle.

The budget check needs an injectable clock or counter. If the patch builds a private sleeper inside a closure you cannot observe, the gate will not see the wait. Instrument that seam first. Until you can, do not claim a call-budget property.

Bilateral replay is the wrong tool when the patch changes the contract on purpose. The oracle file then differs, and the table says reject the run. A reviewer writes a new property and repeats the mutant check. Updating a snapshot inside the agent diff is not that review.

Free-model drafts can be tautologies with serious names. Step 2 is the countermeasure. Skip it, and later lanes will certify whatever sentence was drafted.

A disposable server does not improve a bad fixture. Byte-unstable inputs, live clocks, and unlocked network calls make both the digest and the bilateral compare meaningless. Freeze the fixture before you freeze the test.

Equal call counts can still hide a behavior change if the counter is coarse. A clock that increments once per public call will miss a nested retry that reuses the same tick. Split the counter by dependency name before you trust row 6.

Who should skip this gate

Skip it when no base revision exists. One side cannot separate a flake from a regression. Wait until the parent commit is identifiable.

Skip it for browser tours that cannot pin a fixture digest. The shrink rule needs byte-stable input. A live backend does not provide that.

Skip it as an incident hotfix when waiting would extend an outage. Write down the skipped step, the revision, and a follow-up date. Do not invent a freeze entry afterward and treat it as evidence.

Skip it when every property compares a function to itself. The classifier will return no_freeze. That answer is consistent, and it does not protect the patch.

Closing check

Feed the classifier a mutant that adds one clock call and returns the original dict. The expected lane is property_regression. If you observe flake_queue_eligible or no_freeze, the gate is wrong, and the patch stays closed.

If MonkeyCode supplies the draft or the runner, confirm on that account's console that free model access and the free server option are still listed, then paste those console facts into the run log. The lane rule stands without either offer.