DEV Community: Finley Zhou

Two Authors, One Patch: Don't Let the Agent That Wrote the Code Write the Oracle

Finley Zhou — Sat, 07 Nov 2026 19:59:15 +0000

A patch and its test are two claims, and an agent that writes both has effectively written neither one for you. The test will agree with the code because the same context produced them, not because the behavior was ever specified. If your gate only asks "did the new test pass after the patch", self-confirmation sails through untouched.

The fix is structural, not rhetorical: separate the author of the patch from the author of the oracle, and then prove the separation held by measuring whether each new oracle fails on the unpatched tree. That measurement — base-pass rate — is the number this post is about.

Earlier posts here covered diff-scoped gating and seeded flaky freezes. This one targets a different hole: oracles that were never capable of failing.

1. The failure mode, stated precisely

An oracle is only evidence if it is falsifiable by the pre-patch tree. Three ways that property silently disappears:

Same-context confirmation. The patch author knows which line it changed and asserts exactly that line's new output. Fine as a smoke check, useless as a specification.
Fixture absorption. The patch also edits the fixture, so the assertion compares the new code to the new expectation.
Spec-free tautology. The oracle asserts result is not None because the issue text was never given to the test author — only the diff was.

All three survive a green run. None survive a base-tree replay.

2. The split protocol

Three numbered steps, each mechanically enforceable.

Step 1 — Build a sealed spec. Take the issue text plus the public surface of non-test modules at the base commit: signatures, docstrings, exported names. Strip function bodies. The oracle author must never see the diff, and never see the new implementation.

Step 2 — Author oracles in a fresh session. No shared history with the patch session. Ask for pytest functions only, one behavior per test, no network, no sleeps, no imports of private helpers.

Step 3 — Classify, don't just execute. Run the generated file against a clean worktree of the base branch and against the patched tree. A new oracle that passes on both is base_pass — it was satisfied by unfixed code and cannot be evidence for this patch.

3. A runnable harness

The script below does steps 1–3 and appends verdicts to a ledger. It is plain Python plus git and pytest; nothing else.

#!/usr/bin/env python3
"""oracle_split.py — author oracles under a sealed spec, then classify them."""
import ast, hashlib, json, os, re, subprocess, sys, tempfile, urllib.request
from pathlib import Path

TEST_RE = re.compile(r"(^|/)(tests?|spec)/|(^|/)test_.*\.py$|_test\.py$")
GENERATED = "test_oracle_split_generated.py"

SYSTEM = """You write pytest oracles from a sealed specification.
Rules: one behavior per test; no network; no sleeps; no imports of private
helpers (names starting with _); assert observable behavior only; a test that
asserts 'is not None' alone is invalid. Return only a python file."""

def sh(*cmd, cwd, timeout=1800):
    p = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=timeout)
    return p.returncode, p.stdout, p.stderr

def diff_paths(repo, base):
    _, out, _ = sh("git", "diff", "--name-only", f"{base}...HEAD", cwd=repo)
    return [l for l in out.splitlines() if l.strip()]

def public_surface(repo, base):
    """Signatures + first docstring line only. Bodies are never read."""
    surface = {}
    _, out, _ = sh("git", "ls-tree", "-r", "--name-only", base, cwd=repo)
    for path in out.splitlines():
        if not path.endswith(".py") or TEST_RE.search(path):
            continue
        rc, src, _ = sh("git", "show", f"{base}:{path}", cwd=repo)
        if rc != 0:
            continue
        try:
            tree = ast.parse(src)
        except SyntaxError:
            continue
        sigs = []
        for node in ast.walk(tree):
            if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
                sig = node.name
                if hasattr(node, "args"):
                    a = node.args
                    names = [x.arg for x in a.posonlyargs + a.args + a.kwonlyargs]
                    sig += f"({', '.join(names)})"
                doc = (ast.get_docstring(node) or "").splitlines()
                sigs.append(f"{sig}  # {doc[0] if doc else ''}")
        if sigs:
            surface[path] = sigs
    return surface

def sealed_spec(repo, issue_text, base):
    return {
        "issue": issue_text,
        "public_surface": public_surface(repo, base),
        "forbidden": "The diff and the new implementation are withheld. Do not guess them.",
    }

def author_oracles(spec, timeout=240):
    endpoint = os.environ["ORACLE_ENDPOINT"]      # chat-completions compatible
    model = os.environ["ORACLE_MODEL"]
    payload = {"model": model, "temperature": 0,
               "messages": [{"role": "system", "content": SYSTEM},
                            {"role": "user", "content": json.dumps(spec, indent=2)}]}
    req = urllib.request.Request(
        endpoint, data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ.get('ORACLE_KEY', '')}"})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return json.load(r)["choices"][0]["message"]["content"]

def run_pytest(cwd, rel_path):
    rc, out, err = sh(sys.executable, "-m", "pytest", rel_path, "-v", "--tb=no",
                      "-p", "no:cacheprovider", cwd=cwd)
    status = {}
    for line in (out + err).splitlines():
        m = re.match(r"^(\S+::\S+)\s+(PASSED|FAILED|ERROR|SKIPPED)", line)
        if m:
            status[m.group(1).split("::", 1)[-1]] = m.group(2)
    return status

def classify(repo, tests_src, base):
    with tempfile.TemporaryDirectory() as tmp:
        base_tree = Path(tmp) / "base"
        sh("git", "worktree", "add", "--detach", str(base_tree), base, cwd=repo)
        try:
            target = base_tree / "tests" / GENERATED
            target.parent.mkdir(parents=True, exist_ok=True)
            target.write_text(tests_src)
            base_status = run_pytest(base_tree, f"tests/{GENERATED}")
        finally:
            sh("git", "worktree", "remove", "--force", str(base_tree), cwd=repo)
        live = Path(repo) / "tests" / GENERATED
        live.write_text(tests_src)
        try:
            patch_status = run_pytest(repo, f"tests/{GENERATED}")
        finally:
            live.unlink(missing_ok=True)
    verdicts = {}
    for tid, s in patch_status.items():
        b = base_status.get(tid, "MISSING")
        if s == "PASSED" and b in ("FAILED", "ERROR"):
            verdicts[tid] = "fail_to_pass"    # real evidence
        elif s == "PASSED" and b == "PASSED":
            verdicts[tid] = "base_pass"       # satisfied by unfixed code
        elif s == "FAILED" and b == "PASSED":
            verdicts[tid] = "regression"
        elif s == "FAILED" and b == "FAILED":
            verdicts[tid] = "fails_always"
        else:
            verdicts[tid] = "unknown"
    return verdicts, base_status, patch_status

def oracle_id(tests_src):
    return hashlib.sha256(tests_src.encode()).hexdigest()[:16]

def append_ledger(path, record):
    p = Path(path); p.parent.mkdir(parents=True, exist_ok=True)
    with p.open("a") as f:
        f.write(json.dumps(record, sort_keys=True) + "\n")

if __name__ == "__main__":
    repo, base, issue_file, ledger = sys.argv[1:5]
    issue = Path(issue_file).read_text()
    spec = sealed_spec(repo, issue, base)
    tests_src = author_oracles(spec)
    verdicts, base_status, patch_status = classify(repo, tests_src, base)
    oid = oracle_id(tests_src)
    append_ledger(ledger, {"oracle_id": oid, "base": base,
                           "changed": diff_paths(repo, base),
                           "verdicts": verdicts, "state": "shadow"})
    bad = {k: v for k, v in verdicts.items() if v in ("base_pass", "regression")}
    print(json.dumps({"oracle_id": oid, "counts": {
        s: list(verdicts.values()).count(s) for s in set(verdicts.values())},
        "blocking_findings": bad}, indent=2))
    sys.exit(1 if bad else 0)

Run it as python oracle_split.py . origin/main issue.md tests/oracle_ledger.jsonl. Example report shape (not a measurement from any repo — run it on yours):

{
  "oracle_id": "3f9a1c0d2b7e4a51",
  "counts": {"fail_to_pass": 4, "base_pass": 2, "fails_always": 1},
  "blocking_findings": {"test_parser_returns_list": "base_pass",
                        "test_retry_counts_attempts": "base_pass"}
}

4. The metric that matters: base-pass rate

Track base_pass / (base_pass + fail_to_pass) per patch batch. It is the share of newly authored oracles that the unfixed tree already satisfies. High values mean your spec is too thin, your oracle author is guessing at the diff, or the issue text is a restatement of the code.

Two secondary numbers keep the metric honest: fails_always catches oracles that reproduce a pre-existing bug and can never go green, and unknown catches collection errors, which usually mean the sealed surface omitted something the oracle legitimately needed.

5. Oracle probation instead of a hard gate

A brand-new oracle should not be allowed to block or unblock a merge on its first appearance. Give it a state machine in the ledger:

Ledger state	Gate effect	Promotion rule	Retirement rule
`shadow`	recorded, never blocks	3 consecutive `fail_to_pass` verdicts on distinct patches	14 days with no promotion, or 2 `base_pass`
`blocking`	failures block the merge	—	2 `base_pass`, or 1 `regression` (page a human)
`retired`	skipped, history kept	—	terminal

The retirement rule is what keeps the suite from rotting into regression guards. An oracle that starts passing on the base tree has stopped describing the fix; it either became redundant or was never specific. Retire it, and keep the hash in the ledger so the diff history stays auditable.

6. Where the second session runs

Running two sessions per pull request is the whole cost of this protocol, and it is a cost worth engineering down. Point ORACLE_ENDPOINT at any chat-completions-compatible endpoint you control; if you are already working inside MonkeyCode, the free model access covers the oracle-authoring call, which is the one call you most want to keep cheap because it must happen on every patch, not just the risky ones.

The job itself is mostly waiting: one model round trip, two pytest subprocesses, a git worktree add and remove. That profile suits the free server option well — run the harness in a disposable container there so oracle authoring never consumes CI minutes on your main runner and never touches a developer laptop's working tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you already have a gate that runs diff-scoped tests, adding this split is roughly a two-file change: the harness above plus a ledger path. Start in shadow for a week and look at the base_pass count before you let anything block.

7. Limitations, and who should skip this

The split does not prove spec completeness. Two authors can both miss a behavior. It removes self-confirmation, not uncertainty.
Sealed specs leak. Signature names and docstrings hint at design. If your public surface is one giant run(config), the oracle author is guessing regardless.
Probation delays blocking. A genuinely weak oracle blocks nothing for its first three appearances. That is the trade: fewer false merges against slower detection.
Fixture edits still need their own check. This harness classifies tests, not fixtures; pair it with whatever fixture-integrity rule you already trust.
Skip it entirely if a human writes the tests before the patch exists, or if a single base-tree pytest run already exceeds your per-PR budget — you cannot afford the two-tree replay, and a half-run version produces unknown verdicts that look like coverage.

Your Fixtures Are the Spec: Fixture Mutation as the Missing Gate for Agent Patches

Finley Zhou — Sat, 07 Nov 2026 18:56:59 +0000

An agent patch can change what a fixture means without touching a single assertion. The suite stays green. Your code-mutation job stays green too, because it mutates the code under test — not the hand-built data your tests were quietly calibrated against.

That gap is the one I keep seeing in review: the patch is fine, the tests are fine, and the contract between them drifted. The fix is not more tests. It is one more gate that fails when a fixture stops being load-bearing.

Below is a three-gate harness you can wire into CI this week: fixture mutation, property checks with a hard case budget, and a flake quarantine keyed on failure signature instead of test name. All three emit a single JSON receipt your pipeline can read.

1. The blind spot, concretely

The example below is illustrative, not a benchmark — but it is the shape of the failure.

# src/config.py, before the patch
def load(raw: dict) -> dict:
    return {"db": raw["db"], "cache": raw["cache"]}

# tests/test_config.py
def test_db_host(config):
    assert config["db"]["host"] == "localhost"

The agent patch changes load to return a dataclass. The fixture still builds a plain dict in the new shape, because the fixture was written by hand, not by calling load. The test passes for a reason that has nothing to do with the patch.

Code mutation never finds this. Deleting a line in load may not even be reachable from a test that never calls it.

2. Gate 1 — Mutate the fixture, not the code

Start from the assumption that each fixture is a specification. Then ask the mutation question about the specification.

Step 1: declare mutants next to the fixture. One dict, one callable per mutant, no cleverness.

# gate/fixture_mutants.py
from typing import Any, Callable

Mutator = Callable[[Any], Any]

MUTANTS: dict[str, dict[str, Mutator]] = {
    "invoice": {
        "lines-empty":     lambda fx: {**fx, "lines": []},
        "qty-zero":        lambda fx: {**fx, "lines": [{**l, "qty": 0} for l in fx["lines"]]},
        "total-negative":  lambda fx: {**fx, "total_cents": -abs(fx["total_cents"])},
        "currency-null":   lambda fx: {**fx, "currency": None},
        "created-utc":     lambda fx: {**fx, "created_at": fx["created_at"].replace(tzinfo=None)},
    },
}

Step 2: apply the active mutant in conftest.py, and fail loudly on typos.

# conftest.py
import os
import pytest
from gate.fixture_mutants import MUTANTS

@pytest.fixture
def invoice(base_invoice):
    name = os.environ.get("FIXTURE_MUTANT")
    if not name:
        return base_invoice
    table = MUTANTS["invoice"]
    if name not in table:
        pytest.fail(f"unknown fixture mutant: {name}")
    return table[name](base_invoice)

Step 3: run each mutant in its own process. Shared state between mutants manufactures false kills.

# gate/run_fixture_mutation.py
import json, os, subprocess, sys
from gate.fixture_mutants import MUTANTS

def run(node_ids, mutant):
    env = {**os.environ, "FIXTURE_MUTANT": mutant}
    return subprocess.run(
        [sys.executable, "-m", "pytest", "-q", "-x", *node_ids],
        env=env, capture_output=True, text=True,
    ).returncode

def main():
    node_ids = sys.argv[1:]
    if not node_ids:
        sys.exit("usage: run_fixture_mutation.py <pytest node ids>")
    report = {"killed": [], "survived": []}
    for mutant in MUTANTS["invoice"]:
        bucket = "killed" if run(node_ids, mutant) != 0 else "survived"
        report[bucket].append(mutant)
    report["verdict"] = "pass" if not report["survived"] else "gap"
    print(json.dumps(report, indent=2))
    sys.exit(0 if report["verdict"] == "pass" else 1)

Step 4: triage survivors, do not silence them. A survivor is one of three things: a real gap in the tests, an equivalent mutant that cannot be observed, or a fixture that should not exist. Record which, and why, in the receipt. An unlabeled survivor is a lie you tell your CI.

Step 5: seat the gate on touched fixtures only. Derive the fixture list from the diff; run the full matrix nightly. A five-mutant matrix over one module costs seconds. Over eighty modules it does not.

3. Gate 2 — Property checks with a budget and a shrink log

Fixture mutation proves your examples are load-bearing. It does not prove the input space is covered. Keep property checks narrow and bounded.

from hypothesis import given, settings, strategies as st

@given(st.lists(st.integers(min_value=0, max_value=10_000), min_size=1, max_size=8))
@settings(max_examples=200, deadline=None, derandomize=True)
def test_total_is_never_negative(qty_and_price):
    ...

Three rules that keep this gate honest:

derandomize=True plus a committed seed profile, so a red run is reproducible on the reviewer's laptop.
A fixed max_examples. Unbounded property runs are flaky tests wearing a lab coat.
The shrunk counterexample goes into the receipt verbatim. If the report says "property failed" and nothing else, the gate will be disabled within a month.

4. Gate 3 — Quarantine by signature, not by name

Flaky-test freezes rot because they freeze a name. A name says nothing about whether the failure you froze is the failure you are seeing now.

Key the freeze on a normalized failure signature instead:

{
  "node": "tests/test_invoice.py::test_rounding",
  "signature": "AssertionError|0.01 != 0.010000000000000009|test_invoice.py:88",
  "first_seen": "2026-11-02",
  "expires": "2026-11-16",
  "owner": "@finley",
  "repro": "18/200",
  "reason": "float accumulation in legacy rounding path"
}

The gate fails when: the signature differs from what the run produced, the entry is past expires, the owner is empty, or repro is absent. Four conditions, all checkable mechanically.

5. One receipt, three verdicts

{
  "fixture_mutation": { "verdict": "gap", "survived": ["total-negative"] },
  "properties":       { "verdict": "pass", "cases": 200, "shrunk": null },
  "quarantine":       { "verdict": "pass", "active": 2, "expired": 0 }
}

Emit it as an artifact on every run. Reviewers read the diff and the receipt together; the receipt answers "what did this patch have to survive?" without anyone re-reading the suite.

6. Which gate is mandatory for which patch

Patch class	Fixture mutation	Property checks	Quarantine check
Test-only change	Mandatory	Optional	Mandatory
New branch in pure logic	Recommended	Mandatory	Mandatory
Serialization / schema	Mandatory	Mandatory	Optional
Time, locale, timezone	Mandatory	Optional	Mandatory
Dependency bump	Optional	Recommended	Mandatory
Fixture or helper edit	Mandatory	Optional	Mandatory

If a patch class is missing from the table, default to all three and argue your way down in review.

7. Where a free model tier actually helps

Two steps in this harness are cheap, bounded, and tedious: proposing fixture mutants from a diff, and drafting property generators for a new invariant. Both are good fits for a coding agent, and both fail badly if you hand them an unbounded budget.

This is where MonkeyCode's free model access fits in my loop: I paste the diff and the touched fixtures, ask for at most five mutants per fixture, and treat the output as a proposal list that a human filters before it enters fixture_mutants.py. The surviving-mutant triage stays manual — distinguishing "real gap" from "equivalent mutant" needs the judgment the model does not have. The free server option is useful for the nightly full matrix, which is CPU-bound, long, and does not need to run inside the main CI queue.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Check the project page for current limits, quotas, and server terms — those change, and this article deliberately does not restate them.

8. Limitations, and who should skip this

Cost scales with mutants times suite time. Ten mutants over a four-minute targeted suite is fine. Over a forty-minute suite it is a nightly job, not a PR gate.
Equivalent mutants are the tax. Some fixture changes are unobservable by design. Budget time for triage or the gate becomes noise.
Weak generators pass quietly. A property test over st.integers() proves nothing about your timezone handling. Property checks need adversarial generators, not default ones.
Signature normalization is runner-sensitive. Normalize paths and float formatting before hashing, or you will quarantine the same flake twice.
Quarantine hides bugs if left unowned. An entry without an owner and an expiry is a permanent mute button.
Skip Gate 1 entirely if your tests build data inline per test and never share a fixture — you have already solved the problem structurally.
Do not treat this as a replacement for code mutation. It is an additional gate over the specification layer, not a better version of the existing one.

Start with one fixture

Pick the fixture your last three agent patches touched. Write five mutants. Run the runner above against the tests that use it.

Most suites I have looked at this way survive at least one mutant they should have killed. That single survivor is more useful than another hundred lines of tests written after the fact — and it costs one afternoon.

If you run the loop and want a second read on a survivor you cannot classify, the receipt format above makes that conversation short.

Skip Is Not a Freeze: Revocable Seeded Freezes for Agent Patches

Finley Zhou — Sat, 07 Nov 2026 13:07:49 +0000

A skipped test is not a frozen test. It is a deleted test that still compiles. The difference matters the moment an agent starts shipping patches into the same suite, because a skip has no expiry, no signature, and no obligation to ever come back.

This post describes a freeze that revokes itself: every frozen test carries the seed that reproduces its failure, and every agent patch must re-run that seed and classify the result. If the seeded run now fails deterministically, the freeze is void — that is a regression, not flake.

All code below is a reference implementation. I have not run it against a production repository, so treat the thresholds and the signature function as starting points you calibrate on your own suite.

The timeline that motivates this

An agent patch touches a module next to test_export_roundtrip.
The test fails once in roughly twenty runs, so a reviewer adds @pytest.mark.skip(reason="flaky").
Four patches later, a real regression lands in that path. The test would have caught it, except it has been skipped for eleven days and nobody remembers the reason.

The flake was real. The freeze was not a freeze; it was amnesty. A test that only fails one time in twenty is still the cheapest detector you have for that code path — but only while something forces the question back open.

What a freeze has to contain

A freeze entry is a lease with four obligations:

A reproducer: the interpreter environment plus the seed that turns an intermittent failure into a deterministic one.
A signature: a coarse hash of the failure shape, so you can tell "same flake" from "different break" without reading logs.
Collateral: at least one property check and one pinned fixture that keep partial coverage alive while the test is skipped.
A revocation rule: a condition under which the freeze must be removed, evaluated by a job rather than by memory.

The ledger is human-owned. The gate that reads it runs read-only, and qa/freeze.json should sit under CODEOWNERS. If the agent can edit its own freeze ledger, you have automated the process of hiding failures from yourself.

Step 1: Capture a deterministic reproducer

You do not need to control every source of nondeterminism. You need to control enough that one seed reproduces one failure.

# Minimum environment pins that cover the common cases.
PYTHONHASHSEED=0 TZ=UTC SOURCE_DATE_EPOCH=0 \
  pytest tests/test_export.py::test_export_roundtrip -q -p no:cacheprovider -p no:randomly

If repeated runs at a fixed seed still disagree, stop. That is not a freezable flake — it is scheduling, clock, network, or shared-state nondeterminism, and a seed will not describe it. Say so in the ledger and escalate instead of skipping.

Step 2: Write the ledger entry

{
  "nodeid": "tests/test_export.py::test_export_roundtrip",
  "signature": "9f2c1ab4d7e0",
  "seeds": [0, 7, 13],
  "env": { "PYTHONHASHSEED": "0", "TZ": "UTC", "SOURCE_DATE_EPOCH": "0" },
  "witnesses": [
    "props/test_export_props.py::test_roundtrip_is_idempotent",
    "props/test_export_props.py::test_export_preserves_row_count"
  ],
  "fixtures": { "tests/fixtures/export_small.csv": "sha256:4e1f..." },
  "opened": "2026-11-07",
  "owner": "finley"
}

Three seeds, not one. One seed proves the failure can be reproduced; three seeds give the gate something to distinguish "fails always" from "fails sometimes at fixed seeds." If all three reproduce the same signature, the flake behaves like a deterministic bug under those seeds, which is exactly what you want for a revocation rule.

Step 3: Classify on every agent patch

# gate.py — reference implementation, not executed here.
import hashlib, json, os, subprocess, sys
from pathlib import Path

LEDGER = Path("qa/freeze.json")

def signature(output: str) -> str:
    lines = [l for l in output.splitlines() if l.strip()]
    tail = lines[-1] if lines else "no-output"
    frame = next((l.strip() for l in reversed(lines) if l.strip().startswith("File ")), "no-frame")
    shape = "".join(c if not c.isdigit() else "N" for c in tail)[:120]
    return hashlib.sha1(f"{frame}|{shape}".encode()).hexdigest()[:12]

def run_seeded(entry: dict, seed: int) -> tuple[int, str]:
    env = {**os.environ, **entry["env"], "PYTHONHASHSEED": str(seed)}
    p = subprocess.run(
        [sys.executable, "-m", "pytest", entry["nodeid"], "-q", "-p", "no:cacheprovider"],
        capture_output=True, text=True, env=env, timeout=300,
    )
    return p.returncode, signature(p.stdout + p.stderr)

def classify(entry: dict) -> str:
    results = [run_seeded(entry, s) for s in entry["seeds"]]
    codes = {rc for rc, _ in results}
    sigs = {sig for _, sig in results}
    if codes == {0}:
        return "SEED_STABLE"
    if codes == {1} and sigs == {entry["signature"]}:
        return "STILL_FROZEN"
    if codes == {1}:
        return "DRIFTED"
    return "UNSTABLE_FREEZE"

The classification table is where the policy lives, and it should be written down before the first patch arrives:

Result	Meaning	Gate action
`STILL_FROZEN`	Same failure shape at every recorded seed	Merge may proceed; freeze stays open
`DRIFTED`	Deterministic failure, new signature	Block. The patch changed the failure mode; re-triage, do not skip
`SEED_STABLE`	Passes at all recorded seeds	Do not auto-unfreeze. Queue a soak run unfrozen; remove only after it holds
`UNSTABLE_FREEZE`	Mixed results at fixed seeds	Block. The freeze entry is broken; the flake is not reproducible with a seed

The DRIFTED row is the one that earns its keep. Without it, a patch can convert a flaky assertion into a hard crash and the ledger keeps reporting a benign freeze, because the test is still red in the same place by name.

Keeping coverage alive while the test is dark

A freeze removes one assertion. It should not remove the whole code path from observation.

Each entry needs at least one property check that runs outside the skipped test. From the frozen test's assertions, extract the invariant that must hold for all inputs, not just the fixture row: idempotent roundtrip, row count preserved, ordering stable under sort. Those checks are weaker than the original assertion, and that is the point — weak coverage that runs beats strong coverage that is skipped.

Pin the fixtures by content hash and verify them before running the witnesses. An agent that cannot make the witness pass has an obvious next move: adjust export_small.csv. A hash check on the fixture turns that move into a failed gate instead of a green build.

Limitations and who should not use this

Infra flake is out of scope. If failures correlate with container restarts, disk pressure, or shared database locks, seeds describe nothing. Fix the harness first.
The cost multiplies. Total replay work is patches × frozen tests × seeds. A suite with eighty freezes and three seeds pays 240 extra test invocations per patch. Keep a hard cap on open freezes per repo, and treat hitting the cap as a signal to fix tests rather than to raise the cap.
Time-dependent logic resists pinning. TZ and SOURCE_DATE_EPOCH cover a lot, but code that reads wall-clock deltas will still drift. Freeze those only if you can inject the clock.
Do not use this on a suite with no property layer. If you have no invariants to hold open, a freeze is just a skip with extra JSON, and the ledger becomes the artifact you maintain while coverage decays.
This is not a substitute for fixing the flake. It is a way to stop a fix from being forgotten. If nobody owns the entry, the owner field is decoration.

Where the free tier fits

The gate has two repetitive parts: drafting property invariants from a patch diff, and replaying a seeded matrix on dedicated compute. Neither is intellectually hard, and both are exactly the kind of work that stalls behind a budget conversation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I draft the witnesses with MonkeyCode's free model access — the prompt is the diff plus the frozen test's assertions, and the required output is invariants that do not touch the skipped code path. The useful failure mode to watch for is a model returning the original assertion restated as a "property"; a restated assertion will pass trivially and keep the freeze alive with fake coverage. Then the free server option runs the seeded replay loop, which is the part you cannot run once per patch by hand.

Everything above survives deleting the product name. The ledger, the signature, and the DRIFTED rule are the article; the tooling is convenience.

If you adopt one thing from this post, make it this: track how many freezes revoked themselves in the last month. A ledger where the answer is always zero is not clean — it is unread.

Budget the Gate, Not Just the Tests: Stage Caps and a Calibration Check for Agent Patches

Finley Zhou — Sat, 07 Nov 2026 10:02:49 +0000

A test gate for agent patches fails in two directions. It lets bad patches through, and it gets slow enough that people route around it. This post is about the second failure: how to give each stage of the gate a wall-clock budget, keep that budget honest across machines, and demote checks instead of deleting them when a patch pushes the suite past its cap.

The artifact is two small scripts — a calibration probe and a stage allocator — plus a decision table and a three-strike rule. It assumes you already have checks worth running. It does not re-derive what those checks should be.

The failure this prevents is bypass, not slowness

A slow gate is not a neutral inconvenience. Every extra minute of pre-merge latency raises the chance that someone merges with a local run, or adds a skip marker, or approves a patch they did not read. Agent-authored patches make this worse because they arrive in volume: ten patches a day that each add "one small test" and nobody notices the gate doubled.

Budget creep is measurable, which is the good news. Record the total wall clock of each stage on every run, and you get a curve. The rule I use is boring and effective: if a stage exceeds its cap on three consecutive runs, the next merge window is a budgeting task, not a feature task.

Step 1: Measure per-check cost and normalize it

Absolute seconds do not travel. A runner with a cold cache, a noisy neighbour, or different disk will produce different numbers, and a budget measured on one machine is fiction on another. Normalize instead.

Add a calibration check whose purpose is to measure the machine, not the code. It should be deterministic, CPU-bound, and touch no network or disk:

# calibration_check.py — one number that makes timings portable
import hashlib, json, time

def calibration_seconds(rounds: int = 200_000) -> float:
    payload = b"agent-patch-gate"
    start = time.perf_counter()
    h = payload
    for _ in range(rounds):
        h = hashlib.sha256(h).digest()
    return time.perf_counter() - start

if __name__ == "__main__":
    print(json.dumps({"calibration_s": round(calibration_seconds(), 4)}))

Run it once at the start of the gate, then express every check's cost in calibration units: cost_rel = p95_seconds / calibration_seconds. Use p95, not the mean, because the checks that break your budget are the tail ones.

Two honest caveats. First, this proxy tracks CPU throughput, not I/O or parallel contention; a check that is 80% database round-trips will not normalize properly. Second, run the calibration probe on the same host class you intend to compare, or accept a wide error bar.

Collect the raw numbers into a file you commit:

pytest -q --durations=0 --durations-min=0.5 | tee durations.txt
# then transform to timings.json:
# {"calibration_s": 0.51,
#  "checks": [{"id": "props.core", "p95_s": 41.2},
#             {"id": "fixtures.golden", "p95_s": 12.8},
#             {"id": "audit.flake_ledger", "p95_s": 96.0}]}

The numbers above are illustrative inputs, not measurements of anyone's suite.

Step 2: Fit checks into stage budgets

State the caps once, in calibration units, and let a script assign checks. A cap of 5 minutes with a 0.5 s calibration unit is 600 units. Pick the caps from how long a reviewer will actually wait, not from how long the suite happens to take today.

#!/usr/bin/env python3
"""check_budget.py — place each check in the earliest stage whose cap can hold it.

Exit 2 when a check fits nowhere. Silent truncation is how gates rot.
"""
import json, sys
from pathlib import Path

STAGES = ("pre-merge", "post-merge", "nightly")
BUDGET_REL = {"pre-merge": 600.0, "post-merge": 2400.0, "nightly": 8000.0}
UNMEASURED_REL = 2.0  # conservative charge for a check with no timing

def rel_cost(check, cal):
    p95 = check.get("p95_s")
    return UNMEASURED_REL if p95 is None else float(p95) / cal

def main(src="timings.json", dst="schedule.json"):
    data = json.loads(Path(src).read_text())
    cal = float(data["calibration_s"])
    remaining, schedule, overflow = dict(BUDGET_REL), {s: [] for s in STAGES}, []

    for check in sorted(data["checks"], key=lambda c: rel_cost(c, cal)):
        cost, placed = rel_cost(check, cal), None
        for stage in STAGES:
            if cost <= remaining[stage]:
                remaining[stage] -= cost
                placed = stage
                break
        if placed is None:
            overflow.append(check["id"])
        else:
            schedule[placed].append({"id": check["id"], "cost_rel": round(cost, 3)})

    for stage in STAGES:
        used = BUDGET_REL[stage] - remaining[stage]
        print(f"{stage:11s} {used:7.1f}/{BUDGET_REL[stage]:<7.0f} units  "
              f"{len(schedule[stage])} checks")
    if overflow:
        print("fits no stage:", ", ".join(overflow), file=sys.stderr)
        return 2
    Path(dst).write_text(json.dumps(schedule, indent=2))
    return 0

if __name__ == "__main__":
    sys.exit(main())

The sort is cheapest-first, so the fast stage absorbs as many checks as possible and the expensive ones land where they belong. BUDGET_REL is committed alongside timings.json, so a PR that changes the caps is visible in review.

One more rule makes the ledger self-defending: an agent patch that adds a check must include that check's measured cost. No timing, no pre-merge slot — the check starts in nightly at the conservative charge and earns promotion after three real measurements.

Step 3: Demote, don't delete

When a check exceeds its stage cap, the wrong move is to delete it and the second-wrong move is to raise the cap without saying why. Demotion preserves coverage and moves the cost where latency does not matter.

Check kind	Default stage	First demotion	Deletion requires
Invariant / property checks	pre-merge	split by module, then post-merge	proof the property is subsumed
Golden-file fixture locks	pre-merge	post-merge	owner sign-off in the PR
Order and rerun sampling	post-merge	nightly, sampled	never — sample instead
Quarantine and flake audits	nightly	weekly	never — expire instead
Slow end-to-end replays	nightly	weekly	a dated review ticket

Write the reason into the schedule file. "reason": "overflow" on a nightly entry is the earliest signal that a suite needs splitting, and a bare list of check names tells the next person nothing.

Step 4: Where free model access and a free server actually help

The loop above has two cost centres: authoring the patches and executing the stages. I run agent-authored changes through MonkeyCode, using its free model access for the drafting and the free server option for the post-merge and nightly stages when my own runners are saturated. That keeps the expensive, latency-tolerant half of the gate off the machines people are waiting on.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the product claims as the operator's: free model access and a free server option exist, and I am not going to quote quotas, hardware, or durations I cannot verify. The design consequence is what matters — if you do not control the machine, absolute budgets are meaningless, which is exactly why the calibration unit is in Step 1. Keep timings.json and check_budget.py in git so the numbers and the policy travel with the code instead of living in a CI dashboard nobody reads.

If you want to try the loop without provisioning anything, that combination is a reasonable place to start; run the calibration probe first and re-measure on your own suite before trusting any cap.

Limitations and who should not use this

Suites under roughly 2 minutes of pre-merge time do not need an allocator. The overhead of maintaining the ledger exceeds the latency saved.
I/O-heavy or network-dependent checks normalize poorly; their relative cost will drift, and you will re-measure often.
The calibration proxy is a CPU throughput measure. On shared or throttled hosts the error bar is wide enough that a 10% budget change is noise.
Teams without a place to review a committed budget file will watch it drift. If the schedule is generated but never read, this is ceremony.
If your real problem is checks that never fail, budgeting them is optimizing the wrong axis. Fix the checks first.

The review checklist

Before you trust the gate again, confirm four things: every stage has a cap in calibration units, every check has a measured p95 or an explicit conservative charge, the three-strike rule has an owner, and every demoted check has a written reason and a return path.

Latency budgets are not bureaucracy. They are the reason the gate still gets read on a Friday afternoon.

Stop Writing Test Strategies for Agent Patches. Write a Contract Your CI Can Enforce.

Finley Zhou — Sat, 07 Nov 2026 06:57:36 +0000

A test strategy for agent patches dies the moment it meets a real diff. It lives in a wiki page, the agent never reads it, and the reviewer under time pressure approves a green build. The fix is not a better document. It is a small machine-readable contract that ships in the patch and a linter that rejects the patch when the contract is thin.

Below is the workflow I use: one JSON contract per patch, four linter rules, and a blast-radius table that decides how much checking the patch has earned. The linter is a reference implementation, stdlib only, roughly 60 lines. Run it before you trust it.

Why strategies decay and contracts do not

A strategy says "add property tests where appropriate." Appropriate is a judgment call, and an agent optimizes for closing the ticket. A contract says property_checks must be non-empty and each id must appear as a marker in an added test line. That is a string comparison, and string comparisons do not get tired.

The trade-off is real: contracts encode less than prose. You lose nuance about design intent. You gain a gate that fires without a human in the loop, which is exactly what an agent-generated patch needs before a human looks at it.

1. The contract file

Keep it in the patch, next to the code. JSON because the linter should not drag in a YAML dependency; convert if your repo already ships one.

{
  "patch": "fix-retry-backoff",
  "owner": "finley",
  "blast_radius": "medium",
  "property_checks": [
    {
      "id": "backoff-monotonic",
      "invariant": "delay(n+1) >= delay(n) for all n in [0, 8)",
      "seeds": [0, 7, 42]
    }
  ],
  "fixtures_locked": ["tests/fixtures/clock_frozen.json"],
  "flaky_frozen": ["tests/test_upload.py::test_eventual_consistency"],
  "frozen_until": "2026-11-21",
  "override": null
}

Three fields carry the weight. property_checks[].invariant is prose the linter cannot verify, so a human must own that line — not the agent that wrote the patch. seeds makes the property run reproducible; an unseeded random loop is an unrepeatable test with extra steps. frozen_until gives the freeze an expiry date, so the contract degrades into a loud failure instead of permanent debt.

2. Four rules the linter enforces

Reference implementation, Python 3.11+, no third-party imports:

#!/usr/bin/env python3
"""tools/contract_lint.py - reference implementation, stdlib only."""
import json, re, subprocess, sys, pathlib, argparse

ROOT = pathlib.Path(__file__).resolve().parents[1]


def git(*args):
    return subprocess.run(["git", *args], cwd=ROOT, capture_output=True,
                          text=True, check=True).stdout


def changed_files(base):
    return [p for p in git("diff", "--name-only", f"{base}...HEAD").splitlines() if p]


def added_lines(base):
    out = {}
    for path in changed_files(base):
        diff = git("diff", "-U0", f"{base}...HEAD", "--", path)
        out[path] = [l[1:] for l in diff.splitlines()
                     if l.startswith("+") and not l.startswith("+++")]
    return out


def rule_frozen_touch(c, changed, today):
    frozen_files = {t.split("::")[0] for t in c.get("flaky_frozen", [])}
    guarded = frozen_files | set(c.get("fixtures_locked", []))
    ov = c.get("override") or {}
    valid = bool(ov.get("reason")) and ov.get("expires", "1970-01-01") >= today
    return [f"frozen-touch: {p} is frozen or locked without a valid override"
            for p in changed if p in guarded and not valid]


MARKER = re.compile(r"#\s*property:\s*([\w.-]+)")


def rule_properties(c, added):
    seen = {m.group(1) for p, lines in added.items() if p.startswith("tests/")
            for line in lines for m in [MARKER.search(line)] if m}
    problems = []
    for pc in c.get("property_checks", []):
        if pc["id"] not in seen:
            problems.append(f"property-missing: no added test tagged '# property: {pc['id']}'")
        if not pc.get("seeds"):
            problems.append(f"property-unseeded: {pc['id']} declares no seeds")
    return problems


LIT = re.compile(r'"(?:[^"\\]|\\.)*"|-?\d+(?:\.\d+)?')
ASSERT = re.compile(r"assert\s+.*==\s*(\"(?:[^\"\\]|\\.)*\"|-?\d+(?:\.\d+)?)")


def rule_mirror(c, added):
    """Heuristic: a test literal copied out of the patch proves nothing."""
    src = {lit for p, lines in added.items()
           if not p.startswith("tests/")
           for line in lines for lit in LIT.findall(line)}
    problems = []
    for p, lines in added.items():
        if not p.startswith("tests/"):
            continue
        for line in lines:
            if "literal-ok:" in line:
                continue
            for lit in ASSERT.findall(line):
                if lit in src:
                    problems.append(f"mirror-suspect: {p} asserts {lit}, also added in src")
    return problems


def rule_quarantine_leak(c, added):
    frozen = {t.split("::")[0] for t in c.get("flaky_frozen", [])}
    stems = {pathlib.Path(f).stem for f in frozen}
    problems = []
    for p, lines in added.items():
        if p in frozen:
            continue
        for line in lines:
            if "import" in line or "subprocess" in line:
                problems += [f"quarantine-leak: {p} pulls in frozen module {s}"
                             for s in stems if re.search(rf"\b{re.escape(s)}\b", line)]
    return problems


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", default="origin/main")
    ap.add_argument("--contract", default="patch.contract.json")
    ap.add_argument("--today", default=None)
    a = ap.parse_args()
    import datetime
    today = a.today or datetime.date.today().isoformat()
    c = json.loads((ROOT / a.contract).read_text())
    changed, added = changed_files(a.base), added_lines(a.base)
    found = (rule_frozen_touch(c, changed, today) + rule_properties(c, added)
             + rule_mirror(c, added) + rule_quarantine_leak(c, added))
    if c.get("frozen_until", "9999-12-31") < today:
        found.insert(0, f"contract-expired: frozen_until {c['frozen_until']} has passed")
    strict = c.get("blast_radius") in {"medium", "high"}
    blocking = [f for f in found if strict or not f.startswith("mirror-suspect")]
    print("\n".join(found) or "contract ok")
    return 1 if blocking else 0


if __name__ == "__main__":
    sys.exit(main())

Run it as a pre-CI step:

python tools/contract_lint.py --base origin/main --contract patch.contract.json

The four rules, and what each one actually buys you:

frozen-touch — the patch cannot edit a frozen test or a locked fixture unless the override carries a reason and a future expiry. This is the freeze from your flaky-test policy, made physical instead of advisory.
property-missing / property-unseeded — every declared invariant needs an added test line tagged # property: <id>, and a seed list. The tag is cheap to write and impossible to fake with a passing run.
mirror-suspect — advisory only. If a new assertion compares against a literal that the patch itself introduced, the test is probably restating the implementation. Expect false positives; annotate the honest ones with # literal-ok: and keep the rule noisy rather than silent.
quarantine-leak — nothing outside the frozen file may import its module, so a quarantined test cannot become an accidental dependency of a passing one.

A minimal property test that satisfies rule 2:

# tests/properties/test_backoff.py
import random
from src.retry import backoff_delays

# property: backoff-monotonic
def test_backoff_is_monotonic():
    for seed in (0, 7, 42):
        rng = random.Random(seed)
        for _ in range(200):
            base, attempts = rng.randint(50, 5000), rng.randint(1, 8)
            d = backoff_delays(base, attempts)
            assert all(b <= a for a, b in zip(d, d[1:])), (seed, base, attempts, d)

3. Where free model access and a free server fit

Two parts of this workflow are tedious but low-judgment, and one part is CPU-bound.

Drafting the contract. Writing invariant strings from a diff is a summarization task. Paste the changed function signatures plus git diff --stat into a free model and ask for a candidate contract, then edit the invariant text by hand. Do not skip the edit: an agent that authored the patch should never author the sentence that certifies the patch. That is self-certification with extra steps.

Running the seeded loop. Seed sweeps multiply fast — 3 seeds × 200 iterations × several properties over a shared module. That is the kind of parallel, disposable work the free server option is useful for, so CI minutes stay for the suites that need your real environment. Both availability claims here are operator-supplied; confirm current terms and limits yourself before depending on them, and do not assume anything about hardware, quotas, or persistence.

What not to hand to a model. The linter itself. It is deterministic, fast, and cheap, and a model reviewing a contract it just helped write adds a failure mode rather than removing one. Keep the gate boring.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

4. Sizing the contract by blast radius

Not every patch deserves the same contract. These are policy defaults to start from, not measured detection rates — tune them against your own merged history.

Blast radius	Property checks	Fixtures	Flaky-test handling	Lint mode
low (docs, one leaf util)	≥ 1	none locked	freeze not allowed	advisory
medium (shared module)	≥ 1 per changed public function	all mutated fixtures locked	freeze with expiry, no new deps	strict
high (auth, payments, migrations)	≥ 2, one on the failure path	all touched fixtures locked	freeze requires second reviewer	strict + human sign-off

The point of the table is to stop the argument at review time. Either the contract matches the tier, or the patch waits.

Limitations, and who should skip this

The diff parser is fragile. Renames, binary files, and merge commits will produce wrong changed_files output. Add --find-renames handling or fail closed before you rely on it.
mirror-suspect is a heuristic with a high false-positive rate. It is a prompt for the reviewer, not a proof of a tautological test.
Frozen lists rot. If frozen_until keeps getting bumped, the contract is documenting debt, not paying it down.
Self-authored contracts are the main failure mode. Require a human owner on property_checks text, even if the model drafted it.

Skip this if you are a solo maintainer on a small repo — the contract is more overhead than the diff. Skip it too if your default branch has no stable merge base for origin/main...HEAD, or if a monorepo makes the changed-file list too large to review. The workflow assumes a bounded patch and a shared branch.

Before adding a fifth rule, run this linter against your last five merged agent patches. The violations you find on already-approved work tell you more about your gate than any new rule will.

Run Less, Prove More: Diff-Scoped Test Gating for Agent Patches

Finley Zhou — Sat, 07 Nov 2026 03:50:59 +0000

An agent patch rarely needs your whole suite. It needs a scope you can defend line by line, and a safety net over everything the scope excludes. Most review pipelines get this backwards: they run everything, watch it go red at random, and then learn nothing about which failure the patch actually caused.

This is a selection strategy, not a speed trick. The gate runs a narrow set of tests, emits a receipt explaining why each test was included, and escalates to the full suite whenever a widening trigger fires. Flaky tests are frozen separately — and a freeze is only granted once a property check covers the same behavior.

The cost is a product, not a number

Write the cost down before optimizing it. For a patch p:

cost(p) = |selected(p)| * t_avg + |frozen| * t_skip  + widen(p) * |full| * t_avg

widen(p) is binary here on purpose. Either a trigger fired or it did not. If you make widening a judgment call, reviewers will quietly skip it under deadline, and your escape rate becomes invisible until production finds it.

Measure t_avg on your own repo. Do not reuse numbers from a blog post — including this one.

Why "only run tests that import the changed file" fails

Naive diff selection is a well-known footgun. Four failure modes matter for agent patches specifically:

Shared fixtures. A conftest.py change alters fixtures used by tests that never import the modified module.
Runtime wiring. Entry points, plugin registries, DI containers, and string-based imports create edges no static parser sees.
Contract drift. An agent changes a serializer or CLI flag; every consumer test still passes locally because none of them are in the import closure.
Editable surfaces. Settings modules, migrations, and schema files are imported indirectly by nearly everything.

Each of those becomes a widening trigger in step 3. That is the whole design: be aggressive about narrow selection, be mechanical about widening.

Step 1 — Build a reverse import map

The map is the only expensive part, and it is cacheable per commit.

# scope.py — diff-scoped test selection for a Python repo
import ast, json, pathlib, subprocess

ROOT = pathlib.Path(".").resolve()
SKIP = {".venv", "build", "dist", "node_modules", ".git"}

def changed_files(base: str) -> list[str]:
    out = subprocess.run(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        capture_output=True, text=True, check=True,
    ).stdout
    return [f for f in out.splitlines() if f.endswith(".py")]

def module_name(rel: str) -> str:
    parts = list(pathlib.Path(rel).with_suffix("").parts)
    if parts and parts[-1] == "__init__":
        parts.pop()
    return ".".join(parts)

def imports_of(path: pathlib.Path) -> set[str]:
    try:
        tree = ast.parse(path.read_text(encoding="utf-8"))
    except SyntaxError:
        return set()
    names: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            names.update(a.name for a in node.names)
        elif isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
            names.add(node.module)
    return names

def build_reverse_map() -> dict[str, set[str]]:
    reverse: dict[str, set[str]] = {}
    for path in ROOT.rglob("*.py"):
        if SKIP & set(path.parts):
            continue
        rel = str(path.relative_to(ROOT))
        reverse.setdefault(module_name(rel), set()).add(rel)
        for imp in imports_of(path):
            reverse.setdefault(imp, set()).add(rel)
    return reverse

Store the map keyed by commit SHA. Rebuilding it on every patch is the mistake that makes people abandon the idea.

Step 2 — Compute the importer closure

Walk importer edges upward from the changed modules. Include ancestor packages so pkg.sub also matches pkg.

def importer_closure(seeds: set[str], reverse: dict[str, set[str]]) -> set[str]:
    seen: set[str] = set()
    stack = list(seeds)
    while stack:
        mod = stack.pop()
        for candidate in (mod, *mod.split(".")):
            for f in reverse.get(candidate, ()):
                if f not in seen:
                    seen.add(f)
                    stack.append(module_name(f))
    return seen

def select(base: str = "origin/main") -> dict:
    reverse = build_reverse_map()
    changed = changed_files(base)
    seeds = {module_name(f) for f in changed}
    closure = importer_closure(seeds, reverse)
    tests = sorted(f for f in closure if f.startswith("tests/"))
    return {"base": base, "changed": changed, "selected": tests,
            "widened": None, "reason": []}

A test file is selected because something it imports changed. Nothing else. That is a claim you can audit, which is the point.

Step 3 — Fire widening triggers

Triggers are checked before selection, and any hit short-circuits to the full suite. Keep the list short enough that people actually read it.

Trigger	Why narrow scope is unsafe	Action
`conftest.py`, `tests/fixtures/**`, `pytest.ini` changed	Shared fixtures leak across the whole suite	Full suite
`settings/`, `migrations/`, `schema.*` changed	Imported indirectly by nearly everything	Full suite
Changed file uses `importlib`, `entry_points`, or a plugin registry	Static edges are incomplete	Full suite
Changed file is imported by more than your configured fan-out threshold	Blast radius is structural, not local	Full suite
Public CLI, HTTP route, or serialized payload changed	Contract consumers live outside the closure	Full suite + contract tests
A frozen test's file is in the closure	The freeze suppressed a relevant signal	Unfreeze and rerun

Set the fan-out threshold from your own map: print the top 20 modules by importer count and pick a cutoff below the monoliths. On most repos it is embarrassingly obvious.

Step 4 — Cover the excluded surface with properties

Narrow selection is only honest if something else checks the code you did not run. That is what the property layer is for — not coverage theater, but invariants that hold across inputs the suite never generates.

# tests/properties/test_ledger_props.py — illustrative; assert properties your
# module actually promises before adding this file.
from hypothesis import given, settings, strategies as st
from app.ledger import reconcile

rows = st.lists(
    st.fixed_dictionaries({"amount": st.integers(-10**6, 10**6),
                           "ref": st.text(max_size=8)}),
    max_size=50,
)

@settings(max_examples=200, deadline=None)
@given(entries=rows)
def test_reconcile_is_order_insensitive(entries):
    assert reconcile(entries) == reconcile(list(reversed(entries)))

@settings(max_examples=200, deadline=None)
@given(entries=rows)
def test_reconcile_is_total(entries):
    total = sum(e["amount"] for e in entries)
    assert sum(reconcile(entries).values()) == total

These two properties survive refactors that would break every example-based test. They also fail loudly when an agent "optimizes" a fold into a loop that drops the tail.

Step 5 — Freeze flaky tests behind a substitute

A freeze without a substitute is a coverage withdrawal with better branding. Gate it:

freeze(test) is legal only if
  (a) it has an owner and an expiry date, and
  (b) at least one property check covers the behavior the test asserts.

Record both in the receipt. If (b) is unmet, the correct output is a red gate — not a freeze. This is the single rule that keeps quarantine from becoming silent debt.

Step 6 — Emit a scope receipt

Everything above is worthless if the reviewer cannot see it. Write the selection, the triggers checked, the freezes applied, and the property files run into one JSON artifact attached to the patch.

python scope.py --base origin/main --out scope.json
jq -r '.selected[]' scope.json | xargs pytest -q --timeout=300
jq -r '.properties[]' scope.json | xargs pytest -q

A reviewer reading scope.json should be able to answer "what did this gate not run, and why is that acceptable" in under a minute.

Step 7 — Measure the escape rate

Track one number weekly: full-suite failures that the narrow gate did not catch. Run the full suite on a schedule against the same commits and diff the results. If the escape rate is zero for a month, your triggers are probably too conservative — widen less. If it is nonzero, every escape names the missing trigger. Add it.

This is where a free execution lane helps. The narrow gate belongs on the developer's machine because it must be fast; the scheduled full-suite baseline is throughput work that does not need interactivity. MonkeyCode's free model access and free server option fit that second lane: the server runs the scheduled baseline, and the model handles the boring parts — classifying diff hunks into widening triggers and drafting property candidates for human review.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not let a model approve a trigger decision. Classification is a draft; the trigger list is code, and the trigger list is what the gate obeys.

Limitations, and who should skip this

Static edges only. Dynamic dispatch, getattr chains, and subprocess CLIs are invisible. That is what the trigger table is compensating for, imperfectly.
Python-shaped. The recipe ports to other languages only if you can build a reliable import graph; dynamic languages and macro-heavy code make it fragile.
Needs a stable baseline. Without a trustworthy full-suite run to compare against, the escape rate is unmeasurable and the narrow gate is unfalsifiable.
Skip it for schema migrations, auth and crypto changes, shared integration suites against live external services, and any repo where most tests are end-to-end. Those belong on the full suite every time, and no selection heuristic will save you.
Skip it if your team will not maintain the trigger list. An unmaintained trigger list is worse than no gate, because it looks like coverage.

The takeaway

Scope is a claim, not an optimization. Write the claim down, make widening mechanical, put properties behind every freeze, and measure how often the claim is wrong. Your agent patches will merge faster for the same reason good tests do: the reviewer stops guessing.

If you try one piece, try the receipt. It is thirty lines of JSON and it changes how the whole gate is argued about.

A Green Run Is a Claim, Not Evidence: Re-deriving Agent Patch Test Runs

Finley Zhou — Sat, 07 Nov 2026 00:46:42 +0000

A green test run is a claim about one environment at one moment. An evidence packet is the derivation that lets someone else reach the same conclusion without you. If an agent patch merges with a green badge and nothing else, you shipped a screenshot.

This post builds the smallest packet I trust, a verifier that re-derives it, and a sampling policy so the check stays cheap. The artifact is stdlib Python plus pytest; nothing here depends on a hosted service.

The three ways a green run silently stops being true

Agent patches drift more than human ones because the loop is fast: the patch is edited, rebased, or regenerated between the test run and the merge. Three things break, and all three break quietly.

Tree drift. Someone changes the patch (or a lockfile) after the run. The badge still renders.
Selection drift. Collection changes — a marker moved, a conftest gained a skip, a new test file appeared — so "the suite" is no longer the set of node IDs that ran.
Environment drift. Interpreter patch version, hash seed, timezone, PYTHONPATH, native library versions. Any of these can flip a pass to a fail with zero source changes.

Signal	What it catches	Capture cost	Re-derive cost
Whole-tree SHA-256	tree drift, lockfile edits	one pass over files	same
Hash of sorted collected node IDs	selection drift, silent skips	one `--collect-only`	same
Env whitelist (hash seed, TZ, locale, path)	environment drift	free	free
Interpreter + platform string	version skew	free	free
Full stdout/stderr of the run	everything, at high storage cost	large	large

The first four rows are cheap enough to keep for every patch. The last row is what people usually store, and it is the least useful of the five.

Step 1 — Capture the packet at run time

Capture inside the same process that runs the tests, not from a shell wrapper someone can edit later.

# evidence_packet.py — stdlib only
import hashlib, json, os, platform, subprocess, sys, time

ALLOW_ENV = ("PYTHONHASHSEED", "LANG", "TZ", "PYTHONPATH")
SKIP_DIRS = {".git", "__pycache__", ".venv", ".mypy_cache", "node_modules"}


def tree_digest(root: str) -> str:
    h = hashlib.sha256()
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = sorted(d for d in dirnames if d not in SKIP_DIRS)
        for name in sorted(filenames):
            path = os.path.join(dirpath, name)
            rel = os.path.relpath(path, root)
            h.update(rel.encode())
            with open(path, "rb") as fh:
                h.update(hashlib.sha256(fh.read()).digest())
    return h.hexdigest()


def collect(test_path: str) -> list[str]:
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", test_path, "--collect-only", "-q"],
        capture_output=True, text=True,
    )
    if proc.returncode not in (0, 5):  # 5 = no tests collected
        raise SystemExit(f"collection failed: {proc.stderr[-400:]}")
    return sorted(ln.strip() for ln in proc.stdout.splitlines() if "::" in ln)


def build(root: str, test_path: str) -> dict:
    node_ids = collect(test_path)
    return {
        "schema": 1,
        "captured_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "tree": tree_digest(root),
        "tests_digest": hashlib.sha256("\n".join(node_ids).encode()).hexdigest(),
        "node_count": len(node_ids),
        "env": {k: os.environ.get(k, "") for k in ALLOW_ENV},
        "interpreter": f"{platform.python_version()} {platform.platform()}",
        "argv": [sys.executable, "-m", "pytest", test_path, "-p", "no:randomly", "-q"],
    }

Three decisions matter here. The digest skips caches so it is stable across machines. The node-ID hash sorts before hashing so collection order cannot mask a change. The argv pins the random plugin off, which removes one source of run-to-run variance that a packet should not have to explain.

Step 2 — Re-derive instead of trusting

Verification is not "read the JSON and agree". It is: rebuild the packet from the current tree, diff the cheap fields, and only then re-run the suite.

def verify(packet: dict, root: str, test_path: str) -> dict:
    live = build(root, test_path)
    drift = [k for k in ("tree", "tests_digest", "node_count", "interpreter")
             if live[k] != packet[k]]
    if drift:
        return {"verdict": "STALE", "drifted": drift}
    proc = subprocess.run(packet["argv"], capture_output=True, text=True,
                          env={**os.environ, **packet["env"]})
    return {"verdict": "REPRODUCED" if proc.returncode == 0 else "FAILED",
            "returncode": proc.returncode,
            "tail": (proc.stdout or proc.stderr)[-400:]}

STALE is the verdict that earns its keep. In practice it means one of: the patch was touched after the run, a test was added or deselected, or the interpreter moved. Those are three different bugs with the same symptom, and the drifted list tells you which one you are looking at before you spend a single core-second re-running tests.

Step 3 — Sample re-derivation, do not gate on it

Re-deriving every patch is a tax you will stop paying within a month. Sample instead, weighted by risk:

Patch class	Re-derive	Reason
Touches `conftest.py`, markers, or CI config	always	selection drift surface
Touches dependency locks or interpreter pin	always	environment drift surface
Generated by an agent with >1 edit round	always	tree drift surface
Test-only addition, no prod code	nightly batch	low blast radius
Docs, types, comments	never	no evidence claim worth keeping

The policy is the artifact. A team that cannot state which patches get re-derived is running on vibes with a YAML file attached.

Step 4 — Property probes that survive a re-derivation

A packet proves a run happened as described. It says nothing about whether the run was worth anything. Pair it with a small number of invariant probes that hold across trees, so a stale packet fails loudly rather than politely.

Idempotence probe. Apply the agent's change twice; the second application must be a no-op or must error, never silently double-apply.
Boundary probe. The patched function returns the same value at its documented edges as the pre-patch implementation, or the diff explains why not.
Absence probe. After the patch, the old failure mode is still reachable when the fix is disabled — if it is not, you changed the fixture, not the behavior.

Each probe is one assertion, not a test suite. Three probes that fail for the right reason beat forty that pass for none.

Where free access actually helps

Two pieces of this workflow are off the critical path: the nightly batch re-derivation, and triage of the drifted list when a batch reports a dozen STALE verdicts at once. Those map cleanly onto availability-limited resources rather than your merge gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access is a reasonable fit for the triage half — classifying a batch of drift records into "rebased", "collection changed", "interpreter moved" — because the output is a short label per record and a mislabel only costs a human glance. The free server option fits the batch half: run the verifier on a schedule, write packets and verdicts to an artifact store, and keep the merge gate on your own runners. Neither belongs inline in pre-merge; treat capacity as best-effort and design the workflow so a skipped night weakens your sampling, not your gate.

# nightly, off the critical path
python evidence_packet.py verify --root . --tests tests/ --since 24h \
  --out artifacts/evidence/$(date -u +%Y%m%dT%H%M%SZ).json

Limitations and who should skip this

Slow builds. If one full re-derivation costs more than about ten minutes, sampling shrinks to near zero and the packet becomes decoration. Fix build time first.
Regulated audit trails. This packet is reproducibility evidence, not a signed attestation. It does not satisfy frameworks that require cryptographic signing, retention policy, or third-party attestation.
Single-developer prototypes. One author, one day, no merge queue: the tree digest alone is enough; the rest is ceremony.
Non-deterministic suites. If your suite is genuinely non-deterministic, REPRODUCED versus FAILED is noise. Stabilize or quarantine first; a packet cannot launder a flaky suite into a trustworthy one.

The habit worth keeping is narrow and durable: when a patch reports green, ask what would have to be true for that green to be false, and store the four cheap fields that answer it. If you want to try the batch half on free capacity, the nightly command above is the whole integration surface.

Property Checks Need Enemies: A Degenerate-Oracle Battery for Agent Patches

Finley Zhou — Fri, 06 Nov 2026 21:41:57 +0000

A property check that passes every wrong implementation you hand it is not a verification gate. It is a comment with a stack trace.

When an agent writes the patch and the tests together, both come from the same mental model. CI goes green on the first run, coverage looks respectable, and nothing in the report distinguishes a correct patch from a plausible one. The fix is not more tests. It is a cheaper, adversarial question asked of each property: which wrong implementation does this property actually kill?

The artifact below answers that with a fixed table — a degenerate-oracle battery. Six deliberately broken implementations of the same contract, every property run against all six, and two independent gates on the resulting matrix.

This is not fixture mutation. Fixture mutation asks what inputs a test needs. The battery asks what a property can distinguish, using the implementation as the adversary.

Step 1 — Freeze the contract surface

The battery only pays off on functions with a checkable contract. Require all three before enrolling a target:

Pure or effectively pure — no clock, network, or global state in the return value.
Boundary-level — the agent patch calls it through one named entry point, not through three internal helpers.
Specifiable without the code — you can state the property in one sentence without reading the diff.

If a function fails (1), the battery will produce noise, not signal. Timing-sensitive and stateful code is out of scope; a clock-dependent function has no stable oracle to be wrong about.

Step 2 — Build the degenerate battery

Six adversaries cover most pure-function contracts: empty, constant, identity, partial, sort-only, and off-by-one. Each is a few lines and each represents a real class of agent bug — the patch that returns the input untouched, or merges but never sorts, or sorts but never merges.

# killmatrix.py — degenerate adversaries for merge_intervals(list[(start, end)])

Interval = tuple[int, int]


def degenerate_null(intervals: list[Interval]) -> list[Interval]:
    return []                       # plausible: 'no overlaps to merge'


def degenerate_const(intervals: list[Interval]) -> list[Interval]:
    return [(0, 0)]                 # plausible: hard-coded seed


def degenerate_identity(intervals: list[Interval]) -> list[Interval]:
    return list(intervals)          # plausible: forgot to normalise


def degenerate_sorted_only(intervals: list[Interval]) -> list[Interval]:
    return sorted(intervals)        # plausible: sorted, still overlapping


def degenerate_first_only(intervals: list[Interval]) -> list[Interval]:
    return list(intervals[:1])      # plausible: early return in the loop


def degenerate_off_by_one(intervals: list[Interval]) -> list[Interval]:
    ordered = sorted(intervals)
    return [(lo, hi + 1) if i % 2 else (lo, hi)
            for i, (lo, hi) in enumerate(ordered)]


DEGENERATES = {
    'null': degenerate_null,
    'const': degenerate_const,
    'identity': degenerate_identity,
    'sort1': degenerate_sorted_only,
    'first': degenerate_first_only,
    'obo': degenerate_off_by_one,
}

Keep the set small and stable. A battery that grows every week stops being comparable against last month's baseline.

Step 3 — Write properties that can fail

Use fixed input cases, not randomised search. The battery needs determinism more than it needs reach: the same property must produce the same verdict across machines and across reruns, or the kill matrix is not a gate.

PROPERTIES = {}


def prop(name: str, guard: bool = False):
    def deco(fn):
        fn.guard = guard
        PROPERTIES[name] = fn
        return fn
    return deco


CASE = [(5, 8), (1, 3), (2, 4)]


@prop('sorted_and_non_overlapping')
def _shape(merge):
    out = merge(CASE)
    assert out == sorted(out), out
    assert all(a[1] < b[0] for a, b in zip(out, out[1:])), out


@prop('coverage_preserved')
def _coverage(merge):
    out = merge(CASE)
    assert sum(h - l for l, h in out) == sum(h - l for l, h in CASE), out


@prop('idempotent')
def _idem(merge):
    once = merge(CASE)
    assert merge(once) == once, once


@prop('empty_is_empty', guard=True)
def _empty(merge):
    assert merge([]) == [], 'empty input'

The guard flag matters. Some assertions are cheap sanity checks that will never discriminate much; forcing them through the same threshold produces busywork and teaches reviewers to ignore the gate.

Step 4 — Run the kill matrix

Two axes, two gates. Per property: how many adversaries does it kill. Per adversary: is it killed by at least one property.

def run_matrix(properties, degenerates):
    rows, coverage = {}, {name: 0 for name in degenerates}
    for pname, prop in properties.items():
        kills = []
        for dname, impl in degenerates.items():
            try:
                prop(impl)
            except AssertionError:
                kills.append((dname, 'K'))
                coverage[dname] += 1
            except Exception:
                kills.append((dname, 'E'))   # crash counts as a kill, logged
                coverage[dname] += 1
            else:
                kills.append((dname, '.'))
        rows[pname] = kills
    return rows, coverage

An illustrative run of the harness above, on the four properties in Step 3:

property                      null const ident sort1 first obo  kills
sorted_and_non_overlapping    .    .     K     K     .     K    3
coverage_preserved            K    K     .     .     K     K    4
idempotent                    .    .     .     .     .     K    1
empty_is_empty [guard]        .    K     .     .     .     .    1

killed by >=1 property:      1    2     1     1     1     3

The verdict: idempotent fails the gate at one kill. That is the useful part of the report — merge is idempotent under four of six broken implementations, so idempotence alone proves almost nothing about this patch. Adversary coverage is healthy; no blind spot in the battery itself.

Step 5 — Route each outcome

The gate should produce a task, not a mood. Route by cell, not by overall pass rate:

Property kills	Adversary coverage	Verdict
0	any	Property is decoration — delete or rewrite against the spec
1	any	Too weak — widen the input case or merge with a sibling property
≥2	all adversaries killed somewhere	Acceptable, report as baseline
≥2	some adversary survives everywhere	Battery blind spot — write a property, do not add another example test

Exit non-zero on rows 1 and 2, and on any unkilled adversary. Keep the threshold at two kills as an explicit constant in the file, so a later patch cannot quietly lower it.

Step 6 — Keep flaky freezes inside the battery's scope

A frozen test still runs in the nightly job, and freezing does not exempt it from review. Two rules keep freezes from hiding non-discriminating checks:

A freeze may only be lifted together with a replacement property that passes the battery — otherwise the flake returns, now with a green badge.
The battery runs on the frozen test's module too. A quarantine that removes coverage should not also remove the evidence that the remaining properties discriminate.

Runtime stays bounded: the cost is properties × adversaries fixed-input calls, not a randomised search. For a handful of contracts this is a sub-second job, which is why it is safe to run on every patch.

Where free model access and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The expensive step is Step 2 — authoring plausible-wrong implementations for a new signature. That is drafting work, and free model access makes it cheap: ask for six wrong implementations of the stated contract, each with a one-line rationale explaining which bug it stands for, then prune anything that is wrong for the same reason as another. The nightly matrix itself is a batch job with no database, no browser, and no GPU, so the free server option is sufficient for it.

Neither step is a substitute for the human judgement in Step 1. If the contract cannot be stated without the diff, no amount of generation will fix the battery.

Limitations and who should not use this

The adversaries are bounded. Passing all six is evidence of robustness against six specific bugs, not correctness.
Fixed input cases trade reach for determinism. Numeric edge cases and boundary arithmetic need a separate seeded property run.
Correlated blind spots are real: if the same model drafts the patch, the properties, and the degenerates, weaknesses line up. Draft degenerates from the stated contract, never from the diff.
Not suitable for latency or performance contracts, UI behaviour, or anything whose oracle is a human reviewer.
Do not adopt it for a codebase with no pure boundary functions; the setup cost will exceed the return.

Reproduce it

python killmatrix.py --min-kills 2 --fail-on-blindspot
python killmatrix.py --json report.json      # baseline for the next diff

Commit the JSON report alongside the patch. The next reviewer compares matrices, not opinions — and a property that survived every enemy is visible before merge instead of after the revert.

If you want to try the drafting half without a billing decision, MonkeyCode's free model access and free server option cover exactly this workflow; the harness above needs nothing else.

Route Agent Patches to the Cheapest Gate That Can Catch Them

Finley Zhou — Fri, 06 Nov 2026 17:37:55 +0000

Route Agent Patches to the Cheapest Gate That Can Catch Them

The cheapest reliable verification for an agent patch is a function of the diff's shape, not of who wrote it. Most review pipelines miss this and run the entire matrix on everything. That is not rigor. It is a way of postponing a decision until the queue decides for you.

Patch volume went up. Reviewer attention did not. The only sustainable response is to spend expensive gates where they can actually fail, and cheap gates everywhere else.

The failure mode: uniform verification

A one-line docstring edit and a lockfile bump currently cost the same wall-clock time. Both wait behind the same suite. Both get the same treatment.

The result is predictable. The suite gets slower, the queue gets longer, and people start merging on red because "it's probably the flaky one." Any of that sound familiar? Uniform verification does not buy safety — it buys the habit of ignoring failures.

A patch router fixes the ordering, not the tests. It answers one question before anything runs: what is the minimum set of gates that could plausibly catch a defect in this diff?

Four signals that actually change the gate set

Diff size alone is a weak proxy. These signals are stronger, and all four are extractable with git diff alone.

Signal	Why it changes the tier
Production files changed with no test file changed	Nothing in the diff demonstrates the new behaviour was exercised
Assertions added vs. removed	Tests edited without a new assertion are often tests edited to pass
Dependency or CI surface touched	Blast radius leaves the repo; local gates cannot see it
Change size on production files	A 4-line change and a 400-line change fail differently

Note what is not on the list: commit message, model name, author confidence. None of those are evidence.

The router

Reference implementation below, standard library only, no benchmark claims attached. It reads a diff range and prints the tier plus the gate list.

#!/usr/bin/env python3
"""triage.py - emit the cheapest verification tier that can catch this diff.

Usage: python3 triage.py [base_ref]   (default: HEAD~1)
Output: JSON on stdout.
"""
import json
import re
import subprocess
import sys

TEST_PATH = re.compile(
    r"(^|/)(tests?|spec|__tests__)(/|$)"
    r"|(_test\.py$|^test_.*\.py$|\.test\.[jt]sx?$|\.spec\.[jt]sx?$)"
)
SRC_PATH = re.compile(r"\.(py|ts|tsx|js|jsx|go|rs|java|kt|rb|php|cs)$")
DEP_PATH = re.compile(
    r"(^|/)(package-lock\.json|pnpm-lock\.yaml|poetry\.lock"
    r"|requirements.*\.txt|pyproject\.toml|go\.sum|Cargo\.lock)$"
)
CI_PATH = re.compile(
    r"(^|/)(\.github/workflows/|\.gitlab-ci\.yml|Makefile|tox\.ini|noxfile\.py)"
)
ASSERT_LINE = re.compile(r"\+(?!\+\+).*\b(assert\b|expect\s*\(|require\.\w+\()")

GATES = {
    0: ["lint"],
    1: ["lint", "unit-suite"],
    2: ["lint", "unit-suite", "property-checks", "fixture-lock"],
    3: ["lint", "unit-suite", "property-checks", "fixture-lock",
        "order-permutations", "freeze-audit", "human-signoff"],
}


def git(*args):
    return subprocess.run(("git",) + args, capture_output=True,
                          text=True, check=True).stdout


def numstat(base):
    rows = []
    for line in git("diff", "--numstat", base, "HEAD").splitlines():
        add, dele, path = line.split("\t", 2)
        rows.append({"path": path,
                     "add": 0 if add == "-" else int(add),
                     "del": 0 if dele == "-" else int(dele)})
    return rows


def signals(rows, base):
    src = [r for r in rows
           if SRC_PATH.search(r["path"]) and not TEST_PATH.search(r["path"])]
    tst = [r for r in rows if TEST_PATH.search(r["path"])]
    added = removed = 0
    for line in git("diff", "-U0", base, "HEAD", "--").splitlines():
        if ASSERT_LINE.match(line):
            added += 1
        elif line.startswith("-") and not line.startswith("---") \
                and ASSERT_LINE.match("+" + line[1:]):
            removed += 1
    return {
        "prod_files": len(src),
        "test_files": len(tst),
        "src_churn": sum(r["add"] + r["del"] for r in src),
        "asserts_added": added,
        "asserts_removed": removed,
        "deps_changed": any(DEP_PATH.search(r["path"]) for r in rows),
        "ci_changed": any(CI_PATH.search(r["path"]) for r in rows),
    }


def tier(s):
    if s["deps_changed"] or s["ci_changed"]:
        return 3, "dependency or CI surface moved"
    if s["prod_files"] and not s["test_files"]:
        return (2 if s["src_churn"] > 30 else 1), "production change, no test change"
    if s["prod_files"] and s["test_files"]:
        if s["asserts_added"] == 0:
            return 3, "tests edited but no assertion added"
        if s["asserts_removed"] > s["asserts_added"]:
            return 3, "more assertions removed than added"
        return 2, "production and test change together"
    if s["test_files"]:
        return 1, "test-only change"
    return 0, "docs or comments only"


if __name__ == "__main__":
    base = sys.argv[1] if len(sys.argv) > 1 else "HEAD~1"
    sig = signals(numstat(base), base)
    t, why = tier(sig)
    print(json.dumps({"tier": t, "reason": why,
                      "gates": GATES[t], "signals": sig}, indent=2))

Run it before you run anything else:

python3 triage.py origin/main

Sample output for an agent patch that edited three tests and removed two assertions without adding any:

{
  "tier": 3,
  "reason": "tests edited but no assertion added",
  "gates": ["lint", "unit-suite", "property-checks", "fixture-lock", "order-permutations", "freeze-audit", "human-signoff"],
  "signals": {"prod_files": 0, "test_files": 3, "src_churn": 0, "asserts_added": 0, "asserts_removed": 2, "deps_changed": false, "ci_changed": false}
}

That path escalates on purpose. A patch that only touches tests and only removes assertions is the exact shape you want a human to look at.

Wire it in, in five steps

Land triage.py in scripts/ and make it a one-liner in your CI job, before the test step.
Make the runner read the JSON and short-circuit. Tier 0 runs lint and stops.
Keep tier 3 as a blocking human gate. Do not let it auto-merge on green.
Log tier and reason next to every build. After two weeks you will know whether the thresholds match your repo.
Tune the 30 in src_churn > 30 from that log, not from intuition.

The runner itself is boring, which is the point:

#!/usr/bin/env bash
set -euo pipefail
python3 scripts/triage.py "${1:-HEAD~1}" > /tmp/plan.json
cat /tmp/plan.json
tier=$(python3 -c 'import json;print(json.load(open("/tmp/plan.json"))["tier"])')
case "$tier" in
  0) ./scripts/lint.sh ;;
  1) ./scripts/lint.sh && ./scripts/test.sh ;;
  2) ./scripts/lint.sh && ./scripts/test.sh \
       && ./scripts/properties.sh && ./scripts/fixture-lock.sh ;;
  *) ./scripts/full-matrix.sh ;;
restart_placeholder
esac

Where a free model tier actually fits

One step in this workflow is high-volume, low-stakes, and wrong a lot: drafting the candidate property list for a tier-2 patch. That is a good job for a free model tier, because every draft is validated by an executable check before it counts.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The second fit is the runner. Tier 0 and tier 1 gates are cheap and should never queue behind the main CI fleet, so an always-on free server running the fast path is a reasonable use of one. Both availability claims — the free model access and the free server option — come from the project operator; check the current terms on the project page yourself before you plan capacity around them.

What must stay outside the model's hands: the tier decision, the pass/fail verdict, and the human sign-off. Those are the parts where a plausible wrong answer is expensive.

Property checks by patch shape

A property that cannot be executed is not a property. Keep the list short, executable, and tied to the diff.

Patch shape	Property to try first	Cheap check
Pure transform	Round-trip: decode(encode(x)) == x	200 generated inputs
Parser or tokenizer	Never raises on arbitrary bytes; returns a result or a typed error	1k fuzz iterations
State machine	No transition lands outside the declared state enum	Enumerate all edges
Adapter around an external API	Every timeout path returns the documented error type	Stub the client, inject failure
Cache or memo layer	Read-after-write never returns a stale value for the same key	Handler-level test

If tier 2 cannot produce at least one executable property for the changed file, that is itself a signal. It usually means the patch boundary is unclear, not that the code is safe.

Freezes need an expiry the router can read

An unexpiring flaky-test freeze is just a deleted test with extra ceremony. Give each record an owner and a date, and let the router see it.

{
  "test": "tests/test_billing.py::test_proration_rounding",
  "owner": "finley",
  "reason": "upstream FX feed returns stale rates after 22:00 UTC",
  "frozen_at": "2026-10-30",
  "expires": "2026-11-13"
}

Add one rule on top of the router: if any freeze record whose test path overlaps the diff has an expires in the past, force tier 3. Expired freezes are not noise. They are unverified claims that nobody has revisited.

Limitations and who should not use this

The heuristics read diff shape, not semantics. A four-line change to an auth check can be more dangerous than a 400-line refactor, and this router will happily assign it tier 1. Reviewers are still the last gate.

Path regexes break on monorepos with co-located tests (foo.test.ts next to foo.ts in the same directory is fine; __tests__ with a custom convention is not). Adjust the patterns before you trust the tier. The assertion counter is a lexical match and will miss custom assertion helpers.

Skip this approach if you have a mandatory full-matrix policy you cannot change, if your suite is fast enough that running everything is genuinely cheap, or if you work in a domain where every diff needs full review regardless of shape. In those cases the router adds a layer without removing work.

Start narrow

Run the router in log-only mode for a week. Compare the tier it would have assigned against what you actually ran. You will find two or three misroutes, fix the regexes, and then the short-circuit is safe to turn on.

If you want to try it with a free model drafting the property lists and a small always-on machine running the tier-0 and tier-1 path, the project page for MonkeyCode lists the current free model access and free server option — verify the terms there first, and start with tier 1 on a repository you already trust.

Quarantine Withdraws Coverage: Find the Only Witness Before You Freeze a Test

Finley Zhou — Fri, 06 Nov 2026 14:51:14 +0000

A flaky test freeze is a coverage withdrawal, and most teams approve it without checking the balance. If the test you are about to quarantine is the only check that observes some invariant, every agent patch merged during the freeze is silently approved against one fewer witness. The fix is boring: make witnesses explicit, then refuse freezes that would drop a property to zero coverage.

This is the workflow I use now. It takes about an hour to adopt on a single service and it has one hard rule — no freeze without a second witness.

The rule

A test may be frozen only if every property it witnesses is also witnessed by at least one non-frozen check.

That is the whole policy. The rest of this article is the machinery that makes the policy cheap to enforce, because a policy you have to remember is a policy you will skip on a Friday afternoon.

Why agent patches make this urgent

A human patch usually arrives with intent. An agent patch arrives with a diff and a green CI run. Neither of those tells you which invariants the author had in mind, so CI status becomes the primary evidence — and CI status is exactly what a freeze quietly weakens.

Worse, flake freezes are the cheapest knob an agent can reach for. If an agent can edit the freeze list, it can turn a failing assertion into a skipped assertion without touching the assertion. That means the freeze list is a security boundary, not a housekeeping file, and it deserves the same review as production config.

Step 1 — Turn invariants into seeded property checks

Example tests say "this input works." Properties say "no input breaks this." For freeze decisions you need the second form, because a property keeps witnessing after the example test is gone.

# tests/properties/test_ledger.py
from hypothesis import given, settings, strategies as st

from app.ledger import Ledger, InsufficientFunds

@given(st.lists(st.integers(min_value=-10_000, max_value=10_000), max_size=64))
@settings(max_examples=300, derandomize=True)
def test_balance_never_negative(amounts):
    ledger = Ledger()
    for amount in amounts:
        try:
            ledger.apply(amount)
        except InsufficientFunds:
            pass
        assert ledger.balance >= 0

@given(st.integers(min_value=1, max_value=10_000))
@settings(max_examples=50, derandomize=True)
def test_rejected_withdrawal_is_a_noop(amount):
    ledger = Ledger(initial=0)
    before = ledger.snapshot()
    try:
        ledger.apply(-amount)
    except InsufficientFunds:
        pass
    assert ledger.snapshot() == before

Two details matter. derandomize=True makes the sweep deterministic, so a failure is reproducible without re-running the generator loop. And the property is named after the invariant, not the input — that name is the key you will join on later.

# reproduce a failure locally with a pinned seed
pytest tests/properties -q --hypothesis-seed=12345

If you already have a property library, skip the code and go to the join. The join is the part nobody does.

Step 2 — Record who witnesses each property

Keep a small JSON file next to the freeze list. No database, no dashboard. It has to be reviewable in a diff.

{
  "properties": [
    {
      "id": "prop.ledger.balance_non_negative",
      "check": "tests/properties/test_ledger.py::test_balance_never_negative",
      "witnessed_by": [
        "tests/test_ledger.py::test_overdraft_rejected",
        "tests/test_ledger.py::test_withdraw_more_than_balance"
      ]
    },
    {
      "id": "prop.ledger.rejected_is_noop",
      "check": "tests/properties/test_ledger.py::test_rejected_withdrawal_is_a_noop",
      "witnessed_by": [
        "tests/test_ledger.py::test_overdraft_rejected"
      ]
    }
  ],
  "frozen": []
}

witnessed_by lists the example tests that would catch a violation of the same invariant with concrete, readable inputs. Note the asymmetry in the sample above: prop.ledger.rejected_is_noop has exactly one witness. That property cannot be frozen today, and the gate below will say so.

Step 3 — Gate the freeze in CI

The gate is short enough to read in a review, which is the point.

# scripts/witness_gate.py — stdlib only
import json
import sys


def main(path="properties.json") -> int:
    with open(path) as f:
        model = json.load(f)

    frozen = set(model.get("frozen", []))
    orphans = []

    for prop in model["properties"]:
        witnesses = set(prop.get("witnessed_by", []))
        if not witnesses:
            orphans.append((prop["id"], "no witnesses recorded"))
            continue
        survivors = witnesses - frozen
        if witnesses & frozen and not survivors:
            orphans.append((prop["id"], f"last witness frozen: {witnesses}"))

    if orphans:
        for prop_id, reason in orphans:
            print(f"BLOCK {prop_id}: {reason}")
        return 1
    print("ok: every property keeps at least one live witness")
    return 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:]))

Wire it to the diff so it only bites when the freeze list actually changes:

if git diff --name-only "$BASE"...HEAD | grep -qx properties.json; then
  python scripts/witness_gate.py || exit 1
fi

A blocked freeze is not a dead end. It produces exactly one of three follow-ups: write a property, promote an untracked test into witnessed_by, or delete the invariant on purpose and say why in the PR body. All three are better outcomes than a silent withdrawal.

Step 4 — Lock the generator, not just the fixture

Fixture locks catch edited fixtures. They do not catch a narrowed generator, and narrowing is the cheaper move: max_size=64 becomes max_size=4, everything still passes, and the property now witnesses almost nothing.

# scripts/prop_lock.py — fingerprint what the property actually exercises
import hashlib
import pathlib

PATHS = [
    "tests/properties/conftest.py",
    "tests/properties/strategies.py",
    "fixtures/ledger_golden.json",
]


def fingerprint() -> str:
    h = hashlib.sha256()
    for p in PATHS:
        h.update(p.encode())
        h.update(pathlib.Path(p).read_bytes())
    return h.hexdigest()[:16]


if __name__ == "__main__":
    print(fingerprint())

Commit the output to prop.lock and fail CI when the fingerprint moves without a reviewer ack. This is the same idea as a manifest diff, applied one layer down: the generator config is part of the test contract, so it belongs in the diff you read, not in the diff you skim.

Step 5 — Run the randomized sweep off the critical path

Here is where hosted capacity genuinely helps. Property sweeps are CPU-shaped and bursty: a 300-example sweep across a handful of properties is cheap, but running it on every push competes with the same runners your patch CI needs.

I keep the deterministic witness gate (Step 3) and the fingerprint check (Step 4) on the main runners, because they must be fast and authoritative. The long randomized sweep goes to a free server option so a bigger max_examples does not slow the merge path. In this setup that means MonkeyCode's operator-provided free server, with free model access used for one narrow job: proposing property candidates from a diff, always into a draft file that a human merges or discards.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The division of labor is deliberate. The model proposes invariant names and generator shapes; it never approves a freeze, never edits witnessed_by, and never decides that a property is satisfied. Those are review actions. Treat any free tier as convenience, not as an availability guarantee, and keep the authoritative gate where you control the runner.

Decision table

Freeze request	Witness state	Action
Flaky example test	Two or more properties still witnessed	Approve with an expiry date and an owner
Flaky example test	Last witness of one property	Block; write the property first
Flaky property check	Deterministic failure at pinned seed	Not a flake — it is a bug
Flaky property check	Fails only on unseeded runs	Shrink the generator, then re-triage
Any freeze	`properties.json` untouched by the patch	Reject the patch, not the test

Row three is the one people get wrong. A property that fails at a pinned seed is a reproduced defect, so freezing it converts a known bug into an unowned one.

Limitations, and who should skip this

This workflow assumes your properties are honest. A generator that only emits small positive integers will happily certify a ledger that breaks on zero, and no amount of gate machinery fixes a distribution that never explores the boundary. Review the strategies like you review the assertions.

It also assumes a freeze actually matters to you. Teams with no rerun history, no flake inventory, and no agent patches in the loop should spend the hour elsewhere — a quarantine list with no attribution data is just a skip file with extra steps. And if your suite is mostly end-to-end tests sharing a database, the property-plus-witness split does not map cleanly; fix the shared state first.

Finally, this is a policy layer, not a proof. It tells you which invariants lose their last observer. It cannot tell you whether the invariant was ever the right one to hold.

Wrap

The gate is thirty lines and one JSON file, and it converts a silent coverage withdrawal into a reviewable decision. Start with a single property that maps to a test you are already tempted to freeze. If the gate blocks you on the first try, it is working.

If you want the sweep off your merge path, the free server option is a reasonable place to put it — just keep the witness gate on a runner you own.

Mutate the Fixture, Not the Code: Catching Agent Tests That Only Mirror the Implementation

Finley Zhou — Fri, 06 Nov 2026 11:44:45 +0000

Most agent-written tests pass for the wrong reason: the fixture already contains the answer. The test asserts that the code reproduces a value that was copied out of the code in the first place. Code mutation testing rarely catches this, because the mutant dies on some other assertion in the same file. Fixture mutation does catch it, and it costs one suite run per mutant.

So the conclusion up front: before you spend effort on property checks or a flake freeze, spend one cheap pass mutating the fixtures and see which tests stop failing. A test that survives a semantic change to its own input is not testing behavior — it is comparing two copies of the same string.

One caveat about everything below: the harness is scaffolding I have not executed in this exact form. Treat the code as a starting point, run it against your own repository, and record your own numbers. Do not inherit mine, because I am not publishing any.

What fixture mutation measures, and what it does not

Fixture mutation is a differential test on the inputs. You perturb a field the code is supposed to consume, rerun the suite, and observe whether anything notices. If nothing notices, one of three things is true: the field is unused, the assertion is too loose, or the test hardcodes the expected output instead of deriving it.

It does not measure coverage, and it does not prove correctness. A mutant can be unkillable for a legitimate reason — a field consumed only on a branch your suite never reaches on purpose. That is exactly why the classification step below has three buckets instead of two.

Step 1: Build a mutant set that is semantically plausible

The cheapest mutants are worthless. Reversing a string or nulling every field produces instant kills and tells you nothing. You want mutations a real upstream system could plausibly send.

# fixture_mutants.py — scaffolding, not executed in this article
import copy, json
from dataclasses import dataclass

@dataclass(frozen=True)
class Mutant:
    id: str
    fixture: str
    pointer: tuple
    op: str
    before: object
    after: object

def walk(node, path=()):
    yield path, node
    if isinstance(node, dict):
        for k, v in node.items():
            yield from walk(v, path + (k,))
    elif isinstance(node, list):
        for i, v in enumerate(node):
            yield from walk(v, path + (i,))

def numeric_drift(doc):
    """Plausible drift: rounding boundaries and quantity edits."""
    for path, node in walk(doc):
        if isinstance(node, int) and not isinstance(node, bool):
            yield path, node, node + 1
        elif isinstance(node, float):
            yield path, node, round(node + 0.01, 2)

def drop_optional(doc):
    """Remove one key at a time; the schema may allow it."""
    for path, node in walk(doc):
        if isinstance(node, dict):
            for k in list(node):
                yield path + (k,), node[k], "__ABSENT__"

def reorder_list(doc):
    """Order-dependence is a real bug class, not a contrived one."""
    for path, node in walk(doc):
        if isinstance(node, list) and len(node) > 1:
            yield path, node, list(reversed(node))

Aim for 15–40 mutants over the fixtures your agent patch actually touches. Bound the set deliberately — the runtime below multiplies by it.

Step 2: Run every mutant N times, including a no-op control

This is the part that makes the result trustworthy when your suite is flaky. Each mutant runs N times with different ordering and hash seeds. The no-op control — a mutant with an identical payload — establishes the baseline instability of the suite itself.

#!/usr/bin/env bash
# run_mutants.sh — scaffolding
set -uo pipefail
OUT=mutants/report.jsonl
: > "$OUT"

for m in mutants/cases/*.json; do
  id=$(basename "$m" .json)
  for seed in 1 2 3; do
    cp "$m" "$FIXTURE_DIR/orders.json"
    PYTHONHASHSEED="$seed" FIXTURE_DIR="$FIXTURE_DIR" \
      pytest -q -p no:cacheprovider tests/ >/dev/null 2>&1
    code=$?
    printf '{"mutant":"%s","seed":%d,"exit":%d}\n' "$id" "$seed" "$code" >> "$OUT"
  done
done

Keep set -e off. You are collecting exit codes, not enforcing success.

Step 3: Classify into three buckets, and exclude harness errors

pytest exit codes carry meaning you must not flatten. 0 is a pass. 1 is a genuine test failure. 2, 3, 4, and 5 are interruption, internal error, usage error, and no-tests-collected — those are harness bugs, not kills. Folding a 5 into the killed bucket is how a mutation report silently becomes fiction.

import collections, json

HARNESS_ERRORS = {2, 3, 4, 5}
rows = [json.loads(l) for l in open("mutants/report.jsonl")]
by = collections.defaultdict(list)
for r in rows:
    by[r["mutant"]].append(r["exit"])

control = by.pop("m000-noop", [])
buckets = {"killed": [], "survivor": [], "unstable": [], "error": []}
for mid, exits in by.items():
    if any(e in HARNESS_ERRORS for e in exits):
        buckets["error"].append(mid); continue
    fails = sum(e == 1 for e in exits)
    passes = sum(e == 0 for e in exits)
    key = "killed" if passes == 0 else "survivor" if fails == 0 else "unstable"
    buckets[key].append(mid)

scored = len(buckets["killed"]) + len(buckets["survivor"])
fms = len(buckets["killed"]) / scored if scored else float("nan")

Report fms next to control and the unstable count, never alone. A score of 0.62 on a suite where the no-op control failed 2 of 3 times means nothing.

Step 4: Read the table, not the number

Head outcome	Base outcome	Interpretation	Action
killed	killed	Assertion already sensitive	Leave it
survivor	killed	Patch removed sensitivity — the strongest signal here	Block the patch
survivor	survivor	Pre-existing gap	File, do not block
unstable	any	Suite instability, not a test-strength result	Freeze ledger
error	any	Harness bug	Fix the runner first

The row that matters for agent patches is survivor / killed. Agents are very good at deleting the one assertion that made a fixture edit fatal, because the deletion makes the new test green. That delta is a regression in verification strength, and it is invisible to a pass/fail gate.

Running the base tree doubles your runtime. If that is unaffordable, run base once per module rather than once per patch, and accept the coarser resolution.

Step 5: Freeze unstable mutants without laundering them into kills

An unstable mutant is a flaky-test finding wearing a mutation costume. It must never be counted as killed, and it must never be counted as a survivor either — it sits outside the denominator with an owner and an expiry.

# freeze-ledger, one entry per unstable mutant
- test: tests/test_orders.py::test_total_rounding
  detected_by: fixture-mutant m012 (numeric_drift)
  observed: 1/3 pass on unmutated fixture
  owner: payments
  opened: 2026-11-06
  expires: 2026-12-06
  exit_criteria: 20 consecutive clean runs across 3 seeds, unmutated fixture

Two rules keep the ledger honest. First, an expired entry fails the build; a freeze that cannot rot is a permanent deletion. Second, a frozen test cannot be cited as evidence for the patch that touched it — otherwise the freeze quietly becomes an approval.

Where a free-tier model genuinely helps

Two parts of this workflow are tedious rather than hard: generating semantically plausible mutants, and triaging survivor clusters. That is where I would use MonkeyCode's free model access — feeding it a fixture plus the consuming function's signature and asking for candidate mutations with a one-line rationale each, then grouping the survivors by which assertion they should have tripped.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option matters for a different reason: the mutant matrix is embarrassingly parallel and boring, exactly the workload you do not want consuming your interactive machine. Check the current terms yourself before depending on either capability — I am not asserting quota, hardware, or permanence, and availability at that tier is not something this article can promise. Treat the free tier as a burst resource: run the matrix, export the JSONL, go home. And do not ship private fixtures or production-derived data to any hosted model without your own review.

Keep the model on the generator side, not the judge side. It proposes mutants; the test suite votes. A model that both proposes and evaluates its own mutants reproduces exactly the tautology this workflow exists to catch.

Cost, limits, and who should skip this

Runtime is mutants × seeds suite runs. With 25 mutants and 3 seeds that is 75 runs, which is why you scope the fixture set to files the patch touched rather than the whole repository.

Skip this if any of the following is true:

Your suite already takes more than a few minutes; you need test-level selection before mutation.
Your fixtures are generated at runtime from a schema — mutate the generator's inputs instead, and note that the mapping back to a failing test gets fuzzier.
You have no stable no-op control. Without it, every number below is noise, and the right next task is the flake freeze, not this one.
Your tests assert on opaque outputs you cannot perturb meaningfully; property checks will serve you better here.

Fixture mutation is not a replacement for property-based testing. It is the cheap prior step that tells you whether property checks would be answering a question anyone is asking. If you run the no-op control against a fixture-heavy suite, the unstable count is the number I would want to compare — it caps how much trust anything else in the report deserves.

Three Green Reruns Prove Almost Nothing: Sizing a Flake-Detection Budget for Agent Patches

Finley Zhou — Fri, 06 Nov 2026 08:40:08 +0000

A merge gate that reruns a failing test three times is not testing for flakes. It is testing whether the flake happens to be loud.

Run the arithmetic before you argue about it. At a 2% per-run failure rate, three reruns catch the flake about 5.9% of the time. The other 94% of the time the gate prints green, the agent patch merges, and the flake ships with a clean receipt stapled to it.

This post is about that receipt. It gives you a small script, a decision table, and a gate policy for sizing how many reruns a patch actually needs — and for admitting when reruns are the wrong instrument entirely.

What a rerun is actually measuring

A rerun is a sample. One execution of a test is a Bernoulli trial: pass or fail, with some unknown probability p of failing on any given run. Rerunning k times and reporting green means you observed k passes in a row.

The probability that the gate catches a flake at all is:

detection(k, p) = 1 - (1 - p)^k

That is the whole model. It has one honest assumption — independence — and one uncomfortable consequence: the useful quantity is k, and nobody on your team is currently choosing k on purpose. It is chosen by whatever the CI config said in 2021.

The numbers most teams never print

Failure rate `p` per run	Reruns for 95% detection	Power at 3 reruns
5%	59	14.3%
2%	149	5.9%
1%	299	3.0%
0.5%	598	1.5%
0.1%	2,995	0.3%

Read the middle column slowly. A 1% flake — the kind people describe as "rare, probably infrastructure" — needs roughly 299 reruns to be caught 95% of the time. Three reruns catch it about 3% of the time.

So when a gate goes green after three reruns, the correct statement is not "the test is stable." It is "the test is stable at the resolution this gate can see." Those are different claims, and only one of them is evidence about the patch.

Step 1 — Estimate `p`, and admit how little you know

You rarely have a clean p. You have a history: 4 failures in 120 runs, say. The naive estimate is 3.3%, but with n = 120 the 95% Wilson interval on that estimate is wide — roughly 1.3% to 8.3%.

That width matters, because the budget is steeply nonlinear in p. Planning against 1.3% means ~230 reruns; planning against 8.3% means ~35. If you plan against the point estimate, you will under-provision the gate for the regimes that hurt.

Use the upper bound of the interval for the merge gate. Use the point estimate only for cost forecasting.

Step 2 — Invert the formula to get a rerun budget

Solving for k:

k = ceil( ln(1 - target) / ln(1 - p) )

With target = 0.95, that gives the middle column above. Two practical notes.

First, this is a budget, not a wish. If the required k exceeds what your wall-clock and runner bill can absorb, the answer is not "rerun more anyway." The answer is that reruns cannot reach the frequency band where this bug lives, and you need a different instrument.

Second, the marginal returns collapse fast. At p = 1%, going from 299 to 309 reruns buys about half a percentage point of detection. You are paying ten full test cycles for 0.5%.

Step 3 — The artifact: `flake_budget.py`

Standard-library Python 3.9+, no dependencies. It takes either raw rates or an observed fail/run count and prints the budget, the power you would have had one rerun short, and the marginal gain from the last rerun.

#!/usr/bin/env python3
"""flake_budget.py - size a rerun budget before trusting a green agent patch.

  python3 flake_budget.py --target 0.95 --cost 8 --rates 0.05 0.02 0.01 0.005
  python3 flake_budget.py --target 0.95 --cost 8 --observe 1 120
"""
import argparse
import math


def runs_for(p: float, target: float) -> int:
    """Reruns so P(>=1 failure) >= target, assuming independent runs."""
    if p >= 1.0:
        return 1
    if p <= 0.0:
        return -1  # unreachable at any k
    return math.ceil(math.log(1.0 - target) / math.log(1.0 - p))


def power(p: float, k: int) -> float:
    return 0.0 if p <= 0.0 else 1.0 - (1.0 - p) ** k


def wilson(fails: int, n: int, z: float = 1.96) -> tuple[float, float]:
    if n == 0:
        return (0.0, 1.0)
    ph = fails / n
    d = 1.0 + z * z / n
    c = (ph + z * z / (2 * n)) / d
    h = z * math.sqrt(ph * (1 - ph) / n + z * z / (4 * n * n)) / d
    return (max(0.0, c - h), min(1.0, c + h))


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--target", type=float, default=0.95)
    ap.add_argument("--cost", type=float, default=None, help="seconds per run")
    ap.add_argument("--rates", type=float, nargs="*", default=[])
    ap.add_argument("--observe", type=int, nargs=2, metavar=("FAILS", "RUNS"))
    args = ap.parse_args()

    rows = []
    if args.observe:
        fails, n = args.observe
        lo, hi = wilson(fails, n)
        print(f"observed {fails}/{n} -> p in [{lo:.4f}, {hi:.4f}] (95% Wilson)")
        rows.append(hi)  # plan against the conservative bound
    rows.extend(args.rates)
    if not rows:
        ap.error("pass --rates and/or --observe")

    print(f"\ntarget detection power: {args.target:.0%}")
    print(f"{'p(run)':>9} {'reruns':>7} {'power(k-1)':>11} {'gain':>6} {'secs':>8}")
    for p in rows:
        k = runs_for(p, args.target)
        before = power(p, k - 1)
        secs = f"{k * args.cost:.0f}" if args.cost else "-"
        print(f"{p:9.4f} {k:7d} {before:10.1%} {args.target - before:6.1%} {secs:>8}")


if __name__ == "__main__":
    main()

Run it against your own history before touching the CI config:

python3 flake_budget.py --target 0.95 --cost 8 --observe 1 120
# observed 1/120 -> p in [0.0015, 0.0455] (95% Wilson)
# target detection power: 95%
#    p(run)  reruns  power(k-1)   gain     secs
#    0.0455      65        93.9%   1.1%      520

Sixty-five reruns, about 8.7 minutes of runner time, to make a 4.5% flake visible 95% of the time. That may be a defensible price for one suspicious test. It is not a defensible price per agent patch when the agent is producing thirty patches a day.

Step 4 — Put the budget in the merge gate

A policy that survives contact with a real queue:

Compute the budget first. Every quarantined test carries its Wilson interval and required k. No number, no quarantine.
Compare k to the rerun cap. The cap is a fixed wall-clock number, not a habit. Three is fine as a cap; it is not fine as a justification.
If k > cap, refuse the green. Mark the patch unproven-on-flaky-test rather than passed. That is the whole point: the gate stops laundering absence of evidence into evidence of stability.
Convert, don't rerun. Turn the flaky path into a deterministic check — a property, an invariant, an injected clock, a locked fixture. A property check has k = 1 by construction and costs the same every time.
Escalate with a deadline. If conversion is not possible this week, the test goes to quarantine with an owner and an expiry, and the suite runs it out-of-band rather than gating merges on a coin flip.

Budget `k` for 95%	Reading	Gate action
≤ 10	cheap to observe	rerun to budget inside the gate
11–60	expensive but real	rerun on a nightly lane, block only on nightlies
61–300	beyond gate resolution	refuse green, open a conversion task
> 300	not a rerun problem	quarantine with expiry; property-check or delete

Where the free tier actually fits

Two parts of this loop are cheap and parallel: drafting the property check that replaces the rerun, and running the nightly sweep that measures p in the first place.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option, and those two map onto exactly those two jobs — a model to read the flaky test and the patch diff and propose an invariant worth asserting, and a server to run the out-of-band sweep so the budget measurement does not sit on a developer laptop. Treat both availability claims as operator-supplied and re-verify them against current terms before you wire them into a required check; I have not benchmarked throughput, and a measurement lane that silently slows down is worse than no lane. The honest framing: use the free tier to buy the evidence, not to skip the gate. If you want to try the loop end to end, start with one quarantined test and one property it should have asserted all along.

Limitations worth stating out loud

Independence is doing heavy lifting. Flakes are frequently correlated with seed, test ordering, clock, or network state. Reruns on the same seed do not resample those axes, so the computed power is an upper bound, not a promise. Vary the axis you suspect, or the number lies to you politely.

Small n makes p a rumour. Below roughly 50 observations the interval is so wide that any budget derived from it is a guess with decimal places. Collect first, plan second.

Cost is not only wall-clock. A 300-rerun budget on a 90-second integration test is 7.5 hours per patch. That is a decision about the suite, not about the flake.

Who should not use this

Teams whose suite is small and fast, where a 3-rerun gate genuinely covers the flakes that matter — measure once, and if k ≤ 10 holds, the policy is overhead. Teams whose flakes are deterministic given a seed: the fix is a reproduction harness, not a budget. And anyone tempted to use the budget as a substitute for conversion — the script tells you the price of seeing a bug, never the price of fixing one.