Unproven Is Not Green: Rank Agent-Patch Tests, Then Isolate Flakes

An agent patch that leaves CI green has not been proven. It has only survived the tests that ran. Those are different claims.

Rank every test by what it can actually falsify. Execute in three lanes: in-process properties, digest-locked fixtures, and isolated retries. Flakes leave the critical path as UNPROVEN. They do not become skips. They do not become passes.

If a patch touches a module and the only remaining evidence for that module is UNPROVEN, the merge is not green. It is unfinished.

The failure mode this ranking targets

Agent patches fail CI in boring ways. They also pass CI in boring ways. The second case is worse.

A typical patch keeps existing assertions intact, overfits one checked-in file, or rides a test that already flakes on shared runners. Volume does not fix that. Another hundred tests with the same oracle quality still cannot falsify the change.

Falsifiability is the scarce resource. Spend it on purpose.

A rank, not a skip list

Score each test on three binary questions. Keep the arithmetic visible. Hidden weights become folklore.

Question	Yes	No
Can a wrong patch in the diff make this test fail without editing the test?	4	0
Is every input a file, seed, or literal whose digest is pinned?	2	0
Does a failure name a function or module inside the diff?	1	0

A test that scores 0 cannot block a merge. It is telemetry. A test that scores 4 or more can sit in the critical path. A test that scores 2–3 belongs in the fixture lane only after its inputs are pinned.

The numbers are a starting rubric. Change them in the repo, not in a hallway argument.

Three execution lanes

Map the rank onto a scheduler. Do not map it onto a skip decorator.

1. Property lane. Pure functions, algebraic laws, schema checks. No clock. No network. No shared /tmp. Run in-process. Fail closed.
2. Fixture lane. Golden files, recorded traces, checked-in CSVs. Hash every byte the test reads. If the hash drifts, the lane fails before the assertion runs.
3. Retry lane. Tests with residual environmental noise. New process, fresh working directory, bounded attempts. The lane may return UNPROVEN. It may not return PASS after a flake.

UNPROVEN is the whole point. A freeze-as-skip converts instability into a fake pass. A retry lane records that you did not obtain evidence.

Reference scorer

The following is a self-contained reference implementation. It is a workflow artifact, not a measured production deploy. Save it as lane_score.py and run it against a patch directory.

#!/usr/bin/env python3
"""Rank tests, run three lanes, emit PASS/FAIL/UNPROVEN."""
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
import tempfile
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Callable

@dataclass(frozen=True)
class Spec:
    name: str
    falsifies_diff: bool
    inputs_pinned: bool
    localizes: bool
    lane: str  # property | fixture | retry
    fixture_paths: tuple[str, ...]
    command: list[str]


def rank(spec: Spec) -> int:
    return (
        (4 if spec.falsifies_diff else 0)
        + (2 if spec.inputs_pinned else 0)
        + (1 if spec.localizes else 0)
    )


def digest_tree(paths: tuple[str, ...]) -> str:
    h = hashlib.sha256()
    for raw in sorted(paths):
        p = Path(raw)
        h.update(p.as_posix().encode())
        h.update(p.read_bytes())
    return h.hexdigest()


def run_cmd(cmd: list[str], cwd: Path, env: dict[str, str]) -> int:
    proc = subprocess.run(
        cmd,
        cwd=cwd,
        env=env,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        check=False,
    )
    Path(cwd, "lane.out").write_text(proc.stdout)
    return proc.returncode


def isolate(cmd: list[str], attempts: int = 2) -> str:
    """Retry in a fresh tempdir. Flake => UNPROVEN, never PASS."""
    codes = []
    for _ in range(attempts):
        with tempfile.TemporaryDirectory(prefix="lane-") as tmp:
            code = run_cmd(cmd, Path(tmp), os.environ.copy())
            codes.append(code)
            if code == 0:
                return "PASS" if all(c == 0 for c in codes) else "UNPROVEN"
    if all(c != 0 for c in codes):
        return "FAIL"
    return "UNPROVEN"


def evaluate(spec: Spec, lock: dict[str, str]) -> dict:
    score = rank(spec)
    if score == 0:
        return {"name": spec.name, "rank": score, "status": "TELEMETRY"}

    if spec.lane == "fixture":
        current = digest_tree(spec.fixture_paths)
        expected = lock.get(spec.name)
        if expected is None:
            return {"name": spec.name, "rank": score, "status": "FAIL",
                    "reason": "missing fixture lock"}
        if current != expected:
            return {"name": spec.name, "rank": score, "status": "FAIL",
                    "reason": "fixture digest drift"}

    if spec.lane == "retry":
        status = isolate(spec.command)
    else:
        code = run_cmd(spec.command, Path.cwd(), os.environ.copy())
        status = "PASS" if code == 0 else "FAIL"

    return {"name": spec.name, "rank": score, "lane": spec.lane, "status": status}


def touched_unproven(results: list[dict], critical: set[str]) -> bool:
    names = {r["name"] for r in results if r["status"] == "UNPROVEN"}
    return bool(names & critical)

A property example for a CSV aggregator. The law is conservation of rows, not a golden total that an agent can hard-code.

def test_row_conservation(in_rows: list[dict], out_rows: list[dict]) -> None:
    assert len(out_rows) == len(in_rows)
    assert {r["id"] for r in out_rows} == {r["id"] for r in in_rows}


def test_totals_non_negative(out_rows: list[dict]) -> None:
    for row in out_rows:
        assert float(row["amount"]) >= 0.0

Pin fixtures as digests, not as “the file that happened to be in testdata/ last Tuesday.”

python3 - <<'PY'
from pathlib import Path
import hashlib, json
lock = {}
for p in sorted(Path("testdata").glob("*.csv")):
    lock[p.as_posix()] = hashlib.sha256(p.read_bytes()).hexdigest()
Path("fixtures.lock.json").write_text(json.dumps(lock, indent=2) + "\n")
print(f"pinned {len(lock)} files")
PY

Merge rule

Number the gate so a human can audit it in review.

1. Load the spec list and the fixture lockfile.
2. Drop rank-0 tests from the blocking set. Keep them in the log.
3. Run the property lane to completion. Any FAIL rejects the patch.
4. Run the fixture lane only if every digest matches. Digest drift is a fail, not a skip.
5. Run the retry lane last, with a fresh process per attempt.
6. If the patch touches module M and every test that can falsify M is UNPROVEN, reject.
7. Print a JSON report. Store it next to the patch, not in a chat transcript.

python3 lane_score.py --specs specs.json --lock fixtures.lock.json --out report.json
python3 - <<'PY'
import json, sys
rep = json.load(open("report.json"))
blocking = [r for r in rep if r["status"] in {"FAIL"}]
unproven = [r for r in rep if r["status"] == "UNPROVEN"]
print(f"fail={len(blocking)} unproven={len(unproven)}")
sys.exit(1 if blocking else 0)
PY

The last snippet treats UNPROVEN as visible but not automatically fatal. Pair it with step 6. A report that is all UNPROVEN on the touched surface is a reject even if the process exit code is 0.

Where a local proposer fits

The scorer is only useful if you run it before shared CI. A tight loop looks like: propose a patch, score it, keep the report. Shared runners then become a confirmation, not a discovery tool.

MonkeyCode's free model access and free server option can sit in that loop as the proposer. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The scorer does not depend on that proposer. Any source of diffs can feed the same lanes.

Do not ask the model to write the properties that judge its own patch in the same turn. Split the jobs. Properties are review artifacts. Patches are hypotheses.

What this does not catch

Wrong properties pass wrong patches. A conservation law that omits dropped rows will bless a deleter. Rank cannot save a hollow oracle.

Isolation does not fix a product race. It only stops the test process from sharing a working directory with the previous attempt. If the code under test deadlocks on a real mutex, the retry lane will report FAIL or UNPROVEN depending on timing. That is a signal to write a deterministic reproduction, not to raise the attempt count.

Digest locks do not freeze intended fixture updates. When the contract changes, update the lock in the same commit as the fixture. A lock that never moves is how teams start smuggling behavior changes into “test data cleanup.”

The rank is a heuristic. A test can score 7 and still assert the wrong thing. Read the assertion. The table does not replace that.

Who should not use this

Do not use three lanes as a substitute for failing closed in safety-critical code. If UNPROVEN must mean FAIL, delete the retry lane. The scheduler is for noisy application tests, not for invariants that protect money, health, or access control.

Do not use it when the suite is only live end-to-end checks against shared staging. There is no digest to pin. There is no in-process property. Rank will correctly mark almost everything as telemetry, and you will have built a reporter for a vacuum.

Do not use it to launder a skip list. If a test is flaky because the assertion is racy, rewrite the assertion. The retry lane is for residual runner noise after that rewrite, not instead of it.

A small close

Green means “nothing falsified the hypothesis.” UNPROVEN means “we did not obtain evidence.” Keep those strings distinct in the report. If you already generate candidate patches on a free local server, wire the scorer into that loop before the diff reaches shared CI.