Stop Sharing a Flake Allowlist With Agent Diffs

Agent diffs should not inherit the production flake budget. A human suite that retries, quarantines, or freezes flaky tests is a merge loophole once a model can add both code and tests. Score those diffs in a separate lane that pins fixtures, requires property oracles, and treats non-determinism as a hard fail.

That is the method. The rest is a policy file, a replay harness, and a promotion token the merge gate can parse.

The leak is the allowlist

Production CI is built for people. People negotiate a retry. They freeze a noisy case until the next sprint. They keep an allowlist because the pager is quiet and the product still ships.

Agents do not share that social contract. A patch can add a test that passes twice, fails on a third replay, and still looks green if the job retries. The same patch can rewrite a fixture so the old failure disappears. If scoring shares the human allowlist, the agent inherits every exemption you ever granted.

Do not delete the human suite. Split it. Shared retries are not kindness. They are an API for lucky greens.

Two lanes, two questions

Lane A is production CI. It may keep a small, dated flake budget for known human tests. Lane B is the agent scoring lane. Lane B has no flake budget, no retry-as-pass, and no freeze that outlives the merge request.

Lane B answers one question: does this diff stay correct when I/O is pinned and the new tests are replayed. Lane A answers a different question: does the branch still fit the product. Mixing those questions is how tautological tests and flaky greens reach main.

The scoring lane should be cheaper than the product pipeline. Generate many candidate diffs. Replay only Lane B. Promote survivors into Lane A. Spend production minutes last, not first.

Policy, not folklore

Store the split in the repo. A Slack exception does not survive the next agent run.

# scoring.lane.toml
[lane]
name = "agent-score"
flake_budget = 0
retries_as_pass = false
max_wall_seconds = 180
forbidden_pytest_plugins = ["pytest-rerunfailures"]

[fixtures]
lockfile = "tests/fixtures.lock.json"
require_digest = true

[properties]
module = "tests/oracles.py"
fail_on_missing_oracle = true

[authorship]
treat_new_tests_as_untrusted = true
min_replays = 3

flake_budget = 0 is the rule. Three replays is the measurement. If any replay disagrees, the diff fails scoring even if two of three were green.

Decision table

Observation	Lane B (score)	Lane A (production)
New test fails 1 of 3 replays	Reject the diff	Do not import the test
New test is deterministic but never imports an oracle	Reject when fail_on_missing_oracle is set	Optional follow-up
Fixture bytes changed without a lockfile update	Reject	Not a Lane A concern
Dated freeze on a known human flake	Ignore the freeze if the agent touched the test	Honor freeze until expiry
Property oracle fails, unit tests pass	Reject	Block merge
CI retry plugin enabled on the scoring job	Misconfiguration; fail closed	Allowed only with a dated budget
Both lanes green and score-report.json says pass	Promote	Merge

A green unit job is not a row in this table. The table is the artifact you review in the merge request.

Workflow

1. Classify the diff. List added and modified tests. Treat every new test file as untrusted input, not as evidence that the production code works.
2. Lock fixtures. Hash fixture files. Refuse the patch if bytes moved and tests/fixtures.lock.json did not.
3. Require a property oracle. A unit assertion that repeats the implementation is not an oracle. Bind at least one invariant: round-trip, monotonicity, idempotence, or schema.
4. Replay under a wall clock. Run the new tests min_replays times with hash randomization pinned. Any mismatch is a fail. It is not a quarantine candidate.
5. Forbid retry plugins on this job. pytest-rerunfailures, GitHub retry actions, and "rerun failed jobs" buttons map flakes onto greens. Lane B must not load them.
6. Write score-report.json. The merge gate parses the file. It does not scrape CI logs with a regex in a bot comment.
7. Promote, then run Lane A. Only then spend production minutes.

Step 4 is where teams cheat. They see two greens and ship. Lane B exists so the third replay still happens.

Harness (proposal)

The following scoring harness is a proposal, not a measured benchmark. Wire it as a job that does not share the production retry policy.

# score_lane.py
from __future__ import annotations

import hashlib, json, os, subprocess, sys, time
from pathlib import Path

POLICY = {
    "min_replays": 3,
    "max_wall_seconds": 180,
    "lockfile": Path("tests/fixtures.lock.json"),
    "pytest_args": ["-q", "--maxfail=1", "-p", "no:rerunfailures"],
}

def digest_tree(root: Path) -> dict[str, str]:
    out = {}
    for p in sorted(root.rglob("*")):
        if p.is_file() and p.name != ".gitkeep":
            out[str(p.as_posix())] = hashlib.sha256(p.read_bytes()).hexdigest()
    return out

def assert_fixtures_locked() -> None:
    lock = POLICY["lockfile"]
    current = digest_tree(Path("tests/fixtures"))
    if not lock.exists():
        raise SystemExit("missing fixture lockfile")
    expected = json.loads(lock.read_text())
    if current != expected:
        raise SystemExit("fixture digest drift; refuse scoring")

def run_once(replay: int) -> dict:
    env = os.environ.copy()
    env["PYTHONHASHSEED"] = "0"
    env.pop("PYTEST_ADDOPTS", None)  # refuse inherited rerun flags
    t0 = time.monotonic()
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", *POLICY["pytest_args"], "tests/"],
        env=env,
        capture_output=True,
        text=True,
        timeout=POLICY["max_wall_seconds"],
    )
    return {
        "replay": replay,
        "code": proc.returncode,
        "seconds": round(time.monotonic() - t0, 3),
        "tail": (proc.stdout + proc.stderr)[-2000:],
    }

def main() -> None:
    assert_fixtures_locked()
    runs = [run_once(i) for i in range(POLICY["min_replays"])]
    codes = {r["code"] for r in runs}
    report = {
        "lane": "agent-score",
        "flake_budget": 0,
        "runs": runs,
        "deterministic": len(codes) == 1,
        "pass": codes == {0},
    }
    Path("score-report.json").write_text(json.dumps(report, indent=2))
    if not report["pass"] or not report["deterministic"]:
        raise SystemExit("Lane B reject: fail or flake")
    print("Lane B accept")

if __name__ == "__main__":
    main()

Run it like this:

unset PYTEST_ADDOPTS
python score_lane.py
test -f score-report.json
jq '{pass, deterministic, codes: [.runs[].code]}' score-report.json

If deterministic is false, the merge gate must fail. Do not map that outcome to "flaky, retry." Retry is how Lane A thinks. Lane B measures.

Keep the scoring job itself free of retry wrappers:

# .github/workflows/agent-score.yml
# Proposal only. Do not attach a retry action to this job.
name: agent-score
on: pull_request
jobs:
  lane-b:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest
      - run: python score_lane.py
      - run: jq -e '.pass and .deterministic' score-report.json

A workflow that retries lane-b on failure is Lane A wearing a Lane B badge. Fail closed instead.

Oracles the agent cannot clone

Pair the harness with one oracle module. A patch that copies the implementation into an assert should still fail the static check.

# tests/oracles.py
def oracle_round_trip(encode, decode, blob: bytes) -> None:
    assert decode(encode(blob)) == blob

def oracle_idempotent(apply, state) -> None:
    once = apply(state)
    twice = apply(once)
    assert once == twice

def oracle_schema(payload: dict, required: set[str]) -> None:
    missing = required - set(payload)
    assert not missing, missing

New tests must import that module. A one-line check is enough to start:

git diff --name-only origin/main...HEAD -- 'tests/**/test_*.py' \
  | xargs -r grep -L 'tests.oracles' \
  && echo 'oracle import missing' && exit 1

The grep is weak. An unused import fools it. Tighten later with an AST pass that requires a call, not a name. Start with the import rule anyway. Most agent patches that greenwash never reach for an invariant.

Cheap generation, strict scoring

Candidate generation is noisy. You want many cheap patches, not one expensive guess on the merge queue.

MonkeyCode's free model access and free server option fit as a scoring sandbox: emit diffs off the production runners, execute score_lane.py on a host that does not load the human flake allowlist, and keep Lane A for survivors. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not treat that sandbox as a substitute for production CI. This article does not claim a quota, a hardware profile, a duration, or a model name. If the free server is unavailable, run the same harness on any ephemeral VM you already own. The lane split is the method. The host is interchangeable.

If you generate patches with another tool, keep the same report schema. The gate should read score-report.json, not a vendor string.

Limitations

Zero flake budget will reject legitimate tests that talk to live clocks, live DNS, or an unruly browser. That is correct behavior for Lane B. Those tests belong behind fakes, or they belong only in Lane A after a human accepts the risk.

The harness does not prove properties. It replays pytest and checks fixture bytes. Weak oracles still merge. Static grep for tests.oracles can be spoofed.

PYTHONHASHSEED=0 does not freeze the world. Threads, foreign-language maps, network jitter, and clock reads still move. Pin those at the library boundary or fail the score.

The workflow also assumes you can tell agent diffs from human diffs. If authorship is unlabeled, apply Lane B to every diff. That is slower. It is still safer than a shared allowlist.

Fixture SHA-256 locks assume fixtures are small and committed. Live, multi-gigabyte data needs recorded traces with explicit expiry. That design is out of scope here.

Who should not use this

Do not split lanes if you have no merge gate that can parse a JSON report. A markdown comment is not a gate.

Do not use a zero-flake scoring lane as cover for ignoring production flakes. Lane A still needs hygiene. The split only stops agents from mining the exemptions.

Skip this if the agent only drafts comments and never touches tests or fixtures. The leak is authorship of evidence, not authorship of prose.

Skip it if your organization requires retries on every job for capacity reasons and cannot exempt one workflow. A Lane B job with retries is worse than no split. It produces false confidence.

What to keep

Score first. Do not forgive variance in the scoring lane. Promote only with a report the gate can parse. Keep retries in the human suite, where a person still owns the exception. If you need a sandbox that does not spend merge-queue minutes, run the same harness on a spare host and keep the report format stable.