Did the Patch Cause the Flake, or Find It? A Base/Head Attribution Gate for Agent Patches

Short version: when an agent patch surfaces a flaky test, run that test on the base tree and the patch tree with matched seeds before you touch anything. Only a base-tree failure buys a freeze. If the base tree is clean run after run, the flake is the patch's fingerprint — and freezing the test deletes the cheapest evidence you will ever have.

Most flake policies answer one question: when do we revisit this test? That is the cheap half. The expensive half is who pays for it — the test author, or the agent that just touched the code?

Why a freeze is an escape hatch an agent can pull

An autonomous patch sees the test suite as a constraint, not as a contract. If a test blocks the patch and the test can be classified as flaky, the patch wins by default: the test gets quarantined, the suite goes green, the merge proceeds.

The direction of causality is never checked. A patch that changes a shared fixture's setup order, mutates a module-level cache, or shortens a timeout can make a previously stable test flaky — and that flakiness is a real bug signal about the patch, not noise about the test. Relabelling it as test debt is how a genuine regression enters main with a green checkmark next to it.

So the freeze needs a gate, and the gate needs evidence in both directions.

Step 1: Pin the nondeterminism inputs, then split them into two modes

A flake comparison between two trees is meaningless unless the environments match. Pin these on both sides: PYTHONHASHSEED, TZ, LC_ALL, worker count, and the plugin set (disable ordering plugins for the first pass).

Then run two distinct modes, because they answer different questions:

1. Matched mode — same PYTHONHASHSEED on base and head. This isolates patch dependence: if head fails and base passes under identical inputs, the difference is the diff.
2. Swept mode — seeds 0..N-1 on both trees. This isolates test determinism: a test that fails on seed 7 in both trees has a hash-order bug that predates the patch.

Mixing the two modes is the most common mistake. Randomizing seeds on both trees makes an unstable test look patch-induced roughly half the time.

Step 2: A reference attribution script

The script below is a reference implementation to adapt to your repository — I am not claiming measured results from your suite. It builds two detached worktrees, runs the single node ID on each, and parses JUnit XML rather than exit codes, so timeouts and collection errors are distinguishable.

#!/usr/bin/env python3
# flake_attrib.py - attribute a flaky test to base, head, or harness.
# usage: python flake_attrib.py --test tests/test_orders.py::test_totals --runs 30
import argparse, os, subprocess, sys, xml.etree.ElementTree as ET
from pathlib import Path


def worktree(rev: str, path: Path) -> Path:
    if not (path / '.git').exists():
        path.parent.mkdir(parents=True, exist_ok=True)
        subprocess.run(['git', 'worktree', 'add', '--detach', str(path), rev], check=True)
    return path


def run_once(tree: Path, nodeid: str, seed: int) -> str:
    env = {**os.environ, 'PYTHONHASHSEED': str(seed), 'TZ': 'UTC', 'LC_ALL': 'C.UTF-8'}
    junit = tree / f'.junit-{seed}.xml'
    subprocess.run(
        [sys.executable, '-m', 'pytest', nodeid, '-q',
         '-p', 'no:randomly', '-p', 'no:cacheprovider',
         '--junitxml', str(junit)],
        cwd=tree, env=env, capture_output=True, text=True)
    return classify(junit)


def classify(junit: Path) -> str:
    if not junit.exists():
        return 'harness_error'
    seen = 'not_collected'
    for case in ET.parse(junit).getroot().iter('testcase'):
        if case.find('failure') is not None or case.find('error') is not None:
            return 'fail'
        seen = 'skipped' if case.find('skipped') is not None else 'pass'
    return seen


def verdict(base: list, head: list) -> str:
    b, h = base.count('fail'), head.count('fail')
    if b == 0 and h == 0:
        return 'NO_REPRO'
    if b > 0 and h > 0:
        return 'PRE_EXISTING'
    if b == 0 and h > 0:
        return 'PATCH_INDUCED'
    return 'PATCH_MASKS'  # head clean, base flaky: the patch hides a failure


if __name__ == '__main__':
    ap = argparse.ArgumentParser()
    ap.add_argument('--test', required=True)
    ap.add_argument('--base', default='origin/main')
    ap.add_argument('--head', default='HEAD')
    ap.add_argument('--runs', type=int, default=30)
    a = ap.parse_args()
    base = worktree(a.base, Path('.flakes/base'))
    head = worktree(a.head, Path('.flakes/head'))
    b = [run_once(base, a.test, s) for s in range(a.runs)]
    h = [run_once(head, a.test, s) for s in range(a.runs)]
    print(f'base {b.count("fail")}/{a.runs}  head {h.count("fail")}/{a.runs}')
    print('verdict:', verdict(b, h))

A not_collected result on the base tree is itself information: the patch added the test, so no attribution is possible and the question moves to whether the test fails when the code change is reverted.

Step 3: Turn the verdict into a freeze decision

The table is the whole policy. Anything not in the first row does not get a freeze.

base fails	head fails	verdict	freeze allowed?	action
0/N	k/N	PATCH_INDUCED	no	block patch, or fix the determinism bug in it
m/N	m/N (same seeds)	PRE_EXISTING	yes, with expiry	open test-debt ticket, freeze with ledger record
m/N	m/N (different seeds)	HARNESS/ENV	no	fix the harness or the shared fixture first
0/N	0/N	NO_REPRO	no	increase N, then record the pinned inputs
m/N	0/N	PATCH_MASKS	no	investigate: a masked failure is usually a swallowed exception

Inline values (0/N, m/N) are placeholders, not measurements. Fill them from your own run.

Step 4: Make the freeze a record, not a comment

A quarantine marker in a test file carries no evidence and no owner. Use a machine-readable ledger so CI can audit it:

{
  "test": "tests/test_orders.py::test_totals",
  "verdict": "PRE_EXISTING",
  "evidence": {"base_fail": "<m>/30", "head_fail": "<m>/30", "mode": "swept", "seeds": "0..29", "tz": "UTC"},
  "freeze_until": "2026-11-27",
  "owner": "<handle>",
  "recheck": "python flake_attrib.py --test tests/test_orders.py::test_totals --runs 30"
}

The expiry date is the easy part. The verdict and evidence fields are the part that stops a freeze from becoming a silent deletion.

Step 5: Enforce the ledger in CI

# tools/check_freezes.py
import datetime, json, pathlib, sys
today = datetime.date.today()
bad = []
for f in pathlib.Path('tests/freezes').glob('*.json'):
    rec = json.loads(f.read_text())
    if rec.get('verdict') != 'PRE_EXISTING':
        bad.append(f'{f.name}: verdict {rec.get("verdict")} does not justify a freeze')
    if datetime.date.fromisoformat(rec['freeze_until']) < today:
        bad.append(f'{f.name}: freeze expired on {rec["freeze_until"]}')
if bad:
    print('\n'.join(bad))
    sys.exit(1)

Run it as a required check. An expired freeze should fail the build loudly rather than be quietly renewed.

Where the cheap parallel runs come from

Attribution is embarrassingly parallel and individually cheap: two trees, N seeds, one node ID, plus a second loop where an agent proposes a deterministic replacement for the flaky assertion. That second loop is where MonkeyCode's free model access and free server option are relevant for this workflow — the proposal-and-rerun cycle runs many short iterations, and moving the soak loop off a laptop keeps the local machine usable while the seeds sweep. Availability of the free model access and the free server option is operator-supplied; I am not asserting model names, quotas, hardware, or persistence here, so measure your own iteration budget before designing around it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want to try the loop end to end, start with the --runs 10 smoke pass on one known-flaky test before wiring it into CI.

Limitations you should accept before adopting this

• N=30 cannot prove stability. Under the rule of three, zero failures in 30 runs gives a 95% upper bound near 10%. To bound a 5% flake rate you need roughly 60 clean runs; for 1%, several hundred. Attribution reclassifies risk, it does not eliminate it.
• Both worktrees share one machine. A resource-contention flake will look patch-induced. Run one tree at a time if the test is timing-sensitive.
• Base may not be installable. If the lockfile or a native dependency does not resolve for origin/main, the base pass is unavailable and the verdict must fall back to NO_REPRO rather than a guess.
• The ledger needs an owner per entry, or expiry becomes a renewal ritual.
• A freeze is not a fix. Every PRE_EXISTING entry is test debt with a deadline attached.

Who should not use this

Skip it if your suite runs in under a minute — just rerun everything twice on both trees. Skip it if you cannot build the base revision reproducibly, since an unmatched comparison is worse than no comparison. And skip it if no one owns triage: an attribution gate that nobody reads is just a slower way to merge.

Checklist

1. Pin hash seed, TZ, locale, and plugin set on both trees.
2. Run matched mode to test patch dependence, swept mode to test determinism.
3. Require a base-tree failure before any freeze is granted.
4. Record verdict, evidence, owner, expiry, and a recheck command.
5. Fail CI on expired freezes and on freezes with a non-PRE_EXISTING verdict.

The gate is not there to make flaky tests disappear. It is there to make sure that when one disappears, you know what paid for it.