Is It Flaky or Is It Your Patch? A Base-vs-Head Attribution Matrix

Conclusion first: a test that fails one run in five after an agent patch has not been shown to be flaky. It has been shown to fail. "Flaky" is a guess until you run the same repetitions on the base commit, in the same pinned environment, and see instability there too. The matrix below costs one extra suite run per repetition and turns that guess into a label you can defend in review.

What a head-only stress run cannot prove

Run the patched suite five times. One test fails once. That result is consistent with two very different worlds: the test was always unstable, or the patch made it unstable.

A head-only experiment cannot separate them, and the cheapest available label is "flaky". The cheapest available action is a skip. The patch goes green, the skip becomes debt, and the information that would have blocked the patch is gone.

The asymmetry is worth naming. Agent-proposed patches are exactly the kind of change that introduces nondeterminism — a new cache, a lazily built module-level dict, an added retry, a shared temp path. A human author usually remembers touching something global. A diff rarely says so out loud.

The matrix: 2 commits × K repetitions, one environment

The experiment is deliberately boring. Both commits, same machine, same hour, same pinned fixtures.

1. Pin the environment. PYTHONHASHSEED=0, TZ=UTC, fixed LC_ALL, no network, fixed tmpdir root. If these vary between runs, you are measuring the environment, not the patch.
2. Freeze test order. Disable any order randomizer (-p no:randomly). Order is a separate variable with its own experiment; sweeping it here doubles the run count and blurs attribution.
3. Pick K for detection, not estimation. This is the part people get wrong. Five runs can catch a flake that fires often. Five runs cannot bound a rare one.
4. Record every outcome to a per-run file. A conftest.py hook is enough; no plugin required.
5. Interleave base and head in a single loop so both commits see the same host, the same cache state, the same background load.
6. Join the tallies and bucket each test by the pair (base class, head class), not by the head class alone.

Here is the honest arithmetic for step 3, computed from the Wilson score interval (95%, two-sided) — not measured, just the formula applied:

Failures observed	K = 5, upper bound	K = 20, upper bound
1	0.62	0.24
2	0.77	0.30

One failure in five runs leaves the true failure rate anywhere below roughly 62%. That is not an estimate of flakiness; it is a detection event. If you need a rate — for triage priority, for an SLA, for deciding whether to spend a week on it — collect 20 or more repetitions. If you only need to know whether this exemption is legitimate, K = 5 is usually enough, because the interesting cases fail loudly.

The recorder

One file, no dependencies. The RUN_RECORD env var names the output for this repetition.

# conftest.py — one outcome per test, per run
import json, os

RECORD = os.environ["RUN_RECORD"]   # e.g. reports/head-03.json
_rows = []

def pytest_runtest_logreport(report):
    if report.skipped:
        _rows.append({"nodeid": report.nodeid, "outcome": "skipped"})
    elif report.when == "call" and report.outcome in ("passed", "failed"):
        _rows.append({"nodeid": report.nodeid, "outcome": report.outcome})

def pytest_sessionfinish(session, exitstatus):
    os.makedirs(os.path.dirname(RECORD), exist_ok=True)
    with open(RECORD, "w") as fh:
        json.dump(_rows, fh)

The driver is a shell loop, not a framework. A red suite is data here, so || true is intentional.

set -euo pipefail
export PYTHONHASHSEED=0 TZ=UTC LC_ALL=C.UTF-8
K=5
for i in $(seq -w 1 "$K"); do
  git checkout --detach "$BASE_SHA"
  RUN_RECORD="reports/base-$i.json" pytest -q -p no:randomly || true
  git checkout --detach "$HEAD_SHA"
  RUN_RECORD="reports/head-$i.json" pytest -q -p no:randomly || true
done

The attribution

Join the two globs, classify each side, and map the pair to a bucket. Anything that lands in clean → flaky is the outcome the whole exercise exists to find.

#!/usr/bin/env python3
# attribute.py  (reduced sketch, not a library)
import glob, json, sys

BUCKETS = {
    ("flaky",  "flaky"):  "pre-existing flake — exemption eligible",
    ("clean",  "flaky"):  "patch-induced — block, do not freeze",
    ("clean",  "broken"): "patch regression — block",
    ("broken", "broken"): "already red before the patch — out of scope",
    ("broken", "clean"):  "patch fixes it — keep the test",
}

def load(pattern):
    tally = {}
    for path in glob.glob(pattern):
        for row in json.load(open(path)):
            t = tally.setdefault(row["nodeid"], {"passed": 0, "failed": 0, "skipped": 0})
            t[row["outcome"]] += 1
    return tally

def classify(t):
    total = t["passed"] + t["failed"]
    if total == 0:
        return "untested"
    if t["failed"] == 0:
        return "clean"
    return "broken" if t["passed"] == 0 else "flaky"

base, head = load("reports/base-*.json"), load("reports/head-*.json")
bad = 0
for nodeid in sorted(base.keys() | head.keys()):
    b = classify(base.get(nodeid, {"passed": 0, "failed": 0, "skipped": 0}))
    h = classify(head.get(nodeid, {"passed": 0, "failed": 0, "skipped": 0}))
    label = BUCKETS.get((b, h), "unclassified")
    if label.startswith("patch-induced"):
        bad += 1
    print(f"{label:42s} {b:>7s} -> {h:<7s} {nodeid}")
sys.exit(1 if bad else 0)

Reading the matrix

Base	Head	What it means	Action
flaky	flaky	Nondeterminism predates the patch	Exemption possible; needs an owner and an expiry
clean	flaky	The patch introduced the instability	Block. This is a bug with a friendly name
clean	broken	Ordinary regression	Block
broken	broken	Base was already red	Attribution undefined — fix base first
any	untested	Test disappeared or was skipped at collection	Treat as a removal, not a pass

Two rows carry most of the value. clean → flaky is the one that head-only stress runs systematically mislabel. broken → broken is the one that quietly invalidates the experiment: if base is not green, you have no reference class and no attribution, and the run should be discarded rather than interpreted.

Exemptions are debt, and the debt may not grow

Once attribution is trustworthy, enforcement is small. Keep a committed ledger where each quarantined test carries an owner, a tracking issue, an expiry date, and a named replacement assertion that runs in the default suite. The gate reads the ledger from the base commit and the head commit and fails the patch if the number of entries increased.

The deposit mechanics — why a quarantine needs a replacement assertion rather than a promise — are the subject of an earlier post of mine, A Skipped Test Is a Withdrawal, so I will not restate them here. The point this matrix adds is narrower: no entry should ever be created from head-only evidence, because a fraction of those entries are paying for a bug the patch introduced.

Where the repetitions actually run

The cost center is step 5. Ten full suite executions per patch, on two checkouts, in one pinned shell. That is fine once; it is annoying as a pre-merge habit.

Two operator-supplied notes about the tooling I run this with, stated plainly rather than measured: MonkeyCode offers free model access and a free server option. Both map onto the loop above without stretching. Free model access drafts candidate replacement assertions from the diff — proposals I review before they enter the ledger, never auto-merged. The free server option is where the 2×K loop executes, so a developer machine is not pinned for the duration and metered CI minutes are not spent on an experiment whose whole purpose is to fail.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I have not verified quotas, hardware, model names, or persistence for either offering, and nothing above depends on them. If the free server option disappears tomorrow, the loop still runs on a laptop, only slower. That is the test of whether a tool is load-bearing in a workflow: remove it and the method survives with worse ergonomics.

Limitations and who should skip this

• K = 5 detects; it does not estimate. Do not publish a flake rate from five runs. The table above shows why.
• Pinning hides environment flake. A test that only fails with a real network or a non-UTC clock will look clean here. That is a different experiment — cross-machine portability, not patch attribution.
• A green-on-both test can still be worthless. This matrix says nothing about whether the assertion has content. Tautologies pass every repetition you throw at them.
• Red base invalidates everything. If the base commit does not pass its own suite, stop and fix that first.
• Cost is real. Two checkouts, K repetitions, interleaved. On a suite that takes minutes, this is the difference between a habit and a special occasion.

Who should not use this: solo maintainers with a ten-second suite and no quarantine mechanism at all, where re-running by hand is genuinely cheaper than a driver script. Teams whose suites reach the network or a live database on every run — the base column will not be reproducible, and the fixture problem has to be solved before attribution means anything.

The number worth trading for

If you adopt only one habit from this, make it the second column. Every time a patch arrives with a failing test and the familiar phrase "that one's flaky", run the base commit five times before you believe it.

The metric I would want back from anyone who tries this is not the flake rate. It is the ratio of head-only "flaky" labels that turned out to be patch-induced. If that ratio is near zero, the label was fine and you have a cheap confirmation ritual. If it is not, the matrix just paid for itself on the first patch it blocked.