Two Runs, One Verdict

The patch touched one cache helper. CI came back green on the first run. Two days later, main went red on a test nobody had edited.

That test had been flipping for weeks. The patch did not break it. The patch landed on the lucky side of a coin.

A single suite run cannot tell those two cases apart. Agents produce patches fast enough that luck becomes the main source of noise. The verdict needs more samples than the patch itself.

The aggregate exit code hides the flip

pytest returns a zero or a one. That number collapses hundreds of test outcomes into a single bit. A patch that breaks three tests and fixes three others can still exit zero.

Compare per-test outcomes instead. Run the suite on the parent commit and on the patched commit, twice each. Four runs total. The extra runs pay for themselves the first time you skip a false green.

The harness

The example below is illustrative and unbenchmarked. It shells out to pytest inside two git worktrees and reads the JUnit XML. Keep the reporting boring. Keep the classification explicit.

# paired_run.py — illustrative example, not benchmarked
import json, subprocess, sys
import xml.etree.ElementTree as ET
from pathlib import Path

def run(tree, tag):
    xml = Path(f'/tmp/{tag}.xml')
    subprocess.run(
        [sys.executable, '-m', 'pytest', '-q',
         '-p', 'no:cacheprovider', f'--junitxml={xml}', 'tests/'],
        cwd=tree, check=False,            # exit code is not the verdict
    )
    out = {}
    for case in ET.parse(xml).getroot().iter('testcase'):
        key = case.get('classname') + '::' + case.get('name')
        if case.find('skipped') is not None:
            out[key] = 'skip'
        elif case.find('failure') is not None or case.find('error') is not None:
            out[key] = 'fail'
        else:
            out[key] = 'pass'
    return out

def classify(parent, patch):
    verdict = {}
    for key in set(parent[0]) | set(patch[0]):
        before = {r.get(key, 'missing') for r in parent}
        after = {r.get(key, 'missing') for r in patch}
        if len(before) > 1:
            verdict[key] = 'frozen'        # unstable before the patch
        elif before == after:
            verdict[key] = 'unchanged'
        elif before == {'fail'} and after == {'pass'}:
            verdict[key] = 'repaired'
        elif before == {'pass'} and after == {'fail'}:
            verdict[key] = 'broken'
        else:
            verdict[key] = 'inconclusive'  # missing test or collection error
    print(json.dumps(verdict, indent=2, sort_keys=True))
    return verdict

if __name__ == '__main__':
    parent_tree, patch_tree = Path(sys.argv[1]), Path(sys.argv[2])
    verdict = classify(
        [run(parent_tree, 'parent-1'), run(parent_tree, 'parent-2')],
        [run(patch_tree, 'patch-1'), run(patch_tree, 'patch-2')],
    )
    sys.exit(1 if 'broken' in verdict.values() else 0)

The worktrees come from three commands. No stashing. No branch juggling.

git worktree add /tmp/parent HEAD
git worktree add /tmp/candidate HEAD
git -C /tmp/candidate apply /tmp/candidate.diff
python paired_run.py /tmp/parent /tmp/candidate

Note what the exit code does here. It fires only on broken. Every other label is text for a reviewer to read, not a signal to merge.

The decision table

Verdict	Meaning	Action
broken	passed twice before, fails twice after	reject, no debate
repaired	failed twice before, passes twice after	allow, cite the test
frozen	parent disagreed with itself	exclude from vote, log an owner
unchanged	same outcome four times	no signal, no credit
inconclusive	missing or uncollected	block until collection is fixed

unchanged deserves emphasis. A patch that keeps 400 tests green has not earned trust. It has produced zero evidence about the diff. Only broken and repaired move information from the patch into your decision.

Fixtures need a hash, not a name

Property checks inherit whatever the fixture hands them. A fixture that drifted silently turns a property test into decoration. Pin the inputs by content.

# conftest.py — illustrative example, run first in every paired run
import hashlib, json
from pathlib import Path

PINS = json.loads(Path('tests/fixtures/pins.json').read_text())

def digest(path):
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]

def test_fixtures_are_pinned():
    for name, expected in PINS.items():
        assert digest(Path('tests/fixtures') / name) == expected, name

If that test fails, throw the classification away. A changed fixture invalidates the comparison, and the verdict table is not worth reading after it.

Where the four runs live

Four runs of a fast suite fit on a small box. Four runs of a slow suite do not. MonkeyCode offers free model access and a free server option, which is enough to host the harness above without renting a runner on day one. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The product does not change the method. A model proposes a patch. The paired run classifies it. A human reads the table. Keep those three steps separate and the tooling stays swappable, whatever generates the diff.

Limits, and who should skip this

Double running costs double compute. A suite that takes 20 minutes now takes 40, per patch, and that is a real bill on a busy repository.

Two runs also cannot freeze a flake with a period longer than two runs. A test that flips once in fifty will still slip through. Add a third run when the verdict sits close to your release gate.

Skip this approach if your suite has no deterministic fixtures, or if your CI cannot check out two commits at once. Skip it if collection alone costs more than a human review. And skip it if you intend the table to replace a person reading the diff.

Start with one repo and one flaky file. Run both sides twice, print the table, and count the unchanged rows. That count tells you how much of your green build was ever evidence.

If you want to try the loop without provisioning a box first, the free model access and free server on MonkeyCode are a reasonable place to start.