Gate Flaky Fixes: Run the Test on Base and Patched Worktrees Before Anyone Calls It Fixed

A patch that turns a red test green is not automatically a fix. On a flaky test, it may be luck. The cheapest defence is repetition on two checkouts: the base ref and the patched ref.

This post gives a small worktree-based harness. It repeats one target test N times on both refs and writes a JSON receipt. A model review is only allowed after that receipt exists.

The failure mode this catches

A contributor reports a bug. They add a test, it fails once locally, they patch the code, it passes once. Everyone approves. The test was non-deterministic from the start, and the patch changed nothing relevant.

Three months later the test flakes in CI. The blame lands on the wrong commit.

Repetition separates the two cases. Base fails 4 of 10. Patched fails 0 of 10. That is evidence. Base fails 1 of 10 and patched fails 0 of 10 is not.

Step 1: Fix the target test and the refs

Scope to a single test node, not the whole suite. Ten runs of one test is cheap. Ten runs of a large suite is not.

# pick the narrowest test that still reproduces the report
BASE_REF=origin/main \
PATCH_REF=HEAD \
RUNS=10 \
TEST_CMD='pytest -q tests/test_parser.py::test_nested_alias' \
  ./flakegate.sh

The env vars are the whole interface. Nothing in the repo needs patching.

Step 2: Run the harness

The script creates two detached worktrees from the same clone. Both share one object store, so the setup is fast.

#!/usr/bin/env bash
# flakegate.sh - repeat one test on base and patched worktrees.
set -euo pipefail

REPO=${REPO:-$PWD}
BASE_REF=${BASE_REF:-origin/main}
PATCH_REF=${PATCH_REF:-HEAD}
RUNS=${RUNS:-10}
TEST_CMD=${TEST_CMD:?'set TEST_CMD'}

ROOT=$(mktemp -d)
BASE_WT="$ROOT/base"
PATCH_WT="$ROOT/patched"

git -C "$REPO" fetch --quiet origin || true
git -C "$REPO" worktree add --detach "$BASE_WT"  "$BASE_REF"  >/dev/null
git -C "$REPO" worktree add --detach "$PATCH_WT" "$PATCH_REF" >/dev/null

run_n() {
  local dir=$1 fails=0 i
  for i in $(seq 1 "$RUNS"); do
    if ! (cd "$dir" && eval "$TEST_CMD") >/dev/null 2>&1; then
      fails=$((fails + 1))
    fi
  done
  echo "$fails"
}

BASE_FAILS=$(run_n "$BASE_WT")
PATCH_FAILS=$(run_n "$PATCH_WT")

cat <<JSON
{"base_ref":"$BASE_REF","patch_ref":"$PATCH_REF",
 "base_sha":"$(git -C "$BASE_WT" rev-parse HEAD)",
 "patch_sha":"$(git -C "$PATCH_WT" rev-parse HEAD)",
 "runs":$RUNS,"test":"$TEST_CMD",
 "base_failures":$BASE_FAILS,"patched_failures":$PATCH_FAILS}
JSON

git -C "$REPO" worktree remove --force "$BASE_WT"
git -C "$REPO" worktree remove --force "$PATCH_WT"

Both worktrees point at the same interpreter and site-packages. Build once, not twice. Pin the dependency state before the run, or the two refs get different environments.

Step 3: Read the receipt as a gate

The numbers are a decision, not a report. Use the table below.

base failures / N	patched failures / N	Verdict
0	0	No reproduction. Do not claim a fix.
0	>0	The patch broke something. Stop.
>0	0	Gate passes. Evidence is monotonic.
>0	>0, lower	Partial. Publish both rates, never "fixed".
>0	>= base	No evidence of improvement. Reframe or revert.

The middle row is the only one that unlocks review. Everything else goes back to the contributor with the raw counts.

Step 4: Let a free model review, but only the receipt and the diff

After the gate passes, model review is worth it. Before it passes, model review just adds confident noise.

The review input is small and bounded: the JSON receipt, the diff, and the source file the test covers. No repo, no issue thread, no chat history.

You receive: a flake gate receipt, a unified diff, one source file.
Answer exactly these three questions, one line each.
1. Does the diff modify the function exercised by the named test?
2. Does the diff introduce a new code path the test does not cover?
3. Name one line of the diff you cannot justify from the source file.
If you cannot answer from the provided text, write UNKNOWN.

The third question is the useful one. It forces a concrete citation instead of a summary. UNKNOWN is a valid and frequent answer.

This step is cheap enough to run on free model access, which is how I exercised it. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project advertises free model access and a free server option; treat current allowances and limits as changing, and read the project page for the terms in force.

Step 5: Move the loop onto a throwaway runner

A local machine accumulates state. Sockets, caches, and stale fixtures all shift flake rates. Running the same harness on a clean machine makes the receipt reproducible by someone else.

A free server option is enough for this. The harness needs no GPU. It needs a clean checkout and ten minutes of wall clock.

The receipt should record the runner identity and the commit SHAs. Without that, two contributors get two different answers and neither can be checked.

Limitations worth stating up front

• N=10 is a smoke test. A 2% flake survives ten runs most of the time. Raise N when the counts are close.
• Environment-dependent flakes need the failing environment. A green run elsewhere proves nothing.
• Timing, network, and thread-order bugs are the worst fit. Repetition on a fixed machine may never trigger them.
• Shared state between tests pollutes the measurement. Isolate the test before trusting any count.
• Long suites make N×cost painful. Always narrow to one node, then widen later if needed.

Who should not use this

The gate is wrong for purely deterministic test failures. Run the test once and move on. It is also wrong for reviewer-time emergencies: a one-line revert does not need a ten-run harness.

Skip it when the bug report contains no trigger at all. Repetition without a trigger measures nothing.

What you keep

One script, one JSON boundary, one decision table. The receipt is small enough to paste into a pull request and stable enough to compare across weeks.

That is the real gain. The conversation shifts from "does this look right" to "what did the counts say". Most disagreements end there.

If you want to try the review step without paying for inference or a runner, MonkeyCode publishes free model access and a free server option for experimentation; start with the harness above and see whether the numbers change your review habits.