- Book: Ship It — the pragmatic startup tech stack
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Say the checkout test fails one CI run in fifty. You paste the link into your coding agent and type "fix the flaky checkout test." It runs the test and gets green. It wraps an assertion in a retry, runs the test again, gets green again, and reports the task done.
Two green runs are exactly what the broken version produces most of the time. The agent proved nothing, and you wouldn't have either with the same two runs.
The Claude Code best-practices page names the problem: "Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available." A flaky test is the worst case for "looks done".
So before an agent touches it, you make the test fail on demand and write down the command and seed that did it. Then you hand over a check that has to pass hundreds of times in a row, whether the suite runs on Go, pytest, Vitest or Jest.
Which kind of flake you have
A reproducible failure is a command plus a seed that fails within minutes, not a CI link that fails sometimes. To get one, you need to know which kind of flake you have.
The first kind fails in any order: a data race, a timeout that's too tight, a clock, shared state inside one test. You find it with volume, running the same test hundreds of times. The second kind depends on what ran before it: a global left dirty, a database row from another test, a cache. You find it by running many different orders, each from a seed you can replay.
One failure in hundreds of runs: you have to go looking for it.
Go: -count repeats the order, -shuffle picks it
go test can do both. -count n will "Run each test, benchmark, and fuzz seed n times". -shuffle on randomises the order of tests, and "the seed will be reported for reproducibility". -race turns on the race detector. Start with volume on the one test:
go test ./internal/cart -run '^TestCheckoutTotals$' \
-count=500 -race -failfast -timeout=30m
-failfast stops at the first failure, so the output stays short. The -timeout matters too. The default is 10 minutes, and the race detector docs say that for a typical program "memory usage may increase by 5-10x and execution time by 2-20x". If 500 race-checked runs go past 10 minutes, the test binary panics, and that reads like a failure that has nothing to do with your bug.
If the test alone never fails, it probably depends on another test, and now you need orders. Here Go's source has a detail that changes the command. In testing.go, M.Run shuffles the top-level tests once, prints the seed with fmt.Println("-test.shuffle", n), and only then enters the -count loop. So -count=500 -shuffle=on is one random order, run 500 times. It is not 500 orders.
For many orders, run many test binaries, each with a seed you pick:
log=$(mktemp)
for i in $(seq 1 50); do
seed=$RANDOM
if ! go test ./internal/cart -count=10 \
-shuffle="$seed" -race >"$log" 2>&1; then
echo "FAILED: -shuffle=$seed"
tail -n 40 "$log"
break
fi
done
When it fails, go test ./internal/cart -count=10 -shuffle=<seed> -race replays the same order. Go also caches passing results when you name packages, so an agent that runs go test ./internal/cart twice with no change in between can get a cached pass on the second run instead of a real one. The docs call -count=1 "the idiomatic way to disable test caching explicitly". Put a -count on every command you hand over.
pytest: pytest-repeat for volume, pytest-randomly for order
Python needs two plugins. pytest-repeat adds --count, and its README notes you can pair it with -x "to force the test runner to stop at the first failure." pytest-randomly shuffles the test order, prints the seed at the start (its example is Using --randomly-seed=1553614239), and resets Python's global random seed "before each test setup, run, and teardown". So one seed replays both the order and any data drawn from random.
python -m pip install pytest-repeat pytest-randomly
pytest "tests/test_cart.py::test_checkout_totals" \
--count=500 -x -q
Same catch as Go: one pytest run is one seed, so those 500 copies all run in one shuffled order. For many orders, put this in the loop from the Go section and pass the seed yourself:
pytest tests/ -x -q --randomly-seed="$seed"
--randomly-seed=last reruns the previous seed, as long as pytest's cache provider plugin is enabled.
Then turn the plugin off and give each test the same 50 runs in file order: pytest tests/ -x -q -p no:randomly --count=50. Don't add -p no:randomly to the loop itself, because with the plugin off pytest no longer knows --randomly-seed and every pass exits with a usage error. If it fails shuffled but never in file order, the test depends on another test's leftovers. If it fails either way, look at time, threads, the network or the database.
Vitest and Jest: pick the seed yourself
Vitest 5 can do both from the command line. --repeats <number> will "Repeat every test a specific number of times regardless of the result", and --sequence.shuffle with --sequence.seed gives you a seeded order. The sequence config docs say the seed defaults to Date.now(), and they don't say Vitest prints it. So pass your own, and you always know which one failed:
npx vitest run src/cart --repeats=50 --bail=1
npx vitest run --sequence.shuffle \
--sequence.seed="$seed" --bail=1
Jest 30 has --randomize, which shuffles tests within a file and displays the seed, and only works with the default jest-circus runner. --seed=<num> sets the seed, which Jest also uses to shuffle the order test suites run in, and it must sit between -2147483648 and 2147483647. In bash, $RANDOM stays between 0 and 32767, so it fits. For Jest, the loop does the repeating:
npx jest src/cart --randomize --seed="$seed" --bail
The check script and the brief
Put the loop in one script that exits 0 only when every run passes. You run it, the agent runs it, and nobody has to judge whether "it looked fine".
#!/usr/bin/env bash
# scripts/flaky-check.sh
# Exit 0 only if RUNS seeded runs all pass.
set -u
RUNS=${RUNS:-50}
run_once() {
go test ./internal/cart -count=10 -shuffle="$1" -race
}
log=$(mktemp)
for i in $(seq 1 "$RUNS"); do
seed=$RANDOM
if ! run_once "$seed" >"$log" 2>&1; then
echo "run $i/$RUNS FAILED, seed $seed"
tail -n 40 "$log"
exit 1
fi
done
echo "$RUNS/$RUNS runs passed"
Fifty runs of -count=10 is 500 executions of every test in the package, across 50 orders. For pytest, the body of run_once becomes pytest tests/ -x -q --count=10 --randomly-seed="$1". For Vitest or Jest, use the commands above with "$1" as the seed.
How many runs is enough depends on the failure rate, so measure that first with the loop, counting single executions of the test. If it failed 1 execution in 50 and the "fix" changed nothing, the chance that all 500 executions in the script (50 runs × -count=10) still pass is 0.98^500, about 1 in 24,000. At 1 in 1,000, a test that is still broken passes those same 500 executions 61% of the time (0.999^500). Rare flakes need bigger numbers. An order-dependent flake has no steady per-execution rate, because it fails only in some orders, so for those raise RUNS to try more orders.
Now the brief. 8812 below stands for the seed your loop printed:
Fix the flaky test TestCheckoutTotals in ./internal/cart.
Reproduce first. This fails within a few minutes:
go test ./internal/cart -count=10 -shuffle=8812 -race
Run it before you change anything and confirm it fails.
Failure:
<paste the failing assertion and any race report>
Rules:
- Find the cause. No retries, sleeps, t.Skip or
longer timeouts.
- Don't change the test's assertions or
scripts/flaky-check.sh.
Done when both pass:
go test ./internal/cart -count=10 -shuffle=8812 -race
scripts/flaky-check.sh
Reproducing first is the Claude Code docs' own advice for bugs: "write a failing test that reproduces the issue, then fix it." If you want the gate enforced inside Claude Code itself, the same docs describe a Stop hook that "runs your check as a script and blocks the turn from ending until it passes".
Running the loop next to the agent in Hermes
I build Hermes IDE, so weigh this section with that in mind. Everything above works in any terminal. Hermes puts the agent, your repro loop and the verdict in one window.
Press ⌘N (Ctrl+Shift+N on Windows and Linux) and paste the brief as the task. The agent, Claude Code or Codex for example, starts in a real terminal with your own sign-in, on its own hermes/<name> branch and in its own worktree, with the brief as its first prompt.
The ⌘N New task sheet: the task, chips for Claude Code, the orders-api project, a new hermes/ worktree, Accept edits, model and effort, and the "Hermes will run" preview of the claude command.
Then split the pane, right or down, and open a plain shell beside the agent. git worktree list shows the task's folder. cd there and run the seeded repro yourself, once before the agent's first edit and once after its last. One caution: your loop and the agent's own test runs compete for CPU, and that changes timing. A race can show up more often or less often while both are busy, so run the final check when the agent is idle.
Two sessions in split panes: Codex on orders-types and Claude Code on orders-api, each in its own hermes/ worktree, with the sessions sidebar (status, context use, cost, branch) on the left.
Your script stays the definition of done, so run it in that pane before you land. Then open the Review Desk (⌘G, or Ctrl+Shift+G on Windows and Linux) and read the change turn by turn. You're checking that the rules held: no new sleep, no retry, no loosened assertion. For the check script, plain git answers in one line:
git diff main...hermes/<name> -- scripts/
Empty output means the agent left your check alone.
If this was useful
A flaky test is a CI problem before it's an agent problem: once a pipeline goes red at random, people learn to hit rerun, and the real failures hide in the noise. Ship It covers everything between localhost and production for a small team, from hosting, payments and auth to CI/CD and monitoring, which is where a check like this one ends up living.
Hermes IDE is free to download, source-available under BSL 1.1, runs on macOS, Windows and Linux, and isn't affiliated with Anthropic, OpenAI or Nous Research. If it saves you a rerun, a star on GitHub helps other developers find it.
Top comments (0)