5 Claude Code Patterns I Use to Hunt Down Flaky Tests

TL;DR

Flaky tests are the worst kind of bug — they waste your time, erode trust in CI, and usually hide a real race condition or shared-state bug underneath the "just rerun it" band-aid. I've been using Claude Code as a flaky-test hunting partner for the past few months, and I landed on 5 patterns that consistently turn "sometimes it fails" into "here's exactly why, and here's the fix." This post walks through each one with real prompts and terminal output.

The Problem

Every team has that one test suite where a handful of tests fail maybe 1 in 20 runs, nobody knows why, and the unwritten rule is "just hit re-run." I inherited a suite like that a while back — around 15 tests that would intermittently fail in CI but pass locally almost every time. Classic symptoms:

• A test timed out only under CI's slower, more contended CPU
• Two tests silently shared a database row and stepped on each other when run in parallel
• A "random" UUID generator wasn't seeded, so one in a few hundred runs produced a collision
• An async call resolved in a different order depending on event-loop timing

The hard part with flaky tests isn't fixing them — it's reproducing them reliably enough to even start debugging. If a bug only shows up 5% of the time, you can stare at the code for an hour and see nothing wrong. I wanted a repeatable process, not another "add a sleep(2000) and pray" fix.

The specific incident that pushed me to formalize this: a checkout test started failing about once a day in CI, always on a different assertion, always unreproducible locally. Three different engineers had each spent an afternoon on it, added a retry, and moved on. By the time it landed on my desk the test had retry: 2 quietly baked in and was still failing occasionally — the retry was masking the bug, not fixing it. That's usually the tell that nobody has actually found the root cause yet.

How I Solved It

1. Loop the suspect test until it breaks, capture every failure

The first move is always to stop treating flakiness as a one-off. I ask Claude Code to write a tight loop that reruns the specific test N times and dumps full output (not just pass/fail) on every failure:

for i in $(seq 1 100); do
  echo "=== run $i ==="
  npx jest src/orders/checkout.test.ts -t "applies discount" --silent 2>&1 | tee -a /tmp/flaky-run.log
done
grep -c "FAIL" /tmp/flaky-run.log

Running this against a suspected test tells you two things fast: the actual failure rate (1/100 vs 1/5 completely changes your debugging strategy), and — critically — the stack trace from an actual failure instead of a hypothetical one. I never start hypothesizing before I have at least 3 real failure logs in hand.

2. Diff the passing run against the failing run

Once I have a log with both passes and failures, I hand both to Claude Code and ask it to diff the execution paths — not the code, the behavior:

"Here's a log with 100 runs of this test, 6 failed. Compare the timing and order of the setup/teardown hooks in a passing run vs a failing run. What's different?"

This is where an agent earns its keep over manual debugging — it can hold two long, noisy logs in its head at once and spot the divergence point (e.g., "in failing runs, afterEach for the previous test hadn't finished releasing the DB lock before this test's beforeEach started"). That's a shared-state bug, and once you see it, it's obvious in hindsight — but it's genuinely hard to eyeball in a 400-line log.

For the checkout test from earlier, the actual divergence turned out to be order-dependent: the test suite ran two checkout tests back to back that both wrote to the same orders table row via a hardcoded test fixture ID. On a fast machine, the first test's cleanup finished before the second test's setup ran. Under CI's heavier load, that ordering flipped about 5% of the time, and the second test occasionally read a half-committed row from the first. Nobody had spotted it because the two tests lived in different files, hundreds of lines apart — the kind of thing that's invisible unless you're specifically comparing hook timing across runs.

3. Git bisect flakiness, not just breakage

git bisect is usually pitched for "this used to pass, now it fails." It works just as well for "this used to be flaky at 1%, now it's flaky at 20%" — you're bisecting a failure rate, not a boolean. I wrote a small bisect script that runs the test 30 times per commit and marks the commit bad if the failure rate crosses a threshold:

git bisect start HEAD HEAD~40
git bisect run bash -c '
  fails=0
  for i in $(seq 1 30); do
    npx jest checkout.test.ts -t "applies discount" --silent || fails=$((fails+1))
  done
  [ $fails -lt 3 ]  # exit 0 = good, 1 = bad
'

This found a commit that reordered two middleware functions — nothing to do with the test file itself — that quietly introduced a race. I would not have found that by reading the test.

4. Ask the agent to classify the failure mode, not just "fix it"

Early on I'd ask Claude Code to "make this test stop failing," which — unsurprisingly — sometimes produced a fix that just made the symptom rarer instead of removing the cause (a longer timeout, an extra retry). Now I explicitly ask it to classify first:

"Before proposing a fix: is this a timing/race issue, a shared-state issue, an unseeded-randomness issue, or an environment difference between local and CI? Cite the specific lines that support your classification."

Forcing the classification step out loud changes the fix that comes back. "Shared state" gets a fix that isolates test data; "timing" gets a fix that waits on a real signal instead of a fixed delay. I've caught the agent trying to reach for sleep() more than once — asking for the classification first heads that off.

5. Convert the fix into a regression guard, not just a patch

The last pattern: once the actual bug is found and fixed, I ask for a targeted stress test that would have caught the original bug, not just confirmation that the original flaky test now passes once. Something like a tighter version of pattern #1, checked into CI as a nightly job rather than every PR (running a test 100x on every commit is too slow for the main pipeline):

# .github/workflows/flaky-guard.yml (nightly, not on every PR)
- run: |
    for i in $(seq 1 200); do
      npx jest checkout.test.ts -t "applies discount" || exit 1
    done

If the underlying race ever comes back (say, someone touches the middleware order again), this catches it in a nightly run instead of silently reintroducing 1-in-200 flakiness that nobody notices for months.

flowchart LR
    A[Suspect test] --> B[Loop N times, capture full logs]
    B --> C[Diff passing vs failing run]
    C --> D[Bisect failure rate across commits]
    D --> E[Classify: timing / shared-state / randomness / env]
    E --> F[Targeted fix]
    F --> G[Nightly stress-test regression guard]

Lessons Learned

1. You can't debug what you can't reproduce. The loop-and-capture step (pattern #1) is unglamorous but it's 80% of the value — most of my "flaky test debugging" time used to be spent staring at code with zero real failure data in front of me.
2. Bisect the rate, not just the boolean. A test that went from 1% flaky to 20% flaky is a real regression with a real culprit commit — treat it like one.
3. Forcing a classification step before a fix changes the fix. Asking "why" before "how" stopped the agent (and me) from reaching for timeout bumps and sleeps as a first resort.
4. Shared state is the most common root cause, by a wide margin. Of the ~15 flaky tests I've fixed this way, at least 10 came down to tests silently sharing a DB row, a file, or a global — not "real" timing bugs.
5. A fix without a regression guard is a fix that comes back. The nightly stress-test step feels like overkill until the exact same race resurfaces three months later from an unrelated change.

What's Next

I'm now looking at running pattern #1's loop automatically whenever CI detects a test that passed-then-failed on a rerun, instead of waiting for a human to notice the pattern and go hunting manually — the trigger already exists in most CI systems' retry logs, it's just not wired up to anything today. I'd also like to feed the nightly stress-test results back into a running "flakiness score" per test file, so the tests most worth this treatment bubble to the top instead of getting picked by whoever's most annoyed that week. If that works reliably I'll write it up.

One caveat worth naming: this whole workflow assumes you can run the suspect test in a loop cheaply. For tests gated behind expensive external services or paid API calls, 100–200 iterations isn't free, and I've had to dial the loop count down to 20–30 and accept a noisier signal. It's still better than zero real failure data, but it's the one place this approach gets more expensive than it looks on paper.

Wrap-up

If you've got a flaky test suite gathering dust behind a "just rerun it" habit, try pattern #1 first — 100 loop iterations with full log capture will tell you more in 10 minutes than an hour of reading code. Follow me here on Dev.to for more of these Claude Code build logs, and if you want to try this yourself, grab Claude Code and point it at your worst offender.