Your Flaky Tests Aren't Flaky
I've hit rerun on a red CI build without looking at the logs more times than I want to admit. "Probably flaky." It passes. I merge. Move on.
I did this for years. Then I built an AI agent that reads CI logs across 50+ runs and tells you what's actually failing, and why. I ran it against 21 open-source repos.
56 tests that everyone called "flaky" were real bugs. Same error. Every run. Deterministic failures disguised as flaky tests because nobody read the logs.
What scares me about this isn't the reruns. It's what happens to the signal. Once a red build stops meaning "something is broken" and starts meaning "who knows," engineers merge past it. Real regressions ship. Not because anyone approved them. Because nobody believed the test that caught them. The cost of flaky tests isn't the rerun. It's the trust. Once that's gone, every failure becomes a judgment call, and most people's judgment is "rerun and move on."
The Setup
I built ai-flake-sleuth. LangGraph agent. Fetches GitHub Actions run history, parses CI logs, classifies each failure as a real bug, a flaky test, or infrastructure noise.
12 test frameworks across 5 languages. I had to build 13 new parsers during the study because the original design only handled pytest and unittest. That lasted about 20 minutes before I hit Jest, Vitest, Go test, RSpec, Mocha, PHPUnit, JUnit, Cypress. Each has a completely different log format.
Two models: a local 9B (Qwen3.5-9B, free, runs on my Mac) and a cloud model (GPT-5 Nano, $0.0007 per call). Total cloud spend for the study: $0.22.
One design choice I want to name. The agent classifies and recommends, but it never auto-quarantines. Quarantining a flaky test hides the problem so you can ship faster. It also hides real bugs. The human decides: quarantine and ship, fix now, or bump the retry threshold. But before you can make that call, you need to know whether the test is actually flaky. That's the diagnostic layer.
What Happened
On 4 repos (pytest, langchain, django-rest-framework, jestjs) both models agreed. Everything flaky. No disagreement.
On 3 repos, they disagreed hard.
fastapi: local said 18 flaky, 0 real bugs. Cloud said 8 flaky, 13 real bugs.
vercel/next.js: local said 71 flaky, 0 real bugs. Cloud said 185 flaky, 35 real bugs.
rubocop: local said 88 flaky, 0 real bugs. Cloud said 80 flaky, 8 real bugs.
56 tests the local model called flaky were deterministic failures. Same error every run. Real bugs.
Not a clean "cloud wins" story. The local model found 3 real bugs the cloud model missed (2 in vue, 1 in react) because the cloud run on vue degraded when the prompt blew past the context window. Neither model is perfect. The disagreement is the data.
Was the Cloud Model Right?
I manually checked one. test_tutorial003.py::test_main from fastapi. Raw log:
FAILED test_main - Failed: DID NOT WARN. No warnings of type (DeprecationWarning,) were emitted.
Same error. All 14 runs. Identical. The test expects a deprecation warning the code isn't emitting. It will fail every time until someone fixes the code. Cloud model was right.
Local model said INFRA because the error signature hash was empty. The error message was in the raw log. It just didn't make it through the data pipeline to the classification prompt. A data gap, not a classification error.
I also ran a rules-only baseline. No LLM, just regex and stats. On the 4 repos where both models agreed everything was flaky, rules agreed too. The LLM earned its $0.22 on the 56 disagreements: cases where the error signature was empty or generic, the failure rate was 14/14, and rules said "flaky" while the cloud model said "real bug." Rules see the missing error and hedge. The model sees 14/14 deterministic and commits.
The Bug That Was Producing Fake Verdicts
First time I ran the agent with a Qwen3 reasoning model, every classification came back FLAKY. Every single one. I thought the model was just conservative.
It wasn't.
Qwen3 with thinking enabled spends its entire token budget reasoning before emitting the JSON. I had max_tokens hardcoded at 512. The model used all 512 tokens thinking ("Okay, let's analyze this. The test failed 14 times. The error signature is empty. Wait, let me reconsider...") and never produced the JSON. Response truncated. JSON parse failed. Fallback returned FLAKY.
The classified_by prefix was llm:, the same prefix used for real verdicts. No way to tell a real model verdict from a silent fallback. The tool was lying to me, and the lies looked like real answers.
Fix: configurable max_tokens, finish_reason detection, 4 distinct prefixes for auditability, and a --no-thinking flag that disables Qwen3's reasoning mode entirely. 79 completion tokens instead of 4,096. 16 seconds instead of 214. 13× faster, same or better accuracy.
What Most CI Failures Actually Are
8 of 21 repos had zero test failures in their CI logs. Not because their tests are perfect. Because their CI "failures" were in other jobs. Lint. Build. CodeQL. Dependabot. The test jobs passed.
GitHub Actions runs multiple jobs per workflow. A run can "fail" because a lint job failed while the test job passed. The tool handles this gracefully (clean report, zero tests classified, no crash) but it means you need to download enough runs and the right workflows to find actual test failures.
The Cost
Entire field study: $0.22 in cloud LLM calls. 303 calls across 8 repos. Local model free. Cloud model $0.0007 per call.
For context: developers spend $2,250/month repairing flaky tests (Leinen et al., ICST 2024). Rerun builds across 1,960 GitHub Actions projects accumulated 339 years of waiting time (Ge & Zhang, 2025). The study cost less than a coffee.
What I'd Do Differently
Filter by job type at download time. I downloaded 2,000+ irrelevant log files from lint/build/CodeQL jobs that had nothing to do with tests.
Pass per-test stats, not all-test stats. Two repos produced prompts exceeding any model's context window because the cross-run context included all 14,000 test stats instead of just the one being classified. Django's prompt was 9.8 million characters.
Run models sequentially. The local MLX endpoint is single-threaded. Running rubocop concurrently with other repos caused 74 out of 88 calls to fail with connection pool exhaustion.
Test parsers against real logs before running full analysis. GitHub Actions prefixes every line with a timestamp. Jest uses PASS not PASSED. Vercel has a suite name prefix. I discovered all of these the hard way.
The Honest Part
I only manually validated 1 of the 56 REAL_BUGs. The pattern (14/14 identical failures) strongly suggests the rest are also real bugs, but I haven't checked each one. A proper accuracy study would sample 20+, read the logs, compare against human judgment.
The local model is a 9B parameter model, 4-bit quantized. Not surprising it misses things a larger cloud model catches. But it's free, local, private. Right approach: run local first, then run cloud on the disagreements.
8 repos produced zero results because their CI failures weren't test failures. The tool didn't crash. It reported clean CI. But you need to pick repos where test jobs actually fail.
The Takeaway
56 REAL_BUGs is the headline. The deeper finding: the model you choose to classify CI failures changes what you find. A free local model said everything was flaky. A $0.22 cloud model found 56 real bugs. Same data, same prompts, same parser. Different model, different truth.
If you're rerunning flaky tests without diagnosing them, some of those "flaky" tests are real bugs. You're shipping regressions because the CI signal is noise. The fix isn't more retries. It's reading the logs.
pip install ai-flake-sleuth
Repo: github.com/deghosal-2026/ai-flake-sleuth
When a test goes flaky, what's your first move — quarantine and ship, fix it now, or bump the retry threshold? And have you ever gone back to check whether the thing you quarantined was actually a real bug?
Top comments (0)