3 "Flaky" Tests, 1 Real Segfault: How to Tell a Bug From a Model Failure

A team investigating three flaky CI tests this year didn't find flakiness. They found a race condition that silently dropped a data point, a background task re-inserting deleted rows, and a segfault that reproduced in 7% of runs — but only on Python 3.13. Full writeup here. All three had been living under the label "flaky" for a while, because that label is cheaper than investigating.

That's the exact failure mode this post is about — not "AI is bad at testing," but a sharper question: when a test disagrees with your assumptions, how do you know if the test found a bug, or if the test (or the model that wrote it) is just wrong?

This is the trust gap the data keeps confirming. Stack Overflow's 2026 Developer Survey (30,000+ respondents, published Oct 6) found only 6.6% of developers would trust AI with an important decision outright — but nearly half said they'd trust AI output when they can verify it themselves. Verification, not vibes, is what closes the gap. The 2025 edition of the same survey found 84% of developers now use AI tools, while only 29% trust the output and 46% actively distrust it. Adoption outran trust, and nobody built the missing piece.

Why "the AI said it passed" isn't evidence

There's a structural reason AI-written tests are bad witnesses for their own correctness: a recent bug-validation study (AnyPoC, arXiv 2604.11950) found that naive LLM agents are unreliable validators — they're biased toward reporting success, sometimes producing proof-of-concept reproductions that look plausible but don't actually run, or confidently describing execution traces that never happened. Separately, researchers cataloguing why AI-generated tests mislead point out the core mechanism: an LLM generating a test often writes the expected value by looking at what the code currently does, not what it's supposed to do. If the implementation is wrong, the test enshrines the wrong answer and then passes forever. High coverage, zero fault detection.

That's also why Thoughtworks' Technology Radar now recommends mutation testing over coverage as the more honest signal for AI-assisted suites — coverage tells you code ran, not that anything was actually checked. A suite that can't catch a deliberately flipped >= isn't testing, it's decoration.

A concrete way to tell the difference

You don't need a research paper to apply this — you need a second, independent check that doesn't trust the first one's homework. That's the whole premise of the free lab I built:

• node run-lab.js weak — an AI-written test suite runs against deliberately broken code and reports 3/3 passing.
• node run-lab.js verify-candidate — an independently written verification suite runs against the same broken code and fails, catching all 4 seeded bugs.

Same code. Same "the tests pass" starting point. Opposite verdicts, because one suite actually checks outcomes against the spec and the other checks outcomes against itself. That gap — not whether you used AI — is what determines whether you shipped a real bug or a model failure.

Three things worth doing this week if you work with AI-generated tests:

1. Don't trust a green run from the same model that wrote the test. Reproduce independently, ideally with a suite or a human who never saw the implementation.
2. Check what the test actually asserts, not just that it ran. A test with no meaningful assertion is coverage theater.
3. Run mutation testing on anything AI-generated and load-bearing. If flipping a comparison operator doesn't break the suite, the suite isn't catching real bugs — it's catching nothing.

Try it yourself

The free lab above is open source and takes about 10 minutes to run: ai-test-verification-lab on GitHub.

If the "real bug vs. model failure" problem is one you deal with regularly, I wrote a longer field guide on exactly this — reproducible labs included, not just advice: AI-Assisted Software Testing in Practice ($9.99 Kindle) or the PDF+EPUB+labs bundle on Gumroad ($19). It's a first-edition field guide with runnable proof, not a platform or a certification — the tagline I keep coming back to is: don't trust it, run it.