DEV Community: Nikolay Chernev

Flaky AI tests are usually bad tests: how to stabilize a non-deterministic suite

Nikolay Chernev — Mon, 02 Nov 2026 07:00:05 +0000

Short answer: most AI test flakiness comes from two sources — exact-match assertions on output that was never guaranteed to match, and missing pins on model version and temperature. Fix those two things and 80% of your flapping tests stabilize. The remaining 20% are genuine non-determinism, and the answer there is a different kind of assertion, not a different kind of model.

The "AI is untestable" complaint comes up constantly in QA teams that are six months into an LLM project. The suite runs, half the tests pass, half fail, nobody trusts it, CI goes red, someone marks the failures "known flaky" and disables them. Now you have a test suite that runs green and tells you nothing.

That's not an AI problem. That's a test-design problem that AI exposed.

Why do AI tests flap when nothing has changed?

Two reasons, and they're distinct enough that fixing the wrong one wastes a week.

Reason 1: the assertion is wrong. You called the API once, got a string back, and wrote assert response == "Here are the three items: apples, bananas, oranges." The model gives the same information every time but rephrases it — "The three items are apples, bananas, and oranges." or "I found three items: apples, bananas, oranges." Your assertion fails. The model isn't broken. The test is.

Reason 2: the test isn't pinned. You're running at the default temperature (often 0.7–1.0), and you haven't nailed down the model version. Even at temperature=0, some providers add noise. At temperature=1.0, the same prompt produces genuinely different outputs run-to-run. If your test doesn't control these, you're testing a moving target.

How do you tell which problem you have?

Run the failing test 10 times in a row against the same model, same prompt, with temperature=0 and an explicit model version pinned. Count the failures.

10/10 fail: the assertion is fundamentally wrong — it's asserting something that was never stable.
0/10 fail: your test was running at high temperature, or the model version was drifting. Pinning fixed it.
3/10 fail at temperature=0: you've found genuine residual non-determinism (most providers don't guarantee exact reproducibility even at temp=0). You need a statistical assertion or a threshold check.

The diagnosis tells you which fix to apply.

How do you fix wrong assertions?

Stop asserting the string. Assert the properties that must be true of any valid output.

# Before: brittle, flaps on every rephrasing
def test_item_list():
    response = ai_app.query("List the three items.")
    assert response == "Here are the three items: apples, bananas, oranges."

# After: stable, tests what actually matters
def test_item_list_properties():
    response = ai_app.query("List the three items.")

    # all three items must be present, order doesn't matter
    for item in ["apple", "banana", "orange"]:
        assert item in response.lower()

    # must not invent a fourth item
    unexpected = ["grape", "mango", "pear", "melon"]
    assert not any(u in response.lower() for u in unexpected)

    # must stay concise — this isn't an essay prompt
    assert len(response) < 300

These assertions are deterministic. They never flap, they run at CI speed, and when one fails you know exactly what class of failure occurred. The first version was testing the model's wording. These versions test whether the model was correct.

The same logic applies to structured output. If your app returns JSON, don't assert the whole serialized string — assert the schema and the values.

import jsonschema

ITEM_SCHEMA = {
    "type": "object",
    "required": ["items", "count"],
    "properties": {
        "items": {"type": "array", "items": {"type": "string"}, "minItems": 1},
        "count": {"type": "integer", "minimum": 1},
    },
}

def test_structured_output_shape():
    result = ai_app.query_json("List the items.")
    jsonschema.validate(result, ITEM_SCHEMA)
    assert result["count"] == len(result["items"])

Schema validation never flaps. A miscount between count and len(items) is a real logic bug — and it'll catch it reliably every time.

How do you pin model version and temperature?

This is the simplest fix and the most commonly skipped one.

import anthropic

client = anthropic.Anthropic()

def query_for_tests(prompt: str) -> str:
    """Use this wrapper in all tests — never call the client directly."""
    resp = client.messages.create(
        model="claude-haiku-4-5-20251001",  # explicit version, not "claude-haiku-latest"
        max_tokens=512,
        temperature=0,                       # deterministic mode for unit tests
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.content[0].text

Two things to note. First, pin the full model ID — claude-haiku-4-5-20251001, not a latest alias. When the provider bumps the alias, your test behavior changes silently. Second, run your property checks at temperature=0, but keep a separate eval suite that runs at your production temperature. A bug that only surfaces at temperature=1.0 is still a bug, and you want to find it before users do.

What about tests that still flap after pinning?

At temperature=0, most models produce identical output for identical inputs most of the time. "Most of the time" isn't all the time — some providers are explicit that exact reproducibility isn't guaranteed. If you're still seeing flakiness after pinning, you have two options.

Option A — statistical assertion. Run the query N times and assert a pass rate, not a single result.

def test_sentiment_is_positive_most_of_the_time():
    results = [ai_app.classify_sentiment("Great product!") for _ in range(20)]
    positive_rate = results.count("positive") / len(results)
    assert positive_rate >= 0.90, f"Only {positive_rate:.0%} positive — expected ≥90%"

Twenty runs is usually enough to distinguish a real regression (drops to 60%) from noise (stays above 90%). Don't run this in fast CI — it's for a nightly or pre-release gate.

Option B — property check on the distribution. If your app is supposed to return a confident label 90% of the time, write that as the spec directly. A test that runs once and sometimes passes isn't a spec — it's a guess.

Which flakiness isn't worth fighting?

Genuine creative variation in open-ended generation. If your feature is "give me five marketing tagline ideas," the outputs should vary. Testing that they don't vary is testing the wrong thing. What you can assert: every tagline is under 15 words, none contain the competitor's name, all use imperative tense. Stable properties on variable output. Writing those for LLM and RAG output takes some practice but it's the same skill.

Also not worth fighting: latency variation. A 200ms swing on a generation call doesn't mean anything. Set a hard ceiling (5s, 10s, whatever production SLA is) and assert that — not an exact time.

What this doesn't fix

Bad property coverage. If you write properties that are too loose — "response must be non-empty" — you'll have a green suite that misses every quality regression. Properties have to actually capture the contract of the feature.

The deeper problem: figuring out what properties to assert for your specific system is the design work. There's no universal answer. The properties for a customer-support chatbot are different from those for a code-generation tool. You have to understand what "correct" looks like for your feature before you can write a test that catches when it isn't.

That design work is what AI Software Testing in Practice is built around — working out the right assertion strategy per feature type, with examples across real AI product categories. The free test lab has working property-check templates you can adapt today.

FAQ

Are flaky AI tests a sign that the model is broken?
Rarely. Most AI test flakiness is caused by exact-match assertions on non-deterministic output, or by tests that don't pin temperature and model version. Fix the assertion and the test stops flapping without touching the model.

Should I quarantine flaky AI tests instead of fixing them?
Quarantine for 48 hours while you diagnose — don't leave them skipped permanently. A flaky test is either a bad assertion (fix it now) or real non-determinism you need to understand (build a statistical version). Skipping it forever means a category of bugs stops being caught.

How do I stabilize an AI test that sometimes passes and sometimes fails on the same input?
First check if you're pinning model version and temperature=0. If yes and it still flaps, the test is asserting something that varies legitimately — rewrite it as a property check or a scored threshold instead of an exact match.

From experienced tester to AI testing: the roadmap, in order

Nikolay Chernev — Wed, 28 Oct 2026 19:48:14 +0000

Short answer: learn it in this order — the non-determinism mindset, prompts as test inputs, evals (the new assertion), LLM/RAG failure modes, then verifying AI-written tests with mutation testing. Skip the math-heavy ML theory; you're testing behavior, not training models.

I keep seeing the same post on r/softwaretesting: "10 years of manual and automation under my belt, and the AI stuff makes me feel like a junior again. Where do I even start?" The honest answer is that you're closer than you think. Maybe 70% of what makes you a good tester transfers directly. The other 30% is a handful of new ideas, and they stack in a specific order. Learn them out of order and it feels like chaos. Learn them in order and each one clicks.

Here's the order I'd follow.

1. Get the one mindset shift first: non-determinism

Everything new starts here. In normal testing, one input has one correct output, so you assert equality. In AI testing, one input has a set of valid outputs — ask the same question twice and the wording changes, both answers fine. That single fact breaks every exact-match assertion you've ever written and is why your first naive AI tests will flake.

Don't move on until this feels natural, because the rest of the roadmap is just tools for dealing with it. If a test flaps on identical input, your instinct should flip from "the system is flaky" to "I'm asserting the wrong thing."

2. Learn prompting — as a tester, not a hobbyist

Everyone treats prompting as a party trick. For you it's controlling the test input. A vague prompt is an uncontrolled variable; you can't tell a real regression from your own sloppy setup. Spend a week writing prompts that pin down format, scope, and constraints, and reading back exactly what the model did with them. You're not learning to be clever. You're learning to make the input reproducible enough to test against.

This is the step most testers skip, and it's why their AI tests feel unreliable later.

3. Evals — the assertion for the AI era

This is the real new skill, and it's where I'd spend the most time. An eval replaces "did the output equal X?" with "is this output acceptable?" Three flavors, in increasing effort:

Property checks — rules true for every valid answer. No PII leaked, no invented dollar figures, output parses as JSON. Deterministic, cheap, catch the scary failures.
Reference/similarity — score the answer against a known-good one with embeddings or keyword coverage, assert a threshold, not equality.
LLM-as-judge — have a model grade against a rubric. Powerful for open-ended text, but it's non-deterministic too, so pin its model and pair it with hard property checks.

def test_support_reply():
    out = bot("my card was declined")
    assert out.intent == "billing"              # property: deterministic
    assert "$" not in out.text                   # property: no invented amounts
    assert judge(out.text, rubric="acknowledges, asks one question, no blame") >= 0.8

If you learn one thing from this whole list, learn to write evals. It's the skill job posts are circling around without naming.

4. The LLM/RAG failure modes you'll actually hit

Now get specific about how these systems break, because it's not how normal software breaks — and how to tell a real bug from the model just drifting:

Hallucination — confident, fluent, completely made up. Your test is a grounding check: did the answer stick to the provided context?
Retrieval failure (RAG) — the model's fine, but it fetched the wrong documents. Test retrieval separately from generation, or you'll debug the wrong half for a day.
Prompt injection — user input that hijacks the instructions. This is the new security test, and it's yours now.

You don't need to know how a transformer works to test any of this. You need to know what wrong looks like, which is exactly what testers are good at.

5. Use AI to write tests — then prove the tests work

You'll let Copilot or Claude generate test suites. Fine. But generated tests have a specific trap: they tend to assert that the code does what it currently does — bugs included. A green suite that was written against buggy code is just the bug, notarized.

The fix is mutation testing: deliberately break the code and check your tests scream. If you flip a > to >= and every test still passes, those tests prove nothing. Run this on AI-generated suites especially — it's how you actually earn confidence in AI-written tests. This is the whole point of the book, and the free lab is a 10-minute demo of it — a suite that's green against four real bugs.

What to skip

You do not need: the math behind training, building models, PyTorch, or a data-science course. You're testing behavior at the boundary, the same place you always have. Picking up ML theory first is the most common way people stall for three months and learn nothing testable. Skip it until something forces you to care.

Realistically this is a few focused weeks, not a bootcamp — and you keep every bit of your existing automation, CI, and bug-sense. Don't let anyone sell you "AI testing" as a reset. It's an extension.

FAQ

Do I need to learn Python?
If you already automate, you're set — most eval and LLM tooling is Python-first, but it's the same testing you do now with different assertions.

Is this going to replace traditional QA?
No. It sits on top of it. CI, coverage, API and E2E testing all still apply; AI adds non-determinism and a new set of failure modes to cover.

Where does the book fit in step 3 and 5?
Those two steps — evals and verifying AI-written tests — are the spine of AI Testing in Practice. It walks the full version with runnable examples, so you're not piecing it together from scattered blog posts.

Codex or Claude wrote my tests — how do I actually trust them?

Nikolay Chernev — Wed, 28 Oct 2026 19:37:08 +0000

Short answer: don't trust them because they're green. AI writes tests that assert the code's current behavior — including the bugs. Earn the confidence by mutation testing: change the code on purpose and check the tests fail. The ones that stay green are lying to you.

Last month I let Claude generate a test file for a pricing function. Forty tests, all passing, 94% coverage, took ninety seconds. It felt like a gift. Then I changed a <= to < in the discount tier — a real off-by-one I'd have shipped — and all forty tests still passed. The suite wasn't testing the pricing logic. It was describing it.

That's the trap, and it's specific to generated tests.

Why AI tests pass against broken code

When you write a test, you usually start from "here's what this should do." When a model writes a test, it reads the implementation and writes assertions that match what the code does. Those are not the same job. If the function has a bug, the model faithfully writes a test that locks the bug in place and marks it green.

Three failure patterns show up again and again:

The mirror test. It asserts result == function(input) essentially by re-deriving the output from the same code path. Tautology. It can never fail.
The assertion that isn't. assert response is not None, assert len(result) > 0. Technically an assertion. Catches almost nothing. Models love these because they always pass.
Coverage theater. The line runs, so coverage counts it, but nothing checks the value it produced. 94% coverage, 0% confidence.

Coverage tools can't see any of this. A line is "covered" the moment it executes — whether or not a single assertion cares about the result. This is why I stopped treating coverage as a quality signal years ago, and why it's actively dangerous with AI-generated suites: it gives you a big green number that measures the wrong thing.

The test that proves your tests: mutation testing

Here's the move. Instead of asking "do my tests pass?", ask "if the code were wrong, would my tests notice?" You answer it by deliberately introducing bugs — mutants — and running the suite against each one.

Flip > to >=
Change + to -
Replace a return value with None
Delete a line
Swap and for or

Every mutant is a tiny, realistic bug. A good test suite kills the mutant — at least one test fails. A mutant that survives is a bug your tests would have let through. The score is simple: killed ÷ total. That number, not coverage, is how much your suite is actually worth.

You don't do this by hand. The tooling is mature:

Python — mutmut or cosmic-ray
JavaScript/TypeScript — Stryker
Java — PITest
C# — Stryker.NET

A first run on that AI-generated pricing suite looked like this:

$ mutmut run
# ... a few minutes later
$ mutmut results
Killed: 22  Survived: 18  Timeout: 0

# the survivors are the interesting part:
- discount.py:14  `<=` → `<`     (survived)   ← my off-by-one
- discount.py:27  `return total` → `return None`  (survived)
- tax.py:9        `rate * amount` → `rate + amount` (survived)

Eighteen survivors out of forty mutants. The suite that looked airtight was catching a little over half of injected bugs. Now I had a to-do list: eighteen specific holes, each pointing at the exact line and the exact kind of mistake my tests ignored. I wrote real assertions for those, re-ran, and got survivors down to two (both equivalent mutants — changes that don't actually alter behavior, which you learn to recognize).

The workflow I'd actually recommend

Keep letting the AI write tests. It's genuinely fast and the boilerplate is fine. Just add one gate:

Generate the tests. Let the model do the grunt work.
Run mutation testing on the module, not just coverage.
Treat every surviving mutant as a missing assertion. Fix the test, not the score.
Set a mutation-score threshold in CI for the code that matters (payments, auth, anything that loses money or leaks data). Start at 70%, raise it as you go.

The point isn't to mutation-test your entire repo on every commit — that's slow and you'll turn it off within a week. Run it on the risky modules, and run it especially on anything an AI wrote, because that's where the mirror tests hide.

What this doesn't fix

Mutation testing proves your tests can detect code changes. It does not tell you the original spec was right — if the requirement itself is wrong, you'll have a high mutation score on the wrong behavior. It's also not free: runtimes are minutes, not seconds, so it belongs on critical paths and pre-merge, not on every keystroke. And it says nothing about non-deterministic AI outputs — testing an LLM's answers is a different problem (that's evals, and telling a real bug from model drift).

What it does do is convert "the tests are green" into "the tests would actually catch a bug," which is the only version of green worth having.

This verification layer — proving AI-written tests before you rely on them — is the spine of AI Testing in Practice. If you want to feel it before reading, the free lab is a 10-minute run: a green suite sitting on top of four real bugs, and the mutation pass that exposes them.

FAQ

Isn't high coverage enough if the tests are well written?
Coverage tells you a line ran, not that anything checked the result. You can hit 100% coverage with zero meaningful assertions — and AI-generated suites do this constantly. Mutation score measures whether the assertions actually bite.

How is mutation testing different from just writing more tests?
More tests can all be mirror tests. Mutation testing finds the specific behaviors nothing is checking, so you add the three assertions that matter instead of thirty that don't.

My mutation run is too slow to put in CI. Now what?
Don't run it repo-wide. Scope it to the modules that carry real risk and run it pre-merge on changed files. A 90-second mutation run on the payment module is worth more than a 40-minute run across code nobody will break.

Your AI Wrote 3 Tests. 3 Passed. The Code Was Still Broken.

Nikolay Chernev — Mon, 19 Oct 2026 14:13:54 +0000

Your AI Wrote 3 Tests. 3 Passed. The Code Was Still Broken.

I ran this exact scenario last week: an AI assistant writes a small test suite against a function with a seeded bug. All 3 tests pass. Green checkmarks, done, ship it. Then I ran an independent verification suite against the same code — written to check the spec, not to agree with whatever the implementation already does. It failed, and it caught all 4 bugs I'd planted, including the one the AI's own tests walked right past.

That gap — "the tests pass" vs. "the code is correct" — is the single biggest trust problem in AI-assisted testing right now, and it's not a vibe. Stack Overflow's 2025 Developer Survey put real numbers on it: 84% of developers now use or plan to use AI tools, up from 76% in 2024, but trust in the accuracy of what those tools produce kept falling — Stack Overflow's own analysis puts it at 29%, down from 40% the year before, and 46% of developers say they actively distrust AI output (InfoWorld, LeadDev). Adoption is up. Trust is down. That's not a contradiction, it's a warning sign.

Why "all tests passed" doesn't mean what you think

Here's the mechanism, and it's structural, not a one-off glitch. When the same model writes the implementation and the tests, both inherit the same blind spots. If the code has a bug — say, a counter that increments regardless of whether a request actually succeeded — a model asked to "write tests for this function" will often write an assertion that checks the behavior the code actually has, not the behavior the spec requires. The test and the bug agree with each other. Nothing goes red, because nothing was ever checking against an independent standard.

This is distinct from flakiness, though the two get confused. Flaky tests fail intermittently on correct code. What I'm describing is the opposite and arguably worse: tests that pass reliably on incorrect code. Google's own engineering data shows roughly 16% of tests exhibit some flakiness, and flaky failures account for around 84% of red builds — a huge, well-studied problem. But a test suite that passes consistently on broken code doesn't even give you a red build to investigate. There's nothing alerting you. That's the harder failure mode to catch, because every signal you'd normally rely on says "fine."

A concrete way to tell the difference

The fix isn't "don't use AI to write tests." It's: don't let the same pass of AI-authored assertions be the only check. Verify the verifier.

Separate the test-writer from the test-grader. Don't ask the same prompt/context that explains the implementation to also validate it. Write (or generate) an independent suite from the spec/requirements alone, without showing it the implementation.
Run the AI's suite against a known-broken version of the code. If it still passes 3/3, the suite isn't testing behavior — it's testing "does the code run."
Check what fraction of seeded bugs survive. This is exactly what mutation testing measures (coverage tells you code executed; mutation score tells you whether anything was actually checked) — but you don't need a mutation-testing framework to get the intuition. Plant 3-4 real bugs by hand and see which ones your AI-written suite catches.
Treat a pass from AI-authored tests as a claim, not a verdict. "Don't trust it — run it" against an independent check before you believe it.

I built a small, free, runnable lab that makes this concrete instead of theoretical: a deliberately broken implementation, an AI-generated test suite that passes 3/3 against it, and an independent verification suite that fails and catches all 4 seeded bugs. You can run both in under a minute and watch the gap yourself:

Free lab (GitHub): https://github.com/chernevnikolay86-wq/ai-test-verification-lab

node run-lab.js weak              # AI's own tests: PASS 3/3 on broken code
node run-lab.js verify-candidate  # independent suite: FAILS, catches all 4 bugs

If this distinction — "the model failed" vs. "there's a real bug, and the model's own tests just didn't catch it" — is a question you're currently answering by gut feel, that's the exact problem the lab is built to make visible. It's also chapter one of a longer field guide I wrote on verification-first AI testing for QA/SDET engineers, with reproducible examples instead of slide-deck claims:

Kindle ($9.99): https://www.amazon.com/dp/B0HL4SC5PX
PDF+EPUB+labs bundle ($19): https://nikolaychernev.gumroad.com/l/ai-testing-in-practice

No guarantees, no "best practices" fluff — just a repeatable way to check whether you can trust what the AI just handed you.

100% Coverage, 4% Mutation Score: The Gap That Explains Why Your AI Tests Pass Everything

Nikolay Chernev — Fri, 16 Oct 2026 17:57:42 +0000

100% Coverage, 4% Mutation Score: The Gap That Explains Why Your AI Tests Pass Everything

One team ran mutation testing on an AI-generated suite that reported 100% line coverage. Every statement executed. The mutation score — the share of deliberately injected bugs the tests actually caught — came back at 4%. Shiplight's write-up walks through exactly how that happens, and it's worth sitting with the number: 96% of injected defects slipped straight through a "fully covered" suite.

That gap is the whole problem with trusting an AI's own tests, and it's sharper than the usual "AI hallucinates" complaint. Coverage tells you code ran. It says nothing about whether anything was actually checked.

Why coverage lies

Line coverage counts a statement as tested the moment it executes inside a test function — even if the assertion that follows is expect(result).not.toBeNull() on a function that was never going to return null. Thoughtworks' Technology Radar flags this directly: high coverage percentages routinely mask logically hollow tests, especially generated ones.

There's a structural reason AI-written tests drift this way. When the same model writes the implementation and the test, both inherit the same blind spots. One practitioner account (HackerNoon) describes a counter that increments whether or not a request actually succeeds — and a test suite that treats "counter went up" as proof the request worked, because that's the assumption baked into the code it was copying. The bug and the test agree with each other. Nobody disagrees, so nothing goes red.

Mutation testing is the counter-check: inject a small fault (flip a comparison, drop a line, change a constant) and rerun the suite. If nothing fails, that mutant "survived" — and a surviving mutant means some class of real bug could ship with every test green. In one informal report, 20 AI-written tests against a Laravel controller reported full endpoint coverage but killed only 68 of 95 injected mutants — 27 bugs that full coverage didn't catch.

A reader already found the sharper version of this

Last week I wrote about telling a real bug from a model failure. A reader, Foundry, left a comment describing something worse than a weak assertion: a check that is structurally incapable of failing.

Their example: an automated mobile check hashed a screenshot before and after a swipe to confirm the feed advanced. On a playing video, the hash changes every single frame — so the check passes whether or not the swipe did anything at all. A second check read process age with ps -o etimes=, which doesn't exist on macOS; the command failed silently, the age never crossed the threshold, and the rule never fired, ever. Neither check had ever gone red, so neither got flagged for review. Their question is the one worth stealing: could this check pass if the thing I care about did not happen?

That's mutation testing in one sentence, applied by hand instead of with a tool.

What to actually do about it

You don't need a mutation-testing framework to start. Before trusting any AI-written test — yours or a teammate's — ask it Foundry's question directly: if I introduce the bug this test is supposed to catch, does it fail? If you can't answer that without running it, that's the review item, not "does it pass."

The free lab I maintain is a runnable version of exactly this comparison. node run-lab.js weak generates AI tests against deliberately broken code — they pass 3/3. node run-lab.js verify-candidate runs an independent verification suite against the same code — it fails, and it catches all 4 seeded bugs. The difference between those two runs is the entire argument of this post, reproducible in about two minutes:

Free lab: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

If this is a problem you deal with weekly and you want the fuller method — how to tell a real failure from a model failure, how to build an independent verification layer, worked examples — that's the subject of the book:

Kindle ($9.99): https://www.amazon.com/dp/B0HL4SC5PX
PDF+EPUB+labs bundle ($19): https://nikolaychernev.gumroad.com/l/ai-testing-in-practice

Don't trust it — run it.

3 'Flaky' Tests, 1 Real Segfault: How to Tell a Bug From a Model Failure

Nikolay Chernev — Mon, 12 Oct 2026 18:54:12 +0000

3 "Flaky" Tests, 1 Real Segfault: How to Tell a Bug From a Model Failure

A team investigating three flaky CI tests this year didn't find flakiness. They found a race condition that silently dropped a data point, a background task re-inserting deleted rows, and a segfault that reproduced in 7% of runs — but only on Python 3.13. Full writeup here. All three had been living under the label "flaky" for a while, because that label is cheaper than investigating.

That's the exact failure mode this post is about — not "AI is bad at testing," but a sharper question: when a test disagrees with your assumptions, how do you know if the test found a bug, or if the test (or the model that wrote it) is just wrong?

This is the trust gap the data keeps confirming. Stack Overflow's 2026 Developer Survey (30,000+ respondents, published Oct 6) found only 6.6% of developers would trust AI with an important decision outright — but nearly half said they'd trust AI output when they can verify it themselves. Verification, not vibes, is what closes the gap. The 2025 edition of the same survey found 84% of developers now use AI tools, while only 29% trust the output and 46% actively distrust it. Adoption outran trust, and nobody built the missing piece.

Why "the AI said it passed" isn't evidence

There's a structural reason AI-written tests are bad witnesses for their own correctness: a recent bug-validation study (AnyPoC, arXiv 2604.11950) found that naive LLM agents are unreliable validators — they're biased toward reporting success, sometimes producing proof-of-concept reproductions that look plausible but don't actually run, or confidently describing execution traces that never happened. Separately, researchers cataloguing why AI-generated tests mislead point out the core mechanism: an LLM generating a test often writes the expected value by looking at what the code currently does, not what it's supposed to do. If the implementation is wrong, the test enshrines the wrong answer and then passes forever. High coverage, zero fault detection.

That's also why Thoughtworks' Technology Radar now recommends mutation testing over coverage as the more honest signal for AI-assisted suites — coverage tells you code ran, not that anything was actually checked. A suite that can't catch a deliberately flipped >= isn't testing, it's decoration.

A concrete way to tell the difference

You don't need a research paper to apply this — you need a second, independent check that doesn't trust the first one's homework. That's the whole premise of the free lab I built:

node run-lab.js weak — an AI-written test suite runs against deliberately broken code and reports 3/3 passing.
node run-lab.js verify-candidate — an independently written verification suite runs against the same broken code and fails, catching all 4 seeded bugs.

Same code. Same "the tests pass" starting point. Opposite verdicts, because one suite actually checks outcomes against the spec and the other checks outcomes against itself. That gap — not whether you used AI — is what determines whether you shipped a real bug or a model failure.

Three things worth doing this week if you work with AI-generated tests:

Don't trust a green run from the same model that wrote the test. Reproduce independently, ideally with a suite or a human who never saw the implementation.
Check what the test actually asserts, not just that it ran. A test with no meaningful assertion is coverage theater.
Run mutation testing on anything AI-generated and load-bearing. If flipping a comparison operator doesn't break the suite, the suite isn't catching real bugs — it's catching nothing.

Try it yourself

The free lab above is open source and takes about 10 minutes to run: ai-test-verification-lab on GitHub.

If the "real bug vs. model failure" problem is one you deal with regularly, I wrote a longer field guide on exactly this — reproducible labs included, not just advice: AI-Assisted Software Testing in Practice ($9.99 Kindle) or the PDF+EPUB+labs bundle on Gumroad ($19). It's a first-edition field guide with runnable proof, not a platform or a certification — the tagline I keep coming back to is: don't trust it, run it.

Your AI Wrote Tests With 100% Coverage. They Catch 4% of Real Bugs.

Nikolay Chernev — Sun, 11 Oct 2026 12:01:37 +0000

That's not a hypothetical split — it's a documented test suite: 100% line coverage, 4% mutation score. Every line executed. 96% of seeded bugs slipped through anyway. Source.

Coverage tells you which lines ran. It says nothing about whether the test would notice if the logic were wrong. That gap is exactly where AI-generated tests tend to live: fluent, green, and checking almost nothing.

A fresh data point: what happens when a second AI checks the first one's work

Two weeks ago, First Mate Technologies published results from building QueueMate, a restaurant queue app, with a deliberate builder/checker split: one model writes the implementation, a separate model designs test cases and reviews the code before it merges. The checker produced 554 test cases. The resulting sweep found 38 defects — including a severity-one bug in session handling that the builder's own tests had missed. (developer-tech.com)

That's the whole thesis of verification-first testing in one case study: the model that wrote the code is a bad judge of whether the code is right. You need a second, independent check — and ideally not just "another AI," but something that actually probes behavior.

Also this past week: Momentic launched "Mo," an agent that drives an app the way a human would and reports reproduction steps and video evidence rather than asserting against its own output (SiliconANGLE, Sep 28) — a sign the industry is converging on the same answer: separate the generator from the grader.

The number that exposes the gap: mutation score

Here's the practical tool, and it's not new or exotic — mutation testing has been around for decades, it's just newly relevant:

A mutation testing tool (Stryker, PIT, mutmut, depending on your stack) automatically changes a small piece of your code — flips a > to >=, deletes a line, swaps a boolean.
It reruns your test suite against that mutated code.
If a test fails, the mutant is "killed" — your suite caught it. If every test still passes, the mutant "survives" — your suite didn't actually verify that logic, no matter what the coverage report says.

Gartner's February 2026 insights abstract on this put it plainly: relying on AI to expand test coverage leads to missed defects, because AI-generated tests often lack depth and precision — and recommended mutation testing specifically as the corrective layer (Gartner, Sushant Singhal & Erin Khoo).

A mutation score isn't a vibe. It's a number that tells you, mutant by mutant, whether your tests would actually notice if the logic broke.

Seeing it without installing a mutation framework

You don't need a full mutation testing setup to feel this gap in under ten minutes. I built a free, runnable lab that does the same thing by hand: four bugs seeded into working-looking code.

node run-lab.js weak → the AI-generated test suite passes 3/3. On broken code.
node run-lab.js verify-candidate → an independent verification suite fails, and catches all 4 seeded bugs.

Same mechanism as a mutation score, minus the tooling setup: deliberately broken logic, and a direct answer to "would my tests have caught this?" Repo: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

What to actually do about it

Don't grade an AI's tests by whether they're green. Grade them by whether they'd fail on a plausible wrong version of the code.
If you can run a mutation tool for your stack, run it on AI-generated test files specifically — the survival rate is the honest coverage number.
If you can't, write (or have a second, independent pass write) 2-3 deliberately broken variants of the function under test and check whether the existing suite catches any of them. That's mutation testing with a manual crank.
Treat "the AI's own tests pass" as the start of verification, not the end of it — the question that matters is "real bug, or model failure?", and only an independent check answers it.

This is chapter-one territory in AI-Assisted Software Testing in Practice, a field guide for testers and SDETs who already know the fundamentals and now need a working method for verifying AI-written tests rather than trusting them. No certification, no platform claims — a runnable kit and the reasoning behind it.

Free lab (same mechanism as above, run it yourself): https://github.com/chernevnikolay86-wq/ai-test-verification-lab
Kindle, $9.99: https://www.amazon.com/dp/B0HL4SC5PX
PDF+EPUB+labs bundle, $19: https://nikolaychernev.gumroad.com/l/ai-testing-in-practice

Make AI-generated tests you can actually trust. Don't trust it — run it.

Three Reports Dropped This Week. All Say the Same Thing: Your AI's Tests Are Lying to You.

Nikolay Chernev — Thu, 01 Oct 2026 18:32:18 +0000

89% of organizations had an AI-related production incident. Not "might." Had. That's not a hot take — it's from a report published three days ago.

This week, three separate 2026 industry surveys landed within 48 hours of each other, and they all point at the same hole: AI got good at writing code and good at writing tests for that code — including tests that pass when the code is wrong.

The numbers, stacked up

Qodo's 2026 State of AI Code Quality Report (500 developers, 300 engineering leaders, surveyed Aug 2026): reviewing and validating AI-generated code is now the #1 delivery bottleneck. 89% of orgs had an AI-related production incident. Only 3.7% of engineering leaders think their current process is actually sufficient going forward.
Checksum's State of AI Code 2026 (105 engineering leaders): 78.1% trust AI code more than they did a year ago. Meanwhile, 61% shipped a production incident from AI-generated code in the last 90 days, and 74.3% had to roll back AI code because it failed in a way their own unit tests didn't catch.
DeviQA's 2026 QA Report (4,000 QA professionals): 47% have hit AI-assisted code that passed its primary scenario and broke something else anyway.

Rising confidence. Rising incidents. That gap is the story.

Why this isn't a review problem — it's a trust-the-grader problem

Here's the part that should worry testers specifically: it's not that AI-generated tests are sparse. It's that they can be confidently, fluently wrong — and they're the same agent's own homework, graded by itself.

OWASP's Secure Coding with AI Cheat Sheet now calls this out directly: coding agents can make CI green by deleting failing tests, weakening assertions (an assertEquals quietly becomes assertNotNull), mocking the very unit under test, or just asserting whatever the buggy code already does. Their line, almost verbatim: a passing test suite generated by the same agent that produced the code provides no independent assurance.

That's not a hypothetical. I built a free, runnable lab specifically to make it visible in under 10 minutes.

See it fail, on purpose

The AI Test Verification Lab ships deliberately broken code plus the AI-generated test suite that was written for it. Run it yourself:

node run-lab.js weak
# AI's own tests: PASS 3/3 — on broken code.

node run-lab.js verify-candidate
# Independent verification suite: FAIL 8/12 — catches all 4 seeded bugs.

Same code. Same bugs. One test suite says "ship it." The other says "stop." The difference isn't model quality — it's whether anything independent checked the AI's own claim.

This matches what the Stack Overflow 2025 Developer Survey already told us at scale: 84% of developers use AI tools, but only 3% highly trust the output, 46% actively distrust it, and 45% say they lose real time debugging what it hands back. JetBrains' 2025 ecosystem survey puts regular AI usage at 85%. Adoption is not the problem. Verification is.

The verification-first method (short version)

Never let the AI grade its own test. If the same prompt/session that wrote the code also wrote the test, treat it as a draft, not evidence.
Seed known bugs and check the suite catches them. If it doesn't catch a bug you put there on purpose, it won't catch the one you didn't.
Watch for weakened assertions and deleted tests in diffs, specifically — not just "tests still pass."
Keep an independent suite that doesn't share assumptions with the code generator.

That's the whole method. It's not exotic — it's just discipline that AI-assisted speed makes easy to skip.

Go deeper

If this resonated, the free lab above is the fastest way to see it firsthand — it's the same demo in this post, runnable in 10 minutes. For the full method, worked examples, and a runnable kit built for QA/SDET workflows, the field guide is AI-Assisted Software Testing in Practice ($19, PDF+EPUB+labs).

Make AI-generated tests you can actually trust. Don't trust it — run it.

The 15-second test that tells you an AI-generated test is worthless

Nikolay Chernev — Wed, 30 Sep 2026 09:14:18 +0000

Your AI just generated a test. Before you trust it, run this 15-second check — no tools, just one question.

"What broken version of this code would still make this test pass?"

If you can picture even one realistic wrong implementation that stays green, the test isn't checking behavior. It's decoration — delete it or fix it.

Why this one question works

A test earns trust only if it fails when the behavior is wrong. AI-generated tests reliably do the two easier things: they run, and they agree with the code the model just wrote — bugs included. The 15-second question forces the third, the one that matters: discrimination.

Take a generated test like this:

test('applies discount', () => {
  expect(typeof applyDiscount(order)).toBe('number')
})

What broken applyDiscount would still pass? Almost any of them — wrong rate, wrong rounding, no discount at all. It returns a number every time. Covered, green, worthless.

Now the discriminating version:

test('10% over $2000, exactly at the boundary', () => {
  expect(applyDiscount({ total: 2000 })).toBe(2000)     // boundary: not discounted
  expect(applyDiscount({ total: 2001 })).toBe(1800.9)   // 10% off
})

A > where the spec says >= now turns this red. That's a test.

Make the habit concrete (free, 10 min)

I built a tiny lab (no install, Node 20+) that runs the whole failure mode:

node run-lab.js all

It hands the AI's own suite a pricing function with four seeded defects. The AI's tests pass 3/3 against the broken code; an independent verification suite fails and catches all four:

WEAK TESTS (the AI's suite) .... PASS 3/3
STRONG VERIFICATION ........... FAIL 8/12   <- catches all 4 defects

Open the flawed file and try to write the smallest test that catches each defect before you read the verification suite. That exercise is the whole method.

The loop around the question

The 15-second test is the pocket version of a bigger habit:

Design the discriminating test first, then generate.
Keep the generator separate from the approver — a different model or session reviews it.
Prove the suite catches faults with seeded defects and mutation.
A human owns the release call.

Try it, then go deeper

Free, self-contained lab: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

I turned the full method into a book + runnable kit:

Kindle on Amazon ($9.99): https://www.amazon.com/dp/B0HL4SC5PX
Full bundle (PDF + EPUB + labs) on Gumroad ($19): https://nikolaychernev.gumroad.com/l/ai-testing-in-practice

Honest caveat: the lab systems are simulators with declared defects — they prove the method catches faults, not that any product is certified. Don't trust it — run it.

Coverage theatre: your AI hit 90% coverage and still shipped the bug

Nikolay Chernev — Wed, 30 Sep 2026 09:07:33 +0000

Your AI assistant just generated tests until the coverage bar turned green. 92%. Ship it?

Here's the trap: coverage measures which lines ran, not whether a test would notice when they're wrong. AI made that gap free — you can generate 90% coverage in a minute, all of it happy-path, none of it discriminating. That's coverage theatre: it looks like safety, it measures activity, it proves almost nothing.

Why coverage lies

A line is "covered" the moment a test executes it. But executing a line and asserting the right thing about it are different claims. An AI-generated test that calls applyDiscount(order) and asserts typeof result === 'number' covers the function — and would stay green if the discount math were completely wrong.

Coverage answers "did this code run in a test?" The question that earns trust is "would a test fail if this code were wrong?" Those are not the same, and AI output routinely nails the first while skipping the second.

The antidote is old and boring: break the code on purpose

If a suite can't tell correct behavior from a real defect, its coverage number is decoration. The cheapest check is mutation: change > to >=, flip a boolean, swap two lines — then see if anything goes red. If the suite stays green on broken code, it isn't protecting you.

I put the smallest runnable version of this in a free lab (no install, Node 20+):

node run-lab.js all

It hands the AI's own suite a pricing function with four seeded defects, then runs an independent verification suite against the same broken code:

WEAK TESTS (the AI's suite) .... PASS 3/3
STRONG VERIFICATION ........... FAIL 8/12   <- catches all 4 defects
MUTATION GATE ................. PASS 4/4
LAB GATE: PASS

The AI's tests pass against code that is wrong. Verification is what fails — and that failure is the signal coverage never gave you.

What to do instead of chasing the coverage bar

Before you trust a generated test, ask: what realistic broken version of this code would still make it pass? If you can picture one, the test isn't discriminating.
Add a mutation or seeded-defect check to CI. One flipped operator per critical function exposes a suite full of decoration.
Keep the thing that writes tests separate from the thing that approves them. The generator proposes; an independent reviewer (a different model or session) checks; a human owns the release call.
Report mutation score, not just coverage %. Coverage tells you what ran; mutation tells you what your tests can actually catch.

Coverage isn't useless — low coverage is a real red flag. But high coverage is not evidence of anything on its own, and AI made it trivially easy to manufacture. Treat the green bar as a starting question, not an answer.

Try it, then go deeper

Free, self-contained lab: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

I turned the full method into a book + runnable kit — the operating loop, governance, and labs across unit, web/API, instrument-cluster and CAN/UDS domains.

Kindle on Amazon ($9.99): https://www.amazon.com/dp/B0HL4SC5PX
Full bundle (PDF + EPUB + labs) on Gumroad ($19): https://nikolaychernev.gumroad.com/l/ai-testing-in-practice

Honest caveat: the lab systems are simulators with declared defects — they prove the method catches faults, not that any product is certified. That's the point: don't trust things that merely look right.

Your AI wrote 40 tests. How many would catch a real bug?

Nikolay Chernev — Sat, 26 Sep 2026 09:06:52 +0000

Your AI assistant just generated a full test suite. It's green. Ship it?

Here's the uncomfortable part: a passing run confuses three different claims, and only the last one earns trust.

The test runs. It executes without error. Table stakes.
The test agrees with the code. The assertions mirror the implementation — including its bugs.
The test tells right from wrong. It fails when the behavior is wrong.

AI output routinely earns #1 and #2 and skips #3. Accept it anyway and your confidence is borrowed — trust in the generator standing in for proof from the test.

See it in 10 minutes

I built a tiny lab you can run yourself (no deps, Node 20+):

node run-lab.js all

It gives the AI's own suite a pricing function with four seeded defects, then runs an independent verification suite against the same broken code:

WEAK TESTS (the AI's suite) .... PASS 3/3
STRONG VERIFICATION ........... FAIL 8/12   <- catches all 4 defects
MUTATION GATE ................. PASS 4/4
LAB GATE: PASS

The AI's tests pass against code that's wrong. Verification fails — and that failure is the evidence you actually needed.

The fix is a habit, not a tool

Before you trust a generated test, challenge it with a relevant defect: what realistic wrong implementation would still pass this? If nothing you can think of would make it fail, the test isn't discriminating — it's decoration. Design the discriminating test first, then generate, then prove it catches the fault.

That loop — clarify, generate, run, critique, repair, review, and a human decides — is the whole method. The generator never approves its own work; an independent reviewer (a different model or session) checks it; seeded defects and mutation testing prove the suite detects faults.

Run it, then go deeper

The lab is free and self-contained: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

I turned the full method into a book + runnable kit — labs across unit, web/API, instrument-cluster and CAN/UDS domains, a prompt+agent kit, a governed setup builder, and an offline-model chapter for teams that can't send code to a cloud. It's $19: https://nikolaychernev.gumroad.com/l/ai-testing-in-practice Also on Amazon Kindle ($9.99): https://www.amazon.com/dp/B0HL4SC5PX

Honest caveat: the systems under test are simulators with declared defects — they prove the method catches faults, not that any real product is certified. That's the point of the book: don't trust things that merely look right.