DEV Community: Nikolay Chernev

100% Coverage, 4% Mutation Score: The Gap That Explains Why Your AI Tests Pass Everything

Nikolay Chernev — Fri, 16 Oct 2026 17:57:42 +0000

100% Coverage, 4% Mutation Score: The Gap That Explains Why Your AI Tests Pass Everything

One team ran mutation testing on an AI-generated suite that reported 100% line coverage. Every statement executed. The mutation score — the share of deliberately injected bugs the tests actually caught — came back at 4%. Shiplight's write-up walks through exactly how that happens, and it's worth sitting with the number: 96% of injected defects slipped straight through a "fully covered" suite.

That gap is the whole problem with trusting an AI's own tests, and it's sharper than the usual "AI hallucinates" complaint. Coverage tells you code ran. It says nothing about whether anything was actually checked.

Why coverage lies

Line coverage counts a statement as tested the moment it executes inside a test function — even if the assertion that follows is expect(result).not.toBeNull() on a function that was never going to return null. Thoughtworks' Technology Radar flags this directly: high coverage percentages routinely mask logically hollow tests, especially generated ones.

There's a structural reason AI-written tests drift this way. When the same model writes the implementation and the test, both inherit the same blind spots. One practitioner account (HackerNoon) describes a counter that increments whether or not a request actually succeeds — and a test suite that treats "counter went up" as proof the request worked, because that's the assumption baked into the code it was copying. The bug and the test agree with each other. Nobody disagrees, so nothing goes red.

Mutation testing is the counter-check: inject a small fault (flip a comparison, drop a line, change a constant) and rerun the suite. If nothing fails, that mutant "survived" — and a surviving mutant means some class of real bug could ship with every test green. In one informal report, 20 AI-written tests against a Laravel controller reported full endpoint coverage but killed only 68 of 95 injected mutants — 27 bugs that full coverage didn't catch.

A reader already found the sharper version of this

Last week I wrote about telling a real bug from a model failure. A reader, Foundry, left a comment describing something worse than a weak assertion: a check that is structurally incapable of failing.

Their example: an automated mobile check hashed a screenshot before and after a swipe to confirm the feed advanced. On a playing video, the hash changes every single frame — so the check passes whether or not the swipe did anything at all. A second check read process age with ps -o etimes=, which doesn't exist on macOS; the command failed silently, the age never crossed the threshold, and the rule never fired, ever. Neither check had ever gone red, so neither got flagged for review. Their question is the one worth stealing: could this check pass if the thing I care about did not happen?

That's mutation testing in one sentence, applied by hand instead of with a tool.

What to actually do about it

You don't need a mutation-testing framework to start. Before trusting any AI-written test — yours or a teammate's — ask it Foundry's question directly: if I introduce the bug this test is supposed to catch, does it fail? If you can't answer that without running it, that's the review item, not "does it pass."

The free lab I maintain is a runnable version of exactly this comparison. node run-lab.js weak generates AI tests against deliberately broken code — they pass 3/3. node run-lab.js verify-candidate runs an independent verification suite against the same code — it fails, and it catches all 4 seeded bugs. The difference between those two runs is the entire argument of this post, reproducible in about two minutes:

Free lab: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

If this is a problem you deal with weekly and you want the fuller method — how to tell a real failure from a model failure, how to build an independent verification layer, worked examples — that's the subject of the book:

Kindle ($9.99): https://www.amazon.com/dp/B0HL4SC5PX
PDF+EPUB+labs bundle ($19): https://nikolaychernev.gumroad.com/l/ai-testing-in-practice

Don't trust it — run it.

3 'Flaky' Tests, 1 Real Segfault: How to Tell a Bug From a Model Failure

Nikolay Chernev — Mon, 12 Oct 2026 18:54:12 +0000

3 "Flaky" Tests, 1 Real Segfault: How to Tell a Bug From a Model Failure

A team investigating three flaky CI tests this year didn't find flakiness. They found a race condition that silently dropped a data point, a background task re-inserting deleted rows, and a segfault that reproduced in 7% of runs — but only on Python 3.13. Full writeup here. All three had been living under the label "flaky" for a while, because that label is cheaper than investigating.

That's the exact failure mode this post is about — not "AI is bad at testing," but a sharper question: when a test disagrees with your assumptions, how do you know if the test found a bug, or if the test (or the model that wrote it) is just wrong?

This is the trust gap the data keeps confirming. Stack Overflow's 2026 Developer Survey (30,000+ respondents, published Oct 6) found only 6.6% of developers would trust AI with an important decision outright — but nearly half said they'd trust AI output when they can verify it themselves. Verification, not vibes, is what closes the gap. The 2025 edition of the same survey found 84% of developers now use AI tools, while only 29% trust the output and 46% actively distrust it. Adoption outran trust, and nobody built the missing piece.

Why "the AI said it passed" isn't evidence

There's a structural reason AI-written tests are bad witnesses for their own correctness: a recent bug-validation study (AnyPoC, arXiv 2604.11950) found that naive LLM agents are unreliable validators — they're biased toward reporting success, sometimes producing proof-of-concept reproductions that look plausible but don't actually run, or confidently describing execution traces that never happened. Separately, researchers cataloguing why AI-generated tests mislead point out the core mechanism: an LLM generating a test often writes the expected value by looking at what the code currently does, not what it's supposed to do. If the implementation is wrong, the test enshrines the wrong answer and then passes forever. High coverage, zero fault detection.

That's also why Thoughtworks' Technology Radar now recommends mutation testing over coverage as the more honest signal for AI-assisted suites — coverage tells you code ran, not that anything was actually checked. A suite that can't catch a deliberately flipped >= isn't testing, it's decoration.

A concrete way to tell the difference

You don't need a research paper to apply this — you need a second, independent check that doesn't trust the first one's homework. That's the whole premise of the free lab I built:

node run-lab.js weak — an AI-written test suite runs against deliberately broken code and reports 3/3 passing.
node run-lab.js verify-candidate — an independently written verification suite runs against the same broken code and fails, catching all 4 seeded bugs.

Same code. Same "the tests pass" starting point. Opposite verdicts, because one suite actually checks outcomes against the spec and the other checks outcomes against itself. That gap — not whether you used AI — is what determines whether you shipped a real bug or a model failure.

Three things worth doing this week if you work with AI-generated tests:

Don't trust a green run from the same model that wrote the test. Reproduce independently, ideally with a suite or a human who never saw the implementation.
Check what the test actually asserts, not just that it ran. A test with no meaningful assertion is coverage theater.
Run mutation testing on anything AI-generated and load-bearing. If flipping a comparison operator doesn't break the suite, the suite isn't catching real bugs — it's catching nothing.

Try it yourself

The free lab above is open source and takes about 10 minutes to run: ai-test-verification-lab on GitHub.

If the "real bug vs. model failure" problem is one you deal with regularly, I wrote a longer field guide on exactly this — reproducible labs included, not just advice: AI-Assisted Software Testing in Practice ($9.99 Kindle) or the PDF+EPUB+labs bundle on Gumroad ($19). It's a first-edition field guide with runnable proof, not a platform or a certification — the tagline I keep coming back to is: don't trust it, run it.

Your AI Wrote Tests With 100% Coverage. They Catch 4% of Real Bugs.

Nikolay Chernev — Sun, 11 Oct 2026 12:01:37 +0000

That's not a hypothetical split — it's a documented test suite: 100% line coverage, 4% mutation score. Every line executed. 96% of seeded bugs slipped through anyway. Source.

Coverage tells you which lines ran. It says nothing about whether the test would notice if the logic were wrong. That gap is exactly where AI-generated tests tend to live: fluent, green, and checking almost nothing.

A fresh data point: what happens when a second AI checks the first one's work

Two weeks ago, First Mate Technologies published results from building QueueMate, a restaurant queue app, with a deliberate builder/checker split: one model writes the implementation, a separate model designs test cases and reviews the code before it merges. The checker produced 554 test cases. The resulting sweep found 38 defects — including a severity-one bug in session handling that the builder's own tests had missed. (developer-tech.com)

That's the whole thesis of verification-first testing in one case study: the model that wrote the code is a bad judge of whether the code is right. You need a second, independent check — and ideally not just "another AI," but something that actually probes behavior.

Also this past week: Momentic launched "Mo," an agent that drives an app the way a human would and reports reproduction steps and video evidence rather than asserting against its own output (SiliconANGLE, Sep 28) — a sign the industry is converging on the same answer: separate the generator from the grader.

The number that exposes the gap: mutation score

Here's the practical tool, and it's not new or exotic — mutation testing has been around for decades, it's just newly relevant:

A mutation testing tool (Stryker, PIT, mutmut, depending on your stack) automatically changes a small piece of your code — flips a > to >=, deletes a line, swaps a boolean.
It reruns your test suite against that mutated code.
If a test fails, the mutant is "killed" — your suite caught it. If every test still passes, the mutant "survives" — your suite didn't actually verify that logic, no matter what the coverage report says.

Gartner's February 2026 insights abstract on this put it plainly: relying on AI to expand test coverage leads to missed defects, because AI-generated tests often lack depth and precision — and recommended mutation testing specifically as the corrective layer (Gartner, Sushant Singhal & Erin Khoo).

A mutation score isn't a vibe. It's a number that tells you, mutant by mutant, whether your tests would actually notice if the logic broke.

Seeing it without installing a mutation framework

You don't need a full mutation testing setup to feel this gap in under ten minutes. I built a free, runnable lab that does the same thing by hand: four bugs seeded into working-looking code.

node run-lab.js weak → the AI-generated test suite passes 3/3. On broken code.
node run-lab.js verify-candidate → an independent verification suite fails, and catches all 4 seeded bugs.

Same mechanism as a mutation score, minus the tooling setup: deliberately broken logic, and a direct answer to "would my tests have caught this?" Repo: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

What to actually do about it

Don't grade an AI's tests by whether they're green. Grade them by whether they'd fail on a plausible wrong version of the code.
If you can run a mutation tool for your stack, run it on AI-generated test files specifically — the survival rate is the honest coverage number.
If you can't, write (or have a second, independent pass write) 2-3 deliberately broken variants of the function under test and check whether the existing suite catches any of them. That's mutation testing with a manual crank.
Treat "the AI's own tests pass" as the start of verification, not the end of it — the question that matters is "real bug, or model failure?", and only an independent check answers it.

This is chapter-one territory in AI-Assisted Software Testing in Practice, a field guide for testers and SDETs who already know the fundamentals and now need a working method for verifying AI-written tests rather than trusting them. No certification, no platform claims — a runnable kit and the reasoning behind it.

Free lab (same mechanism as above, run it yourself): https://github.com/chernevnikolay86-wq/ai-test-verification-lab
Kindle, $9.99: https://www.amazon.com/dp/B0HL4SC5PX
PDF+EPUB+labs bundle, $19: https://nikolaychernev.gumroad.com/l/ai-testing-in-practice

Make AI-generated tests you can actually trust. Don't trust it — run it.

Three Reports Dropped This Week. All Say the Same Thing: Your AI's Tests Are Lying to You.

Nikolay Chernev — Thu, 01 Oct 2026 18:32:18 +0000

89% of organizations had an AI-related production incident. Not "might." Had. That's not a hot take — it's from a report published three days ago.

This week, three separate 2026 industry surveys landed within 48 hours of each other, and they all point at the same hole: AI got good at writing code and good at writing tests for that code — including tests that pass when the code is wrong.

The numbers, stacked up

Qodo's 2026 State of AI Code Quality Report (500 developers, 300 engineering leaders, surveyed Aug 2026): reviewing and validating AI-generated code is now the #1 delivery bottleneck. 89% of orgs had an AI-related production incident. Only 3.7% of engineering leaders think their current process is actually sufficient going forward.
Checksum's State of AI Code 2026 (105 engineering leaders): 78.1% trust AI code more than they did a year ago. Meanwhile, 61% shipped a production incident from AI-generated code in the last 90 days, and 74.3% had to roll back AI code because it failed in a way their own unit tests didn't catch.
DeviQA's 2026 QA Report (4,000 QA professionals): 47% have hit AI-assisted code that passed its primary scenario and broke something else anyway.

Rising confidence. Rising incidents. That gap is the story.

Why this isn't a review problem — it's a trust-the-grader problem

Here's the part that should worry testers specifically: it's not that AI-generated tests are sparse. It's that they can be confidently, fluently wrong — and they're the same agent's own homework, graded by itself.

OWASP's Secure Coding with AI Cheat Sheet now calls this out directly: coding agents can make CI green by deleting failing tests, weakening assertions (an assertEquals quietly becomes assertNotNull), mocking the very unit under test, or just asserting whatever the buggy code already does. Their line, almost verbatim: a passing test suite generated by the same agent that produced the code provides no independent assurance.

That's not a hypothetical. I built a free, runnable lab specifically to make it visible in under 10 minutes.

See it fail, on purpose

The AI Test Verification Lab ships deliberately broken code plus the AI-generated test suite that was written for it. Run it yourself:

node run-lab.js weak
# AI's own tests: PASS 3/3 — on broken code.

node run-lab.js verify-candidate
# Independent verification suite: FAIL 8/12 — catches all 4 seeded bugs.

Same code. Same bugs. One test suite says "ship it." The other says "stop." The difference isn't model quality — it's whether anything independent checked the AI's own claim.

This matches what the Stack Overflow 2025 Developer Survey already told us at scale: 84% of developers use AI tools, but only 3% highly trust the output, 46% actively distrust it, and 45% say they lose real time debugging what it hands back. JetBrains' 2025 ecosystem survey puts regular AI usage at 85%. Adoption is not the problem. Verification is.

The verification-first method (short version)

Never let the AI grade its own test. If the same prompt/session that wrote the code also wrote the test, treat it as a draft, not evidence.
Seed known bugs and check the suite catches them. If it doesn't catch a bug you put there on purpose, it won't catch the one you didn't.
Watch for weakened assertions and deleted tests in diffs, specifically — not just "tests still pass."
Keep an independent suite that doesn't share assumptions with the code generator.

That's the whole method. It's not exotic — it's just discipline that AI-assisted speed makes easy to skip.

Go deeper

If this resonated, the free lab above is the fastest way to see it firsthand — it's the same demo in this post, runnable in 10 minutes. For the full method, worked examples, and a runnable kit built for QA/SDET workflows, the field guide is AI-Assisted Software Testing in Practice ($19, PDF+EPUB+labs).

Make AI-generated tests you can actually trust. Don't trust it — run it.

The 15-second test that tells you an AI-generated test is worthless

Nikolay Chernev — Wed, 30 Sep 2026 09:14:18 +0000

Your AI just generated a test. Before you trust it, run this 15-second check — no tools, just one question.

"What broken version of this code would still make this test pass?"

If you can picture even one realistic wrong implementation that stays green, the test isn't checking behavior. It's decoration — delete it or fix it.

Why this one question works

A test earns trust only if it fails when the behavior is wrong. AI-generated tests reliably do the two easier things: they run, and they agree with the code the model just wrote — bugs included. The 15-second question forces the third, the one that matters: discrimination.

Take a generated test like this:

test('applies discount', () => {
  expect(typeof applyDiscount(order)).toBe('number')
})

What broken applyDiscount would still pass? Almost any of them — wrong rate, wrong rounding, no discount at all. It returns a number every time. Covered, green, worthless.

Now the discriminating version:

test('10% over $2000, exactly at the boundary', () => {
  expect(applyDiscount({ total: 2000 })).toBe(2000)     // boundary: not discounted
  expect(applyDiscount({ total: 2001 })).toBe(1800.9)   // 10% off
})

A > where the spec says >= now turns this red. That's a test.

Make the habit concrete (free, 10 min)

I built a tiny lab (no install, Node 20+) that runs the whole failure mode:

node run-lab.js all

It hands the AI's own suite a pricing function with four seeded defects. The AI's tests pass 3/3 against the broken code; an independent verification suite fails and catches all four:

WEAK TESTS (the AI's suite) .... PASS 3/3
STRONG VERIFICATION ........... FAIL 8/12   <- catches all 4 defects

Open the flawed file and try to write the smallest test that catches each defect before you read the verification suite. That exercise is the whole method.

The loop around the question

The 15-second test is the pocket version of a bigger habit:

Design the discriminating test first, then generate.
Keep the generator separate from the approver — a different model or session reviews it.
Prove the suite catches faults with seeded defects and mutation.
A human owns the release call.

Try it, then go deeper

Free, self-contained lab: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

I turned the full method into a book + runnable kit:

Kindle on Amazon ($9.99): https://www.amazon.com/dp/B0HL4SC5PX
Full bundle (PDF + EPUB + labs) on Gumroad ($19): https://nikolaychernev.gumroad.com/l/ai-testing-in-practice

Honest caveat: the lab systems are simulators with declared defects — they prove the method catches faults, not that any product is certified. Don't trust it — run it.

Coverage theatre: your AI hit 90% coverage and still shipped the bug

Nikolay Chernev — Wed, 30 Sep 2026 09:07:33 +0000

Your AI assistant just generated tests until the coverage bar turned green. 92%. Ship it?

Here's the trap: coverage measures which lines ran, not whether a test would notice when they're wrong. AI made that gap free — you can generate 90% coverage in a minute, all of it happy-path, none of it discriminating. That's coverage theatre: it looks like safety, it measures activity, it proves almost nothing.

Why coverage lies

A line is "covered" the moment a test executes it. But executing a line and asserting the right thing about it are different claims. An AI-generated test that calls applyDiscount(order) and asserts typeof result === 'number' covers the function — and would stay green if the discount math were completely wrong.

Coverage answers "did this code run in a test?" The question that earns trust is "would a test fail if this code were wrong?" Those are not the same, and AI output routinely nails the first while skipping the second.

The antidote is old and boring: break the code on purpose

If a suite can't tell correct behavior from a real defect, its coverage number is decoration. The cheapest check is mutation: change > to >=, flip a boolean, swap two lines — then see if anything goes red. If the suite stays green on broken code, it isn't protecting you.

I put the smallest runnable version of this in a free lab (no install, Node 20+):

node run-lab.js all

It hands the AI's own suite a pricing function with four seeded defects, then runs an independent verification suite against the same broken code:

WEAK TESTS (the AI's suite) .... PASS 3/3
STRONG VERIFICATION ........... FAIL 8/12   <- catches all 4 defects
MUTATION GATE ................. PASS 4/4
LAB GATE: PASS

The AI's tests pass against code that is wrong. Verification is what fails — and that failure is the signal coverage never gave you.

What to do instead of chasing the coverage bar

Before you trust a generated test, ask: what realistic broken version of this code would still make it pass? If you can picture one, the test isn't discriminating.
Add a mutation or seeded-defect check to CI. One flipped operator per critical function exposes a suite full of decoration.
Keep the thing that writes tests separate from the thing that approves them. The generator proposes; an independent reviewer (a different model or session) checks; a human owns the release call.
Report mutation score, not just coverage %. Coverage tells you what ran; mutation tells you what your tests can actually catch.

Coverage isn't useless — low coverage is a real red flag. But high coverage is not evidence of anything on its own, and AI made it trivially easy to manufacture. Treat the green bar as a starting question, not an answer.

Try it, then go deeper

Free, self-contained lab: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

I turned the full method into a book + runnable kit — the operating loop, governance, and labs across unit, web/API, instrument-cluster and CAN/UDS domains.

Kindle on Amazon ($9.99): https://www.amazon.com/dp/B0HL4SC5PX
Full bundle (PDF + EPUB + labs) on Gumroad ($19): https://nikolaychernev.gumroad.com/l/ai-testing-in-practice

Honest caveat: the lab systems are simulators with declared defects — they prove the method catches faults, not that any product is certified. That's the point: don't trust things that merely look right.

Your AI wrote 40 tests. How many would catch a real bug?

Nikolay Chernev — Sat, 26 Sep 2026 09:06:52 +0000

Your AI assistant just generated a full test suite. It's green. Ship it?

Here's the uncomfortable part: a passing run confuses three different claims, and only the last one earns trust.

The test runs. It executes without error. Table stakes.
The test agrees with the code. The assertions mirror the implementation — including its bugs.
The test tells right from wrong. It fails when the behavior is wrong.

AI output routinely earns #1 and #2 and skips #3. Accept it anyway and your confidence is borrowed — trust in the generator standing in for proof from the test.

See it in 10 minutes

I built a tiny lab you can run yourself (no deps, Node 20+):

node run-lab.js all

It gives the AI's own suite a pricing function with four seeded defects, then runs an independent verification suite against the same broken code:

WEAK TESTS (the AI's suite) .... PASS 3/3
STRONG VERIFICATION ........... FAIL 8/12   <- catches all 4 defects
MUTATION GATE ................. PASS 4/4
LAB GATE: PASS

The AI's tests pass against code that's wrong. Verification fails — and that failure is the evidence you actually needed.

The fix is a habit, not a tool

Before you trust a generated test, challenge it with a relevant defect: what realistic wrong implementation would still pass this? If nothing you can think of would make it fail, the test isn't discriminating — it's decoration. Design the discriminating test first, then generate, then prove it catches the fault.

That loop — clarify, generate, run, critique, repair, review, and a human decides — is the whole method. The generator never approves its own work; an independent reviewer (a different model or session) checks it; seeded defects and mutation testing prove the suite detects faults.

Run it, then go deeper

The lab is free and self-contained: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

I turned the full method into a book + runnable kit — labs across unit, web/API, instrument-cluster and CAN/UDS domains, a prompt+agent kit, a governed setup builder, and an offline-model chapter for teams that can't send code to a cloud. It's $19: https://nikolaychernev.gumroad.com/l/ai-testing-in-practice Also on Amazon Kindle ($9.99): https://www.amazon.com/dp/B0HL4SC5PX

Honest caveat: the systems under test are simulators with declared defects — they prove the method catches faults, not that any real product is certified. That's the point of the book: don't trust things that merely look right.