Your AI wrote 40 tests. How many would catch a real bug?

Nikolay Chernev — Sat, 26 Sep 2026 09:06:52 +0000

Your AI assistant just generated a full test suite. It's green. Ship it?

Here's the uncomfortable part: a passing run confuses three different claims, and only the last one earns trust.

The test runs. It executes without error. Table stakes.
The test agrees with the code. The assertions mirror the implementation — including its bugs.
The test tells right from wrong. It fails when the behavior is wrong.

AI output routinely earns #1 and #2 and skips #3. Accept it anyway and your confidence is borrowed — trust in the generator standing in for proof from the test.

See it in 10 minutes

I built a tiny lab you can run yourself (no deps, Node 20+):

node run-lab.js all

It gives the AI's own suite a pricing function with four seeded defects, then runs an independent verification suite against the same broken code:

WEAK TESTS (the AI's suite) .... PASS 3/3
STRONG VERIFICATION ........... FAIL 8/12   <- catches all 4 defects
MUTATION GATE ................. PASS 4/4
LAB GATE: PASS

The AI's tests pass against code that's wrong. Verification fails — and that failure is the evidence you actually needed.

The fix is a habit, not a tool

Before you trust a generated test, challenge it with a relevant defect: what realistic wrong implementation would still pass this? If nothing you can think of would make it fail, the test isn't discriminating — it's decoration. Design the discriminating test first, then generate, then prove it catches the fault.

That loop — clarify, generate, run, critique, repair, review, and a human decides — is the whole method. The generator never approves its own work; an independent reviewer (a different model or session) checks it; seeded defects and mutation testing prove the suite detects faults.

Run it, then go deeper

The lab is free and self-contained: https://github.com/chernevnikolay86-wq/ai-test-verification-lab

I turned the full method into a book + runnable kit — labs across unit, web/API, instrument-cluster and CAN/UDS domains, a prompt+agent kit, a governed setup builder, and an offline-model chapter for teams that can't send code to a cloud. It's $19: https://nikolaychernev.gumroad.com/l/ai-testing-in-practice

Honest caveat: the systems under test are simulators with declared defects — they prove the method catches faults, not that any real product is certified. That's the point of the book: don't trust things that merely look right.

DEV Community: Nikolay Chernev

Your AI wrote 40 tests. How many would catch a real bug?

See it in 10 minutes

The fix is a habit, not a tool

Run it, then go deeper