AI is great at writing tests fast, and good at writing tests that look real but verify the wrong thing. Here's the line between useful scaffolding ...
For further actions, you may consider blocking this person and/or reporting abuse
Really enjoyed this — "it inferred a contract from the shape of the code, which is
exactly the wrong direction" is the whole ballgame.
Let me add one thing that turns your diagnostic into a power tool. Your test —
change the implementation without changing behavior; if the tests fail, they test
implementation — is mutation testing done by hand, once. Mutation testing automates
it: it makes thousands of tiny changes to your code and reports which ones your suite
failed to notice. Every surviving mutant is a test that was asleep on the job. It's
the honest answer to "but every test was passing," because it stops trusting the
green bar and asks the only question that matters: who's testing the test?
And that's exactly where AI-generated suites get exposed. Implementation-shaped tests
inflate line coverage while tanking mutation score — 96% coverage / 34% mutation
score is a real and depressingly common gap. The AI wrote a pile of tests that assert
the code's shape, so naturally they survive when you mutate the shape's behavior.
Mutation score is the lie detector this whole article is circling.
There's a nice synergy on top: a surviving mutant is a coordinate — it tells you
where the suite is blind. Property-based / diverse-data generation is the search
that kills it — it tells you what inputs to throw at that spot. Mutants point,
generated data shoots. I wrote that pairing up here:
High-Confidence Testing with Mutation Analysis and Diverse Test Data.
Last thing, on your "shallow edge cases." Timezone boundaries, idempotency
collisions, soft-delete cascades — those aren't input edges, they're schedule
edges. They don't live in a function body, so there's no shape for an AI (or a human
reading code) to infer them from; they only exist in the ordering of events across
the system. That's the beat for deterministic simulation testing: fork
run the same real code across thousands of event orderings, and the rare-schedule bug
stops being a cryptid that vanishes when you look at it. Wrote that on
VOPR: The Multiverse Machine That Kills Production Bugs.
Great piece.
This is a brilliant addition "who's testing the test?" is the question the whole piece was circling without ever naming it. You're right that mutation score is the missing lie detector: the green bar only proves the tests ran, mutation testing proves they were awake. That 96% coverage / 34% mutation gap is basically the article compressed into two numbers.
And your input-edge vs schedule-edge distinction is sharper than how I framed it. Timezone boundaries, idempotency collisions, soft-delete cascades really don't live in a function body for an AI (or a human) to infer, they only exist in the ordering of events. "Mutants point, generated data shoots" is a great way to say it. Both your pieces are going on my reading list.
Thanks for the thoughtful comment! 🙏
The refactor diagnostic at the end is good, but it has a failure mode worth naming: it rewards behavioral tests that are still verifying the wrong behavior. A test can survive every implementation change you throw at it and still be asserting your buggy assumption faithfully — the SUMMER25 test passes through any rename, and it's exactly the test that encodes "discounts are always positive" without ever checking it. Surviving refactors proves the test is coupled to behavior, not that it's coupled to the right behavior.
The stronger mutation to run isn't renaming a method — it's changing the outcome. Flip a boundary the way a bug would: return
amountwhen the code is expired instead of erroring, or let the discount go negative. If the suite still passes, you've learned something the refactor test can't tell you, which is that your contract has a hole in it, not just that your tests are implementation-shaped. That's basically hand-rolled mutation testing, and it's the one check that catches the closed-loop problem you spend the whole piece describing — because it attacks the shared assumption directly instead of the code shape around it.Great point, Wren! Thanks for this. You're right, surviving a refactor only proves the test isn't implementation-coupled, not that it's asserting the right behavior. Flipping a real boundary condition (mutation-style) is the stronger check. Might fold that distinction into the piece.
Thank you Nazar, have a great day!!!
The lie in AI test generation is subtle: it writes tests that pass, which feels like coverage but often just encodes the code's current behavior, bugs included. It's great at the mechanical cases you'd have skipped out of boredom. It's weak exactly where tests matter most, the edge case you didn't think of, because it's reasoning from the same blind spots as the code. Tests generated from the implementation prove it does what it does, not what it should. The value is real for breadth; the trap is mistaking that breadth for correctness.
"Reasoning from the same blind spots as the code" that's the sharpest one-line version of what I was trying to say the long way around. That's the whole trap: the AI infers what to test from the implementation, so it can only ever confirm the code does what it does, never what it should. The breadth is real and genuinely useful; the danger is reading that breadth as correctness.
Thanks for reading!
The "tests what the code currently does, not what it's supposed to do" point is the one I'd underline. Generated tests love to pin the current implementation, so they pass on day one and then block every refactor while catching none of the regressions that matter. Capturing fixtures from recorded real responses instead of letting the model invent them is the step most people skip, and it's what keeps the mocks honest.
Exactly, that's the trap. A green suite on day one feels like progress but it's really just a snapshot of whatever the code happened to do, bugs included. Recording real responses for fixtures is underrated precisely because it's the boring, unglamorous part, and that's where the honesty comes from. Once the mocks drift from reality the whole safety net turns into theater.
Hey, this article appears to have been generated with the assistance of ChatGPT or possibly some other AI tool.
We allow our community members to use AI assistance when writing articles as long as they abide by our guidelines. Please review the guidelines and edit your post to add a disclaimer.
Failure to follow these guidelines could result in DEV admin lowering the score of your post, making it less visible to the rest of the community. Or, if upon review we find this post to be particularly harmful, we may decide to unpublish it completely.
We hope you understand and take care to follow our guidelines going forward!
Hey, thanks for the heads-up and the link to the guidelines.
The ideas, structure, and opinions in this piece are my own. It's based on real experience writing and reviewing tests, and I checked it for factual accuracy before publishing. I do use AI to help me draft and tidy up my writing, so I'm glad to be transparent about it. I've added a disclosure line to the post per the guidelines. Thanks for keeping the standards high.
happy to adjust anything else if needed.
AI saves time, but humans provide the context, critical thinking, and final confidence before anything reaches production.
Exactly, that's the line I was trying to draw. AI can write the tests fast, but it can't decide what "correct" means for your system. That last mile of context and judgment stays with us, and it's the part that actually keeps things safe in prod. Thanks for reading! 🙏
Really strong framing. The line that stuck with me was “the AI is allowed to generate the body of the test, not the intent of the test.” That matches what we see in production: model-generated tests look great until a domain invariant shifts and you realize the suite was only mirroring the implementation. I also liked the distinction between obvious edge cases and domain edges — the DST / idempotency examples are exactly the kind of failures that never show up in generic “add more tests” advice. Feels like the practical workflow is: human writes the contract in plain English, AI expands coverage around it, and reviewers stay ruthless about business semantics.