Three weeks ago I watched a test suite go green while the feature underneath it was completely broken. Not partially broken. Not an edge case. The ...
For further actions, you may consider blocking this person and/or reporting abuse
The “would this test fail if the logic were wrong?” question is probably the best filter for AI-generated tests. Coverage can tell you which lines executed, but it cannot tell you whether the assertions encode the behavior that actually matters.
A useful extension is to test the negative space of the requirement: deliberately mutate the implementation so it violates a business invariant and see whether the suite catches it. At IT Path Solutions, that kind of verification is especially valuable when stabilizing AI-generated applications the goal isn’t more tests, but tests that can distinguish a correct implementation from a plausible-looking wrong one.
There’s also an important trust-boundary here: let the AI translate human-defined behavioral requirements into test code, but don’t let the same generation process define what “correct” means. Otherwise the implementation and its oracle can quietly agree with each other while both are wrong.
On gating every PR: if you move the check into an agent hook instead of CI, watch the timeout. We tried the new Mods hooks in Claude Code, gave one a 10 s budget, and it got cut at 10.6 s with the edit going through anyway. A mutation run that slow fails open there, so CI is the safer place for it.
yeah that tracks with everything I've read about how these hooks behave. the timeout doesn't actually block, it just gives up and lets the call go through like nothing happened, which is such a bad failure mode for anything that's supposed to be a guardrail. the one time it's too slow is exactly the time you needed it to say no, and instead it says nothing
good way to put it too, fails open is really the problem, not the speed itself. CI can sit there and take its time before saying no. a hook stuck in the middle of the flow doesn't get that luxury, so yeah, anything with actual teeth probably stays in CI regardless of how fast you make the mutation run
the version of this that gets me is when the tests are actually good, mutation-tested even, and it still ships broken because the environment it runs in during CI is not the environment it lands in. different env var, a migration that never ran, a secret that went stale. the suite proved the logic was right, it never proved the deploy would look anything like what the tests assumed. that gap doesn't show up in coverage or mutants either, it shows up as a 2am page.
the "confidence gap" framing is the right name for something i didn't have good language for. we hit this on a RAG pipeline last quarter: the AI wrote 20 tests for the retrieval step, every one passed, and it took a production bug to realize none of them were asserting on the actual ranked order. just that results were returned.
your third question (what would a tired tester think of) is the one we now require as an explicit prompt instruction before any AI generated test lands in a PR. doesn't fully fix it, but it catches ~60% of the shallow cases.
what's your rule on regeneration? if a test fails your three questions, do you delete and reprompt, or manually patch it?
the fix that actually held up for us is dumb simple. write the test from a spec a human typed, before the AI touches the function. once the same model writes the code and the check, of course they agree with each other, that is not a passing test, that is the model agreeing with itself.
the pattern you're naming (tests that confirm structure rather than semantics) shows up at every layer of the AI toolchain. I see it in agent eval harnesses too: checking that the agent produced output, not that it solved the task.
the mechanism is identical. the model writes tests against its own mental model of the implementation, so they pass by construction — they're just a restatement of what the model already believes.
the fix I've landed on: write the expected output by hand before generating any test. did that end up being your approach too?
Mutation testing is the cleanest way I know to measure this confidence gap. Deliberately change comparison operators, delete side effects, or replace returned values, then count which mutants survive. Coverage tells you the line ran; surviving mutants tell you the test had no opinion about whether the behavior was right. I would also keep the plain-language cases versioned beside the code so an agent cannot quietly rewrite both the implementation and its oracle in the same change.
Honestly you just named the thing I was circling around the whole post without landing on it. Coverage tells you the line ran, mutants tell you whether anyone actually cared what it did. Two completely different questions and we've been reporting the wrong one for years 😅
But your second point is the one that got me. Versioning the plain-language cases next to the code is the guardrail I completely missed, because the failure I wrote about can still happen even with a decent spec. Agent changes the implementation, test goes red, agent helpfully "fixes" the test in the same commit, everything green again. Nobody notices. That's so much worse than the shallow assertion problem and it never shows up anywhere.
Curious how you run the mutation side in practice though. Is it fast enough to gate every PR for you, or more of a weekly audit thing? That's where I keep getting stuck once the suite gets big.
"Spot on! 🎯 The silent agreement between a broken implementation and a self-adjusted AI test suite is the ultimate blind spot. If the agent can silently rewrite the test to match its flawed logic in the same commit, green checks become pure theater. Mutation testing and human-defined intent are the only real anchors left."
AI is adding to the work.
Yeah, it shifts more than it removes. It takes the typing away and hands you review work instead, and review is the slower part. The trap is that it feels like time saved because the output appears instantly, so nobody counts the hour spent reading it properly afterward. Still worth it for me, but only if I actually do that reading.