The code compiled The tests passed The PR was green So I merged it.
Days later I was looking at something completely unrelated when I noticed it: a state update that had been technically correct in exactly the way I'd asked for, had quietly broken an assumption the rest of the system depended on. The update itself was fine. The problem was that another piece of state was supposed to change along with it and that relationship wasn't obvious from the function I was editing. The tests covered the updated state. They didn't cover the relationship between the two Nothing had crashed. No test had failed. No reviewer, including me, had caught it at the time.
The scary part wasn't that AI wrote the code. I'd expected AI-written code to have bugs, the obvious kind, the kind a test catches immediately.
The scary part was that everything, every single signal I normally trust, told me the code was correct.
That's the actual problem with AI-generated code. Bad code is easy to distrust. Code that passes every test you throw at it is much harder to question, and that's exactly backwards from how risk should work.
But the Tests Passed
I know what you're thinking, because I thought it too: if the tests passed, what's actually wrong here?
Tests verify behavior you thought to check. They don't automatically prove your architecture is sound, that a business invariant survived the change, that there's no race condition, that an edge case is handled, or that the abstraction the AI generated actually makes sense for where the codebase is headed.
A passing test tells you that one expectation passed. It doesn't tell you that your understanding of the system is correct.
That gap, between "the thing I checked worked" and "the thing I understand is true," is where my bug was hiding the entire time.
Everything Looked Green
Here's what my review actually looked like before I merged that PR.
CI: green. Tests: green. Lint: green. Build: green. The AI's explanation of what it changed: confident, clear, reasonable-sounding. The diff itself: sensible, nothing that looked out of place.
There was nothing obviously wrong with the code.
And that's exactly why it was dangerous. I wasn't ignoring red flags. There weren't any red flags to ignore. Every signal I'd normally use to decide "does this need a closer look" said no.
The state update genuinely did what I'd asked. It just also did something I hadn't asked about at all, and nothing in my review process was built to catch that category of problem.
Tests Can Only Protect the Things You Thought to Test
Here's the technical core of what actually happened, stripped down to the general shape of it.
Say you think the rule is: "if a condition is true, allow the update." You write a test for the true case. You write a test for the false case. Both pass. Looks solid.
But the real invariant, the one that actually needed to hold, was something neither test captured: that this piece of state couldn't be safely updated independently of another piece of state elsewhere in the system. The AI's change satisfied the specification I gave it completely. The specification itself just didn't include the constraint that mattered.
AI is very good at satisfying the specification you give it. The problem is that developers often give AI a specification without realizing they've left out the most important constraint. The dangerous part is when the specification is incomplete, and you don't find out it was incomplete until after you've already trusted the result.
AI Doesn't Make This Worse Because AI Is Bad. It Makes It Worse Because AI Is Convincing.
I don't think this is an argument against AI-generated code. I think it's an argument about what confidence actually means.
AI-generated code usually comes with clean naming, comments, reasonable-looking abstractions, tests, and a confident explanation of what it did and why. Your brain reads all of that and concludes: this looks professional, this looks like it was thought through.
But readable code can still encode the wrong assumption just as easily as messy code can. More tests can create more confidence without creating more actual understanding of the system. The polish isn't evidence of correctness. It's just polish.
The Worst Code Isn't Broken Code
Broken AI code is, in a strange way, the safe kind. It's easy to catch, because something exposes it immediately: the compiler, a failing test, an obvious error.
Working-but-wrong AI code is the dangerous kind. It's harder to catch, it builds false confidence exactly when you should be most careful, it gets merged, and it quietly becomes the kind of technical debt nobody knows exists yet.
| Code | Tests | Risk |
|---|---|---|
| Obviously broken | Fail | Low, visible |
| Incomplete | Fail | Visible |
| Working happy path | Pass | Medium |
| Wrong assumption | Pass | High |
| Wrong business rule | Pass | Very high |
My bug lived in the bottom two rows. That's exactly where you don't feel like you need to be looking.
Stop Reviewing the Summary. Review the Assumptions.
Here's the shift that actually changed how I review AI-generated PRs now.
An AI agent's summary of its own change reads well almost by default: implemented the state update as requested, added corresponding tests. That sentence sounds complete. It also tells you nothing about what the change assumed to be true elsewhere in the system.
Instead of reviewing what the summary says changed, I've started asking a different set of questions: What changed? Why did it need to change? What assumption does this code make about the rest of the system? What invariant has to remain true for this to be safe? What happens outside the path I explicitly tested? What existing behavior could this quietly affect?
Don't just review what the agent changed. Review what it assumed.
My 5-Minute Check Before I Trust AI-Generated Code
This is the practical part, the thing I actually run through now before merging anything AI generated.
1. What problem is this code actually solving? If I can't state it clearly, I stop before going further.
2. What assumption is this solution making? Especially the assumptions that aren't written down anywhere, the ones that just feel obvious until they aren't.
3. What invariant has to remain true? State consistency, an API contract, authorization, ordering, idempotency, data integrity. Something on this list is usually the thing that actually breaks.
4. What happens on the unhappy path? Not "does it work," but specifically "how does it fail, and does that failure mode matter."
5. Can I explain this change without reopening the AI conversation? If the explanation only exists inside the chat history, I don't understand it yet. I just have access to an explanation.
A Small Example
setUser({
...user,
status: "active"
});
Tests: updates the status, returns an active user, handles an invalid user. All green.
None of that tells you that status and permissions were supposed to change together. The AI updated status correctly, exactly as asked. Permissions just stayed stale, because nothing in the function, or the tests, said they were connected.
The code isn't wrong. It's just answering a narrower question than the one that actually needed answering.
Don't Stop Using AI. Change What You Ask It to Do.
Fix this bug gets you an answer. Here's my hypothesis for what's causing this, try to disprove it gets you a stress test of your own thinking, not just a patch.
Implement this feature gets you an implementation. Here are the constraints, what assumptions am I missing" gets you a second set of eyes on the part that actually tends to fail.
Is this code correct gets you reassurance. Find cases where this implementation could violate this specific invariant gets you something you can actually act on.
The Goal Isn't Just More Tests. It's Better Questions.
I want to be clear about something, because it would be easy to walk away from this thinking tests aren't worth trusting. They absolutely are. They're essential. They're just not the entire verification process, and treating them as if they were is exactly how a bug like mine slips through everyone's review at once.
Tests verify behavior. Developers own the assumptions. AI can help challenge both, but it can't take responsibility for either.
AI Doesn't Own the Bug. We Do.
AI generated the code. AI suggested the approach. AI wrote the tests. AI explained the change confidently, in a way that made sense.
But I merged it.
Which means the more code AI generates, the more important human verification becomes, not less. The volume going up doesn't shrink the responsibility. It just makes it easier to feel like it has.
So, Would You Merge It?
The code compiles. The tests pass. The PR is green. The AI says the implementation is correct.
Would you merge it?
Maybe. But first I'd want to know what assumptions are hiding behind that green checkmark, because I've already learned, the slow and embarrassing way, that green doesn't mean what I used to think it meant.
The dangerous code isn't always the code that fails.
Sometimes it's the code that passes everything you remembered to test.
Have you ever merged something because "all tests passed," only to find out later the tests were proving the wrong thing entirely? What's the most convincing AI-generated change you've seen pass everything and still turn out to be wrong? Genuinely want to hear your version of this. 👇
Top comments (6)
Great that you didn't just signal the problem, but also added actionable advice!
Tests remain a great (essential, even) tool, but the challenge has always been in answering the question what to test ...
Thanks Lob Exactly I think what to test is often where the real engineering judgment lives. AI can help generate tests, but if we give it the wrong target, it can give us a very complete answer to the wrong question. That’s why I like using AI to challenge my assumptions, not define them.
This:
"That’s why I like using AI to challenge my assumptions, not define them"
That seems a great approach/attitude ...
Thanks Leob That’s exactly the mindset I’m trying to build around AI-assisted development use it to pressure-test the reasoning, not outsource the reasoning.
The better results is in the better understanding of the problem. I couldn't help but react. Even the best programmers make mistakes but being able to "reevaluate" when you think everything is perfect is priceless 🙌
Thanks Ekong Absolutely Re-evaluate is the part that’s easy to skip when everything looks green. Even when the code, tests, and review all look right, stepping back and questioning the original assumption can reveal what the checks never covered.