AI writes most of my code now. I'm not mad about it — it's faster, and most of the time it's right.
But after months of reviewing what these agents produce all day, I've noticed they fail in a very specific, very consistent way. Not random garbage — plausible, well-formatted, test-passing code that's wrong in ways the test suite never notices.
Here are the seven I now look for on every AI-assisted PR, and how to catch each one before it reaches production.
1. The breaking change to your own API
The agent "cleans up" a response DTO — renames a field, drops one that looks unused, makes something optional. Clean diff. CI green.
Except the field lived in a contract another team, a mobile app, or a partner integration depends on. The break isn't in this repo, so nothing here can see it — and the tests pass because the agent updated them in the same commit.
Catch it: diff your OpenAPI/schema against the version that's actually deployed, in CI, and fail on removed/renamed/narrowed fields. Humans miss a renamed key in a 600-line diff. A diff tool never does.
2. The N+1 query that passes every test
The agent writes an ORM loop that looks perfectly idiomatic and fires one query per row. On your 3-row test fixture it's instant. On 50,000 production rows it melts the database.
Tests run on tiny fixtures, so this is invisible until a customer with real data hits it.
Catch it: assert query counts in your tests (assertNumQueries(2) and friends), load-test the hot paths, and turn on slow-query logging in staging.
3. Happy-path-only code
Agents optimize for the example in your prompt. You showed it the success case, so it wrote the success case. Null inputs, empty lists, a timeout, a 500 from a downstream service — silently unhandled.
Catch it: property-based testing (throw a thousand weird inputs at it), and explicitly review for the unhappy path. "What happens when this is null/empty/slow/down?" is still your job.
4. The concurrency bug
The code is correct — single-threaded. Check-then-act on a shared resource, an unguarded counter, a missing transaction. Your test suite runs serially, so it never reproduces the race.
Catch it: concurrency tests that actually run in parallel, design for idempotency, and lean on database constraints as the backstop the application logic forgot.
5. The confidently outdated API
The model's training data has a half-life. It'll reach for a deprecated method, a flag that was removed two versions ago, or a function signature that's almost right. It compiles, it runs, and it quietly rots.
Catch it: pin your dependencies, lint for deprecations, and trust the official docs over the model's confidence. "It worked" is not "it's current."
6. The security footgun
String-interpolated SQL. A missing authorization check. A secret that ends up in a log line. CORS set to * because that made the error go away. Your tests assert behavior, not safety, so all of these pass.
Catch it: SAST in CI, a dedicated security-focused review pass, and a rule that an agent's "it works" is never the bar for auth or input-handling code.
7. The yes-man test
This is the subtle one, and the most dangerous.
When the same agent writes the code and the test in one pass, the test stops being an independent check. It asserts whatever the code happens to do — including the bug. You get a green checkmark and zero protection, because the test and the code were written to agree with each other.
- assertThat(json).contains("\"currency\":\"USD\"");
+ assertThat(json).doesNotContainKey("currency"); // "fixed" the test to match the change
That test didn't catch the breaking change. It ratified it.
Catch it: write the test from the spec before the agent touches the implementation, and treat any test modified in the same commit as the code it covers with deep suspicion.
The pattern underneath all seven
Every one of these has the same root: AI made writing code nearly free, but it can't see the things that live outside the function it's editing — your other services, your production data volume, your concurrency, your consumers, your threat model.
The bottleneck moved from typing to verifying. And a test the agent wrote for its own code isn't verification — it's the agent agreeing with itself.
So the leverage now isn't a cleverer prompt. It's guardrails the agent can't fake: contract checks, query budgets, security scans, and tests written independently of the code they guard.
What's the dumbest thing an AI agent has confidently shipped for you? I collect these. 👇
Top comments (1)
The "plausible and test-passing" category is the one that changes review work the most.
For AI-generated code, tests are still necessary, but they are not enough. I have found the extra checklist is usually about intent: did the model change a boundary, weaken a guard, invent a default, or make the code look simpler by moving risk somewhere else?