A few weeks ago I started a small experiment: every night, I ran automated scans over AI-generated code files — code written by Copilot, Cursor, ChatGPT and friends, collected from real projects.
The results after scanning 75 files:
- 81% contained at least one issue
- 2,277 findings in total
- That's an average of 30 issues per file
Not style nitpicks. Real, shippable bugs.
The most common offenders
1. Hallucinated APIs. The AI confidently calls a function that doesn't exist — in the library version you actually have installed. It compiles in your head, it doesn't compile in CI.
2. Placeholder URLs. https://api.example.com/... left in "finished" code. It looks intentional. It ships. It 404s in production at 2am.
3. SQL built with string formatting. f"SELECT * FROM users WHERE id = {uid}" — the AI knows this is wrong. It writes it anyway, because it's the pattern that most often "works" in training data.
4. Silent fallbacks. A try/except that swallows every error and returns a default. Your code never crashes. It also never tells you it's broken.
5. Hardcoded secrets. API keys and tokens pasted straight into source. The AI saw it a thousand times in tutorials, so it reproduces it.
Why AI keeps making these mistakes
AI code assistants optimize for plausible code, not correct code. They reproduce the statistical center of GitHub — including its bugs, its outdated APIs, and its bad habits from 2019 tutorials.
The better the AI gets at writing code that looks right, the worse our review habits get. When 200 lines appear in 10 seconds, nobody reads line 137.
So I built a thing
I got tired of finding these by hand, so I turned the nightly scanner into a VS Code extension: CodeVouch. It scans the AI-generated code in your workspace for exactly these patterns — hallucinated APIs, placeholder URLs, SQL string formatting, silent fallbacks, hardcoded secrets — and shows a quality report before you ship.
- Marketplace: https://marketplace.visualstudio.com/items?itemName=codevouch-ai.codevouch
- Website: https://codevouch.codes
- GitHub (open detection rules — star us!): https://github.com/8rabbits/CodeVouch
It's free for scanning; a Pro tier adds deep review. But honestly, the point of this post isn't the tool — it's the data.
The takeaway
If you ship AI-generated code without reviewing it, you're shipping bugs at AI speed. Review it — with your eyes, with a scanner, with anything. Just don't skip it.
I'm continuing the nightly scans. If people are interested, I'll publish a full breakdown of the 2,277 findings — which AI models produce which bugs, and which categories are getting worse.
Top comments (4)
I think one subtle shift AI has introduced is that code review has become evidence review. Most of the issues you listed hallucinated APIs, silent fallbacks, placeholder URLs, swallowed exceptions share the same root cause: the code looks complete but doesn't prove it's correct. That's why I'm seeing mature teams move beyond "LGTM" reviews toward requiring evidence: API version references, failing-then-passing tests, security scans, and runtime validation before merge. At IT Path Solutions, we've found that AI can generate implementations remarkably fast, but production quality comes from the verification pipeline around it, not the generation itself. As models improve, I suspect the biggest competitive advantage won't be who generates the most code it's who builds the fastest feedback loop to distinguish plausible code from trustworthy code.
"Evidence review" is a great way to put it — I'm stealing that phrase for the follow-up post 😄
And you're right about the shared root cause. I actually learned this the hard way: I was building a customer management system with AI, and it came together scarily fast — looked complete, ran fine. Then the bugs started. Endless ones. And the worst part? The AI that wrote them couldn't find them. Every check came back "looks good." That's the whole reason I ended up building this scanner — so people can catch this stuff before their AI-built app goes live, not after.
That's also why it became an editor extension instead of a report tool — the closer the check sits to the moment the code is written, the cheaper the fix. A nightly CI scan is too late if you've already built three features on top of a hallucinated API. Ask me how I know 😅
Do your teams run the verification pipeline on every PR, or only on AI-heavy ones?
Hello greetings
Check my bio for remote job