I Scanned 75 AI-Generated Code Files Every Night. 81% Had at Least One Issue.

Ida Zhang — Wed, 29 Jul 2026 09:09:37 +0000

A few weeks ago I started a small experiment: every night, I ran automated scans over AI-generated code files — code written by Copilot, Cursor, ChatGPT and friends, collected from real projects.

The results after scanning 75 files:

81% contained at least one issue
2,277 findings in total
That's an average of 30 issues per file

Not style nitpicks. Real, shippable bugs.

The most common offenders

1. Hallucinated APIs. The AI confidently calls a function that doesn't exist — in the library version you actually have installed. It compiles in your head, it doesn't compile in CI.

2. Placeholder URLs. https://api.example.com/... left in "finished" code. It looks intentional. It ships. It 404s in production at 2am.

3. SQL built with string formatting. f"SELECT * FROM users WHERE id = {uid}" — the AI knows this is wrong. It writes it anyway, because it's the pattern that most often "works" in training data.

4. Silent fallbacks. A try/except that swallows every error and returns a default. Your code never crashes. It also never tells you it's broken.

5. Hardcoded secrets. API keys and tokens pasted straight into source. The AI saw it a thousand times in tutorials, so it reproduces it.

Why AI keeps making these mistakes

AI code assistants optimize for plausible code, not correct code. They reproduce the statistical center of GitHub — including its bugs, its outdated APIs, and its bad habits from 2019 tutorials.

The better the AI gets at writing code that looks right, the worse our review habits get. When 200 lines appear in 10 seconds, nobody reads line 137.

So I built a thing

I got tired of finding these by hand, so I turned the nightly scanner into a VS Code extension: CodeVouch. It scans the AI-generated code in your workspace for exactly these patterns — hallucinated APIs, placeholder URLs, SQL string formatting, silent fallbacks, hardcoded secrets — and shows a quality report before you ship.

Marketplace: https://marketplace.visualstudio.com/items?itemName=codevouch-ai.codevouch
Website: https://codevouch.codes
GitHub (open detection rules — star us!): https://github.com/8rabbits/CodeVouch

It's free for scanning; a Pro tier adds deep review. But honestly, the point of this post isn't the tool — it's the data.

The takeaway

If you ship AI-generated code without reviewing it, you're shipping bugs at AI speed. Review it — with your eyes, with a scanner, with anything. Just don't skip it.

I'm continuing the nightly scans. If people are interested, I'll publish a full breakdown of the 2,277 findings — which AI models produce which bugs, and which categories are getting worse.

DEV Community: Ida Zhang

I Scanned 75 AI-Generated Code Files Every Night. 81% Had at Least One Issue.

The most common offenders

Why AI keeps making these mistakes

So I built a thing

The takeaway