DEV Community

Cover image for I Scanned Hundreds of AI-Generated Codebases. Here's What Keeps Showing Up.
Paschal Ugwuanyi
Paschal Ugwuanyi

Posted on

I Scanned Hundreds of AI-Generated Codebases. Here's What Keeps Showing Up.

#ai #webdev #javascript #security

I Scanned Hundreds of AI-Generated Codebases. Here's What Keeps Showing Up.

There's a moment every vibe coder knows.

You've been building for six hours straight. Cursor has been finishing your sentences. The app actually works , you can click through it, the data saves, the auth flow runs. You push to production feeling like you've unlocked a cheat code for software development.

Then a user emails you.

Or worse , they don't. They just leave.

I've spent the last several months doing something most developers skip: scanning AI-generated codebases before they ship. Not after. Before. AI analysis followed by human engineers who know exactly what to look for. And the same vulnerabilities keep appearing not occasionally, not in bad codebases, but constantly, across projects built by smart people using the best tools available.

This isn't a piece about AI being bad. I use AI tools every day. This is about a specific, predictable blind spot one that's costing founders users, trust, and in some cases, their entire product.

Let me show you what I keep finding.

The Problem Isn't That AI Writes Bad Code

That's the misconception worth killing first.

AI coding tools Cursor, Bolt, v0, GitHub Copilot, Claude write remarkably functional code. That's exactly what makes this dangerous. The code compiles. The tests pass if you wrote any. The happy path works beautifully.

But Veracode found that 45% of AI-generated code introduces security vulnerabilities. CodeRabbit found that AI co-authored projects have 2.74× more security issues than human-written code. Forrester projects $1.5 trillion in technical debt by 2027 from AI-generated code alone.

These aren't fringe cases. These are the statistical outcomes of a tool optimized for plausible, functional output not defensive, adversarial-resistant output.

The distinction matters. A lot.

What I Actually Keep Finding

1. SQL Injection The Classic That AI Keeps Reinventing

I see this in probably 60% of codebases with a database layer. Here's what AI generates:

const user = await db.query(
  `SELECT * FROM users WHERE email = '${req.body.email}'`
);
Enter fullscreen mode Exit fullscreen mode

It works perfectly until someone sends ' OR '1'='1 as their email address and reads your entire users table. That string interpolation isn't a style choice. It's an open door.

The parameterized version isn't even harder to write:

const user = await db.query(
  'SELECT * FROM users WHERE email = $1',
  [req.body.email]
);
Enter fullscreen mode Exit fullscreen mode

AI knows this pattern exists. It just doesn't consistently choose it unless you specifically ask and most people don't know to ask.

2. Secrets in Source Code

This one is quiet and catastrophic.

const stripe = require('stripe')('sk_live_424242...');
const db = new Client({ connectionString: 'postgresql://admin:prod-password@...' });
Enter fullscreen mode Exit fullscreen mode

AI completes code based on context and comments. When your comment says // Connect to Stripe and your variable is named stripeKey, it fills in something that looks right. Sometimes it pulls from patterns in your own codebase including things you typed once and deleted.

The secret is now in your source. If it reaches your git history, it's effectively permanent. Secret scanners run on public repos constantly. This is how API keys get harvested within hours of a push.

The .env pattern isn't complicated. It's just not what AI defaults to.

3. Unprotected Admin Routes 

app.get('/admin/users', async (req, res) => {
  const users = await db.query('SELECT * FROM users');
  res.json(users);
});
Enter fullscreen mode Exit fullscreen mode

AI generates the route. It works. You test it. You move on.

What's missing: any check that the person hitting /admin/users is actually an admin. Or even logged in. It's not that AI doesn't know about auth middleware it's that you didn't ask for a protected admin route, so it gave you a route that works.

I've seen full user databases exposed this way. Real email addresses. Real data. In apps that had been live for months.

4. File Upload Path Traversal

This one is subtle enough that even experienced developers miss it:

filename = file.filename
file.save(f'./uploads/{filename}')
Enter fullscreen mode Exit fullscreen mode

Looks reasonable. Works in testing. But what if filename is ../../etc/passwd? Or ../app.py? The AI gave you the functionality you asked for. It didn't model what an attacker would do with it.

5. Missing Input Validation Everywhere 

app.post('/transfer', async (req, res) => {
  const { amount, toAccount } = req.body;
  await transfer(amount, toAccount);
  res.json({ success: true });
});
Enter fullscreen mode Exit fullscreen mode

What happens when amount is -10000? Or null? Or a string? Or 999999999999?

AI generates the optimistic path the code that works when users do what you expect. It doesn't naturally generate the pessimistic path the code that survives what users actually do.

Why This Keeps Happening (And Why Being More Careful Isn't the Answer)

Here's the structural reality: LLMs are trained to predict what code should look like, not to think like an attacker.

Security thinking is fundamentally adversarial. It requires asking: what could go wrong here? what would someone malicious do with this input? what happens at the edges? These aren't questions an autocomplete model naturally asks. They're questions that come from experience watching things break in production.

The instinct most developers have "I'll just be more careful" doesn't scale. Not when you're shipping fast. Not when you're using an AI tool that generates 200 lines in the time it takes you to review 20.

You need a different layer in your process.

What That Layer Looks Like

After scanning enough codebases to see these patterns become statistical certainties, I built Assayer to do this properly not just with AI, but with the combination that actually works.

Here's how it works:

You connect your GitHub repo. The AI scanner maps your codebase , your stack, your patterns, your actual vulnerabilities and flags every finding with the exact file and line number. Then a human senior engineer reviews what the AI found.

That last part matters. AI catches things fast. Humans understand context. Together they catch what either alone would miss.

You get two paths depending on what you need:

Path A — Review: Full report with exact fix instructions. You implement them.

Path B — Review + Fix: The engineers do the work directly. You ship.

It's not a linter. It's not a generic SAST tool. It's the layer between your AI-built codebase and your real users , staffed by people who think adversarially about code for a living.

The scan is free. Connect your repo here →

(Beta ends April 5 — after that, scans move to a credit system.)

The Checklist (For Right Now, Before You Read Further)

If you have something shipping soon, run through this before you push:

  • [ ] All database queries use parameterized inputs — no string interpolation
  • [ ] Passwords are hashed with bcrypt or argon2 — never compared in plain text
  • [ ] Secrets live in .env files — never hardcoded in source
  • [ ] Every admin route checks authentication AND authorization
  • [ ] File uploads sanitize filenames and validate file types
  • [ ] Every external input is validated before it touches your database
  • [ ] Error messages don't expose stack traces or internal details to users
  • [ ] Rate limiting exists on auth endpoints

Most AI-generated codebases fail three or more of these. Not because the developer is careless , because the tool optimizes for different things than this list does.

The Bigger Point

Vibe coding isn't going away. The speed is too good, the leverage is too real. We're not going back to writing everything by hand.

But we're entering a phase where the gap between "the AI wrote it and it works" and "the AI wrote it and it's safe" is becoming expensive. Users are smarter. Attackers are faster. The standards for what a launched product means are going up.

Build fast. That's the point. Just put something between your AI tools and your users that thinks adversarially about what they built.

Your users shouldn't be your QA team.

If you're building on Cursor, Bolt, v0, Replit, Windsurf, or any AI stack run the free scan at Assayer.dev before April 5. Takes about two minutes to connect your repo.

Drop questions or your own AI code horror stories in the comments , genuinely curious what patterns others are seeing.

https://assayer.dev/blog

Top comments (0)