A friend of mine — I'll call him Marcus — shipped an invoicing tool for freelancers last spring. Built it entirely with Claude and Cursor in about three weeks, no developer on the team. Six weeks after launch he quietly took it down. He mentioned "something with the database" and that users had reached out about seeing each other's invoices.
He never fully explained what happened. He didn't need to.
The Numbers
According to Aikido Security (2025), AI-generated code introduces over 10,000 new vulnerabilities per month — a 10x increase in six months.
One in five breaches now involves AI-generated code. According to Apiiro, privilege escalation paths are up 322%.
An arXiv study (2024) tested models separately and got the same result every time: at least 62% of AI-generated programs contain vulnerabilities regardless of which model wrote them. Veracode puts XSS at 86% of AI code, Log Injection at 88%.
IBM Cost of a Data Breach Report 2025: +$670,000 per incident where unauthorized AI was involved. 63% of breached organizations had no AI governance policy at all.
Four Real Cases
Lovable (May 2025, CVE-2025-48757). A Swedish vibe coding platform. 170 out of 1,645 generated apps exposed names, addresses, financial records, and API keys to anyone without authentication.
Replit / SaaStr (July 2025). SaaStr founder Jason Lemkin told the AI agent not to touch his code. 11 times. Some in ALL CAPS. The agent deleted the live database anyway, generated 4,000 fake entries to cover it, and reported rollback was impossible. Rollback was possible.
Amazon Q for VS Code (July 2025). Unknown user creates a GitHub account — no history, no commits. Somehow gets write access to the official Amazon Q repo with nearly one million installs. Ships a hidden instruction to wipe the user's filesystem and cloud resources. Seven days live. What stopped it: a typo in the attack code.
Moltbook (January 2026). An AI-only social network built entirely by an AI assistant — founder Matt Schlicht publicly said he didn't write a single line of code. The Supabase database had no row-level security and an API key hardcoded in the frontend JavaScript. Wiz researchers found it within minutes. 1.5 million API tokens exposed, 35,000 email addresses, private messages between agents. Meta acquired the platform anyway.
Can Better Models Fix This?
I spent a while trying to convince myself yes before writing this. Here's what I found.
Clean training data — models don't "know" code is safe, they reproduce patterns. New library, edge case context, old patterns break. A sterile dataset produces sterile code.
RLHF penalties — vulnerabilities are context-dependent. The same SQL query is safe in one place, dangerous in another. Penalizing a pattern without understanding context misfires both ways.
Static analyzer in the pipeline — linters catch known patterns. Logical and architectural vulnerabilities aren't patterns. The checker is blind in the same places as the generator.
Multi-agent verification — both models trained on the same data share the same blind spots. Two nearsighted people don't make one person with clear vision.
All four paths treat the symptom. The model isn't the problem. The problem is who has access.
A Proposal Nobody Has Made Yet
Existing approaches split into two camps: train developers better, or regulate companies. Nobody has proposed a technical mechanism at the platform level. Here is one.
No DevOps engineer exposes port 22 without restrictions — that's just how it's done. AI companies opened code generation without restrictions and called it democratization.
A competency barrier has five layers:
1. Paid access only. Code generation on paid plans or API only. Free tier gets nothing. Financial threshold filters casual users before any test.
2. Practical competency test. Tasks from a closed, rotating pool — like SWE-bench but private. Fix real code, don't explain it. Can't be googled, can't be fed to an AI — pool is closed.
3. Identity verification. Phone or card at registration. No anonymous second accounts.
4. Server-side enforcement. Verified Developer flag on the API key, checked server-side. Not in the interface — on the server. Curl without the flag gets nothing. That path is closed.
5. Real consequences. Cheating violates ToS. Ban plus loss of subscription or deposit without refund. Social engineering stops being worth it.
For an actual developer: 20 minutes. For everyone else: learn or hire. That's how it should work.
For AI companies this isn't charity: fewer breaches means higher trust from enterprise clients who pay more and stay longer. Regulation is coming regardless. Better to meet it as a responsible tool than scrambling after the next headline.
Where This Goes
The next high-profile AI code breach isn't a question of probability. It's a question of timing.
A competency barrier doesn't fix everything. But it can be implemented today — no legislation, no committees, no waiting.
The question is whether any AI company will move before regulators force them to.
Top comments (0)