I built VibeDoctor, a scanner for AI-generated code. It looks for the stuff generic scanners miss - hallucinated imports, XSS patterns specific to AI output, N+1 queries in agent loops, memory leaks from copy-pasted React hooks.
Once I had it working, I had to know: how does the code from the tools everyone is using actually hold up? So I pointed it at three of the biggest AI coding repos on GitHub.
The leaderboard
- open-lovable (Firecrawl): 50/100, 190 issues, 24 critical
- devika: 66/100, 81 issues, 16 critical
- bolt.new (StackBlitz): 64/100, 77 issues, 0 critical security findings
Not a single one cleared 70. The tools millions of people are using to ship production apps are themselves scoring "poor" to "fair" on the patterns they generate.
Let me show you the specific things that stood out.
open-lovable: 22 XSS vectors in one repo
open-lovable is Firecrawl's open-source Lovable clone. It's a great project and I like the team. But the scanner found 22 separate XSS vectors, mostly dangerouslySetInnerHTML and direct innerHTML assignments across ASCII animation components, tooltips, and the main generation page.
It also found 4 unbounded while loops with .push() inside the apply-ai-code route. No break, no limit, no max iterations. If the AI response ever loops, the Node process runs out of memory.
And one file, app/generation/page.tsx, is 3,958 lines long. A single function, AISandboxPage, is 3,888 of those lines. This is what AI-assisted development produces when nobody stops to refactor.
Report: https://app.vibedoctor.io/shared/fb2ce6f3-c32f-4f6a-9af4-e43fb0ac698b
devika: the agent loop is doing N+1 queries
devika was a viral AI developer agent that hit 18k stars in its first week. The scanner found 5 separate N+1 query patterns inside src/agents/agent.py and src/agents/runner/runner.py. Every iteration of the reasoning loop fires off individual database queries instead of batching. On a long agent run this is brutal.
It also has a hardcoded api_key="no..." in src/llm/lm_studio_client.py:12. Placeholder or not, that pattern gets copied into forks and nobody notices when someone swaps in a real key.
Zero test files for 86 source files. The requirements.txt has 33 unpinned dependencies. No lock file.
Report: https://app.vibedoctor.io/shared/62e9432d-0786-4a00-9ff4-243571b8bb22
bolt.new: clean code, dirty dependencies
Here's the plot twist. bolt.new scored best of the three on the code itself. Zero critical security findings in their own code. Good structure, reasonable function lengths, proper Remix patterns.
But their package.json is pinning react-router@6.24.1, which has four separate CVEs against it right now including path traversal and SSR XSS. Plus svelte@4.2.18 (three CVEs), nanoid@3.3.6, jsondiffpatch@0.6.0. None of these are bolt.new's fault, they're supply chain. But they ship with every user's generated project until someone updates.
This is the hardest problem in vibe coding and nobody is solving it: the AI tool generates clean code, pins an old dependency it was trained on, and every app built with it inherits the CVE.
Report: https://app.vibedoctor.io/shared/77e28919-a345-4743-b982-cf943295fed2
The meta-point
AI coding tools are trained on code, then they generate code in the patterns of that training data. When the reference implementations have XSS, N+1s, god files, and stale dependencies, every app built with them starts from that baseline. Generic scanners don't know to look for the AI-specific patterns. Human reviewers don't read 3,888-line functions.
Something has to catch this before it ships. Right now almost nothing does.
Firecrawl, StackBlitz, and the devika team are shipping at a pace most of us can't match. None of this is a dunk. It's a pattern worth naming because the apps built with these tools inherit it.
Try it on your own code
VibeDoctor is free for your first scan. Connect your GitHub, pick a repo, get a report like the ones above. Signup required, scan is free: https://vibedoctor.io
If you're from Firecrawl, StackBlitz, or the devika team and want to chat about any specific finding, 911@vibedoctor.io. Happy to be wrong on specifics and I'll update the post.
Top comments (0)