SystAgProject

Posted on Apr 20

I Audited 21 Public Vibe-Coded Apps in 48 Hours. Here Are the 5 Patterns That Keep Showing Up.

#security #ai #webdev #supabase

Over the last 48 hours I've run VibeScan — my LLM-powered security audit for AI-generated SaaS — against 21 public apps built on Lovable, Bolt, v0, Cursor, Replit, and Windsurf. I wanted to check whether the 5 patterns I found in 9 apps earlier this week were a small-sample fluke or a real signal.

At 21 apps the signal is unmistakable.

Total findings across the corpus: 20 critical + 84 high + 58 medium = 162 real issues. Every single app had at least one. The most egregious had 13 (1 critical / 8 high / 4 medium). The "cleanest" still had 3 mediums.

Here's the ranked list of what keeps showing up — hit rate as unique codebases affected out of 21, not total finding count. So "9/21" means that pattern appeared in 9 distinct apps, which is basically half.

1. Missing authentication / auth bypass on serverless endpoints

Hit rate: 9/21 codebases (43%). Most common severity: critical / high.

The defining pattern: the app's server-side code (Supabase Edge Function, Next.js API route, Express route, Cloudflare Worker) trusts whatever the client says without verifying a JWT. Real finding titles from the corpus:

"Every API and WebSocket endpoint is completely unauthenticated" (backend-mock/server.js)
"Phone OTP verify accepts any Firebase UID without verification" (supabase/functions/auth-phone-otp/index.ts)
"Sensitive Supabase Edge Functions have JWT verification disabled"
"Calling, contacts, and notification routes accept any user ID with no auth"
"send-daily-email edge function has JWT verification disabled"

The scariest sub-class I saw: a function that accepts user_id from the request body and uses it with a service-role DB client (which bypasses Row-Level Security). Net effect: one curl command lets anyone impersonate any user.

Fix: every sensitive function needs ~20 lines of supabase.auth.getUser() at the top — verify the Authorization header, reject if invalid, use the verified user.id (not the request body) as identity. I wrote the full pattern + curl test vectors here.

2. No rate limiting on sensitive endpoints

Hit rate: 9/21 codebases (43%). Most common severity: high.

Every endpoint that touches email (/api/contact), authentication (/api/signup, /api/forgot-password), LLM calls (/api/scan, /api/chat), or any mutable action takes unlimited requests. Real finding titles:

"No rate limiting on message send or any endpoint"
"No input validation or rate limiting on QR login endpoint"
"No rate limiting on /api/scan endpoint" (LLM call — direct cost exposure)
"Public stats endpoint has no rate limit"
"No rate limiting on signup endpoint"

A single script with 10 parallel connections can exhaust your Supabase egress, SendGrid credits, OpenAI quota, or Stripe webhook window in under 60 seconds. The attacker doesn't even need a bug — just availability.

Fix: Upstash Ratelimit is 5 lines per endpoint. Ratelimit.slidingWindow(20, "60 s") keyed on IP (or user.id after auth is added). Their free tier covers most side projects.

3. Dangerous CORS / wildcard origin

Hit rate: 7/21 codebases (33%). Most common severity: high.

Either Access-Control-Allow-Origin: * with credentials (browsers will silently ignore the wildcard but the intent is to allow everything), OR a localhost+production origin list that includes dev-server origins in prod. Real findings:

"CORS wide-open on all API and WebSocket endpoints"
"Wide-open CORS on all API endpoints"
"Local API server accepts requests from any website (wildcard CORS)"

The common scaffolding pattern: CORS gets turned on loose during dev (because it's annoying), then never tightened before deploy. The cors({origin: '*'}) line survives into production.

Fix: replace the wildcard with an allowlist of your actual production domain(s) + localhost for dev. If you genuinely need public API access, don't send credentials with the requests.

4. Client-side trust / admin-by-flag

Hit rate: 6/21 codebases (32%). Most common severity: high / medium.

The pattern: the client reads a profiles.is_admin column (or equivalent) and toggles admin UI based on it. The backend then trusts that same flag for gatekeeping admin actions — but because the profile row is user-writable (via an insufficiently restrictive RLS policy), the user can flip their own flag and become admin.

Real finding titles:

"Admin status determined by client-side profile read"
"Client trusts is_admin flag from profiles table"
"Users can set their own order total and status"
"Admin 'list all sessions' endpoint called from browser with no auth check"

Fix: admin state lives in a separate table users can't write to, OR in a Supabase JWT custom claim set by a trusted admin-grant function. Never read admin status from a column the user's session can update.

5. Unsafe file upload (missing MIME check / size cap)

Hit rate: 3/21 codebases (14%). Most common severity: medium / high.

User-uploaded files (profile photos, document attachments, product images) land in a public storage bucket with no MIME-type whitelist, no size cap beyond the default 50 MB, and often no path-sanitization. Real findings:

"plant-images storage bucket is public with no file-type or size restrictions"
"Product image uploads have no MIME type or size restrictions"
"Request body parsed as JSON with no size limit or schema validation"

The attack: upload a 4 GB file disguised as a PNG. Exhaust storage quota, inflate your bill, maybe get the file served with the wrong MIME type (if Supabase Storage trusts the upload extension) and use it as a vector.

Fix: whitelist MIME types explicitly in the Supabase storage policy. Cap file size at the use-case-realistic upper bound (500 KB for profile photos, 5 MB for attachments). Validate the magic bytes server-side, don't trust the extension.

The remainder

Lower-hit-rate patterns that still showed up in the corpus:

Weak password / credential handling — 3/21 (14%) — plain SHA-256, minimum length 6 chars, password derived from phone number.
Permissive or missing RLS — 2/21 (10%) — world-readable tables, rate-limit tables with no RLS.
Missing input validation / no schema — 2/21 (10%).
Environment / debug / error leak — 2/21 (10%) — stack traces in prod, console.log with secrets.
Committed secrets — 1/21 (5%) — service-role key in the client bundle.
SQL injection / query construction risk — 1/21 (5%) — ILIKE wildcard injection allowing data exfil.
Secret in URL query-param — 1/21 (5%) — yes, caught this in my own tool before I shipped it.

Every single app had at least three patterns from the above list. The average was 7.7 findings per app, the worst was 21 findings in a single codebase.

What to do this week

If your app was scaffolded by Lovable, Bolt, v0, Cursor, Replit, or Windsurf in the last 6 months, the odds that at least one of the top-5 patterns above applies to you are high — the #1 and #2 patterns alone cover 43% of codebases each, and the top 4 collectively hit 85%+ of the corpus at least once.

Three ways to check:

Free preview scan. Email mike.j.kaplan+scan@gmail.com with your GitHub repo URL. Subject line VibeScan preview, body the URL. I'll run a preview scan (~25% of your code, highest-risk files) and reply with the top finding within ~5 minutes. No signup, no catch, no upsell unless the scan finds something.
Full audit. $49 one-time at systag.gumroad.com/l/vibescan. Every finding severity-graded, every finding with the exact file + line + copy-paste fix. PDF delivered in ~10 minutes. 7-day refund, no questions.
Do the top-5 yourself. Links above to detailed writeups on patterns #1 (edge function auth) and the broader 12-issue checklist. Budget 30-60 minutes and you're in the top decile of vibe-coded apps.

Methodology (for the skeptics)

21 public apps from GitHub search for "Built with Lovable", "Made with bolt.new", "v0.dev", "Built with Cursor", "Built with Replit Agent", "built with Windsurf" in README.
Cloned each to a disposable sandbox. Audited via Claude Opus 4.7 against a tuned system prompt (12 failure categories: leaked secrets / missing auth / SQL injection / unvalidated input / unsafe file handling / CORS gaps / rate limits / race conditions / weak hashing / env leaks / logging gaps / client-trust bugs).
8 preview-tier audits (1 batch, ~12 files each, ~$0.15/run). 13 full-tier audits (up to 4 batches, ~50 files each, ~$1/run). Total corpus cost: $12.
Severity bar: critical = auth bypass / secrets exposure / SQL injection with user-controlled input / arbitrary file read-write. high = missing rate limit on sensitive endpoint / unvalidated persisted input / dangerous CORS / admin-by-client-flag. medium = weak error handling / theoretical but plausible attack paths / cryptographic oddities.
Findings clustered into pattern buckets via substring matching on titles. Some findings didn't match any bucket ("Other" category, excluded from the top-5 table).
No repo was cherry-picked. First 21 results from the queries that passed the size-and-star filters (size 100 KB - 30 MB, stars < 800) were audited in order. One clone failed with no scannable files (excluded).
Pattern bucket definitions + the full pipeline code are public in the VibeScan repo under scripts/bulk_research_audit.py and scripts/research_aggregate.py — run them yourself if you want to verify or expand the corpus.

— Michael

DEV Community