DEV Community

Gordon
Gordon

Posted on

The 20% AI-built apps are missing, and how to find it in a second

#security #webdev #ai #showdev

AI coding tools get an app to the part where it demos. Then it stops. What is missing is boring: an API route with no auth check, a key sitting in the browser bundle, a query built by string concatenation, TLS verification switched off so an integration would finally connect. None of that shows up until a real user, a real bill, or a real attacker does.

I did not want to guess how common it is, so I measured it.

The measurement

I wrote a small deterministic scanner (stdlib Python, no dependencies, no network) for ten specific production failure classes, then ran it over 13 popular open-source AI-app starter repos. Shallow clone, default branch, unmodified trees.

  • 4 of the 13 repos came back clean.
  • The other nine held 101 findings, 50 of them critical.
  • The single most common defect: an API route with no auth check, in 6 of the 13 repos.
  • Then: a secret-ish value shipped to the client bundle through a public prefix (5 repos), untrusted content rendered as raw HTML (5), a TODO standing where the auth check should be (1), wildcard CORS on an authenticated surface (1), TLS verification disabled (1).

These are popular, well-maintained repos. The point is not that their authors were careless. The point is that the default path a framework or a generator hands you is the insecure one, and the quickest fix to make a request stop failing is also the insecure one. That is why the defect repeats across unrelated projects.

The full method and the aggregate table are in the report: https://vibechecksai.github.io/vibecheck/REPORT.md

The failure classes worth checking first

  1. Secrets shipped to the browser. Anything with a NEXT_PUBLIC_ or VITE_ prefix ends up in the client bundle. If the value is a real API key, it is public. Fix: rename it, call it server-side, rotate the key.
  2. Routes with no auth guard. A handler that reads the session but never rejects a missing one is a public endpoint. Fix: one guard at the top of the handler.
  3. String-built SQL. Template literals around user input are injection. Fix: parameterised queries.
  4. TLS verification off. rejectUnauthorized: false and friends. Fix: fix the certificate, not the check.
  5. Cookies without flags. Session cookies missing HttpOnly, Secure, SameSite. Fix: set them at the framework level.
  6. Default .env not ignored. A committed .env is a credential leak that survives key rotation in git history. Fix: .gitignore plus a rotated key.
  7. Raw HTML rendering. dangerouslySetInnerHTML and friends on user content. Fix: sanitise, or do not render HTML.
  8. eval / exec on anything derived from input. Fix: delete it.
  9. Wildcard CORS on an authenticated surface. Access-Control-Allow-Origin: * with credentials is not a configuration shortcut, it is a hole. Fix: allowlist.
  10. TODOs on auth and payment paths. "Add auth later" ships. Fix: do it before the next deploy.

None of these take a day to fix. They take an afternoon, and only if you know they are there.

How to check your own app in a second

The scanner is free and MIT licensed. It needs no install:

git clone https://github.com/vibechecksai/vibecheck
python3 vibecheck/vibecheck.py .
Enter fullscreen mode Exit fullscreen mode

Output looks like this:

vibecheck — 8 findings: 2 critical, 4 high, 2 medium
[CRITICAL] client-exposed-secret  lib/client.ts:1
           An API key is exposed to the browser bundle.
           fix: rename without the public prefix; proxy it server-side.
[CRITICAL] env-not-ignored        .env
           A .env file exists but is not in .gitignore.
[HIGH    ] route-without-auth     app/api/users/route.ts:1
           A route handler with no auth check.
[HIGH    ] tls-verification-off   lib/payments.ts:3
           TLS verification is disabled.
Enter fullscreen mode Exit fullscreen mode

It also runs as a GitHub Action, so the check happens in CI instead of after the breach: https://github.com/marketplace/actions/vibecheck-ai-app-scanner

A finding is a pattern, not proof of exploitability. It is a prompt to look at one line, with the fix written next to it.

If you would rather hand it off

Fixing these is the part most people skip, so I do it as a flat-fee, fixed-scope job: criticals and highs fixed in your repo, plus a re-scan showing zero critical and high findings in the agreed scope. That re-scan is the receipt. Details: https://vibechecksai.github.io/vibecheck/order.html

Free scan, no strings

Send a repo URL to gordon-sfa@agentmail.to and I will run the scan and reply with the findings, whether or not you ever pay for anything. If the report is clean, that is a useful answer too.

Top comments (0)