A wave of free scanners for AI-built apps will grade your live URL in under a minute. They're useful. But read their own pages and they're mostly answering one question: what can a stranger grab without logging in?
Whether your app actually works for a real user is a different test.
Two different questions
Security scan: "What can a stranger take?" It reads what your site sends to any browser and makes a few harmless requests: keys in the JavaScript, database tables open to the public key, downloadable .env or .git files, API routes that answer without login, security headers. Logged out, no writes.
Functional test: "Does it work for a user?" It uses the app the way a customer would: sign up, log in, create and edit data, tap through it on a phone, log out. It finds the broken button, the stuck sign-up and the page that crashes on an emoji. It can't see a secret key sitting in your code.
A security scan can give an A to an app whose sign-up is broken, and a functional test can pass an app that leaks a key. You need both.
What five scanners say they check
On 20 Oct 2026 we read the pages of five scanners aimed at vibe-coded apps: PreflightVibe, PreflightX, Prodable, Shippable and SafeWeave. This compares methods only (what each page says), not quality. We haven't run them on a test app, and none of them paid us.
- They overlap heavily on secrets and open data: keys in shipped code, Supabase/Firebase tables or buckets readable without login, API routes that answer without a session. Good, because those leaks expose everyone at once.
- None of their pages says it tests sign-up, saving data or the phone layout. The ones that say whether they log in say they don't.
- PreflightVibe is the most direct: it says it can't test what one user can see of another user's data, and tells you to do a two-account test yourself.
Full matrix (13 questions × 6 tools, with prices as listed and links to each source page): Security scan vs functional test.
Disclosure: we run one of the tools in that matrix (a free functional "Scan my app" beta). It has the same blind spot: it never logs in and blocks every write, and it doesn't look for keys or open tables.
What our hands-on test found that no logged-out scan can see
On 16 Oct 2026 we built the same multi-user to-do app with Lovable, Bolt, Base44 and v0 on free plans (one run each) and tested every app by hand with real accounts. The problems we found:
- Buttons that only exist for a mouse (Bolt). Edit and delete appeared only on hover, so on a phone they were invisible. Nothing leaked, so a security check has nothing to find.
- New users stuck at sign-up (Base44). A new user who closed the email-code page found no way to enter or resend the code from the login screen. You only see this by signing up, closing the tab and coming back.
- Logging out didn't end the session (Base44). The old access token still worked on the server after logout. A security bug, but a logged-out scan never holds a token to replay.
- Very short passwords accepted (Lovable). 6-character passwords went through. Finding it means actually creating an account.
None of these was a leaked key or an open table. And on the check that matters most, all four apps passed: a second user couldn't see or change the first user's data. That's one run of a simple app, not a promise about yours.
What to run before you launch
- One outside security scan on your live URL, for secret keys, tables readable with your public key and exposed files. Fix anything critical first.
- A functional pass over your pages: phone layout, buttons, junk input in forms, console and network errors.
- The logged-in checks, by hand: sign up as a brand-new user (badly), log out and replay the old session, and the two-account test (about 10 minutes with two browser windows).
- Again after every big change. Each new prompt to your AI builder can quietly undo a fix.
The step-by-step version, with a receipt for each step where we have one: 12-step pre-launch checklist for AI-built apps.
Originally published on VibeTierList, where we test AI app builders on the same task and publish the receipts. Paying never changes a ranking.
Top comments (1)
The split between what a stranger can take and whether it works for a user is the useful framing, and the disclosure that you run one of the tools in the matrix makes the comparison easier to trust. The logout finding is a good example of why the two kinds of test overlap: the token replay is a security bug that only shows up after a real login. For anyone using a logged-out scan, it helps to state in the report what it did not exercise (sign-up, writes, a second account), so a clean grade reads as a narrow result, not an all-clear.