DEV Community: Novruz Veliev

Lovable Deep Scan vs a manual check: what the scan finds and what it can't see

Novruz Veliev — Mon, 12 Oct 2026 09:15:29 +0000

Originally published on shipclarity.app.

Lovable now ships with its own security scanner. A quick scan runs on every publish, and a Deep scan reviews your whole codebase when you ask for it. Founders often ask me whether that makes a manual check pointless. Short answer: run the scan first, it's free and it's good. Then look at the list below of what it was never built to see.

Everything I say about Lovable's feature here comes from its own documentation and blog, as of October 2026. Lovable changes fast, so if a detail below looks out of date, the docs win.

What Lovable's scans are

There are two built-in scans. Both are free to run and don't use credits.

The Quick scan runs by itself every time you publish and takes about 10 to 15 seconds. It reads your database access rules and settings, including Row Level Security, and checks your npm packages against known vulnerabilities. Only packages rated critical become findings.

The Deep scan does not run by itself. You start it from the security view, and it takes a few minutes. It is an AI review of all your application code. Lovable's docs list what it looks at: access control and authorization, unauthenticated endpoints, unsafe input and injection, leaked secrets, payments and billing, authentication and account security, and exposed personal data. Lovable suggests running it before a public launch, when you handle sensitive data, or after big changes.

Findings come in three levels: Critical, Warning and Info. Some findings can be fixed with one click. Deep scan findings are not auto-fixed, so you fix those yourself or ask Lovable in chat, which costs credits.

Lovable's own docs say these tools find common security issues but "cannot guarantee complete security". That is a fair description, and worth keeping in mind when the scan comes back clean.

What the scan is good at

Credit where it's due. A table with RLS switched off, a policy that lets anyone read everything, a secret key pasted into frontend code, a package with a known critical hole: these are exactly the mistakes AI builders make, and the scan catches them in seconds. It also never gets tired. It reads every file, every time.

The Deep scan goes further than most founders expect. It reads your code with your app in mind, so it can flag an Edge Function that anyone can call without logging in, or a price that comes from the browser and could be changed by the buyer. A human reviewer reading the same code would need hours to do that.

If you have never run it, stop here and run it. Fix what it flags. Paying anyone to find what a free tool finds in four minutes is a waste of money.

Why it can't see some problems

Both built-in scans read. They read code, database rules and package lists. Lovable's docs describe them as static analysis. They don't open your live app, sign up, wait for an email, or log in as anybody.

That matters, because a lot of launch bugs don't live in the code at all. They live in the email provider's settings, in the Stripe dashboard, in a redirect URL saved in Supabase, in what your pricing page promises. Code can look right and the app can still fail when a real person uses it.

To be fair, Lovable also lists optional third-party connectors, including an AI penetration test that sends requests to your running app and tries to break authentication and API flows. That is closer to testing, and worth a look for apps with sensitive data. It is still a security tool. It doesn't ask whether a customer can get through your product.

Seven things only a person with accounts finds

The sign-up email that never arrives

The code calls the email function correctly. The scan is happy. Meanwhile the sending domain isn't verified, or the default sender has hit its limit, and new users sit waiting for a confirmation link. You only see it by signing up with a fresh inbox and waiting. More on this in why Lovable reset emails don't arrive.

The password reset that leads nowhere

Even when the reset email does arrive, the link often opens an old preview address or a page that doesn't handle the token. The user clicks, sees an error, and gives up. Nothing about this is a security flaw, so no security scan will report it. It still loses you customers.

One user seeing another user's data

The scan checks that RLS policies exist and aren't wide open. It can't tell you whether the policy matches what you meant. A policy can be valid and still let a team member see every team, or let a user read rows through a view nobody thought about. The way to know is practical: two accounts, one creates data, the other tries to reach it through the screens, the URL and the API. The do-it-yourself version is in Supabase RLS and the anon key.

Test keys on the live site

The Deep scan reviews payment logic in your code. It doesn't see which Stripe mode your live site actually runs in, whether the live webhook exists, or whether the price IDs in your secrets belong to test mode. Those settings live in dashboards, outside the codebase. The 15-minute check is in Stripe test mode vs live mode.

The pricing page that disagrees with the terms

Your pricing page says "cancel anytime" and the terms say annual plans aren't refundable. The plan card says 5 projects and the app stops you at 3. This is a mismatch between words on two pages and behaviour in a third, and it only shows up when someone reads them side by side and then tries the limit.

The layout on a real phone

A button hidden under the keyboard, a modal you can't close on a small screen, a table that scrolls the whole page sideways. Code review can't predict how Safari on an iPhone renders your page. Someone has to hold the phone.

Broken paths through the app

A link to a page that was renamed two prompts ago. A Back button that drops you on a blank screen. An onboarding step that only works if you click in the expected order. Each piece of code is fine on its own. The path between them is broken, and a path only exists when someone walks it.

The order that makes sense

Run the Quick scan on every publish. It happens anyway, so read what it says instead of clicking past it. Run the Deep scan before launch and after big changes, and fix what it flags.

Then spend an hour as your own first customer. Use a new email address. Sign up, reset the password, pay, and try to open data that belongs to a second account. Do it once on your phone. You'll find some of the gaps above yourself, and that hour is worth more than any tool.

For a sense of what shows up from the outside alone, I scanned 190 live Lovable apps without logging in. Behind the login there is more, and that part needs accounts.

If you'd rather have someone else be the first customer, that manual pass is what I do at ShipClarity: real inboxes, two accounts, test payments, real phones. Run Lovable's scans first. I start where they stop.

This post was written with AI assistance.

Stripe test mode vs live mode: a 15-minute check before your first real payment

Novruz Veliev — Mon, 05 Oct 2026 09:01:15 +0000

Originally published on shipclarity.app.

Stripe gives you two copies of everything: one for testing, one for real money. Keys, products, prices, webhooks, payment links and the customer portal all exist twice. Most launch-day payment bugs I see in Lovable and Bolt apps come from one piece still pointing at the test copy while the rest has moved to live.

The symptoms are quiet. Checkout works for you, because you test with the card that always works. A real customer gets an error, or pays and never gets access, and you hear about it days later, if at all. Below is the check I run before an app takes its first real payment. It takes about 15 minutes and one small real purchase.

Find out which key your live site actually uses

Every Stripe key says which copy it belongs to. Publishable keys start with pk_test_ or pk_live_, secret keys with sk_test_ or sk_live_. The publishable key sits in your frontend, the secret key must only live on a server - in a Lovable or Bolt app that usually means Supabase Edge Function secrets or your hosting's environment variables.

How to check

Open your live site, then DevTools → Sources, and search the loaded JavaScript for pk_test_. If it's there, your checkout still starts in test mode. While you're in there, search for sk_ too: a secret key in the browser is a bigger problem than test mode, and it needs rotating in the Stripe dashboard today.

Then check the server side: in Supabase, Edge Functions → Secrets (or wherever your project keeps them), look at the Stripe secret. It's common to switch the publishable key and forget the secret one, or the other way round. Both have to say live.

Prices and products don't move to live by themselves

A product or price you created in test mode does not exist in live mode. Its ID (price_...) is only valid in the copy where it was created. If your code or your AI builder hardcoded test price IDs, live checkout fails with an error like "No such price", even though the key is right.

How to check

Stripe dashboard, test mode switched off → Product catalog. Every plan your pricing page sells should be there with the same amount, currency and interval. Then compare the price IDs your app uses (search your code or Edge Functions for price_) with the IDs shown in live mode. They will be different strings - that's expected - and every one in your code must be the live one.

Check the amounts while you're there. A $19 plan in test mode and a $190 plan in live mode, or monthly in one and yearly in the other, is a mistake that nobody catches with a test card.

Webhooks: a separate endpoint and a separate secret

Webhooks are how Stripe tells your app "this person paid". They're set up per mode. A webhook endpoint you added in test mode receives nothing from live payments, and its signing secret (whsec_...) is different from the live one. The result is the worst kind of bug: the customer is charged, and your app never unlocks their account.

How to check

Live mode → Developers → Webhooks. There must be an endpoint pointing at your production URL, listening to the events your app uses (typically checkout.session.completed, and the subscription events if you sell plans). Copy its signing secret and make sure your server uses that one, not the test one. After the real purchase below, open the endpoint and check that the event shows as delivered, not failed.

Payment links and the customer portal

If you sell through Stripe Payment Links, look at the link behind every "Buy" button. Test links contain test_ in the address, and the page they open shows a "Test mode" badge. A live site with a test payment link takes no money at all.

The customer portal - where subscribers update a card or cancel - is also configured separately for each mode. If "Manage subscription" opens an error for a real customer, the portal was only set up in test mode.

One real purchase, then a refund

This is the step people skip, and the only one that proves the whole chain. Buy your cheapest plan with a real card, as a new user. Then check four things: the charge appears in the live dashboard, the webhook shows as delivered, your app gave the account what it paid for, and the receipt email arrived (Stripe doesn't send receipts for test payments, so this is the first time you'll see one). Refund yourself from the dashboard when you're done.

One more quick test: on your live checkout, try the test card 4242 4242 4242 4242. It should be declined. If it goes through, your live checkout is still running in test mode.

After the switch: keep testing without touching live

Once you're live, keep test mode (or a Stripe sandbox) for everything you try next, with its own keys in a staging copy of the app. The mistake to avoid is the reverse of launch day: switching the live app back to test keys "for five minutes" to debug something, and forgetting. If you ever do, repeat the key check before you log off.

Want someone to run the money path with you?

The checks above cover the switch itself. What they don't cover: upgrades and downgrades between plans, failed renewals, coupons, refunds that should remove access, and what a second user sees after the first one pays.

The payment path, test and live, is one of the manual checks in a Pre-Launch Audit, done with real test accounts and screen recordings of every step.

This post was written with AI assistance.

What breaks in Lovable apps: data from 190 live apps

Novruz Veliev — Tue, 22 Sep 2026 09:19:18 +0000

Originally published on shipclarity.app.

I ran the same outside-in scan against 190 live apps listed as built with Lovable. No login, nothing a normal visitor's browser could not see. 189 of them had at least one flagged issue. Here is what came up most, what almost never came up, and a two-minute self-check for each.

Aggregate data only: no app names or domains are published, and none will be. The scan ran on 11 September 2026.

The numbers

189 of 190 (99%) had at least one flagged issue (average 4.1, median 4, out of 17 checks)
179 (94%) send no Content-Security-Policy header
172 (91%) send no X-Frame-Options header
150 (79%) answer 200 OK for a page that does not exist
127 (67%) have no DMARC record, or one set to p=none
3 ship a Google/Firebase API key in the bundle; none shipped a Supabase service_role key or a Stripe live secret key
0 exposed a .env, .git or backup file, and 0 of 125 probed Supabase projects returned their schema to the public key

How the data was collected, and what it cannot see

The starting point was a public directory of projects built with Lovable: 512 listings. 428 of them were still live, and 395 of those were in English. After removing portfolios, games, agency pages and pages with no product surface (no pricing, payments, or signup), 190 apps were left, all scanned without errors. 155 run on their own domain; 35 are on a platform subdomain such as *.lovable.app. Some have since moved to Vercel or Netlify, so read this as "built with Lovable", not "hosted by Lovable".

Each app got the same 17 automatic checks that the free ShipClarity scanner runs: exposed config and repo files, secret keys in the JS bundle, Supabase schema exposure to the public key, source maps, SPF, DKIM and DMARC, HTTPS and the HTTP redirect, four security headers, the viewport tag, robots.txt and noindex, the status of a missing page, and first-response time.

What this data cannot tell you

The scanner never logs in. Signup, password reset, one user reading another user's rows, checkout and refunds are not measured here.
It does not run JavaScript. It reads the server HTML and up to 8 script files referenced from the home page, so a key in a lazily loaded chunk would be missed.
Headers are read from the home page only. A CSP set through a <meta> tag is counted as missing.
DKIM cannot be confirmed from outside (the selector is provider-specific), so there is no DKIM number here.
The sample is a directory of projects whose makers chose to list them, not a random sample of every Lovable app.

1. 189 of 190 had at least one flagged issue

Only one app out of 190 came back clean. The average was 4.1 flagged checks out of 17, the median 4, the worst 8. Most of that count comes from the four items below: configuration defaults nobody revisited, not code bugs. "99% have issues" does not mean 99% are broken for users; it means almost nobody went back over the defaults.

2. 94% send no Content-Security-Policy

179 of 190 apps send no Content-Security-Policy header. CSP tells the browser which scripts may run on your page. Without it, any injected script (a compromised third-party widget, or unescaped user content) runs with full access to the page, including whatever the logged-in user can see.

It is a hosting or proxy setting, not app code, which is why it is missing almost everywhere.

Check yours in 2 minutes

curl -sI https://yourapp.com | grep -i content-security-policy

No output means no header (or check DevTools, Network tab, first request, Response Headers). If your host does not let you set headers, a proxy in front of the domain (Cloudflare, or Netlify/Vercel header rules) can add them. Start with Content-Security-Policy-Report-Only so a strict policy does not break your own scripts on day one.

3. 91% send no X-Frame-Options

172 of 190 apps send no X-Frame-Options header. 171 of them send no CSP either, so there is no frame-ancestors rule covering the gap. Another site can load the app in an invisible iframe and trick a logged-in user into clicking something real: a delete button, a plan change, a share toggle.

From the same family: 44 of 190 (23%) lack X-Content-Type-Options: nosniff, and 23 of 190 (12%) lack HSTS.

Check yours in 2 minutes

curl -sI https://yourapp.com | grep -i -E "x-frame-options|frame-ancestors"

You want X-Frame-Options: DENY (or SAMEORIGIN) or a CSP with frame-ancestors 'none'. frame-ancestors is ignored in a <meta> CSP tag; it only works as a response header.

4. 79% return 200 for a missing page

150 of 190 apps answered 200 OK when asked for a random URL that does not exist. It is the single-page-app default: the host serves index.html for every path and the front-end router decides what to show. Visitors may see "not found", but search engines and uptime monitors see a healthy page.

Search engines may index junk URLs as soft 404s, and monitors keep reporting "up" for deleted routes.

Check yours in 2 minutes

curl -s -o /dev/null -w "%{http_code}\n" https://yourapp.com/this-page-does-not-exist

If it prints 200, open the URL in a browser too: a blank page or the home page is worse than a "not found" screen. The minimum fix is a catch-all route that shows a clear not-found page and adds <meta name="robots" content="noindex">. A real 404 status needs a host-level rule.

5. 67% have no enforcing DMARC

127 of 190 apps have either no DMARC record at all (50) or one set to p=none (77), which blocks nothing. All 127 are on their own domain: 82% of the 155 custom domains. The 35 on a platform subdomain inherit its record. And 52 custom domains (27% of all apps) have no SPF record either.

Without an enforcing policy, anyone can send email as your domain, and your own signup and password-reset emails are more likely to land in spam. For the second part there is a separate guide: Lovable password reset email not arriving.

Check yours in 2 minutes

dig TXT _dmarc.yourdomain.com +short          # macOS / Linux
nslookup -type=TXT _dmarc.yourdomain.com      # Windows

No answer means no record; p=none is monitor-only. The target is p=quarantine or p=reject, but only after SPF and DKIM are set up for the provider that sends your mail, or you will block your own emails.

6. 3 apps ship an API key in the bundle

3 of 190 apps had a key matching a known secret pattern in the JavaScript sent to every visitor. All three were Google/Firebase API keys (AIza...). This type of key is designed to sit in client code and is fine if restricted by HTTP referrer and by API in the Google Cloud console. An unrestricted one can be used on someone else's site at your expense.

The more useful number is the zero: none of the 190 shipped a Supabase service_role key, a Stripe live secret key (sk_live_), or an AWS access key in the scanned bundles, the keys that would hand over the database or the payment account. The Supabase anon key is public by design and was not counted.

Check yours in 2 minutes

In DevTools on your live app, press Ctrl+Shift+F (Cmd+Option+F on a Mac) and search all loaded files for service_role, sk_live_, AKIA and AIza. A hit on the first three means rotate the key now and move the call server-side. A hit on AIza means confirm in Google Cloud Console, Credentials, that the key has both a referrer and an API restriction.

7. What held up

The scary items were rare. Out of 190 apps:

0 exposed a .env, .git/config, config or backup file at a guessable path.
190 load over HTTPS. 3 also still answer on plain HTTP without redirecting.
140 (74%) use Supabase. For 125 of them the scanner could test the public key against the database's REST root, and none returned the schema. The other 15 could not be probed.
3 publish source maps in production, 3 have the home page set to noindex, and 19 (10%) serve no robots.txt.

A closed schema endpoint is a good sign, not a clean bill of health. Whether user A can read user B's rows depends on each table's row-level security, testable only with two real accounts.

Check yours in 2 minutes

curl -s -o /dev/null -w "%{http_code}\n" https://yourapp.com/.env
curl -s -o /dev/null -w "%{http_code}\n" http://yourapp.com/

The first should not be 200 with real file contents (a 200 with your normal app page is the soft 404 from point 4, not a leak). The second should be a 301 or 308 redirect to HTTPS.

What to take from this

Almost everything above is configuration: headers, DNS records, a catch-all route. An afternoon covers it. None of it says whether the app works for a paying user. The failures that cost customers live behind the login: a signup that silently fails, a reset link pointing at the old preview domain, one account seeing another's data, a checkout that charges twice. This dataset cannot say how common those are.

If you want to see where your own app stands, the free outside scanner runs the same 17 checks on your URL in about 20 seconds, with no signup.