DEV Community: AuditAI

We applied 243 generated Supabase security migrations to real schemas. Here is what broke.

AuditAI — Thu, 24 Sep 2026 07:29:58 +0000

Our free check reads a snapshot of a Supabase database and, where the fix follows from the schema, writes the migration that closes the hole, plus the lines that undo it. Until this week we had only tested those migrations with unit tests. So we ran them against real schemas.

The setup

155 public Next.js + Supabase repositories from our earlier precision samples.
A database built the way a new Supabase project has it: the supabase/postgres image, then GoTrue's own migrations (auth migrate) and one start of storage-api. Without those you have no auth.jwt(), no storage.objects, and, most important, no Supabase default privileges, so every new table looks private when it is not.
Each repository's supabase/migrations applied in order as postgres, the way the CLI does. 59 of 133 rebuilt cleanly; most of the rest reference a table no migration creates.
Everything ran on a Docker network with no route out, since migrations can call pg_net or http.

For every finding with a migration, on its own copy of the database:

as the stranger the finding names (anon, or any signed-in user), try what it claims: read the table, add a row, call the function;
apply the fix as postgres;
snapshot again and re-run the check: the finding must be gone and nothing new may appear;
try again as the stranger;
apply the rollback and compare the snapshot with the first one.

Numbers


Findings with a migration	243 (on 34 databases)
Applied without error	243
Gone on re-check, nothing new	243
Rollback restored the same access	243
All fixes of a project in one file	24 projects, 0 errors, 0 findings left

Before each fix, the stranger really could do what the finding said in 239 of 243 cases: 152 of 152 SECURITY DEFINER functions ran for anon or a signed-in user, 34 of 34 tables without RLS were readable and writable.

What broke

A rollback handed out a right that was never there. Our fix for an open SECURITY DEFINER function revokes EXECUTE from public, anon, authenticated and grants it to service_role. The rollback treated that last grant as a no-op, because Supabase's default privileges give service_role EXECUTE on every new function. One project had revoked it on purpose. After "undo", the service role could run three functions it could not run before. The rollback now takes it back when the snapshot shows it was not held.

Two cards, one line. A function with two overloads produced two findings that quoted the same fact line. They now show the argument list.

Four headlines named the wrong stranger. They said "anyone can change every row" for an UPDATE policy open to anon. Through the Data API, anon could change nothing: Supabase's authenticator role preloads safeupdate, so an UPDATE needs a WHERE, and a WHERE that reads a column makes Postgres apply the SELECT policies too. These tables had no read policy for anon, so the update matched no rows. In three of the four, signed-in users could read the table, so the true headline is "anyone who signs up can change every row". The rule now works out who can change which rows from what they can read.

What this is not

Re-checked is not verified. We did not run anyone's app, and a security invoker fix can break an app that relied on the owner's rights. That is what the rollback lines are for.

The check is free at https://auditai.sh/check?utm_source=devto&utm_medium=social&utm_campaign=fix-loop. It reads names and rules, never a row of your data.

Three blind tests of our security scanner: 36%, 46%, 31%

AuditAI — Mon, 14 Sep 2026 06:31:03 +0000

Most security tools publish detection rates measured on benchmarks their own authors built. We wanted a number we could not quietly improve by tuning, so we measure Audit AI's scanner blind, on repositories it has never seen. We have run that test three times. This post is the protocol, all three results, and what they taught us.

What the scanner reports

Audit AI reads a Next.js + Supabase codebase together with its SQL migrations and reports authorization bugs: a service-role query filtered only by an id from the URL, a table without row level security, a write policy open to anon, a SECURITY DEFINER function that never checks its caller. Every finding is a claim about code, so every finding can be checked by reading that code.

The protocol, committed before each sample

Before choosing a single repository, we commit the protocol to our repository:

Selection. Public Next.js + Supabase repositories from GitHub code search, taken in the order the search returns them, excluding copies and anything in a corpus we had already tuned the engine on.
Size. 20 repositories and 100 findings, with at most 20 findings from one repository in the later samples.
Sampling. A seeded pseudo-random draw from all findings. The sample is reproducible, and nobody picks the interesting ones.
Labels. A person reads each finding in the code and marks it real, false positive or unsure. The rules for edge cases are written before labelling. Unsure findings stay out of the denominator.

The results

Sample	Real / decided	Precision	95% interval
First	36 / 100	36%	27–45%
Second	44 / 95	46%	37–56%
Third	31 / 99	31%	23–41%

Intervals are Wilson intervals. The drop from the second sample to the third is statistically significant (p ≈ 0.03). The difference between the first and the third is within noise (p ≈ 0.48).

Each build had fixes made after reading the previous sample's labels, and the new repositories still moved the number more than any of those fixes did. That is the main lesson: a precision figure measured on one set of repositories says little about the next set.

Where the scanner was right, and where it was not

SECURITY DEFINER functions without a caller check were real in 12 of 16, 12 of 30 and 18 of 42 findings. In the second sample, twelve of the false positives were functions in one repository that expose public data on purpose.
Write policies open to anon were real in 16 of the 20 findings across all three samples.
Service-role object access without a tenant scope was the weakest rule: real in 19 of 50, 2 of 18 and 1 of 22. Most false positives were apps where the tenant check lives in code or a sign-in the scanner does not follow.
The third sample was concentrated: three repositories gave 60 of its 100 findings, with precision of 75%, 15% and 15%.

What we do with it

A finding in Audit AI stays likely until a sandbox reproduces it: two synthetic tenants, the real request, and a check that the owner's own access still works. Only then is it confirmed, and a fix counts as verified only if the same attack fails afterwards.
Every number is on a public page, including the one that went down: https://auditai.sh/stats?utm_source=devto&utm_medium=social&utm_campaign=blind-precision
Any fix made after reading these labels makes them no longer blind. The next honest number needs new repositories again.

Try it

The scanner is open source under Apache-2.0: https://github.com/audit0/auditai-scanner

You can also paste a public repository at https://auditai.sh/?utm_source=devto&utm_medium=social&utm_campaign=blind-precision and press Prove it on a finding. If it gets something wrong on your code, an issue with the smallest snippet that reproduces it is the most useful thing you can send us.

The bug every AI coding tool ships, and how to prove it is gone

AuditAI — Sat, 12 Sep 2026 17:24:38 +0000

The scanner is open source and the rules are readable before you trust a single finding: github.com/audit0/auditai-scanner. This post is about the one bug it was built for.

The shape of the bug

Ask any AI coding tool for an invoicing app and you will get something close to this route handler.

// app/api/invoices/[id]/route.ts
import { createClient } from "@supabase/supabase-js";const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_SERVICE_ROLE_KEY!);

export async function GET(_req: Request, { params }: { params: { id: string } }) {
  const { data } = await supabase.from("invoices").select("*").eq("id", params.id).single();
  return Response.json(data);
}

It works. It passes review if the reviewer is reading for "does this return an invoice". The service-role key bypasses row level security, and the query filters on id alone, so Alice can read Bob's invoice by changing a number in the URL.

This is not an exotic bug. It is the default outcome of asking for a feature and not asking who is allowed to see it, and it shows up over and over in apps built with Lovable, Bolt, v0, Cursor and Claude Code.

Why a warning is not enough

A scanner that prints "possible authorization issue" at this line has told you almost nothing. You still have to open the file, work out whether the route is reachable, whether some middleware already blocks it, and whether the id is scoped somewhere you did not read. Most teams do that work once, find two false positives, and stop reading the tool's output.

"The model says it fixed it" is worse. A model that both writes the fix and grades the fix is not evidence of anything.

So we picked a harder bar. A finding stays likely until something outside the model reproduces it.

What reproduction looks like

We copy the app into a sandbox with no internet access, seed two synthetic tenants, and send the same request twice.

proof · AUDIT-001
GET /api/invoices/42 as Alice
✗ before fix: 200 OK
✓ after fix:  403 Forbidden

Alice is not supposed to see invoice 42. Before the fix the endpoint hands it over. That is the evidence. It is also, conveniently, a test.

The fix and the regression test

The fix is the smallest one that closes the path, not a refactor:

-const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_SERVICE_ROLE_KEY!);
+import { createServerClient } from "@/lib/supabase/server"; export async function GET(_req: Request, { params }: { params: { id: string } }) {
-  const { data } = await supabase.from("invoices").select("*").eq("id", params.id).single();
+  const supabase = await createServerClient();
+  const { data, error } = await supabase.from("invoices").select("*").eq("id", params.id).single();
+  if (error || !data) return new Response("Forbidden", { status: 403 });
   return Response.json(data);
 }

The request now runs as the signed-in user, so the table's row level security policy decides the answer instead of the route.

The regression test is the reproduction, kept:

it("does not let one tenant read another tenant's invoice", async () => {
  const res = await fetch(`/api/invoices/${bobInvoiceId}`, { headers: aliceAuth });
  expect(res.status).toBe(403);
});

It failed before the fix. It passes after. If someone reintroduces the service-role client next quarter, it fails again.

The honest part

Every result ends with a coverage line rather than a score:

checked 12 routes · verified 2 · confirmed 1 · unverified 0

unverified is not a failure to hide. It is the set of findings we could not reproduce, and we would rather show you the number than dress it up. Findings move from candidate to likely to confirmed only on evidence, and a fix is verified only when the regression test failed before it and passes after it, the existing suite still passes, and a deterministic rescan no longer sees the path.

We measure precision on repositories we have never seen, and publish the number instead of claiming one.

Try it

npx auditai-scan .

It runs locally, needs no account, and sends nothing anywhere. Source, rules and the eval fixtures are here: github.com/audit0/auditai-scanner. The hosted product that reproduces, fixes and proves is at auditai.sh.

If you have a Next.js + Supabase app, scan it now at https://auditai.sh: no sign-up, no waitlist. For a private repository or a fix proven on your own app, write to hello@auditai.sh.