Three blind tests of our security scanner: 36%, 46%, 31%

Most security tools publish detection rates measured on benchmarks their own authors built. We wanted a number we could not quietly improve by tuning, so we measure Audit AI's scanner blind, on repositories it has never seen. We have run that test three times. This post is the protocol, all three results, and what they taught us.

What the scanner reports

Audit AI reads a Next.js + Supabase codebase together with its SQL migrations and reports authorization bugs: a service-role query filtered only by an id from the URL, a table without row level security, a write policy open to anon, a SECURITY DEFINER function that never checks its caller. Every finding is a claim about code, so every finding can be checked by reading that code.

The protocol, committed before each sample

Before choosing a single repository, we commit the protocol to our repository:

• Selection. Public Next.js + Supabase repositories from GitHub code search, taken in the order the search returns them, excluding copies and anything in a corpus we had already tuned the engine on.
• Size. 20 repositories and 100 findings, with at most 20 findings from one repository in the later samples.
• Sampling. A seeded pseudo-random draw from all findings. The sample is reproducible, and nobody picks the interesting ones.
• Labels. A person reads each finding in the code and marks it real, false positive or unsure. The rules for edge cases are written before labelling. Unsure findings stay out of the denominator.

The results

Sample	Real / decided	Precision	95% interval
First	36 / 100	36%	27–45%
Second	44 / 95	46%	37–56%
Third	31 / 99	31%	23–41%

Intervals are Wilson intervals. The drop from the second sample to the third is statistically significant (p ≈ 0.03). The difference between the first and the third is within noise (p ≈ 0.48).

Each build had fixes made after reading the previous sample's labels, and the new repositories still moved the number more than any of those fixes did. That is the main lesson: a precision figure measured on one set of repositories says little about the next set.

Where the scanner was right, and where it was not

• SECURITY DEFINER functions without a caller check were real in 12 of 16, 12 of 30 and 18 of 42 findings. In the second sample, twelve of the false positives were functions in one repository that expose public data on purpose.
• Write policies open to anon were real in 16 of the 20 findings across all three samples.
• Service-role object access without a tenant scope was the weakest rule: real in 19 of 50, 2 of 18 and 1 of 22. Most false positives were apps where the tenant check lives in code or a sign-in the scanner does not follow.
• The third sample was concentrated: three repositories gave 60 of its 100 findings, with precision of 75%, 15% and 15%.

What we do with it

• A finding in Audit AI stays likely until a sandbox reproduces it: two synthetic tenants, the real request, and a check that the owner's own access still works. Only then is it confirmed, and a fix counts as verified only if the same attack fails afterwards.
• Every number is on a public page, including the one that went down: https://auditai.sh/stats?utm_source=devto&utm_medium=social&utm_campaign=blind-precision
• Any fix made after reading these labels makes them no longer blind. The next honest number needs new repositories again.

Try it

The scanner is open source under Apache-2.0: https://github.com/audit0/auditai-scanner

You can also paste a public repository at https://auditai.sh/?utm_source=devto&utm_medium=social&utm_campaign=blind-precision and press Prove it on a finding. If it gets something wrong on your code, an issue with the smallest snippet that reproduces it is the most useful thing you can send us.