I filtered Bandit's false positives with a deterministic gate -- and published where it fails

Blacktrace Foundry — Tue, 14 Jul 2026 05:10:27 +0000

Scanner noise isn't an annoyance. It's a mechanism: after enough files you stop reading the flags, and that's exactly how the one real finding ships. AI-generated code makes it worse — fluent code at volume means more flags, faster numbness.

I built a gate for that, then did something most tool posts don't: I pre-registered six hypotheses before the data existed, and I'm publishing the three that failed alongside the two that held.

What the gate is

Noise Eraser is a fixed AST oracle. Paste flagged Python; it keeps findings in exactly 8 CWE classes and drops everything else:

eval/exec (CWE-95)
unsafe deserialization — pickle, yaml.load (CWE-502)
string-built SQL (CWE-89)
shell injection — shell=True, os.system (CWE-78)
TLS verification bypass (CWE-295)
Flask debug server (CWE-489)
weak hash over a credential (CWE-327)
hardcoded credentials (CWE-798)

KILL keeps the alert. PASS drops it. No LLM, no temperature: same input, same verdict, same SHA-256 receipt, every run. It runs in your browser via Pyodide — your code never leaves the page.

Honest scope, stated before the numbers: this is a focused checker, not a scanner. PASS means "none of my 8 classes fired," not "safe." I would rather it undersell.

The part that failed

On 304 real, cited Python CVEs, the gate catches 0.41 at a 0.06 false-kill rate. The three hypotheses that needed that catch rate to clear a model's discrimination — H1, H2, H3 — are all NOT PROVEN as registered, and the page says so.

Owning the miss properly: on the held-out test split, gate-v1 caught 50 of 122 real findings and missed 72 — 59%. The misses are named, by CVE and CWE class, in the repo. The biggest holes: CWE-502 and CWE-94 (12 each), CWE-798 (9), CWE-22 and CWE-295 (8 each). Path traversal (CWE-22) isn't one of my 8 classes, and the split shows what that costs.

Why publish this? Because a 41% catch with the misses named is worth more than an unfalsifiable 99%. Catch is a commodity — every scanner and every LLM can flag things. False-kill is the scarce good.

The part that held

On real production code (H4): 20,147 parseable files from 173 of the top-200 PyPI packages plus stdlib. The gate false-kills 4.14%, CI [3.87, 4.42]. And not just the mean — the full per-package distribution is published: 76 packages at exactly 0% false-kill, and a small-N tail (a 2-file package with 1 flag reads as 50%) flagged as noise rather than hidden.

Chained after Bandit (H5), which is the actual deployment: Bandit alone false-kills 35.78% on this corpus. Bandit filtered through the gate: 3.70%. That removes 89.67% of Bandit's flags at zero LLM calls and zero marginal cost.

One correction I had to make against my own pre-registration: it cited a secondhand "94–98%" band for LLM-hybrid FP-reduction. Checking the primary papers, QASecClaw reports 88.6% and SAST-Genius ~91% — the confirmed band is 86–92%, and the higher figure didn't check out, so it's dropped. My 89.67% sits inside the confirmed band, deterministically.

The cost frame, so you can read your own row: at a false-kill:miss weight of 1:1 the model wins. At the registered 5:1 the gate wins. The crossover is w* = 2.946 — if a blocked clean deploy costs you more than ~3 missed findings, the deterministic gate beats a 70B model on this corpus.

One file, traced end to end

redis-8.0.1/redis/connection.py. Bandit flags it 8 times — all B110, try/except/pass, lines 954 through 3711. The gate drops all 8.

The exact why: a swallowed exception is a style smell, not an injection, deserialization, or credential-exposure primitive. It's not one of the 8 declared classes, so it clears — by declaration, not by bug. That's the whole design: a scope stated in writing, enforced the same way every run.

The condition that would kill this

Stated before anyone asks: the headline dies — not gets qualified, dies — if a fresh, frozen, unmodified sample of real production Python reproduces a false-kill rate whose 95% CI does not overlap [3.87%, 4.42%], or if the filter is shown silently dropping a confirmed true-positive flag on an actually-vulnerable file. Either one kills the number, not a footnote.

The metric itself is reproducible in under 5 seconds, no key, no deps — ./repro.sh recomputes the published CIs from the published counts. If they match, the statistics aren't fudged.

Break it

Demo (free, in-browser, nothing installed): https://blacktrace.co/noise-eraser
The method, the nulls, the miss list: https://blacktrace.co/kruc

Paste something it gets wrong — a PASS on a real issue inside its 8 classes, or a KILL on something out of scope. That's the only signal I care about.

Disclosure: the same gate exists as a paid CI version — pre-commit hook + GitHub Action, $39/mo per repo. The browser gate is free and stays free.