My Scanner Missed 93% of the Bugs — and That Was the Right First Result

The first time I ran my vulnerability scanner against the industry-standard
benchmark, the bottom line of the scorer's report was this:

$ python scripts/score_benchmark.py --findings out/java.findings.json \
        --truth benchmark-java/expectedresults-1.2.csv

OVERALL   precision 0.60   recall 0.07   F1 0.13    # abridged

Three numbers, and here is what each one means. Precision 0.60 — of all
the alarms the scanner raised, 60% pointed at real bugs: when it spoke, it
was right more often than not. Recall 0.07 — of all the real bugs in the
benchmark, it found 7%. In the four vulnerability classes my scanner covers,
the benchmark contains 777 real, labeled vulnerabilities; it missed 93% of
the bugs it exists to find. F1 0.13 — precision and recall combined into
one score (their harmonic mean), dragged down to almost nothing by that
recall.

My first instinct was to fix it before anyone saw it. Instead I saved the
output, wrote the number into my benchmark log, and kept it — because that
number was always going to be published, and this is the article that
publishes it.

The Context

For the past months I've been deep in AI — reading, building, measuring. One
of the projects that came out of it is an AI vulnerability scanner. The
design in one sentence: deterministic static-analysis rules do
all the searching, and an LLM judges each finding — is this a real bug or a
false alarm? The full architecture gets its own article. This one is about
the first measured number.

The test set is the OWASP Benchmark — 2,740 labeled Java test cases, the
standard exam for Java security scanners. In my scanner's four vulnerability
classes (SQL injection, command injection, path traversal, XSS) there are
1,478 cases: 777 real vulnerabilities and 701 cases deliberately designed to
bait scanners into raising false alarms. Every tool I compare against —
Semgrep, CodeQL — takes the same exam, scored by the same scoring code. Same
rules for everyone.

---

New to this? Three words carry this article. A source is where
untrusted input enters a program (request.getParameter("id") — anything an
attacker can type). A sink is where that input becomes dangerous
(executeUpdate(sql) — running it as a database command). A vulnerability
is data flowing from a source to a sink without being cleaned on the way;
that flowing data is called tainted, and tracking it is taint
analysis — the scanner's whole job is finding those flows. Already know
all that? Skip this box.

Why 0.07 Was the Right First Result

Here's what I had actually run: a spike. A deliberately minimal first version —
one source pattern, getParameter, wired to a handful of sinks, pushed
end-to-end through the whole pipeline: parse 2,740 test cases into a code
graph, run the taint queries, emit findings, score them against the answer key.

The spike's job was never to score well. Its job was to answer one simple
question: does the machinery work at all? And the ugly number, read
carefully, answered it:

• Precision 0.60 — when the scanner did raise an alarm, it was usually right. The taint engine was tracking real flows correctly.
• Recall 0.07 — it was blind to 93% of the bugs. The engine wasn't broken; its vocabulary was tiny. I was listening at one door of a building with many doors.

That's not a broken idea. That's a correct, simple diagnosis of a too-narrow
source list — delivered before I had invested weeks in the wrong layer. If the
first number had been precision 0.10, I'd have had an engine problem, which is
a rebuild. A recall problem is a list problem. Lists are fixable.

Why the Obvious Moves Are Both Wrong

Obvious move #1: don't tell anyone. Fix it quietly, publish only the final
number, look competent. Almost everyone building in public does a version of
this — the "overnight" success graph that starts at the first good result.

The problem: every result in this series is a number from my own benchmark
runs. There is no referee here — no third party checks my work before it
ships; it is me, a scorer script, and you. A reader has exactly one way to
judge numbers like that: the author's track record with results that hurt
him. If every number I show you is a win, you have no reason to trust any of
them. So the bad numbers ship too — and they ship first. I'm going to publish
a head-to-head against Semgrep and CodeQL later, and when I claim a result
there, I want the reader thinking "this is the person who published his own
0.07." Honesty is not a virtue here; it's infrastructure.

Obvious move #2: carpet-bomb the rules. Recall too low? Add patterns!
Match more names, loosen the regexes, taint everything — recall will climb. It
will also destroy precision, and worse: after twenty simultaneous changes you
cannot say which one did what. You've traded a measured system for a vibes
system.

What I did instead was slower and duller: read the benchmark's actual code,
find what it really calls, add sources in order of how often the code uses
them, and re-measure after every change. One variable at a time, one
number per change.

The Climb

Fix 1: Learn the benchmark's vocabulary — recall 0.07 → 0.83

I surveyed which input methods the benchmark's code actually uses, counting
files: getRequestURI in 724 files, getCookies in 664, getParameter in
538, getParameterValues in 510, getHeaders in 400, and on down the HTTP
request surface. My spike had covered exactly one entry in that list.

So sources became one shared definition — a single regex over fully-qualified
method names, so each getter is bound to the type that makes it
attacker-controlled. Condensed here; the full rule table gets its own article:

// Queries run on Joern, an open-source code-analysis engine (Scala DSL).
".*(HttpServletRequest|ServletRequest)\\.(getParameter|getParameterNames|getHeader|" +
  "getHeaders|getCookies|getQueryString|getRequestURI|getInputStream|…)\\b.*" +
"|.*Cookie\\.(getValue|getName)\\b.*" +
"|.*SeparateClassRequest\\.(getTheParameter|getTheValue)\\b.*" // the benchmark's request wrapper

(.*Cookie\.getValue.* matches only the cookie's getter — a bare match on
getValue would match every getValue in existence.)

Two more problems were hiding inside this step, and they were different
problems:

• 117 of the benchmark's real SQL injection cases never touch java.sql — they go through Spring's JdbcTemplate instead. One new sink row took SQLi recall from 0.57 to 0.86.
• 6,060 XSS sink calls were invisible for a reason outside my rules. Without the servlet library on the analysis path, the engine cannot work out what type response.getWriter() returns, so those calls could never match a type-based .*Writer.* pattern. The fix: also accept a sink when the receiver text — the response.getWriter() part as literally written in the code — matches getWriter|getOutputStream. That single change took XSS recall from 0.03 to 0.73.

New score: precision 0.53, recall 0.83.

Fix 2: 97 of the 130 remaining misses shared one missing source — 0.83 → 0.95

There were still 130 real bugs missing. I diffed the misses against the
benchmark code, and 97 of them — three quarters of everything left — took
their taint from a single method I hadn't listed: getParameterNames().

It's easy to see why it gets skipped. getParameter("id") returns a value
the user typed — obviously dangerous. getParameterNames() returns the
parameter names — and names feel like structure, not data. But the client
chooses the names too. ?<script>alert(1)</script>=x is a query string any
client can send, and then the name is exactly as attacker-controlled as the
value.

One name in a regex. Adding it recovered 93 real vulnerabilities on the spot:
recall 0.83 → 0.95. (The other 4 of the 97 were also blocked by a second,
separate problem — they return in Fix 3.) And here's the part that still
bothers me: nothing ever crashed, warned, or looked wrong. A missing source
fails silently. Without a labeled benchmark I would never have known.

Fix 3: The taint bridge — 0.95 → 1.00

That left 37 misses — the 33 that never used getParameterNames, plus the 4
from Fix 2 that had a second problem — and every single one contained
.split(...). Instead of guessing, I measured the taint chain link by link
on one failing case: the variable being split was reachable from the source.
The split call itself — reachable. The array-index access on its result,
param.split(" ")[0] — not reachable. Taint flowed correctly through
split and died at the index operation. The engine ships a default rule for
that operator, and overriding it changed nothing — the gap was in how the
engine applies rules to that operator, not in anything I could configure
away.

So I built a bridge, with one condition that keeps it honest: an index access
over a split-style call is promoted to an additional source only when the
array it indexes is itself reachable from a real source. Indexing an
untainted array stays untainted — that condition is the difference between a
targeted fix and blanket over-tainting. All 37 misses recovered, at a cost of
exactly three new false positives and about 84 seconds of extra work per
scan — a fix that raised recall and precision at the same time.

The Numbers

The whole climb, one measured change at a time:

stage	precision	recall	F1
spike (getParameter only)	0.60	0.07	0.13
+ broadened sources, receiver-text sinks	0.53	0.83	0.65
+ getParameterNames	0.55	0.95	0.70
+ index-access taint bridge	0.56	1.00	0.72

Zero false negatives. All four classes land at recall 1.00 — SQL
injection 272 of 272 real bugs found, command injection 126 of 126, path
traversal 133 of 133, XSS 246 of 246 — the same recall CodeQL achieves on the
same 1,478 cases, scored by the same code. From a seven-row rule table, with
no build step.

Now the uncomfortable part, because this is an engineering log and not a
launch post. Precision 0.56 means 614 false alarms, and this layer falls into
88% of the benchmark's designed traps — more than either incumbent (the
head-to-head article prints the full comparison). Perfect recall with weak
discrimination is uncomfortably close to a tool that says "maybe" about
everything, and I'd rather write that sentence myself than have a reader
write it for me.

Three objections, answered before you raise them

"You fixed recall by reading the benchmark's own code and adding whatever
it calls. That's tuning on the test set — recall 1.00 means nothing."
Partly right, and worth being precise about. The benchmark is public, and it
is open-book for every tool measured on it — the incumbents tune against it
too. But look at what the fixes actually were: the standard
HttpServletRequest input surface — getHeader, getCookies,
getRequestURI — not benchmark-specific hacks; the one special case, the
benchmark's own wrapper class, is disclosed in the regex above. And the traps
argue against gaming: I fall into 88% of the designed false-alarm cases,
worse than CodeQL — if I were fitting to the answer key, that is the first
number I would have fixed. What recall 1.00 honestly claims is narrower:
given a vocabulary, the engine misses nothing that speaks it — and the loop
that built the vocabulary (survey what the code actually calls, add sources
by count, re-measure after every change) is the same loop that would onboard
any real codebase. Generalisation to real repositories is unmeasured; when I
measure it, it gets published, good or bad.

"Why not just use CodeQL? Same recall, better precision." True, and the
head-to-head article prints that row in plain text. Two reasons this project
exists anyway. First, this discovery layer is a seven-row rule table with no
build step — CodeQL brings years of modeled libraries and wants your build.
Second, the precision problem is deliberate surplus: discovery over-reports
so that the judge — the LLM layer — has material to remove, and in
controlled tests, the best judge model I tried removed half the false
alarms — 52% in the latest verified run — at a cost of 2% of the real
bugs. The comparison that decides the design is judged output against the
incumbents, and that number gets published either way.

"An LLM in the pipeline means the results can't be reproducible." The
non-determinism is quarantined by design. Everything in this article is the
deterministic layer: same code in, same flows out — the whole 0.07 → 1.00
climb involved no LLM at all. The model never searches; it only rules on
findings the rules already produced. And its verdicts are not trusted — they
are measured against labeled ground truth, on a fixed-seed sample, with
confidence intervals. How that measurement went — and why the choice of
judge model mattered far more than I expected — is a later article.

---

Going deeper (skip freely — statistics only)

Recall 1.00 against a false-positive rate of 0.88 gives this layer a
Youden's J — recall minus false-positive rate, a one-number score that
punishes saying "maybe" to everything — of just 0.12. Weak discrimination,
stated plainly; raising that number is precisely the judge's job. And the
judge figures quoted above are estimates from a 200-candidate stratified
sample, which is why they carry confidence intervals — sample-level and
full-census numbers never share a table in my logs, because mixing the two
silently shifts precision by a point.

What I Learned

1. Ship the embarrassing number. Publishing 0.07 cost me nothing — the
scanner improved just as fast either way. What it bought is the right to be
believed later: when the head-to-head against Semgrep and CodeQL comes out,
every number in it is backed by the fact that I publish the ones that hurt.

2. A spike's job is to be wrong, cheaply. The minimal version told me in
one run which layer was weak (vocabulary, not engine) — before I had built
anything expensive on top of it. If your first measurement of a new system
isn't a little embarrassing, you probably measured too late.

3. Coverage failures are silent — only measurement finds them. 97 of my
130 misses traced back to one absent source, without a single crash, warning,
or odd log line. This goes far beyond security: the error path nobody tests,
the market segment nobody surveys, the input case nobody generates — absence
never announces itself. If you aren't measuring against ground truth, your
blind spots don't feel like blind spots.

What can you do with this?

The lessons above are mine; this section is yours. Take it by who you are:

• You build software — any software. Two habits you can apply this week: check whether your input validation covers the names of inputs, not only their values (the client chooses both — that gap hid 97 bugs from me); and for any system you own, ask "where is my labeled answer key?" A missing capability produces no error and no log line. Only measurement against known truth makes blind spots visible.
• You build with LLMs or any detection system. Steal the loop wholesale: ship a minimal spike end-to-end, measure it against ground truth, change one variable, re-measure, keep every number. It located my weakest layer in a single run, and it will locate yours.
• You're learning security. You now hold the four ideas that organize static analysis — source, sink, tainted flow, and the precision/recall trade. Everything here is reproducible: the deterministic layer runs completely free, with no API key needed.

That's the first number. Coming in this series: the architecture the whole
scanner stands on — why the rules do all the searching and the AI only
judges — the judge experiments, and the full head-to-head against Semgrep
and CodeQL. Every number gets published, especially the bad ones. (Code and
benchmark artifacts go public alongside the head-to-head article.)

---

I'm Ali Afana — AI builder and security researcher, writing from Gaza. I
build systems in public, measure them against ground truth, and keep the
receipts. This scanner is one project on a longer road — follow for what
comes next.

GitHub · X · LinkedIn