DEV Community: hamelin123

The same detector scores 45.5 or 100 on the OWASP Benchmark. Both are 'true.'

hamelin123 — Wed, 19 Aug 2026 09:45:32 +0000

Every SAST vendor quotes an accuracy number. Almost none of them tell you the three things you'd need to reproduce it. We took a single, fixed detector and showed that its "accuracy" on the OWASP Benchmark swings by more than 50 points depending on choices that have nothing to do with the detector itself.

TL;DR

A SAST accuracy score is not reproducible unless three things are disclosed: the benchmark version, the CWE subset that was averaged, and the matching rule.
Same detector, same detections — score comes out 45.5 across all 11 categories, or 100.0 on the injection-only subset. A 54.5-point swing from subset selection alone.
The OWASP Benchmark's official macro-average weights the smallest category 14.4× more than a case-weighted average would. We computed all of this from the suite's own public ground truth; the script is in the study.

Why one number hides four decisions

The OWASP Benchmark is the closest thing the SAST world has to a shared ruler. But "score = X" collapses at least four independent choices into one figure, and each one moves the result:

1. Which version of the benchmark.

Version	Test cases
v1.1	21,041
v1.2	2,740

v1.2 is 7.7× smaller than v1.1. A false-positive rate computed over 21,041 cases and one computed over 2,740 are not the same measurement, and "OWASP Benchmark score" rarely says which population it's over.

2. Macro-average vs. case-weighted.

The official scorer takes a macro-average — every category counts equally, regardless of how many test cases it has. That sounds fair until you look at what it does to the small categories:

XPath injection has 35 test cases. Under the macro-average it gets 9.1% of the score (1 of 11 categories). Weighted by its actual case count it would get 1.3%. That's a 14.4× inflation of the smallest category.
SQL injection has 504 cases — it gets the same 9.1% under macro, but 18.4% when weighted by cases. So the official average makes SQLi count roughly half as much as its share of the suite.

Neither weighting is wrong. But they answer different questions, and a headline number doesn't tell you which one you're looking at.

3. Which CWE subset got averaged.

Here's the part that should make you suspicious of any single figure. Take a neutral probe that scores perfect detection on the injection families and nothing else:

Averaged across all 11 categories: 45.5
On the injection-only subset: 100.0

Same probe. 54.5 points of difference, entirely from which categories you chose to report.

4. How you match a finding to an expected result. Line-level, method-level, or file-level matching each produce a different true/false-positive count from identical scan output.

How it was computed

Nothing here is an estimate. We ran it against v1.2 ground truth using the OWASP BenchmarkUtils scorer with the Youden index — (sensitivity + specificity) − 1, normalized to 0–100 — and derived the category counts and weightings straight from the suite's public expectedresults-1.2.csv. Change one of the four knobs, keep the detector fixed, and watch the headline move.

# reproduce the weighting + subset math from the official file:
OWASP Benchmark  ->  expectedresults-1.2.csv   (2,740 rows)
# our parser + the full 45.5-vs-100 derivation:
zennoxa.com/research/sast-benchmark-version-drift-2026

What to actually ask for

When a tool (ours included) quotes an OWASP Benchmark score, ask three questions before you believe it:

Which version — v1.1 or v1.2?
Which categories were averaged, and macro or case-weighted?
What matching rule turned scan output into true/false positives?

A score that can answer all three is a measurement. A score that can't is a billboard.

The full derivation — every category's weight, the four knobs, and the reproducible script — is here:

→ Is a published SAST 'accuracy' score even reproducible?

Published by Zennoxa Research. We hold our own numbers to this bar — if a claim on our site can't answer the three questions, it's a bug; tell us.

There are 755 static-analysis tools. Only 42 are open-source security scanners.

hamelin123 — Wed, 19 Aug 2026 08:02:37 +0000

If you run a linter on your code today, you have a lot of choices. If you want one that actually looks for security bugs — and is open source — you have far fewer than the ecosystem's size suggests.

We parsed the public analysis-tools.dev catalog (MIT-licensed, 755 tools) to get an actual count instead of a vibe. Here's what fell out.

TL;DR

Of 755 static-analysis tools in the catalog, 86% are open source — but only 42 are open-source tools whose job is finding security bugs.
21 languages that already have an active linting ecosystem (five or more general static-analysis tools each) have zero open-source security scanner. SQL, Shell, PowerShell, Dart, Elixir, Lua, R, Groovy, Clojure and Haskell are on that list.
Every number here is reproducible from one public dataset. No survey, no vendor list, no "leaders" quadrant.

Why the gap is easy to miss

"Static analysis" and "security scanner" get used interchangeably, and the catalog is full of tools — so it feels like the security-scanner space is crowded. When you split the catalog by what each tool is actually for, the picture changes:

Slice	Count	Share
All static-analysis tools	755	100%
Open source (any purpose)	653	86.5%
Security-focused (SAST)	69	9.1%
*Open-source and* security-focused**	42	61% of the security tools

A general linter tells you a function is too long. A security scanner tells you that a value from request.args reached a SQL string without going through a parameterizer. Both are "static analysis." Only one is looking for the bug that ends up in a CVE. The catalog has 653 of the first kind and 42 of the second.

The 21-language hole

The more interesting finding isn't the total — it's where the 42 aren't.

We took every language that has at least five general static-analysis tools (i.e. a language with a real tooling community, not a niche one) and checked how many have an open-source security scanner. Twenty-one of them have none:

SQL, Shell, Groovy, R, Lua, Dart, PowerShell, Elixir, Clojure, Haskell — and eleven more.

Some of those are load-bearing. Shell and PowerShell run your CI and your production boxes. SQL is where injection lives. Dart ships to phones. If you write those and want an open-source tool that reasons about security specifically, the catalog says you're on your own.

Where the 42 do cluster is unsurprising: Python (13 scanners), Go (12), Java (10), JavaScript (9). The languages with the most security tooling are the ones that already had the most tooling.

How it was measured

Nothing here needs to be taken on faith. The method is three steps against one file:

License — split proprietary vs. open source using the catalog's license field.
Security scope — keep only tools tagged security in the catalog taxonomy.
Language coverage — read each tool's language tags and count per language.

The catalog is a single JSON file in a public MIT repo. We publish the parser and the raw counts alongside the study, so you can re-run it and get the same 42 — or disagree with a classification and see exactly which tool moved the number.

# the dataset:
github.com/analysis-tools-dev/static-analysis  (data/*.yml)
# our parser + raw counts:
zennoxa.com/research/state-of-oss-sast-2026  ->  data.json

Why we bothered

We build a security scanner (Shield), so the honest disclosure is that this map has a self-interested corner. But the count doesn't change based on who's counting — 42 is 42, and the 21-language hole is in the public data whether or not you've heard of us. We'd rather publish the census than a quadrant.

The full breakdown — every language, the license splits, and the reproducible data.json — is here:

→ State of Open-Source SAST 2026

Published by Zennoxa Research. If you think a tool is misclassified, the raw counts are right there — tell us which one.