Every SAST vendor quotes an accuracy number. Almost none of them tell you the three things you'd need to reproduce it. We took a single, fixed detector and showed that its "accuracy" on the OWASP Benchmark swings by more than 50 points depending on choices that have nothing to do with the detector itself.
TL;DR
- A SAST accuracy score is not reproducible unless three things are disclosed: the benchmark version, the CWE subset that was averaged, and the matching rule.
- Same detector, same detections — score comes out 45.5 across all 11 categories, or 100.0 on the injection-only subset. A 54.5-point swing from subset selection alone.
- The OWASP Benchmark's official macro-average weights the smallest category 14.4× more than a case-weighted average would. We computed all of this from the suite's own public ground truth; the script is in the study.
Why one number hides four decisions
The OWASP Benchmark is the closest thing the SAST world has to a shared ruler. But "score = X" collapses at least four independent choices into one figure, and each one moves the result:
1. Which version of the benchmark.
| Version | Test cases |
|---|---|
| v1.1 | 21,041 |
| v1.2 | 2,740 |
v1.2 is 7.7× smaller than v1.1. A false-positive rate computed over 21,041 cases and one computed over 2,740 are not the same measurement, and "OWASP Benchmark score" rarely says which population it's over.
2. Macro-average vs. case-weighted.
The official scorer takes a macro-average — every category counts equally, regardless of how many test cases it has. That sounds fair until you look at what it does to the small categories:
- XPath injection has 35 test cases. Under the macro-average it gets 9.1% of the score (1 of 11 categories). Weighted by its actual case count it would get 1.3%. That's a 14.4× inflation of the smallest category.
- SQL injection has 504 cases — it gets the same 9.1% under macro, but 18.4% when weighted by cases. So the official average makes SQLi count roughly half as much as its share of the suite.
Neither weighting is wrong. But they answer different questions, and a headline number doesn't tell you which one you're looking at.
3. Which CWE subset got averaged.
Here's the part that should make you suspicious of any single figure. Take a neutral probe that scores perfect detection on the injection families and nothing else:
- Averaged across all 11 categories: 45.5
- On the injection-only subset: 100.0
Same probe. 54.5 points of difference, entirely from which categories you chose to report.
4. How you match a finding to an expected result. Line-level, method-level, or file-level matching each produce a different true/false-positive count from identical scan output.
How it was computed
Nothing here is an estimate. We ran it against v1.2 ground truth using the OWASP BenchmarkUtils scorer with the Youden index — (sensitivity + specificity) − 1, normalized to 0–100 — and derived the category counts and weightings straight from the suite's public expectedresults-1.2.csv. Change one of the four knobs, keep the detector fixed, and watch the headline move.
# reproduce the weighting + subset math from the official file:
OWASP Benchmark -> expectedresults-1.2.csv (2,740 rows)
# our parser + the full 45.5-vs-100 derivation:
zennoxa.com/research/sast-benchmark-version-drift-2026
What to actually ask for
When a tool (ours included) quotes an OWASP Benchmark score, ask three questions before you believe it:
- Which version — v1.1 or v1.2?
- Which categories were averaged, and macro or case-weighted?
- What matching rule turned scan output into true/false positives?
A score that can answer all three is a measurement. A score that can't is a billboard.
The full derivation — every category's weight, the four knobs, and the reproducible script — is here:
→ Is a published SAST 'accuracy' score even reproducible?
Published by Zennoxa Research. We hold our own numbers to this bar — if a claim on our site can't answer the three questions, it's a bug; tell us.
Top comments (0)