DEV Community: hamelin123

Popular projects SHA-pin GitHub Actions 67.6% of the time — but Docker base images only 7.6%

hamelin123 — Mon, 24 Aug 2026 08:27:46 +0000

Pinning a dependency to something that can't change underneath you is the same idea whether it's a CI action or a container base image. We already measured how often popular projects pin their GitHub Actions to a commit SHA. So we asked the identical question one layer down: how often do they pin their Docker base images to an immutable @sha256 digest?

TL;DR

Across 25 popular infra/app projects (250 external base-image references), only 7.6% are pinned to an immutable @sha256 digest (19 refs).
81.6% use a mutable version tag like node:20.1.2 (204 refs); 10.8% are :latest/untagged (27 refs).
Only 5 of 22 projects that ship Dockerfiles use @sha256 pinning for any base image.
Same supply-chain principle, measured before: GitHub Actions pinned to a commit SHA = 67.6%. That's roughly 9× more adoption for Actions than for container digests.
Point-in-time (2026-07-21), reproducible across all 25 repos with public tooling.

The numbers

Pin type	Refs	Share
Digest (`@sha256`, immutable)	19	7.6%
Version tag (`node:20.1.2`)	204	81.6%
`:latest` or untagged	27	10.8%
Total external base images	250	100%

The 5 projects that digest-pin any base image: posthog, Ghost, elasticsearch, moby, n8n.

Side by side with the Actions study:

Immutability practice	Adoption across popular repos
GitHub Actions pinned to a commit SHA	67.6%
Docker base images pinned to a digest	7.6%

Why it matters

FROM node:20 (or worse, FROM node:latest) pulls whatever image that tag resolves to at build time. Tags are mutable — the same node:20 can be a different image tomorrow — so a reproducible, tamper-evident build pins to the content digest: FROM node@sha256:…. It's the exact same risk that made SHA-pinning a CI checklist item after the 2025 GitHub Actions compromises (tj-actions, reviewdog) — just one layer down, and adopted ~9× less often.

Our read on the gap: the Actions ecosystem had a forcing function (those compromises) plus a one-click habit (Dependabot). The base-image equivalent hasn't had its forcing function yet — and digest strings are ugly to write by hand, so most teams stop at a version tag.

A fair caveat

Version tags are not "wrong." node:20.1.2 is readable, gets patch updates, and is fine for most teams. Only a @sha256 digest is truly immutable, so we report it as the strict gold standard for reproducible/tamper-evident builds — not as a claim that everyone on a version tag is insecure. The clearly-risky category is :latest/untagged, which makes builds non-reproducible and lets the base change silently. Some pinned/unpinned images are also a project's own first-party images, where the trust model differs.

How to pin

# mutable — resolves at build time, can change underneath you
FROM python:3.13-slim

# immutable — the exact image content, human-readable tag kept in a comment
FROM python:3.13.13-slim-bookworm@sha256:355bfa66770995d7e9a0da4b3473b44d0cb451f6b56f5615ad9c39e3c4eca03f

Then let Dependabot (package-ecosystem: docker) bump the digest so you stay patched without tracking a moving tag.

How we measured it

Corpus: 25 popular infra/app repos; 22 ship Dockerfiles with external base images.
Per repo: sparse-checkout the Dockerfiles, parse every external base-image FROM (excluding internal stage aliases, --platform flags, build-args, and scratch), and classify as digest, version tag, or :latest/untagged.
Cross-check: Shield's CONTAINER-002 (:latest) rule against an independent FROM parser — the :latest counts agreed (16 vs 15, one build-stage edge case). Digest classification is unambiguous (a reference either contains @sha256: or it doesn't), and we hand-verified samples at both ends.

Reproduce it yourself:

git clone --depth 1 --filter=blob:none --sparse https://github.com/grafana/grafana g
cd g && git sparse-checkout init --no-cone && git sparse-checkout set '**/Dockerfile' && git checkout
# a base image is immutable only if the ref contains @sha256:
grep -rhE '^\s*FROM\s' --include='*Dockerfile*' . | grep -v '@sha256:'

Limitations

25 popular repos, point-in-time (2026-07-21) — not a random sample; large projects skew more disciplined, so the ecosystem-wide digest rate is plausibly lower.
We classify references, not unique images; a reused base weights a repo's count.
Stage aliases, --platform flags and build-args are excluded; one residual cross-file alias (~0.4%) may remain and doesn't move the headline.

Full per-repo dataset, method, and the cross-study comparison are on the canonical page: Do popular projects pin their base images? We checked 25. Zennoxa Research publishes reproducible security data on public projects with public tooling — no third-party scanner involved.

Vulnerability advisories grew 2.3x-6.6x in 5 years; open-source tool count stayed flat

hamelin123 — Fri, 21 Aug 2026 03:33:57 +0000

We bucketed every advisory in OSV.dev's public per-ecosystem archives by its record-publication year for 11 package ecosystems, and overlaid the count of open-source, security-tagged tools per language from the analysis-tools.dev catalog. The question: is the pool of open-source tools that covers a language keeping pace with how fast advisories enter the record?

TL;DR (every number recomputes from the public archives)

Known-vulnerability advisories entering the record grew 2.3x-6.6x over five years for most ecosystems: PHP 6.6x, Java 4.8x, .NET 4.2x, Rust 3.0x, Python 2.3x.
The open-source tool count per language is roughly flat — it is a property of the catalog, not a curve that rises with advisory volume.
The gap is a flow: recent years (2023-2025 average) add up to 157 new advisories per open-source tool per year for Rust, 121 for PHP, 112 for Go — against just 2-13 tools per language.
Across the 11 ecosystems, ~4,900 new vulnerability advisories land per year, served by 42 catalogued open-source security tools.
npm is the honest exception (0.8x): its explosive growth is in malicious-package reports, which we excluded. Counting malware, npm's raw volume is 223,802 total, 96.9% MAL-; counting real vulnerable-dependency advisories, it isn't even growing.

Growth: advisories entering the record per year

Known-vulnerability advisories (malware excluded), by publication year:

Ecosystem (language)	2020	2021	2022	2023	2024	2025	Growth (2023-25 vs 2019-20)
Packagist (PHP)	156	430	2,326	771	1,063	712	6.6x
Maven (Java)	151	394	3,009	951	565	711	4.8x
NuGet (.NET)	15	103	313	72	106	103	4.2x
crates.io (Rust)	168	487	339	260	321	363	3.0x
PyPI (Python)	469	1,417	1,878	849	1,020	1,010	2.3x
RubyGems (Ruby)	57	71	238	91	103	67	1.5x
npm (JavaScript)	934	597	669	400	436	632	0.8x

(2026 is a partial year, inflated by an OSV ingestion wave, so we lead with complete years through 2025. Pub/Hex/Swift are omitted from the growth table — their OSV histories start ~2022 with small counts.)

The flow gap: new advisories per open-source tool, per year

Recent-years (2023-2025) average new advisories per year, against catalogued open-source tools:

Ecosystem (language)	New advisories / yr (2023-25 avg)	Open-source tools	New advisories / tool / yr
crates.io (Rust)	315	2	157
Packagist (PHP)	849	7	121
Go	1,342	12	112
Maven (Java)	742	10	74
PyPI (Python)	960	13	74
npm (JavaScript)	489	10	49
NuGet (.NET)	94	7	13
RubyGems (Ruby)	87	8	11

The thinnest-covered ecosystems face the steepest flow. Rust adds ~157 new advisories per open-source tool per year against just 2 catalogued tools; PHP ~121; Go ~112.

Why it matters

Defenders have finite triage capacity. If yearly inflow keeps climbing while the tool count per language stays flat, teams face more signal per tool and per human-hour every year. That is a structural argument for prioritization — ranking by exploitation likelihood (EPSS, CISA KEV) and reachability — rather than trying to hand-review a growing pile. This study measures the flow (how fast advisories enter each year); a companion study, The SCA Coverage Gap, measured the stock (total advisories per tool).

How it was measured (reproducible)

For each ecosystem we download OSV's all.zip, read every advisory's published RFC3339 date, bucket by 4-digit year, and exclude MAL- (malicious-package) advisories so the curve tracks vulnerable-dependency advisories, not malware reports. Tool counts are the open-source, security-tagged tools per mapped language from the analysis-tools.dev catalog.

for eco in npm PyPI Maven Go RubyGems crates.io NuGet Packagist Pub Hex SwiftURL; do
  curl -s "https://osv-vulnerabilities.storage.googleapis.com/$eco/all.zip" -o e.zip
  echo "== $eco (vulnerability advisories by year, MAL- excluded) =="
  unzip -p e.zip -x '*MAL-*' | grep -aoE '"published"[[:space:]]*:[[:space:]]*"[0-9]{4}' \
    | grep -oE '[0-9]{4}$' | sort | uniq -c
done

OSV data is public (per-source CC-BY-4.0 / CC0 / MIT / Apache-2.0, with attribution to OSV.dev and the underlying advisory databases); the analysis-tools.dev catalog is MIT.

Honest limitations

OSV's published field is when an advisory entered the record, not when the vulnerability was introduced or first disclosed — so this measures advisory-publication velocity, not vulnerability incidence. Bulk imports show up as spikes (a log4j-era Java backfill in 2022, a large 2026 ingestion across ecosystems), which is why we lead with complete years through 2025 and use a 2023-2025 average for the "recent flow" figures. Tool counts reflect catalog coverage, not capability — recall/precision of any tool is out of scope, and commercial tools aren't counted. No scanner is named or scored; the tool axis is an aggregate per-language count.

Full study, the complete table (including Go, 2026 partial-year columns, and the FAQ) and the reproduction script: Advisory Growth vs. Tooling Coverage on zennoxa.com

Rust has 1,332 vulnerability advisories for every open-source scanner. Dart has zero scanners.

hamelin123 — Thu, 20 Aug 2026 03:38:49 +0000

We keep hearing that dependency scanning is a solved problem — pick a tool, wire it into CI, done. That's true if you write JavaScript or Python. If you write Rust, Dart, or Elixir, the map has holes you can measure.

We counted the vulnerability advisories in each package ecosystem (from OSV.dev) against the number of open-source tools that can actually scan it (from the analysis-tools.dev catalog). The mismatch is not subtle.

TL;DR

Across 11 ecosystems: 47,422 vulnerability advisories, and only 42 open-source security tools to cover all of them.
Rust has 1,332 advisories per open-source scanner (2,663 advisories, 2 tools). Python's ratio is 999:1, PHP 979:1, Go 715:1.
Dart (Pub) and Elixir (Hex) are "SCA deserts": real advisories exist (13 and 194), and zero catalogued open-source scanners cover them.
Reproducible from two public datasets. The counting has one honest twist — see npm below.

Advisories per open-source scanner

The signal (known vulnerabilities) and the capacity to scan for it (tools) are, in the study's words, "only weakly aligned." Here's the ratio for the busiest ecosystems:

Ecosystem	Vuln advisories	OSS tools	Advisories per tool
Rust (crates.io)	2,663	2	1,332 : 1
Python (PyPI)	12,992	13	999 : 1
PHP (Packagist)	6,851	7	979 : 1
Go	8,577	12	715 : 1
JavaScript (npm)	6,998	10	700 : 1
Java (Maven)	6,859	10	686 : 1

A high ratio doesn't mean the tools are bad — the good ones scale fine. It means the diversity of coverage is thin: if the two Rust scanners miss an ecosystem-specific advisory format or a new source, there's no third option to catch it.

The deserts: signal with no open-source scanner

Two ecosystems have vulnerability advisories and, per the catalog, no open-source security scanner at all:

Dart (Pub): 13 advisories, 0 tools.
Elixir (Hex): 194 advisories, 0 tools.

194 known-vulnerable-package advisories with zero open-source coverage is not a rounding error for a team shipping Phoenix apps. This is exactly the kind of gap that stays invisible until someone asks "why didn't our scanner catch that?"

The npm twist (why raw counts lie)

npm looks terrifying if you take OSV at face value: 223,802 advisories. But 96.9% of those are malicious-package advisories (the MAL- prefix — typosquats, protestware, install-script miners), not vulnerabilities in legitimate packages. Lump them together and npm looks like the riskiest ecosystem on earth; split them out and the real vulnerability count is 6,998.

That split matters for honesty. A vendor motivated to make a number look big would quote the 223,802. We separated malicious-package advisories from vulnerability advisories precisely so the per-ecosystem comparison isn't distorted — and so we're not the vendor quoting the scary number.

How it was measured

Two public datasets, no survey:

OSV.dev per-ecosystem archives — the JSON advisory feeds, counted per ecosystem, with MAL- advisories separated out.
analysis-tools.dev catalog — open-source tools tagged security, counted per language.

# advisories:  OSV.dev per-ecosystem archives (split MAL- from vuln)
# tools:       analysis-tools.dev catalog, security-tagged, per language
# full parser + the 47,422-advisory / 42-tool derivation:
zennoxa.com/research/sca-coverage-gap-2026

The complete table — all 11 ecosystems, the malicious-vs-vuln split, and the commands to reproduce it — is here:

→ The SCA coverage gap: where vulnerability signal outruns open-source tooling

Published by Zennoxa Research. If your ecosystem has an open-source scanner the catalog missed, that changes the ratio — tell us and we'll correct it.

The same detector scores 45.5 or 100 on the OWASP Benchmark. Both are 'true.'

hamelin123 — Wed, 19 Aug 2026 09:45:32 +0000

Every SAST vendor quotes an accuracy number. Almost none of them tell you the three things you'd need to reproduce it. We took a single, fixed detector and showed that its "accuracy" on the OWASP Benchmark swings by more than 50 points depending on choices that have nothing to do with the detector itself.

TL;DR

A SAST accuracy score is not reproducible unless three things are disclosed: the benchmark version, the CWE subset that was averaged, and the matching rule.
Same detector, same detections — score comes out 45.5 across all 11 categories, or 100.0 on the injection-only subset. A 54.5-point swing from subset selection alone.
The OWASP Benchmark's official macro-average weights the smallest category 14.4× more than a case-weighted average would. We computed all of this from the suite's own public ground truth; the script is in the study.

Why one number hides four decisions

The OWASP Benchmark is the closest thing the SAST world has to a shared ruler. But "score = X" collapses at least four independent choices into one figure, and each one moves the result:

1. Which version of the benchmark.

Version	Test cases
v1.1	21,041
v1.2	2,740

v1.2 is 7.7× smaller than v1.1. A false-positive rate computed over 21,041 cases and one computed over 2,740 are not the same measurement, and "OWASP Benchmark score" rarely says which population it's over.

2. Macro-average vs. case-weighted.

The official scorer takes a macro-average — every category counts equally, regardless of how many test cases it has. That sounds fair until you look at what it does to the small categories:

XPath injection has 35 test cases. Under the macro-average it gets 9.1% of the score (1 of 11 categories). Weighted by its actual case count it would get 1.3%. That's a 14.4× inflation of the smallest category.
SQL injection has 504 cases — it gets the same 9.1% under macro, but 18.4% when weighted by cases. So the official average makes SQLi count roughly half as much as its share of the suite.

Neither weighting is wrong. But they answer different questions, and a headline number doesn't tell you which one you're looking at.

3. Which CWE subset got averaged.

Here's the part that should make you suspicious of any single figure. Take a neutral probe that scores perfect detection on the injection families and nothing else:

Averaged across all 11 categories: 45.5
On the injection-only subset: 100.0

Same probe. 54.5 points of difference, entirely from which categories you chose to report.

4. How you match a finding to an expected result. Line-level, method-level, or file-level matching each produce a different true/false-positive count from identical scan output.

How it was computed

Nothing here is an estimate. We ran it against v1.2 ground truth using the OWASP BenchmarkUtils scorer with the Youden index — (sensitivity + specificity) − 1, normalized to 0–100 — and derived the category counts and weightings straight from the suite's public expectedresults-1.2.csv. Change one of the four knobs, keep the detector fixed, and watch the headline move.

# reproduce the weighting + subset math from the official file:
OWASP Benchmark  ->  expectedresults-1.2.csv   (2,740 rows)
# our parser + the full 45.5-vs-100 derivation:
zennoxa.com/research/sast-benchmark-version-drift-2026

What to actually ask for

When a tool (ours included) quotes an OWASP Benchmark score, ask three questions before you believe it:

Which version — v1.1 or v1.2?
Which categories were averaged, and macro or case-weighted?
What matching rule turned scan output into true/false positives?

A score that can answer all three is a measurement. A score that can't is a billboard.

The full derivation — every category's weight, the four knobs, and the reproducible script — is here:

→ Is a published SAST 'accuracy' score even reproducible?

Published by Zennoxa Research. We hold our own numbers to this bar — if a claim on our site can't answer the three questions, it's a bug; tell us.

There are 755 static-analysis tools. Only 42 are open-source security scanners.

hamelin123 — Wed, 19 Aug 2026 08:02:37 +0000

If you run a linter on your code today, you have a lot of choices. If you want one that actually looks for security bugs — and is open source — you have far fewer than the ecosystem's size suggests.

We parsed the public analysis-tools.dev catalog (MIT-licensed, 755 tools) to get an actual count instead of a vibe. Here's what fell out.

TL;DR

Of 755 static-analysis tools in the catalog, 86% are open source — but only 42 are open-source tools whose job is finding security bugs.
21 languages that already have an active linting ecosystem (five or more general static-analysis tools each) have zero open-source security scanner. SQL, Shell, PowerShell, Dart, Elixir, Lua, R, Groovy, Clojure and Haskell are on that list.
Every number here is reproducible from one public dataset. No survey, no vendor list, no "leaders" quadrant.

Why the gap is easy to miss

"Static analysis" and "security scanner" get used interchangeably, and the catalog is full of tools — so it feels like the security-scanner space is crowded. When you split the catalog by what each tool is actually for, the picture changes:

Slice	Count	Share
All static-analysis tools	755	100%
Open source (any purpose)	653	86.5%
Security-focused (SAST)	69	9.1%
*Open-source and* security-focused**	42	61% of the security tools

A general linter tells you a function is too long. A security scanner tells you that a value from request.args reached a SQL string without going through a parameterizer. Both are "static analysis." Only one is looking for the bug that ends up in a CVE. The catalog has 653 of the first kind and 42 of the second.

The 21-language hole

The more interesting finding isn't the total — it's where the 42 aren't.

We took every language that has at least five general static-analysis tools (i.e. a language with a real tooling community, not a niche one) and checked how many have an open-source security scanner. Twenty-one of them have none:

SQL, Shell, Groovy, R, Lua, Dart, PowerShell, Elixir, Clojure, Haskell — and eleven more.

Some of those are load-bearing. Shell and PowerShell run your CI and your production boxes. SQL is where injection lives. Dart ships to phones. If you write those and want an open-source tool that reasons about security specifically, the catalog says you're on your own.

Where the 42 do cluster is unsurprising: Python (13 scanners), Go (12), Java (10), JavaScript (9). The languages with the most security tooling are the ones that already had the most tooling.

How it was measured

Nothing here needs to be taken on faith. The method is three steps against one file:

License — split proprietary vs. open source using the catalog's license field.
Security scope — keep only tools tagged security in the catalog taxonomy.
Language coverage — read each tool's language tags and count per language.

The catalog is a single JSON file in a public MIT repo. We publish the parser and the raw counts alongside the study, so you can re-run it and get the same 42 — or disagree with a classification and see exactly which tool moved the number.

# the dataset:
github.com/analysis-tools-dev/static-analysis  (data/*.yml)
# our parser + raw counts:
zennoxa.com/research/state-of-oss-sast-2026  ->  data.json

Why we bothered

We build a security scanner (Shield), so the honest disclosure is that this map has a self-interested corner. But the count doesn't change based on who's counting — 42 is 42, and the 21-language hole is in the public data whether or not you've heard of us. We'd rather publish the census than a quadrant.

The full breakdown — every language, the license splits, and the reproducible data.json — is here:

→ State of Open-Source SAST 2026

Published by Zennoxa Research. If you think a tool is misclassified, the raw counts are right there — tell us which one.