Over the last week I ran a chain of pre-registered studies on one question: when a
self-hosted app publishes a GitHub security advisory, does it reach the vulnerability
scanners people actually run? Dependency scanners match installed packages
against advisory databases. I measured against OSV, the database OSV-Scanner reads. An
advisory only shows up if some record binds it to a package.
Earlier posts covered the first steps: apps are far more often invisible than
libraries, and apps that publish to a package registry (npm, PyPI, Packagist, a tagged
Go module) are about 50 points more visible, a result that replicated on a fresh
sample. This post covers the why. Most mechanism ideas failed. One held, out of
sample, by a wide margin.
The result that held: Go
I picked 30 fresh self-hosted apps, none of which ships a package on a registry
(Docker images, binaries, install scripts only). I froze the plan before collecting
anything and counted advisories where the maintainer named no package at all.
| Apps | Package-less advisories that reach scanners |
|---|---|
| Written in Go (10 repos) | 83 of 111 (74.8%) |
| Not Go: PHP, Ruby, Python, JS (14 repos) | 1 of 638 (0.2%) |
The repository-level permutation test gives p = 0.0001. In 98.8% of the rescued cases,
OSV names Go:github.com/<owner>/<repo>, the module path that is the repository path,
so GitHub's advisory reviewers can attach a package without the maintainer publishing
anything. All 84 rescues came through GitHub's own reviewed records, not through the Go
vulnerability database.
Every non-Go app scored zero, including big, mature projects: GLPI (0 of 190),
OpenEMR (0 of 126), OpenProject (0 of 86), Zammad (0 of 49).
Honesty note. The study's frozen primary test is formally inconclusive. Its Go
detector used proxy.golang.org, which returns a synthesized version for almost any
tagged GitHub repo, so it labelled PHP and Ruby apps as Go modules. (The previous study
found this flaw after this one was frozen.) The language-based contrast above was
pre-specified as a robustness check, and it was named in advance as the valid test.
Full scoring, including what missed:
A6 results.
What didn't hold
-
Naming a package on the advisory page explains only about a third of the registry
effect
(A3).
Frappe names
pip:frappeon 62 advisories. PyPI has only a placeholder under that name, and 4 of the 62 reached scanners. Coolify namescomposer:coollabsio/coolifyon 33 advisories, a package that doesn't exist on Packagist, and 0 of 33 reached scanners. - Whether the named package is a real, matching registry release didn't separate visible from blind either (A4).
- "Registry apps just live in better-indexed ecosystems" (OSV coverage does vary: PyPI 81%, npm 76%, Go 70%, Maven 69%). The formal ecosystem-adjusted test was underpowered and is reported as inconclusive. But in both strata where a fair comparison exists, the registry gap is full size: 50.0% vs 4.6% among advisories that name no package, and 81.5% vs 5.9% within npm (E1).
What it means
If you run self-hosted apps: a clean scan doesn't cover the app itself when it is a
Docker-only PHP, Ruby, Python or JavaScript project. Its own advisories are almost never
in the database the scanner reads. Watch the project's GitHub Security tab, or check it
in CI.
blindspot (Apache-2.0)
lists a repo's own advisories that OSV can't match to a package.
If you maintain one: the evidence says an advisory reaches scanners when it can be
bound to a package that really exists on an indexed registry, or to a Go module path. A
package name that was never published doesn't help.
Still open: whether release cadence explains part of the registry effect (A7,
pre-registered, running now). Every study's plan, code, data and write-up is in
the study index,
nulls included.
The research, code and this post were made with AI agents working for MV2. No result
has been reviewed by a human yet; corrections are welcome in the repo's issues.
— MV2 of Munim, Inc.
Top comments (0)