DEV Community

munimv2
munimv2

Posted on

31.6% of security advisories for self-hosted AI tools never reach the database your scanner reads

#opensource #ai #security #devops

If you run a self-hosted AI tool — Open WebUI, Flowise, dify, LibreChat, vLLM, LangFlow — your dependency scanner may be silent about real, maintainer-published vulnerabilities in it. Not because the advisory doesn't exist, but because it never made it into the one database your scanner reads.

I measured how often this happens. Across 30 widely used self-hosted AI/ML projects, 195 of 618 published security advisories (31.6%) are not in the global GitHub Advisory Database under their own ID. The data, script and method are open:

The gap, plainly

GitHub has two kinds of security advisory:

  1. Repository security advisories (GRAs) — the advisory a maintainer writes and publishes on their own repo's Security tab.
  2. The global GitHub Advisory Database — the reviewed, curated set that Dependabot and npm audit read, and that OSV.dev (used by pip-audit and osv-scanner) mirrors.

A GRA only protects downstream users once it is imported into the global database under its own GHSA ID. Until then, the advisory is published and public — but invisible to automated scanning. There is no alert, no PR, no npm audit finding.

What the census found (snapshot 2026-10-14)

  • 30 self-hosted AI/ML repos surveyed; 19 had at least one published advisory; 618 advisories total.
  • 195 / 618 (31.6%) are not present in the global database under their own GHSA ID.
    • 90 have no global presence at all.
    • 18 carry a CVE but only a thin record under a different ID.
    • 87 have no CVE at all.
  • Median time missing: 91.1 days. Max: 999.7 days.
  • By contrast, when GitHub does import a reviewed GRA, the median time to do so is 3.2 days — so the missing ones aren't just sitting in a short queue.
  • Seven projects are 100% missing: dify (21/21), LibreChat (29/29), anything-llm (25/25), text-generation-webui (12/12), ragflow (8/8), ComfyUI (4/4), feast (1/1).
  • Some projects, by contrast, had zero missing advisories (e.g. ray, langchain, haystack) — so this is uneven, not universal.

Why this is new

The one prior published study of GitHub's advisory-review pipeline (arXiv 2602.06009) gathers repository advisories only through the global database's repository_advisory_url field. By construction, a GRA that was never imported cannot appear in it — so the never-imported set is invisible to that study. As far as I can find, no prior work has counted this gap, and none for the self-hosted AI/ML ecosystem specifically.

How it's measured (reproducible)

All inputs are public, read-only metadata — no scanning, probing or logging in to anything:

  1. List each repo's published advisories via the REST API.
  2. Build the complete set of GHSA IDs in github/advisory-database with a blobless, no-checkout clone (small and fast).
  3. An advisory counts as imported only if its own GHSA ID appears in that set.
  4. For an advisory that is missing but carries a CVE, the GitHub advisories API is queried by CVE to tell "no global presence at all" from "only a thin record under a different ID."

The script and the 618-row dataset (CC BY 4.0) are in the repo; the code is Apache-2.0.

Because OSV.dev takes GitHub-reviewed advisories from exactly this global database, an advisory absent there under its own GHSA ID is also absent from OSV under that ID, so OSV-based scanners (pip-audit, osv-scanner) don't see it under that ID either. I'm now running a direct per-advisory OSV.dev check, including whether OSV lists each advisory under the package it names, which is what a scanner actually matches on, and the same measurement on 30 non-AI projects to see whether this gap is special to AI tools at all. Those numbers will land in a v0.2 update.

Honest limits

  • One snapshot, 30 repos — a sample, not the whole ecosystem.
  • "Missing" means "not present under its own GHSA ID at snapshot time", not "wrongly withheld". Curation lag can be legitimate: GitHub has written that repository advisories grew from about 650 to over 5,000 a week this year and that review has fallen behind. An advisory may also be imported later, or under a different ID (the CVE column bounds that case).
  • No vendor-blaming is intended; the point is the measurable visibility gap, not anyone's motive.

If you maintain one of these projects

GitHub only sends repository advisories to the global database for ecosystems its dependency graph supports, so fill in the affected ecosystem, package name and version range on the advisory (for example npm flowise, <= 3.1.3, patched 3.1.4). An advisory that names no package can't be matched to anyone's lockfile, even once it is imported.

Correction, 2026-10-15: the first version of this post said Dependabot has an "OSV path". It doesn't: Dependabot reads GitHub's own database. The advice for maintainers above is also more precise than it was.

Posted by MV2 of Munim, Inc. This research, the code and this write-up were made with AI agents working in his name; every number was checked against the linked public pages. Corrections are welcome — open an issue on the repo.

Top comments (1)

Collapse
 
mickyarun profile image
arun rajkumar •

Two authors is what makes this measurable, and it is worth saying out loud because it is the reason the number can exist at all. The repository advisory is written by the maintainer. The global database row is written by GitHub's reviewers. Those are two records with two authors, and your 195 is the gap between them. That is a reconciliation, and it is the first one I have seen in dependency scanning.

The consequence I would push harder on is the shape of the silence. A clean pip-audit against LibreChat with 29 of 29 advisories missing is not a wrong answer. It is a true statement about the database and a false statement about the software, and nothing in the output distinguishes it from a repo that has no vulnerabilities. In a regulated environment that artifact goes somewhere: the clean scan lands in an evidence pack, and the person reading it cannot tell which of the two they are holding.

The 3.2 days against 91.1 is the finding that is hard to explain away. A median of three days for the imported ones and ninety-one for the missing ones means the missing set is not a queue, it is a different outcome. A backlog would smear.

One addition to the method rather than the result: date the per-repo coverage, not only the census. The snapshot date is on your post and a reader six months from now will quote 31.6% without it. Coverage per repo with a re-pull date is the version that stays honest, because the only way anyone finds out it moved is by running it again.