Someone replied on Mastodon to my launch post with a question I didn't have a good answer for:
curious what the security check actually verifies. known CVEs / package provenance is one bar, "this maintainer won't quietly change hands in 8 months" is a much harder one and honestly the one that's bitten people more. still, a tool suggesting the right library beats finding out mid-project it was the wrong one.
I went to check what "security check" actually meant in my own skill, expecting to defend it. Instead I found the weakest rule in the whole file.
What Was Actually There
Phase 3.5's hard gate had this line:
DISCARD (set security: "BLOCKED") if:
- Has a dependency with a known CVE
No data source. No API. Nothing telling the model where to look. In practice this depended entirely on whatever a web search happened to surface for that candidate, which is a fancy way of saying it usually did nothing. And the commenter's actual point, maintainer handoff, wasn't checked at all. Contributors were extracted in Tier 1 and then never used anywhere.
Two gaps, and the second one was the sharper of the two. A known CVE gets patched, gets a CVSS score, gets a GitHub Security Advisory. A quiet maintainer handoff has none of that until someone notices the malware. In event-stream's case, below, that took months.
Building the Real CVE Check
OSV.dev is free, unauthenticated, and covers npm, PyPI, Go, and crates.io from one endpoint:
curl -s -X POST "https://api.osv.dev/v1/query" \
-d '{"package":{"name":"lodash","ecosystem":"npm"},"version":"4.17.15"}'
Straightforward, except for one trap I only found by testing against a real vulnerable package. Query requests==2.19.0 on PyPI and you get advisories back labeled PYSEC-2018-28 with severity: UNKNOWN. Looks like nothing to worry about. But aliases links it to GHSA-x84v-xcm2-53pg, and that one is rated HIGH. PyPI's advisory feed duplicates every GHSA entry under its own ID, and the PyPI-native copy just doesn't carry a severity field. Skip the dedupe and the one advisory that matters hides behind a label that says "unknown."
Building the Publisher-Continuity Check, Badly, Three Times
This is the one the commenter actually asked for, and getting it right took three attempts, each one falsified by testing against real npm data instead of my assumptions about what would work.
Attempt 1: flag any human-to-human publisher change. Tested it against express. False positive. It's not a handoff, it's a healthy team rotating releases among four maintainers over a decade. Same failure on lodash and chalk.
Attempt 2: require that the old publisher never publishes again. This is the technically correct definition of a handoff, and it's useless. Tested against 13 well-known packages: it fired on 8 of them, including react, debug, and semver. Nearly every long-lived package has some handoff somewhere in its history. A rule that fires on most of npm isn't a signal, it's noise, and noise trains people to stop reading warnings.
Attempt 3: same rule, but only report the most recent handoff, and only if it's under 12 months old. This is what shipped. Swept it across 98 popular packages:
n=98 FIRE=7 (7.1%) STALE=50 (51.0%) CLEAN=41
51% of those packages carry a stale handoff somewhere in their past. The recency filter is doing the real work. Without it, this thing is worthless.
I broke it once more trying to "improve" it. I replaced the hardcoded bot exclusion list (GitHub Actions, npm, etc.) with a regex, thinking it would generalize better to automation accounts I hadn't thought of. Fire rate on the same 98 packages jumped from 7.1% to 33.7% overnight. The bug: GitHub Actions has a space in it, my character class only covered [-_], so the single most important filter silently stopped matching and every CI-migrated package started reading as a handoff.
Fixed the regex, then measured it head-to-head against the plain hardcoded list on the same list again. Identical result, 7.1% both. The regex bought nothing. Its one apparent win was a rule I'd written specifically to catch a package I'd just watched it fail on, which is fitting my sample, not generalizing. I threw it out and kept the simpler thing.
Does the Recency Filter Still Catch event-stream?
The real test is event-stream, the 2018 incident where right9ctrl took over publish rights from the original maintainer and shipped a targeted credential-stealing payload three versions later. I ran the check as if I were adopting the package at different points in time:
as of 2018-08-01 (before handoff) -> no handoff
as of 2018-10-01 (before malware disclosed) -> WARN: dominictarr -> right9ctrl
as of 2018-11-26 (disclosure day) -> WARN: dominictarr -> right9ctrl
as of 2026-07-17 (today) -> 94mo old, not reported
The handoff happened on 2018-09-05. My 12-month recency window means the check would have flagged it any time up to 2019-09-05, and specifically on 2018-10-01, a full seven weeks before the malicious code was publicly disclosed on 2018-11-26. On the day it mattered, OSV would have returned nothing. Advisories don't exist before disclosure. That's the exact complementarity the commenter was pointing at: one check catches what's already known, the other catches something the industry hasn't found yet.
Real fires from the 98-package sweep, none of them noise: dougwilson stepping back from cors and morgan, and wikirik picking up both sequelize and validator in the same window. That second one isn't evidence of anything wrong, but it's exactly the kind of pattern worth a human glance instead of a silent PASS.
What It Doesn't Catch
Worth saying plainly, because a security gate that oversells its coverage is worse than no gate. ua-parser-js, rc, and coa are three real npm supply-chain compromises, and in all three the attacker published under the legitimate maintainer's own account. Publisher continuity reads completely clean on all of them. Only the OSV check catches those, and only after someone disclosed it.
And publisher continuity is npm-only. PyPI's package index has no per-release uploader field at all, just an upload timestamp. Python candidates get an explicit N/A (unsupported ecosystem) instead of a number that implies something was checked.
It Found One For Real
I ship checks, then I re-run whatever eval I already had to make sure I didn't just add a rule that looks right and does nothing. I have a paired skill-on/skill-off comparison that keeps recommending vector databases for the same test project, so I re-ran that same prompt to see if the new OSV check actually changed anything.
It did. chromadb, the library that had won every previous run of that comparison, came back with CVE-2026-45829:
curl -s -X POST "https://api.osv.dev/v1/query" \
-d '{"package":{"name":"chromadb","ecosystem":"PyPI"},"version":"1.5.9"}'
CRITICAL, pre-authentication code injection, and no patched version at any point in the current release history. I didn't take OSV's word for the "no fix" part, I checked the GitHub Advisory API directly too: first_patched_version: null. Chroma got discarded before it ever reached scoring, for the first time in every run I'd done. The rule wasn't decoration, it changed a real recommendation the moment it had a real vulnerability to work with.
Reading the actual advisory, it only fires in Chroma's server mode with trust_remote_code set to true, a flag most embedded, in-process usage never sets. I still discarded it unconditionally, that's the point of a hard CRITICAL-with-no-fix rule, but a block that doesn't say why isn't much more useful than no explanation at all. So the DISCARD rule now also carries the advisory's stated trigger condition into security_note, quoted straight from OSV instead of summarized by me:
DISCARD (set security: "BLOCKED", add security_note) if:
- OSV returns a CRITICAL or HIGH advisory ... with no `fixed` version available
— set security_note to the advisory's summary, including any trigger
condition (specific mode, flag, or endpoint) verbatim from OSV
Still blocks regardless of the condition. The note doesn't narrow the rule, it just means the person reading the block can tell "unsafe for me" apart from "unsafe in a mode I don't use," instead of taking a bare BLOCKED at face value.
What Changed
Live in v1.4.6, across all four adapters. Phase 3.5 now runs both checks against live data before scoring anything:
npx skills add Magithar/SKILLmama -a claude-code
github.com/Magithar/SKILLmama, Apache 2.0.
The lesson from this one isn't really about supply-chain security. It's that "make it more general" is not automatically an improvement, and the only way I found that out was by measuring the before and after on the same data instead of trusting that the more sophisticated version was the better one. Twice in this round the simpler rule beat the smarter-looking one, and both times, only running it and counting told me which. And the check itself only proved it was more than decoration once I went back and re-ran the same eval that had been sitting there the whole time, which is exactly what caught Chroma.
Top comments (0)