Building verifiable security checks, source-linked scores, and honest verdicts in SKILLmama v1.10.0–v1.12.0.
After Part 9 went out, someone replied on X with a simple question:
Which version did the security check actually look at, and when?
I didn't know.
The result said Security: PASS. It didn't say which release had passed, or when the check happened.
A new CVE could be published the following morning, and someone reading a saved result a month later would have no way to tell when that PASS was valid.
And it wasn't just security. Popularity: 10/10 looked convincing, but where did that number come from? How many stars? How many downloads? Which source?
That question changed the direction of the next three releases. From v1.10.0 through v1.12.0, the focus became simple:
Every claim in a SKILLmama result should be something the reader can verify.
1. A PASS Needs a Version and a Timestamp
The first change was straightforward. Every security verdict now includes the exact package version checked and a UTC timestamp.
Security: PASS · checked qdrant-client 1.20.0, 2026-11-18 19:32 UTC
SKILLmama still performs live checks. There's no result cache or stored snapshot.
But when somebody copies the result into a pull request, an issue, or a document, the verdict now carries its own context: which version was checked, and when.
That's not a guarantee the package remains safe. It's a record of what was checked at that moment.
Two related changes followed.
Install the version that was checked
Previously:
pip install qdrant-client
Now:
pip install qdrant-client==1.20.0
The previous command could install a different release from the one that passed the security check. Pinning the installation command removes that mismatch.
Blocked candidates shouldn't disappear
Before, a blocked candidate could simply vanish from the shortlist.
Now blocked packages appear under Also Considered, along with their security finding, checked version, and timestamp.
If a popular package is missing from a recommendation, the reader should be able to understand why.
2. Every Score Needs a Source
A timestamp answers when. It doesn't answer where the number came from.
So I updated SKILL.md to require inline source links for Popularity, Maintenance, and Security claims.
That includes:
- GitHub repositories for stars and recent commits
- npm or pypistats for downloads
- OSV listings and individual advisories for security findings
- Registry metadata for publisher history
The most important rule is the fallback:
If a number cannot be linked to a source, it must be reported as N/A (unverified).
The output also distinguishes between a source being unreachable and a check never being attempted.
An unlinked number is a guess with good formatting.
In the npm package, evidence records from gatherSecurityEvidence() and gatherFactorEvidence() now include a sourceUrl. Failed checks retain the source link where one is available, so users can investigate manually.
Running a check produces a Sources section like this:
Sources:
osv ok https://osv.dev/list?ecosystem=PyPI&q=qdrant-client
publisher-continuity unverified none (unsupported-ecosystem)
github-stars ok https://github.com/qdrant/qdrant
pypi-downloads ok https://pypistats.org/packages/qdrant-client
github-last-commit ok https://github.com/qdrant/qdrant
The links aren't historical snapshots, and some source pages will change over time. But at least readers now know where the evidence came from.
3. Adding Evidence Exposed a Bug in My Own Verdicts
This was unexpected.
While testing PyPI packages, I noticed that every package returned WARN, even packages with no reported advisories.
The problem was publisher continuity.
SKILLmama checks whether package publishing ownership has changed. That check works on npm and crates.io, but PyPI and Go don't provide the same per-release uploader information.
The CLI treated this unsupported check as unverified. And the existing rule said an unverified security check must downgrade the overall verdict to WARN.
So every PyPI package was being penalized for something the registry couldn't provide.
There was another problem. SKILL.md already handled this situation differently. It allowed a PASS with N/A (unsupported ecosystem).
The skill and the CLI were producing different verdicts for the same package.
My Part 9 conformance test hadn't caught it because the rule existed in prose, not in one of the structured tables the test compared.
The fix
I separated two situations that shouldn't have been treated as equivalent.
| Situation | Meaning | Verdict impact |
|---|---|---|
| Unsupported ecosystem | The registry doesn't provide the required information | No automatic downgrade |
| Unreachable source | A supported check was attempted but couldn't complete | Downgrade to WARN |
That distinction now exists in both the implementation and SKILL.md.
Not supported is different from not verified.
It's a small distinction in code, but it changes what a security verdict actually means.
4. PyPI Finally Gets Download Data
Before v1.12.0, PyPI popularity scoring used GitHub stars without PyPI download data.
The output included:
npm-downloads: N/A (no-npm-package)
Technically correct, but not much use for evaluating a Python package.
Now SKILLmama fetches the previous week's download counts from pypistats.org and includes them in the Popularity band alongside GitHub stars.
One implementation detail caused a surprising failure. The pypistats endpoint is case-sensitive: Flask can return a 404 while flask succeeds.
The fix was to normalize package names using PEP 503 conventions before requesting the data: lowercase the name and collapse consecutive hyphens, underscores, or periods into a single hyphen.
If the endpoint fails or rate-limits the request, SKILLmama reports N/A (unverified). It doesn't invent a download count.
Testing against ChromaDB
Here's a real check against ChromaDB 1.5.9:
npx skillmama check chromadb \
--ecosystem PyPI \
--version 1.5.9 \
--repo chroma-core/chroma
The result, condensed:
Verdict BLOCKED
OSV blocks:
HIGH GHSA-2wm9-hf6c-p5cr
CRITICAL GHSA-36p7-vc44-83pf
CRITICAL GHSA-f4j7-r4q5-qw2c
HIGH GHSA-xph7-9rjv-w5fr
publisher-continuity:
N/A (unsupported-ecosystem)
Popularity 10/10
Maintenance 10/10
github-stars:
29,715 stars -> 10
pypi-downloads:
1,973,568 last week -> 10
Figures and advisories are from the check described here; they can change after publication.
ChromaDB was popular and actively maintained. It was still blocked.
That's intentional. Popularity cannot override the security gate. The security check runs before scoring, not after it.
5. The README Example Was Made Up
This was probably the most uncomfortable discovery.
The README contained an end-to-end example with illustrative numbers and plausible-looking output. It was meant to demonstrate what SKILLmama could produce.
But after adding the requirement that every number must have evidence, the example became a contradiction. The project was asking users not to trust unverified scores while displaying invented ones in its own documentation.
So I replaced it.
The new example comes from a live run on November 18, 2026. The test project was a small FastAPI + SQLAlchemy + PostgreSQL application, and the task was to find a suitable vector database.
The result ranked:
- Qdrant
- Weaviate
- pgvector
pgvector lost Compatibility points because the project's postgres:16 image didn't include the extension. Rather than just reporting a lower score, the result explained what needed to change to make pgvector compatible.
ChromaDB was blocked because of the security advisories found during the check.
The example links its numerical claims to the sources used at the time. Those values will drift, and the README explicitly says so.
I'd rather show an old, traceable number than a timeless-looking invented one.
6. My Security Skill Failed a Security Audit
SKILLmama is listed on skills.sh, which runs security audits against listed skills.
The Agent Trust Hub audit flagged SKILLmama for a pattern associated with downloading and executing remote code. The reason was a command in the publisher-continuity logic. It used a pattern like:
curl ... | python3 -c '...'
The Python code was a fixed inline parser. It wasn't Python downloaded from the network.
But I understood the warning. Piping network output directly into an interpreter resembles a dangerous command pattern, and an automated auditor shouldn't need to reconstruct the programmer's intentions to assess it.
I changed the implementation to save the registry response to a temporary file and parse that file separately. Same functionality, without the suspicious pipeline.
This doesn't remove every risk associated with untrusted data. It removes that particular execution pattern.
The second warning: untrusted content
Snyk also flagged SKILLmama with warning W011 for reading third-party content.
This is harder to eliminate because reading repository READMEs, documentation, and registry metadata is central to how SKILLmama works.
The meaningful fix was to establish a strict trust boundary. SKILL.md now explicitly states that fetched content is data, never instructions.
A third-party README, search result, or another skill's SKILL.md cannot change SKILLmama's rules, scoring, or execution behavior. Candidates that attempt to inject instructions are discarded.
The existing skills.sh badges still reflect the earlier audit. I've requested a re-audit in vercel-labs/skills#2793, and that remains the top roadmap item until the updated audit is available.
7. A Few Smaller Improvements
The three releases also included changes outside the evidence pipeline.
Trusted Publishing
npm releases now use Trusted Publishing with provenance, linking each published version to the GitHub Actions run that built it.
A quieter CLI
Running:
skillmama
now shows a short quick-start instead of the entire help output.
Interactive terminals also get a one-time welcome message. It never shows in CI, in piped output, or with --json.
Colored security verdicts
check now displays red, yellow, or green verdicts where terminal color is supported. NO_COLOR disables the styling.
A green PASS still means only that the checks performed found no blocking issue. Any checks that weren't completed are disclosed.
Stickers
Five SKILLmama PNG stickers are now included.
skillmama stickers
They're free to print, but not to sell. And yes, stickers are slightly outside the evidence-verification roadmap.
Tests and releases
The npm package moved from 0.1.1 to 0.5.0 over this stretch, and the test suite now contains 233 tests.
8. What's Still Missing
The next gap is already clear: a package version isn't necessarily a source commit.
npm packuments contain a gitHead field for many releases. The next step is to expose the commit associated with a published version when that information is available.
If it isn't available, SKILLmama should say:
commit unknown
It shouldn't guess using the latest repository tag.
There are also limits that haven't changed since Part 9.
Compatibility and Simplicity remain judgment-based scores evaluated by the agent running the skill. The standalone skillmama check command cannot read a package's documentation or code to calculate those dimensions, and the CLI makes that limitation explicit.
The goal isn't to make every dimension look automated. It's to make the boundary between measured evidence and agent judgment visible.
What I Learned
I started this round trying to answer one question from an X reply:
Which version passed, and when?
That led to source URLs, timestamps, pinned installs, real PyPI downloads, and more precise handling of unsupported checks.
But the most important changes weren't planned features.
Requiring evidence exposed a made-up README example. It exposed a disagreement between the skill and the CLI. It exposed that PyPI downloads weren't being collected at all.
None of those problems were discovered by specifically looking for them. They became obvious once every reported number needed a verifiable source.
A result that says PASS is only worth as much as the reader's ability to check it.
And once every number had to carry a link, a number with no link behind it had nowhere to hide.
SKILLmama is open source under Apache 2.0.
GitHub: https://github.com/Magithar/SKILLmama
Try it:
npx skillmama
Top comments (0)