I Wrote Code for a Skill That's Supposed to Require Judgment. Here's Where I Drew the Line.

A few readers of Part 8 asked variations of the same question once they noticed the repo had a packages/ folder sitting next to skillmama/SKILL.md: didn't I already build this? SKILL.md runs, it works, agents install it and it scores candidates. What's the code for?

Fair question, and the honest answer took me longer to land on than I expected. Most of what SKILLmama does is genuinely judgment: deciding whether a library's docs make it "one-liner install" or "clear docs, under 30 minutes," deciding whether a search hit is even a relevant candidate, deciding whether a project already has a tool covered. None of that is code. It can't be, without either faking the judgment or quietly narrowing what the skill claims to do.

But some of it isn't judgment at all. It's a lookup table, or an HTTP call, or arithmetic. And once I could point at exactly which parts those were, I had a second problem I hadn't expected: nothing stopped the code and the markdown spec from drifting apart the moment I touched either one.

Drawing the Line

SKILL.md scores four factors: Compatibility, Popularity, Maintenance, Simplicity. I went through each looking for one honest question: given the same inputs, would two people computing this by hand get the same number?

Compatibility and Simplicity: no. Compatibility asks whether a library fits this project's actual stack, constraints, and existing tools — that's reading, comparing, and weighing, the same work a person does. Simplicity asks how long integration would realistically take from reading the docs. Different readers, same docs, different numbers. These stay judgment forever. Not "not yet automated" — genuinely can't be, without inventing a proxy metric that isn't actually measuring what SKILL.md means.

Popularity and Maintenance: yes, mostly. SKILL.md's own table says >10k stars OR >1M weekly downloads scores 10. That's not an opinion, it's a comparison. GitHub's API returns a star count; npm's downloads API returns a number. Feed both into the stated bands and the output is reproducible.

So the package computes exactly those two, live:

gatherFactorEvidence() -> one GitHub request (stars, last-push, archived)
                        -> one npm request (weekly downloads)
mapPopularityBand()    -> SKILL.md's band table as a lookup
mapMaintenanceBand()   -> same, for last-commit recency

Two HTTP calls, deterministic band, and the caller — an agent, still — picks the actual point inside the band. Popularity comes back 7-9, not 8. That last step stays judgment on purpose; the code's job ends at "which band," not "where in it."

Refusing to Fake the Rest

The tempting shortcut here is to let the injected judge pick anything it wants once it has the evidence — max out every band, hand-wave the disagreements. I didn't want a package that looks deterministic while quietly trusting whatever comes back. So discoverCapabilities(), the function that composes the whole pipeline, enforces the boundary instead of just documenting it: if a caller returns Popularity 10 against evidence that verified 7-9, it throws. Not a warning, not a clamp — a thrown error, at the seam between mechanical evidence and injected judgment.

That was also the moment I noticed nothing had composed the phases at all. Each phase existed — evidence gathering, band mapping, the security gate — but a caller had to remember the right order, remember to drop BLOCKED candidates before scoring, remember the required companion-skills search couldn't be skipped. discoverCapabilities() owns those rules now instead of leaving them as things a caller has to get right from memory every time.

The Drift I Didn't See Coming

Here's the part I didn't plan for. packages/core reimplements pieces of SKILL.md's logic in TypeScript — the 40/30/15/15 weighting, the Maintenance bands, the npm bot-publisher list Check 2 uses to avoid mistaking a CI migration for a handoff. All of that already existed as prose and a reference script in SKILL.md, written for a human or an agent to follow by hand.

Two independent copies of the same logic, one in English, one in code. The obvious failure mode: someone edits SKILL.md's band table six months from now, the tests stay green because they test the code's table, and the two silently disagree forever.

So skill-conformance.test.js doesn't hardcode SKILL.md's numbers into assertions. It parses skillmama/SKILL.md at test time — reads the actual Maintenance table, the actual bot list, the actual tier headings — and asserts the package's code produces the same answer:

const rows = maintenanceRows(); // extracted from SKILL.md's own markdown
for (const row of rows) {
  assert.deepEqual(mapMaintenanceBand(daysAgo(row.days)).band, row.band);
}

I mutation-tested it before trusting it: ten separate hand-edits to SKILL.md — moved a weight, moved a band edge, renamed a tier, dropped a word from a search recipe, changed a bot name. All ten failed the suite. If the extractor can't find what it's looking for, it throws instead of silently matching nothing, which is its own failure mode I had to guard against separately — a parser that finds zero rows and calls that a pass is worse than no test at all.

It Found a Real Gap, Not Just a Missing Test

The conformance test's job is to catch drift between two things that already agree. It found something better: a place where SKILL.md's own table disagreed with itself. Maintenance:

10:  ≤30 days, active releases
7-9: ≤90 days
4-6: ≤180 days
1-3: >365 days or archived

A repo last pushed 200 days ago falls in a hole. Not ≤180, not >365 — nowhere. I found this the same way as the PyPI severity trap in Part 8: not by reading the table carefully, but by trying to write code that covers every case and discovering there wasn't a case for this one.

The instinct is to just pick a number and move on. I didn't, for the same reason the package refuses to fake judgment elsewhere: reporting a band SKILL.md never actually specified is inventing an answer and presenting it with the same confidence as a verified one. So the mapper returned an explicit "SKILL.md doesn't define this range" outcome instead of a guessed score, and the gap sat in the roadmap as its own line item until I had an actual answer: 3-5, splitting the difference between the neighboring bands rather than rounding to either edge. Once that was a real decision instead of a placeholder, the band closed, the special-case branch came out, and the conformance test now tripwires if the gap ever reopens instead of tripwiring on it existing.

Extending the Boundary Without Guessing

Check 2, publisher continuity, shipped npm-only in Part 8, with a specific reason: PyPI's package index has no per-release uploader field. I'd assumed the other three OSV ecosystems — PyPI, Go, crates.io — were all in the same boat, and left them all as unsupported-ecosystem.

I hadn't actually checked crates.io. When I did:

"published_by": {
  "login": "dtolnay",
  "name": "David Tolnay"
}

Real per-version publisher, right there in the version payload. PyPI genuinely has nothing equivalent; Go modules are proxied straight from source control with no publish-account concept at all; crates.io does. So I added crates.io, and only crates.io — same four rules as npm's check (sort by time, old guard must never return, most recent handoff under 12 months, drop non-human publishers), adapted to a registry with no separate time map to join and no literal bot list, because a null published_by already means "not a human account" whether that's CI trusted-publishing or a version from before crates.io tracked publishers at all.

Verified against rustls, which rotates releases among three maintainers (ctz, djc, cpu) the same way express and lodash rotate among npm teams — exactly the shape Part 8's Attempt 2 got wrong the first time. Reports no handoff, correctly. This is the same lesson from that post, pointed the other direction: adding coverage without checking whether the data actually supports it would have been the same mistake as generalizing the bot list into a regex that quietly stopped matching.

---

What Changed

The two packages that used to exist side by side — a core package and a thin cli wrapper around it — are one package now, published as skillmama, because a user installing the command shouldn't have to learn about an internal split that only mattered to me:

npx skillmama check lodash --version 4.17.15

Runs the live-data half of Phase 3.5 and Phase 4 standalone, no agent required — OSV, publisher continuity (npm or crates.io now), the two live-data score bands. It says plainly what it didn't check, too: no LLM means no Phase 3.5 content-reading, so a clean result there means "the mechanical checks found nothing," not "this package is safe." Verified end to end against flatmap-stream@0.1.1, the actual event-stream backdoor payload — comes back BLOCKED, exit code 3.

skillmama/SKILL.md stays the sole source of truth for the full skill; nothing in packages/ is wired into it yet, and that's a separate decision for later. What changed here is narrower and, I think, more durable: the deterministic 30% of the pipeline that genuinely doesn't need judgment now has real code behind it, a test suite that reads the spec instead of a copy of it, and a boundary that throws instead of guessing the moment something crosses from evidence into judgment.

github.com/Magithar/SKILLmama, Apache 2.0.