DEV Community

iamthat
iamthat

Posted on Edited on

We linted 9 public agent skills: 8 PASS, 1 FAIL (and 2 false positives we fixed)

We downloaded the SKILL.md files of nine public agent skills and ran our open-source linter over them. Eight passed. One failed.

This is not a benchmark of those projects — it is a check of one thing: whether a skill is structurally usable by an agent that has to decide when to load it.

Correction, published with the first version of this post: the first run reported 7 PASS / 1 WARN / 1 FAIL. The WARN was our own linter's false positive, not a defect in the skill. We fixed two rules and re-ran the same bytes. The numbers below are the second run, and the details of what we changed are in the last section — because a linter that reports its own mistakes is worth more than a linter that never does.

What we ran

skilllint is a small local linter for agent skills. Eleven rules, no network, no telemetry. It reads a SKILL.md, parses the frontmatter, and answers three questions:

  1. Can an agent tell when to use this skill? (name, description, trigger phrasing)
  2. Is the package self-consistent? (dead markers, references to files that do not exist, oversized files)
  3. Does it behave like an instruction, or like a trap? (unlimited autonomy, embedded secrets, prompt-injection patterns)

Every verdict comes with the SHA-256 of the exact bytes checked, so the result is reproducible.

The skills we checked

Skill Verdict Score Findings
anthropics/skills → skills/claude-api/SKILL.md PASS 83 4
anthropics/skills → skills/doc-coauthoring/SKILL.md PASS 90 2
anthropics/skills → skills/pdf/SKILL.md PASS 95 1
upstash/ratelimit-js → skills/SKILL.md PASS 95 1
upstash/qstash-js → skills/SKILL.md PASS 95 1
upstash/workflow-js → skills/SKILL.md PASS 95 1
better-auth/better-icons → skills/SKILL.md PASS 100 0
xmtplabs/xmtp-agent-examples → skills/SKILL.md FAIL 50 4
kid-sid/claude-spellbook → skills/go/skill.md PASS 95 1
  • R-001 (critical) - frontmatter (--- ... ---) not found
  • R-002 (high) - name is missing or malformed: ''
  • R-003 (high) - description is missing
  • R-008 (medium) - referenced file not found: skills/building-agents/

Nine of ten planned downloads succeeded; the tenth path returned 404 and we dropped it instead of quietly replacing it with something easier.

What the FAIL found

xmtplabs/xmtp-agent-examples has no frontmatter at all. No name, no description.

That is not a style complaint. A skill without a description cannot advertise its trigger, so an agent has no reliable way to know the skill exists or when to reach for it. The package may still be excellent as documentation — as an agent skill it is unreachable.

The two false positives we found in our own linter

Both were found by linting our own skill package, which came back WARN instead of PASS:

  1. Trigger wording was only recognised in Russian. The rule looked for "use when" but not for "Triggers -", so a perfectly good English description was reported as having no trigger. We widened the cue list (EN + RU).
  2. A skill that documents a rule was flagged by that rule. Our own reference/rules.md lists the phrase "without asking" as an example of unbounded autonomy — and the linter flagged the documentation of the rule. We now ignore markdown tables and fenced code blocks, and treat a match inside quotation marks as a citation rather than an instruction.

Both fixes are in the rules table, and both changed real verdicts: anthropics/skills/claude-api moved from WARN (78) to PASS (83) on the same bytes.

This is the point of publishing the method rather than just the results: a checker that cannot be argued with is worth less than one that shows its work.

Honest caveats

  • We fetched only SKILL.md, not the whole package. Every "referenced file not found" finding is therefore not conclusive — the file may exist in the repo and simply was not in our download. We report the raw output rather than editing it into a nicer story.
  • Eleven rules is an opinionated minimum, not a security assessment. A PASS means "structure is sound", never "safe".
  • These are static checks. They cannot tell you whether the instructions work; only execution can.

Why this matters more every week

Skills are executable instructions that get injected into an agent's context. Two failure modes dominate:

  • Silent non-loading. No description, no trigger → the skill simply never fires. Nothing errors. You just never get the capability.
  • Silent over-loading. The description fires on anything → the agent drags a 100 KB file into every relevant context and spends your budget on it.

Both are invisible until you measure them. That is the same reason we built the rest of the stack: a signed receipt for the artifact, a mandate check before the action, and an actual-spend ledger afterwards.

Reproduce it

skilllint is MIT-licensed and runs locally:

python3 -m skilllint path/to/skill      # verdict + sha256 + findings
Enter fullscreen mode Exit fullscreen mode

Or try it on your own skill before publishing it — 20 checks a day, free, no account, no upload.


Need the document, not the explanation? AILegal turns a verified situation into a breakdown, options and a plan — from $2.99, paid with Telegram Stars or USDC. No card: https://ailegal.legal/pricing

Top comments (1)

Collapse
 
skillselion profile image
Skillselion •

The per-rule breakdown is what makes this checkable, so I pulled the same file. Two of the five findings look like rule artifacts.

The claude-api description carries a block starting "TRIGGER" that enumerates when to read it, and a second starting "SKIP" with the overriding conditions. The rule looks like it is matching surface phrasing rather than whether an agent could route.

Your caveat covers the agent-openai.py finding, but it is stronger than "not conclusive": that string sits inside a sentence listing filename patterns to look for in the user's project, as a marker that the file belongs to another provider. There is no such bundled file to miss, so a full-package download would not clear it either. Our pull is 105,146 bytes, which lines up with your 105 KB, so we are on the same version.

That puts 2 of the 5 findings behind the 78 on rule behaviour. Worth printing which rule fired per finding, since the score is what people will quote.