Agent skills are becoming a package format for behavior.
A SKILL.md file can teach an agent how to review code, operate a browser, call an MCP server, publish a release, or work with local files. That is useful, but it also creates a trust question that package ecosystems have seen before:
What should I inspect before I install someone else's skill?
I built SkillPreflight, an open-source static scorecard for agent skills, to make that first review easier. Instead of only looking for obviously dangerous strings, it scores seven areas: security, permission restraint, token efficiency, footprint, maintainability, reliability, and compatibility.
Recently I wanted to test the score model against more than hand-written fixtures, so I froze a public sample of 40 skills from GitHub and published the raw results.
How the sample was built
This is a convenience sample, not a claim about the entire skill ecosystem.
I used two GitHub code-search queries:
SKILL.md path:.claude/skillsSKILL.md path:skills
I selected 20 eligible results from each query after filtering to public, non-fork repositories and one skill per repository. Every source URL was pinned to the exact commit returned by GitHub search.
The scan used SkillPreflight 0.4.0 with its default rules. The committed sample file makes the run reproducible even when repositories change later.
The full methodology, frozen URLs, JSON, and CSV are in the benchmark directory.
Results
All 40 selected skills scanned successfully.
| Metric | Result |
|---|---|
| Average score | 80.8/100 |
| Median score | 85.5/100 |
| Score range | 49-90 |
| High-risk review recommended | 2 |
| Median activation tokens | 840.5 |
| 90th percentile activation tokens | 2,699 |
The grade distribution was:
| Grade | Skills |
|---|---|
| A | 1 |
| B | 29 |
| C | 5 |
| D | 3 |
| F | 2 |
What stood out
1. The biggest gaps were not always dramatic security findings
The lowest category averages were reliability at 29% of its maximum and maintainability at 40%.
Within the selected skill directories, the scanner frequently found no local examples, tests, license, or README. That does not mean the parent repository had none. The benchmark intentionally scanned the selected skill directory rather than treating unrelated repository files as part of the package.
That distinction matters. A skill may be safe enough to read while still being difficult to validate, reuse, or maintain independently.
2. Token cost varied substantially
The median estimated activation size was about 841 tokens, while the 90th percentile reached 2,699 tokens. Four skills crossed the scanner's very-large SKILL.md threshold.
Long instructions are not automatically bad, but always-loaded background material competes with the user's task and other context. Progressive disclosure is useful here: keep activation instructions concise and move detailed references into files that are loaded only when needed.
3. Finding counts need context
One security rule produced 119 occurrences, but those occurrences came from only five skills. Repeated references to credentials or tokens can be legitimate in a security or deployment workflow.
That is why the report now separates "skills affected" from "occurrences." Static analysis should help a reviewer find the right lines; it should not turn pattern counts into an automatic accusation.
4. A high score is not a safety certificate
Static analysis cannot tell whether a remote service is trustworthy, whether downloaded content changes later, or whether a seemingly reasonable workflow will be misused at runtime.
SkillPreflight therefore does not claim that an A grade proves safety. It is a pre-install triage tool: surface risky commands, secret access, prompt-injection language, broad permissions, unpinned dependencies, oversized instructions, and packaging gaps before the skill is trusted.
Try it without installing globally
Scan a local skill:
npx skill-preflight@latest scan ./my-skill
Scan a GitHub skill before cloning it:
npx skill-preflight@latest scan https://github.com/owner/repo/blob/main/path/to/SKILL.md
Generate SARIF for GitHub Code Scanning:
npx skill-preflight@latest scan ./my-skill --format sarif --out skill-preflight.sarif
The scanner reads files but does not execute scripts from the target. It can also run as a GitHub Action so skill authors can apply a score or severity gate on pull requests.
What I would like feedback on
The score weights and static rules are intentionally visible rather than hidden behind a hosted service. The questions I am working through now are:
- Which findings should affect the score, and which should remain informational?
- How should a standalone skill inherit documentation from its parent repository?
- Which permission patterns are too broad across Claude Code, Codex, Cursor, and Gemini CLI?
- What is the most useful way to measure token cost without pretending the estimate is exact?
The project is MIT licensed, and the rule catalog, benchmark data, and implementation are all public:
I would especially value examples of false positives or important pre-install risks that the current rules miss.
Top comments (0)