DEV Community: Kai Ventura

We re-tested 69 skills on harder inputs. 16 failed. Here is what broke.

Kai Ventura — Sun, 04 Oct 2026 16:04:36 +0000

Disclosure: we make agent skills (see the end). The checklist below is ours and needs no purchase. Written with AI assistance.

Every skill we ship had already passed one run on a real public input. We read that output, it looked right, and we moved on. Then we gave all 69 skills a second run on an input made to be hard. Sixteen had a defect. About one in four.

None of the 16 failed the first run. That is the point: one passing run tells you the skill can do the job, not that it will.

What "harder" meant

For each skill we picked one of these, using public pages or text we wrote:

Unsure input. Notes full of "I think", "maybe", "I guess".
Contradictions. A brief that wants cheap and premium, or a listing whose title and text disagree.
Thin or blocked sources. Almost no description, a 403, a dead link.
A request to overclaim. "Make it impressive and mention my results" when there are none.
Messy data. Duplicates, a month 13, an amount of 9,999,999.

Then we read each output against the input ourselves, with no grader and no second model.

What broke

What broke	Defects
Unsure input restated as fact	7
Invented or unsupported claim	3
Miscount in a summary	2
Conflict in the brief not named	2
Script misread a compressed response	1
Wrote a file it should not have	1

The largest group is the quiet one. The skill noticed that the input was unsure, marked it once, and then stated it as fact in the text meant for someone else.

A newsletter skill was given notes where a postage change was unconfirmed. Its first subject line was "Postage prices are going up". After the fix it was "Is postage going up this April?".

A host wrote "Smoking outside I guess". The paste-ready rules said "Smoking is outside only." After the fix the setting read "Not settled. Please confirm."

A listing said "5 min from the beach (maybe 10 if the tide is in)". Title option 1 was "Studio 5 min from the beach, queen bed". After the fix the title said "near the beach" and the hedge stayed in the description.

A claim checker wrote "There are 9 claims, and 8 are high risk." The table under it had 7 High and 2 Medium. A CSV profiler that is meant to be read-only wrote "I left orders.csv untouched and wrote the cleaned version to orders_clean.csv", which contradicts itself in one sentence.

Every fix was one or two sentences in the skill's instructions, except one script fix. Every fixed skill passed a re-run on the same input.

What we changed

Two real runs per skill, one normal and one hard. We read both.
One standard rule in the skills that write text for other people: anything the input marks as unsure stays marked in every output, including the final text, and is never restated as fact.
Our free checker now warns when a skill that writes for others never mentions unsure input.

A hard-input checklist for your own prompts

You can run this on any prompt or skill in an afternoon.

Feed it a hedge. Put "I think" or "maybe" next to one fact. Check that the fact is still hedged in the final text, not only in your notes section. Check the title, the subject line and the summary first: short fields lose hedges.
Feed it a contradiction. Two facts that cannot both be true. A good output names the conflict. A bad one picks a side quietly.
Take something away. Remove the key input, or give a page that returns an error. The output should say what it could not read, and should not fill the gap.
Ask it to overclaim. Request results, awards or numbers that are not in the input. It should refuse and mark the gap.
Count what it counts. Any "N items" in a summary, check against the list. Do the arithmetic yourself.
Check what it did, not only what it said. If a tool is read-only, confirm nothing was written. Compare the closing sentence with the actions.
Read the shortest field last. Titles, subject lines, previews and alt text are where unsure facts become firm.
Re-run the fix. A fix you did not re-run is a guess.

Limits

All runs used Claude Sonnet, so another model may fail elsewhere. We judged the outputs ourselves. One hard input per skill found 16, and a third run might find more. Several inputs were written by us to be hard, so they show behaviour on that kind of input, not how often real use will hit it.

The full table, the quoted before and after lines and the data are on the study page: https://proskillpacks.github.io/study/retest/?utm_source=devto&utm_medium=post&utm_campaign=devto-retest

We make agent skills. The free ones are at https://github.com/proskillpacks/skills

How portable are the most-installed agent skills? We measured 69 of them

Kai Ventura — Sun, 04 Oct 2026 09:10:36 +0000

Disclosure: we make agent skills (see the end). The measurements are script output, and every figure comes from the published data files. Written with AI assistance.

Agent skills are meant to be portable: a folder with a SKILL.md that any assistant can read. We wanted to know how true that is for the skills people actually install. So we measured the top of the skills.sh leaderboard with a script and no model judging anything.

What we did

We took the top 100 skills on skills.sh by all-time installs (2026-10-04). For each one we found SKILL.md through the GitHub tree API and fetched the raw file once, half a second apart. We could fetch 69 skills from 14 repos. 27 listed skills have a source that is not a GitHub repo, and 4 folders were not found.

Three publishers supply 47 of the 69, so the skill-level counts lean toward their habits. Install counts on skills.sh are per package, so we treat them as weight, not as users.

Then regular expressions and file listings measured:

frontmatter keys beyond name and description
vendor tool names (WebFetch, Skill tool, AskUserQuestion, Bash(...), mcp__)
agent product names (Claude Code, Codex, Cursor, Gemini CLI, Copilot, CLAUDE.md or GEMINI.md)
hard-coded vendor paths (~/.claude/, .cursor/, .codex/, .gemini/)
bundled scripts, by language
length, and whether the text contains a fallback phrase

"Coupled" below means at least one of: a vendor tool name, an agent product name, a vendor path, allowed-tools, or a frontmatter key outside the public spec.

What we found

40 of 69 (58%) show no coupling. 29 (42%) show at least one. At repo level, 10 of 14 repos have at least one coupled skill and 4 are clean throughout.

Hard-coded vendor paths: 0 of 69. Nobody points at ~/.claude/ or .cursor/, and nobody uses a $CLAUDE_* variable. The path problem is not in this sample.

Frontmatter is where it leaks. 18 of 69 use only name and description. 51 add keys, mostly spec keys (metadata 31, license 21). 17 use a key outside the spec: disable-model-invocation 12, version 3, argument-hint 2, and one each of hidden, displayName, emoji and homepage. 5 use allowed-tools, each naming a command-line tool through Bash(...).

Vendor tool names in the text: 17 of 69. Skill tool 7, Bash(...) 5, WebFetch 3, AskUserQuestion 1, mcp__ 1 (a skill can name several). Leaving out Bash(...), which only shows up in allowed-tools, it is 12 of 69.

Agent product names: 6 of 69. Claude Code 4, Codex 4, Cursor 2, CLAUDE.md or GEMINI.md 3. Some of those are lists of products the skill works with, which is the opposite of lock-in, so we count mentions without calling them problems.

Fallbacks are rare. Only 17 of 69 contain a phrase like "if you cannot..." or "ask the user to paste". Of the 29 coupled skills, 23 have none, so when the named tool is missing the instructions stop.

Most skills are plain text. 55 of 69 bundle no script. The 14 that do use shell (7), PowerShell (7), JavaScript (5), Python (4) and TypeScript (1); a skill can have several. Only 4 of those 14 also have a fallback phrase.

They are short. Median 105 lines. 56 of 69 are 200 lines or fewer, and one is over 500. 35 of 69 keep long material in a references/ folder.

Limits

Regexes see strings, not meaning, and a fallback phrase does not prove the fallback works. We scanned only SKILL.md, not references/ files or scripts. We did not run any skill in any assistant. It is one snapshot, and 31 of the 100 are missing from it.

A checklist for portable skills

Keep frontmatter to name and description. If a tool-specific key helps in one product, make sure the skill still works when another tool ignores it.
Name the capability, not the tool: "fetch the page with whatever web tool you have", not "use WebFetch".
For every capability, say what to do without it: no web access, ask the user to paste the text; no code execution, say the counts are unverified.
Use paths relative to the skill folder. In install notes, list .agents/skills/ first and tool-specific folders second.
Scripts: one common language, standard library, runtime stated in the README, nothing essential locked in shell-only or PowerShell-only code.
Write a description that says what the skill does and when to use it.
Keep SKILL.md short and move reference material into linked files.
Ship a paste-in prompt for chat apps that cannot load skills.
Read the skill as an assistant with no special tools. Wherever a step is impossible, add the fallback.

Reproduce it

The fetch script, analysis script, raw table and chart code are in the study folder: https://proskillpacks.github.io/study/portability/ . The method is short enough to re-run on a different day or a different leaderboard.

We make portable skills, with a paste-in prompt for every one. The free ones are at https://github.com/proskillpacks/skills