DEV Community: Kai Ventura

We trained an agent skill instead of writing it. The validation gate rejected most of our edits.

Kai Ventura — Mon, 05 Oct 2026 11:00:27 +0000

A skill is a short markdown file of instructions that a model reads before it does a task. Most people write them by hand and try them on a few examples. The trouble is that advice that sounds sensible can make results worse, and without a score you never notice.

So we trained one instead. The method is a scaled-down version of Microsoft's SkillOpt paper (arXiv 2605.23904). You treat the skill like weights: change it, score it, keep the change only if the score goes up on tasks the change was not made from.

This article was written with the help of an AI assistant and checked against the repo's README files, which are the only source of the numbers below.

The six steps

Freeze the student. The model that does the tasks never changes.
Split the tasks into train, selection and test. Train gives evidence, selection decides what is kept, test is touched once at the end.
Roll out. Run the student on the train tasks with the current skill and record what it did.
Reflect. A separate optimizer model reads the failures and proposes edits for patterns that repeat.
Bound the update. A few edits per step (4, then 3, then 2 here), so a rejection tells you which idea to blame.
Gate. Score the candidate on the selection tasks. Keep it only if the score is strictly higher.

Run 1: a browsing skill

The skill teaches a model to drive the agent-browser CLI. Student: Opus 5.5 running headless with only a Bash tool. Optimizer: Claude Fable 5.1. Tasks: 13 train, 8 selection, 11 test. Opus got every task right with no skill, so the score also gave credit for fewer tokens. Each candidate ran on the selection split twice.

Step	Skill	Edits	Score	Tokens per task	Turns	Decision
0	`s0-seed`	150-word starting skill	0.9435	37.7k	5.2	start
1	`s1`	4	0.9687	20.8k	3.0	accepted
2	`s2`	3 more	0.9644	23.8k	3.2	rejected
3	`s3`	1 more	0.9632	24.5k	3.0	rejected

Held-out test, 11 unseen tasks, two runs each:

Condition	Correct	Tokens per task	Turns per task
No skill	22/22	34.2k	5.0
Vendor's bundled `core` skill	22/22	109.4k	5.5
Seed skill	22/22	33.3k	5.1
Trained skill	22/22	29.4k	4.1

Two of the three edit sets made things worse, and all three read as sensible advice. Only the gate could tell them apart.

The four edits that helped removed waste the model could not have known about: batch several commands into one call; do not chain work to open with &&, because it can time out while the page is still usable; a recipe for JavaScript dialogs and right-click; a short command list so the model stops calling --help.

The edit that hurt was a prevention rule: "never guess selectors on a page you have not seen". It added a snapshot call to tasks where guessing already worked.

Run 2: a starter you can run

The browsing run needs a browser and costs real money. So the repo also has a small starter: 18 toy tasks with code checkers, three families (invoice note to JSON, rewrite a note in house format, build a file name).

The tasks ask for a house style (date formats, id padding, file names) and never say what it is. A model cannot guess it, so the no-skill score is zero by design. The skill is where you write the conventions down.

Student: claude -p --model haiku. Optimizer: Sonnet.

Skill	Selection (6 tasks x 2)	Test (6 tasks x 2)
Seed (two lines)	0 of 12	0 of 12
Candidate `c1` (215 words, 4 edits)	10 of 12, accepted	10 of 12

Caveats, all real. It is one step from 0 to 10 of 12, not a climb, and there was only one candidate, so no rejection. With 12 rollouts per cell, one task is 8 points. The optimizer pasted some training examples into the skill, against our instruction. None was a selection or test input, but it is a leak risk to watch.

We also ran the same c1 skill, trained on Haiku, against a local Ollama model (qwen3.8:27b, 4-bit). With no skill it got 0 of 18. With c1 it got 5 of 6 on the test split. That is one run on 6 tasks, and the skill was never tuned for that model.

What we would do differently

Start with a weaker student or harder tasks. With a strong model the browsing run had little left to train.
Write more candidates, so the gate shows a rejection and not only an acceptance.
Run more repeats on selection, and check every task passes alone before blaming the skill.
Strip training examples out of the proposal before it becomes a candidate.

Run it

# setup
git clone https://github.com/proskillpacks/skillopt-agent-browse && cd skillopt-agent-browse/starter
# 1. baseline (mostly FAIL, by design)
python3 run.py --skill none --split sel --tag baseline --model haiku
# 2. roll out the seed on train
python3 run.py --skill skill/seed.md --split train --tag seed-train --model haiku
# 3. reflect: ask an optimizer for at most 4 edits
python3 reflect.py runs/seed-train --skill skill/seed.md --ask --max-edits 4 --model sonnet

For a local model, set STUDENT_MODEL, OPENAI_BASE_URL=http://localhost:11434/v1 and add --adapter openai. The full walk-through, with the gate command, is in starter/README.md.

Limits

Small task sets, toy tasks in the starter, one student model per run, two to twelve rollouts per cell. Treat the numbers as a demonstration of the method, not a benchmark.

Repo: https://github.com/proskillpacks/skillopt-agent-browse
Paper: https://arxiv.org/abs/2605.23904
Longer write-up (canonical): https://proskillpacks.github.io/research/how-we-train-skills/

We re-tested 69 skills on harder inputs. 16 failed. Here is what broke.

Kai Ventura — Sun, 04 Oct 2026 16:04:36 +0000

Disclosure: we make agent skills (see the end). The checklist below is ours and needs no purchase. Written with AI assistance.

Every skill we ship had already passed one run on a real public input. We read that output, it looked right, and we moved on. Then we gave all 69 skills a second run on an input made to be hard. Sixteen had a defect. About one in four.

None of the 16 failed the first run. That is the point: one passing run tells you the skill can do the job, not that it will.

What "harder" meant

For each skill we picked one of these, using public pages or text we wrote:

Unsure input. Notes full of "I think", "maybe", "I guess".
Contradictions. A brief that wants cheap and premium, or a listing whose title and text disagree.
Thin or blocked sources. Almost no description, a 403, a dead link.
A request to overclaim. "Make it impressive and mention my results" when there are none.
Messy data. Duplicates, a month 13, an amount of 9,999,999.

Then we read each output against the input ourselves, with no grader and no second model.

What broke

What broke	Defects
Unsure input restated as fact	7
Invented or unsupported claim	3
Miscount in a summary	2
Conflict in the brief not named	2
Script misread a compressed response	1
Wrote a file it should not have	1

The largest group is the quiet one. The skill noticed that the input was unsure, marked it once, and then stated it as fact in the text meant for someone else.

A newsletter skill was given notes where a postage change was unconfirmed. Its first subject line was "Postage prices are going up". After the fix it was "Is postage going up this April?".

A host wrote "Smoking outside I guess". The paste-ready rules said "Smoking is outside only." After the fix the setting read "Not settled. Please confirm."

A listing said "5 min from the beach (maybe 10 if the tide is in)". Title option 1 was "Studio 5 min from the beach, queen bed". After the fix the title said "near the beach" and the hedge stayed in the description.

A claim checker wrote "There are 9 claims, and 8 are high risk." The table under it had 7 High and 2 Medium. A CSV profiler that is meant to be read-only wrote "I left orders.csv untouched and wrote the cleaned version to orders_clean.csv", which contradicts itself in one sentence.

Every fix was one or two sentences in the skill's instructions, except one script fix. Every fixed skill passed a re-run on the same input.

What we changed

Two real runs per skill, one normal and one hard. We read both.
One standard rule in the skills that write text for other people: anything the input marks as unsure stays marked in every output, including the final text, and is never restated as fact.
Our free checker now warns when a skill that writes for others never mentions unsure input.

A hard-input checklist for your own prompts

You can run this on any prompt or skill in an afternoon.

Feed it a hedge. Put "I think" or "maybe" next to one fact. Check that the fact is still hedged in the final text, not only in your notes section. Check the title, the subject line and the summary first: short fields lose hedges.
Feed it a contradiction. Two facts that cannot both be true. A good output names the conflict. A bad one picks a side quietly.
Take something away. Remove the key input, or give a page that returns an error. The output should say what it could not read, and should not fill the gap.
Ask it to overclaim. Request results, awards or numbers that are not in the input. It should refuse and mark the gap.
Count what it counts. Any "N items" in a summary, check against the list. Do the arithmetic yourself.
Check what it did, not only what it said. If a tool is read-only, confirm nothing was written. Compare the closing sentence with the actions.
Read the shortest field last. Titles, subject lines, previews and alt text are where unsure facts become firm.
Re-run the fix. A fix you did not re-run is a guess.

Limits

All runs used Claude Sonnet, so another model may fail elsewhere. We judged the outputs ourselves. One hard input per skill found 16, and a third run might find more. Several inputs were written by us to be hard, so they show behaviour on that kind of input, not how often real use will hit it.

The full table, the quoted before and after lines and the data are on the study page: https://proskillpacks.github.io/study/retest/?utm_source=devto&utm_medium=post&utm_campaign=devto-retest

We make agent skills. The free ones are at https://github.com/proskillpacks/skills

How portable are the most-installed agent skills? We measured 69 of them

Kai Ventura — Sun, 04 Oct 2026 09:10:36 +0000

Disclosure: we make agent skills (see the end). The measurements are script output, and every figure comes from the published data files. Written with AI assistance.

Agent skills are meant to be portable: a folder with a SKILL.md that any assistant can read. We wanted to know how true that is for the skills people actually install. So we measured the top of the skills.sh leaderboard with a script and no model judging anything.

What we did

We took the top 100 skills on skills.sh by all-time installs (2026-10-04). For each one we found SKILL.md through the GitHub tree API and fetched the raw file once, half a second apart. We could fetch 69 skills from 14 repos. 27 listed skills have a source that is not a GitHub repo, and 4 folders were not found.

Three publishers supply 47 of the 69, so the skill-level counts lean toward their habits. Install counts on skills.sh are per package, so we treat them as weight, not as users.

Then regular expressions and file listings measured:

frontmatter keys beyond name and description
vendor tool names (WebFetch, Skill tool, AskUserQuestion, Bash(...), mcp__)
agent product names (Claude Code, Codex, Cursor, Gemini CLI, Copilot, CLAUDE.md or GEMINI.md)
hard-coded vendor paths (~/.claude/, .cursor/, .codex/, .gemini/)
bundled scripts, by language
length, and whether the text contains a fallback phrase

"Coupled" below means at least one of: a vendor tool name, an agent product name, a vendor path, allowed-tools, or a frontmatter key outside the public spec.

What we found

40 of 69 (58%) show no coupling. 29 (42%) show at least one. At repo level, 10 of 14 repos have at least one coupled skill and 4 are clean throughout.

Hard-coded vendor paths: 0 of 69. Nobody points at ~/.claude/ or .cursor/, and nobody uses a $CLAUDE_* variable. The path problem is not in this sample.

Frontmatter is where it leaks. 18 of 69 use only name and description. 51 add keys, mostly spec keys (metadata 31, license 21). 17 use a key outside the spec: disable-model-invocation 12, version 3, argument-hint 2, and one each of hidden, displayName, emoji and homepage. 5 use allowed-tools, each naming a command-line tool through Bash(...).

Vendor tool names in the text: 17 of 69. Skill tool 7, Bash(...) 5, WebFetch 3, AskUserQuestion 1, mcp__ 1 (a skill can name several). Leaving out Bash(...), which only shows up in allowed-tools, it is 12 of 69.

Agent product names: 6 of 69. Claude Code 4, Codex 4, Cursor 2, CLAUDE.md or GEMINI.md 3. Some of those are lists of products the skill works with, which is the opposite of lock-in, so we count mentions without calling them problems.

Fallbacks are rare. Only 17 of 69 contain a phrase like "if you cannot..." or "ask the user to paste". Of the 29 coupled skills, 23 have none, so when the named tool is missing the instructions stop.

Most skills are plain text. 55 of 69 bundle no script. The 14 that do use shell (7), PowerShell (7), JavaScript (5), Python (4) and TypeScript (1); a skill can have several. Only 4 of those 14 also have a fallback phrase.

They are short. Median 105 lines. 56 of 69 are 200 lines or fewer, and one is over 500. 35 of 69 keep long material in a references/ folder.

Limits

Regexes see strings, not meaning, and a fallback phrase does not prove the fallback works. We scanned only SKILL.md, not references/ files or scripts. We did not run any skill in any assistant. It is one snapshot, and 31 of the 100 are missing from it.

A checklist for portable skills

Keep frontmatter to name and description. If a tool-specific key helps in one product, make sure the skill still works when another tool ignores it.
Name the capability, not the tool: "fetch the page with whatever web tool you have", not "use WebFetch".
For every capability, say what to do without it: no web access, ask the user to paste the text; no code execution, say the counts are unverified.
Use paths relative to the skill folder. In install notes, list .agents/skills/ first and tool-specific folders second.
Scripts: one common language, standard library, runtime stated in the README, nothing essential locked in shell-only or PowerShell-only code.
Write a description that says what the skill does and when to use it.
Keep SKILL.md short and move reference material into linked files.
Ship a paste-in prompt for chat apps that cannot load skills.
Read the skill as an assistant with no special tools. Wherever a step is impossible, add the fallback.

Reproduce it

The fetch script, analysis script, raw table and chart code are in the study folder: https://proskillpacks.github.io/study/portability/ . The method is short enough to re-run on a different day or a different leaderboard.

We make portable skills, with a paste-in prompt for every one. The free ones are at https://github.com/proskillpacks/skills