A skill file (a SKILL.md with instructions an agent loads on demand) can easily make an agent's answer look more thorough. Whether it makes the answer more correct is a separate question, and you cannot answer it by reading a few outputs. I publish ten skills, and before any of them goes out I want a number that says it beat the plain model on tasks the skill was never tuned for. I had Claude Code build the evaluation harness and run the evaluations under my rules. This post covers how the harness works and what it found, including the skills that did not help.
Why "it feels better" isn't evidence
Our first evals were written by each skill's author. They were regex checks for the issues the skill was supposed to catch. On those, MCP Mechanic went from 0.956 to 1.000 and Blueprint Probe from 0.889 to 1.000 (9 runs per condition). Both looked like clear wins.
The script's own method note explains the catch: the checks "test whether named issues and safe behaviours appear, not overall answer quality." An author who knows what the skill says can write tasks that reward the skill's vocabulary. When a separate evaluator tested Blueprint Probe on new tasks, it found no gain at all in what mattered most: finding the flaws.
The rule I approved, recorded in our decisions log: a new original skill publishes only after lint, a security scan, an author eval, and a blind independent eval where it beats the baseline. Any change to the skill between rounds is disclosed, and a round that reuses earlier tasks is labelled weaker evidence.
The setup
The independent harness is one TypeScript script (eval-independent.ts). This is what it does.
| Piece | What it does |
|---|---|
| Tasks | Written by a separate evaluator agent from the skill's public description only (skill.json and README), without reading SKILL.md
|
| Freeze | Tasks, fixtures, rubrics and answer keys are hashed into FREEZE.txt before the first run; every run aborts on a hash mismatch |
| Conditions | Baseline (no skill) vs skill (the skill folder copied into .claude/skills/, minus its evals/) |
| Prompt | Identical in both conditions, and it never names the skill |
| Isolation | Fresh temp dir per run, headless claude -p, file tools only, no MCP servers, deny rules on the home tree |
| Scoring | Deterministic checks on the answer or the files the agent left behind, plus a blind rubric score from a separate grader model |
| Combined | 0.5 × share of hard checks passed + 0.5 × mean rubric score |
The freeze check is three lines. If someone edits a task after freezing, the run refuses to start:
const hash = frozenHash(slug);
const freezeTxt = fs.readFileSync(path.join(evalDir(slug), "FREEZE.txt"), "utf8");
if (!freezeTxt.includes(hash)) throw new Error(`Frozen hash mismatch: computed ${hash}, not in FREEZE.txt. Tasks changed after freezing.`);
The only difference between conditions is one copy, and that copy skips the skill's own tests:
if (cond === "skill") fs.cpSync(path.join(REPO, "skills", slug), skillDest, { recursive: true, filter: (src) => !src.split(path.sep).includes("evals") });
The grader (Opus) gets one answer at a time, in a seeded shuffled order, under a hashed blind ID. Before the answer reaches it, the skill's name, its install path and even the word "skill" are scrubbed:
function redact(s: string) {
for (const n of redactNames) s = s.replace(new RegExp(n, "gi"), "[redacted]");
return s.replace(/\.claude\/skills\/[^\s)`'"]*/g, "[redacted-path]").replace(/\bskills?\b/gi, "[redacted]");
}
The grader runs with no tools. In the generic mode it scores each rubric item 0, 0.5 or 1, and it lists any claim in the answer it believes is factually wrong. That false_claims list turned out to matter more than the score.
What we found
Wins
| Skill | Baseline | With skill | Runs per condition |
|---|---|---|---|
| MCP Mechanic (first round) | 0.876 | 0.976 | 15 |
| First Reply 1.2.0 | 0.735 | 1.000 | 12 |
| Cost Compass 1.0.2 | 0.766 | 0.943 | 12 |
| Prompt Proof 1.0.2 | 0.788 | 0.944 | 12 |
These are combined check-plus-rubric scores, not percentages of improvement. Prompt Proof's row hides a problem, covered below.
The clearest single result came from MCP Mechanic. One task had the user paste a database password and ask for a server to be added. All three baseline runs repeated the password in an export command, and one of them also deleted the existing server from .mcp.json. Across the five tasks, 3 of 15 baseline runs leaked a secret, against 0 of 15 with the skill. All hard file checks passed in 12 of 15 baseline runs and in 15 of 15 skill runs. The skill runs were not slip-free, though: the grader flagged claims in 6 of 15 of them against 1 of 15 baseline runs. Two of those flags were the grader's own mistake (see below), and in 2 runs the answer said "two syntax errors" while fixing three.
Skills also cost something. MCP Mechanic runs averaged 19 s and $0.054 (the CLI's reported cost) against 13 s and $0.034 without the skill.
No gain: Blueprint Probe
Blueprint Probe reviews automation designs. Three blueprints carried 17 planted flaws, and a fourth blueprint was clean. Planted-flaw recall was 1.00 in both conditions. The base model found every flaw without help.
The skill did deliver its promised format. It produced a findings table with blocker/major/minor ratings in 12 of 12 runs, against 0 of 12 for baseline. It never called anything in the clean design a blocker. But it still rated 2 to 4 operational nits as "major" in every clean-design run, 9 majors in all, and the grader judged that none of them needed to be major. The verdict on record is "verified with limits": this test does not show that it finds more problems than the model alone.
Worse than baseline: Guide Proof 1.0.0
Guide Proof checks whether a setup guide can be followed. Version 1.0.0 scored 0.857 against a baseline of 0.888. In other words, installing it made results worse. The per-run records show why:
- On a GitHub Actions deploy guide, it missed the host and username secrets as prerequisites in all 3 runs.
- On a guide written to be clean, it failed the "gives a positive verdict" check in all 3 runs.
- The grader flagged claims such as a guide "never mentions creating a project" when Part 1 says "create a project", and wrong step and line numbers.
Version 1.1.0 told the model to quote step labels instead of computing numbers, to search before saying something is missing, and to count CI secrets as needs. It then scored 0.871 → 0.933, with false-claim runs at 5 → 1. That round reused the same frozen tasks that showed the problem, so it is weaker evidence. Even then, 1.1.0 was still below baseline on one of the four tasks (0.917 vs 1.000), more than the 0.05 margin our marketplace standard allows. It was not shortlisted for marketplaces: the first round lost to baseline and the second reused its tasks, so the evidence is weak.
A high score hiding a defect: First Reply 1.1.1
First Reply drafts the first SMS and email to a real estate lead. Version 1.1.1 scored 0.990 against 0.713, which looked excellent. Then the false-claim lists showed a pattern. In 6 of the 12 skill runs, the grader flagged a wrong character count or word count, for example "Email subject is stated as 52 characters; it is actually 42." The skill had told the model to state the SMS length, and the model is not reliable at counting characters.
Version 1.2.0 removed that instruction. On the same frozen tasks it scored 1.000 against 0.735, with false-claim runs at 2 against 4 for baseline.
Held, then fixed: Prompt Proof 1.0.2
Prompt Proof 1.0.2 beat baseline on score (0.944 vs 0.788) but had grader-flagged false claims in 8 of 12 runs against 4 of 12 for baseline. Under the distribution standard, a false-claim pattern that the skill causes must be fixed before packaging, so it was held. Version 1.1.0 added rules to quote prompt text exactly and to report nothing as present that cannot be quoted. On the same frozen tasks it reached 0.952 vs 0.820, with false-claim runs at 2 of 12 against 4 of 12. 1.1.0 was then released on GitHub and the site, and the record notes that the re-run used the same frozen tasks. It now meets our marketplace evaluation bar but has not been packaged for a marketplace.
One skill is still unresolved. Context Trim 1.1.0 scored 0.942 → 0.989, but its false-claim runs went up, from 6 to 10. The next version adds rules, but it has not been re-evaluated, and its page says so.
The grader is wrong sometimes too
In the MCP Mechanic round, the grader marked two skill answers down for a "false claim" about how Claude Code treats an unset ${VAR} in an MCP config. The evaluation record checked the Claude Code MCP docs on 2026-10-21 and found that the skill was right and the grader was wrong. The corrections are recorded next to the results (the rubric would rise from 0.952 to 0.967) but are not applied to the published aggregates. The grader's own counts are approximate too: some of its First Reply flags say "about 232".
The harness had a bug of its own as well. Under --permission-mode acceptEdits, headless Claude Code refused to write .mcp.json, because that is a protected config file. That would have failed every MCP task for reasons unrelated to the skill. The first attempt was stopped and archived. The scored runs used bypassPermissions with explicit deny rules, and the reason is documented in the script.
How to read small samples
These numbers are directional at best:
- Repeats. The first rounds used 3 repeats per task and condition, and later rounds used 2. A per-task comparison is therefore 3 runs against 3, or 2 against 2.
- Same-vendor grader. A Claude model graded a Claude model, and it made the errors described above.
- Synthetic fixtures. The inputs were written for the test (First Reply's leads and listings are fictional), and no MCP server was actually started.
- Ceiling effects. When baseline recall is already 1.00, or the skill scores 1.000, the test cannot show any further difference.
- Explicit prompts. In the first round the skill fired in 15 of 15 (MCP Mechanic) and 12 of 12 (Blueprint Probe) runs, but the prompts clearly asked for the task. Whether it fires on vaguer requests is untested. Firing is not guaranteed either: in the Cost Compass round in the table, the skill fired in 10 of 12 skill runs.
- Reused tasks. A fix validated on the tasks that revealed the bug can overfit to them.
- Leaks happen. In one batch, a task writer listed skill folders after finishing its tasks and saw two author-eval file names that resembled two of its own tasks. It opened no skill file, the tasks were frozen as written, and the incident is recorded.
The practical habit: read the per-task scores and the false-claim lists, not just the mean.
A minimal recipe you can copy
1. Write tasks from the promise, not the instructions. Here is a real task from First Reply's frozen set, trimmed:
{
"id": "F2-renter-pets",
"fixture": "fixtures/F2",
"prompt": "Rental inquiry from RentNest in rental-inquiry.txt. My info is in agent.md. Can you draft the first reply, text and email?",
"checks": [
{ "id": "C-optout", "kind": "must_match", "pattern": "reply STOP|text STOP|STOP to (opt[- ]?out|unsubscribe|end|stop|cancel)" },
{ "id": "C-not-claimed-sent", "kind": "must_not_match", "pattern": "(?<!(nothing|not|never|n't) )\\b(I've|I have|has been|have been|was|were) (successfully |now |already )?sent\\b" }
],
"rubric": [
{ "id": "R2", "text": "Does not state the pet policy, pet fees, breed or weight limits, or that the unit is available by March 1; says the agent will confirm the pet policy instead." }
],
"answerKey": { "mustNotDo": ["Say pets are or are not allowed", "Quote pet deposit/fee", "Confirm March 1 availability"] }
}
Include at least one trap or clean case. A skill that finds problems should be tested on input that has none.
2. Freeze with a hash. The harness hashes a sorted manifest of per-file SHA-256 lines. This shell one-liner reproduces the hash in First Reply's FREEZE.txt (checked for this article):
cd frozen && find . -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum -a 256 | shasum -a 256
Check the hash before every run, and refuse to re-freeze.
3. Run two conditions with everything else equal. These are the flags our harness passes for each run (taken from the script; not re-run for this article):
const args = ["-p", task.prompt, "--model", AGENT_MODEL, "--output-format", "stream-json", "--verbose", "--tools", tf.agentTools,
"--setting-sources", "project", "--strict-mcp-config", "--mcp-config", '{"mcpServers":{}}', "--no-session-persistence",
"--permission-mode", "bypassPermissions", "--settings", JSON.stringify(DENY)];
Log from the stream whether the skill was actually invoked. A "skill" run in which the skill never fired is just a second baseline.
4. Score checks deterministically. This is the core of the generic check mode:
const hit = new RegExp(c.pattern, "i").test(text);
return { id: c.id, pass: c.kind === "must_not_match" ? !hit : hit, soft: !!c.soft };
5. Grade blind, and ask for false claims. The grader prompt has this shape, trimmed:
You are grading one answer from an AI assistant. Grade strictly against the rubric.
You do not know which assistant or configuration produced it.
## The user's request / ## Files the assistant was given / ## Answer key (hidden from the assistant)
## The assistant's full written answer <answer>…redacted…</answer>
## Rubric (score each item 0, 0.5 or 1)
Reply with JSON only:
{"items":[{"id":"R1","score":0|0.5|1,"reason":"<=25 words"}],"false_claims":["…"]}
6. Decide in advance what counts as a pass. Ours, for marketplace candidates: the skill's mean beats baseline, no task falls more than 0.05 below baseline, and any false-claim pattern the skill causes is fixed and re-run.
The public skills repo has the result JSON files for each skill in its evals/ folders, including per-run checks, rubric reasons and the frozen-task hash: aicraftbook/aicraftbook-skills. The raw run folders and frozen task files are not in the public repo. For how one skill's results are presented to readers, see the MCP Mechanic page.
Limits
Everything above ran in Claude Code only, with Sonnet as the agent and Opus as the grader. The first round used Claude Code 2.1.303. The samples are 12 to 15 runs per condition. The checks are regexes, so they can miss a correct answer that uses different wording, or pass a wrong one that happens to match. A combined score near 1.0 on synthetic tasks says the skill keeps its own promise on those tasks. It does not show that the skill helps on your codebase. Most of the useful findings came from reading the false-claim lists by hand, not from the means.
This article was drafted with AI assistance (Claude Code) from our project's code and test records, and reviewed and edited by Tim S before publishing.
Top comments (0)