There's a whole ecosystem of "agent skills" now — reusable instruction files you drop into Claude Code (or Cursor, or Copilot) to make the model write cleaner code, debug more carefully, use fewer tokens, and so on. Some of these repos have tens of thousands of GitHub stars. Almost none of them ship a single number telling you whether the skill actually does anything.
That bothered me, because "adding a plausible-sounding instruction" and "adding an instruction that works" look identical until you measure them. So I built a benchmark with one rule, committed before I ran anything:
No skill gets merged unless it beats both a no-instruction baseline AND a placebo prompt on its pre-registered target metric, measured on hidden hold-out tests, with accuracy not allowed to drop. Skills that fail are published anyway, with their numbers.
The placebo arm is the part almost nobody runs, and it turned out to be the most important one.
Why a placebo
Most "battle-tested" skill collections that measure anything at all compare skill-on vs skill-off. The problem: that comparison can't separate "this skill works" from "adding any confident-sounding text changes the model's behavior." LLMs are suggestible. If you want to claim your skill did something, you have to show it beats a comparable-length instruction that contains no actual mechanism — just vibes.
So every result here is a three-way comparison — off / placebo / on — run K=5–8 times per task per arm, in isolated git workspaces, graded by hold-out acceptance tests the agent never sees, with every raw run log committed to the repo and the README regenerated from those logs in CI. 496 runs total, all on claude-opus-4-8.
Here's what came out.
Finding 1: the placebo often made code bigger
My anti-over-engineering skill (underkill, ~20 lines) cut source LOC by −23.8% vs baseline at identical accuracy (60/60 hold-out passes). Good. But the interesting column is the placebo: a style-only "write clean, minimal, professional code" instruction (156 words to underkill's 250) didn't reduce code on a single one of the 12 tasks — it increased it on 7 and tied on the other 5 (median +18.8% across the tasks that moved, +6.9% across all 12).
Read that again: telling the model to write clean code made it write more code. The generic exhortation added nothing but tokens, and the model padded to match the vibe. If I'd only compared on-vs-off, underkill would have looked good. Compared against the placebo, it looks good and I can prove the effect isn't just instruction-presence.
Finding 2: a 196k-star skill barely beat "no instructions"
One of the most-installed general behavioral rulesets is the "Karpathy Guidelines" (196k stars as of 2026-07-25) — four rules derived from Andrej Karpathy's observations on LLM coding pitfalls, one of which is "Simplicity First: if you write 200 lines and it could be 50, rewrite it." That's a direct, falsifiable claim on the exact metric underkill targets, so I ran it as a comparison arm on the same 12 tasks.
| arm | median src LOC | hold-out accuracy |
|---|---|---|
| off (no instructions) | 10.5 | 60/60 |
| placebo | 12.5 | 60/60 |
| underkill (mine, ~20 lines) | 8.0 | 60/60 |
| Karpathy Guidelines (~196k★) | 10.0 | 60/60 |
The famous skill beats the placebo clearly (−20%) — so its mechanism is real, not just prompt-presence. But against a no-instruction baseline it barely moves (−4.8%), because on small tasks Opus is already close to minimal. And my focused 20-line skill writes 20% less code than the 196k-star one, at equal accuracy and slightly lower cost.
This isn't a dunk on the Karpathy skill — it addresses four different failure modes and only one is on this axis. The point is narrower and more useful: a rule aimed at one specific failure mode beats a broad, general ruleset that merely mentions it. Instruction presence is cheap; instruction mechanism is what pays.
Finding 3: two of my own four skills failed the gate
Publishing only wins would make the whole exercise worthless, so here are the rejects with their numbers:
-
tests-that-bite(a test-writing discipline skill) hit a ceiling: baseline Opus already killed 100% of my mutation-testing mutants on 3 of 4 tasks. My mutants were too coarse to differentiate. Rejected. -
thrift(a token-spend skill) cut turns by 14% but landed $0.005 above baseline on median cost per run. Under a co-primary "cost AND turns" rule, that's a rejection, not a rounding call.
The thrift sweep had the most counterintuitive result of the whole project: I ran the 92k-star "caveman" terse-mode ruleset as a comparison arm, and it measured more expensive per run than no instructions at all on every cost task. Agentic cost is input-dominated (files read, tool results, repeated context), so an instruction that compresses the model's output can't recoup the input tokens its own prompt adds on every turn. Output-compression advice that's true for chat responses can invert on agentic runs.
What this doesn't prove
The tasks are small and self-contained (the committed reference solutions run 6–63 non-blank source LOC, median 14) — enough to gate a skill on, not enough to generalize to large codebases. K=5–8 gives you evidence, not proof. Pre-registration here is commit-ordering in a repo I control, self-attested (though the harness carries an earlier multi-day record). And hold-out material lives in the same repo, so it's reachable in principle — I run a CI check that scans every committed run log for references to the task material (516/516 committed logs clean). All of this is written up in the repo's "Known Limitations," because a results log that only contains good news isn't worth trusting.
Try it / break it
Two skills passed the gate (underkill, repro-first). They install as a Claude Code plugin or via npx skills add sjh9714/skill-receipts, and the installed snippet is byte-identical to what the benchmark measured (CI enforces that).
If you have a skill you think works — yours, or a famous one you want audited — the gate is the same for everyone: open an issue. I'd genuinely like to be proven wrong on some of these.
Repo + all raw logs: https://github.com/sjh9714/skill-receipts
Top comments (4)
The placebo arm and publishing rejects are exactly the right incentives. I’d make the next version paired: same task, repository snapshot, and sampling seed across arms, then report the paired effect distribution or bootstrap interval instead of only medians. Pre-register a minimum useful effect as well as the metric, and rerun on a second model/version. That separates a real skill mechanism from sampling noise, task-specific ceiling effects, and behavior that disappears after a model update.
Thanks — this is the most useful kind of pushback.
On pairing: the arms already run adjacently per (task, trial) from an identical workspace snapshot, so they share time-of-day / rate-limit / model-load conditions. What I can't pair is the sampling draw itself — the API doesn't expose a deterministic seed for these models — so you're right that some of the spread is unpaired sampling noise. The harness does compute bootstrap CIs internally; I led the published tables with medians for readability, but surfacing the paired effect distribution / interval per arm is a fair ask and probably the highest-value next change.
Pre-registering a minimum useful effect (not just the metric and direction) is the gap I feel most — right now the gate is "beats both controls," which lets a tiny-but-real effect through. And the second-model rerun is exactly the test for "does this survive a model update," which is the failure mode I trust least. Both are going on the list. Thanks again.
Almost nobody runs the placebo arm, and it's the part I'd force into every skill repo README. One methodological question: your placebo was 156 words against underkill's 250, and your own finding says instruction presence pads output. Is the padding dose-dependent? A length-matched placebo with the same token count and no mechanism would tell you whether part of underkill's -23.8% is offsetting padding its own presence induces. Stricter still: a shuffled-sentence variant of underkill itself, same vocabulary, no coherent mechanism. On tests-that-bite, coarse mutants are fixable with real history. Mine your git log for actual bug-introducing diffs and replay those as mutants; they cluster where the code is genuinely fragile, which synthetic operators never do. I run a directory that ranks these skills by install count, and your numbers are the best evidence yet that install counts tell you what got distributed, and nothing about what works. A 196k-star ruleset moving the target metric all of 4.8% against baseline should unsettle people.
This is the best question I've been asked about it, and checking the arithmetic to answer you turned up that I had the number wrong in my own favour.
Counted the way the harness actually injects them — placebo file with its provenance comment stripped, skill snippet verbatim — the shared placebo is 124 words against underkill's 275, not 156 against 250. So the control was running at 45-50% of the skill's length across all four skills, not the ~62% I'd published. The filter I used counted only lines starting with
<!--, so the comment's prose was being counted as placebo text. Corrected in the repo with a test that now asserts the ratio, since I clearly can't be trusted to eyeball it.Which makes your point sharper than I acknowledged: my own finding is that instruction presence pads output, so a control at half the length is a weak control, and part of the -23.8% could be underkill offsetting padding it introduced itself.
So I'm running it. A
placebo-matchedarm, 271 words to underkill's 275, style dimensions only — nothing about size, scope, count, abstraction, validation, or deletion — on the same 12 tasks, K=5. A test asserts the arm contains no size/scope rule, because "trust me, it's mechanism-free" is not good enough for the one file the whole design rests on.Pre-registered the three readings before starting, so I can't pick a story afterwards:
I'm also re-running the 124-word arm in the same window instead of comparing against its numbers from eight days ago, since those were on an older CLI build and I'd rather not fold drift into a length effect.
Result goes up either way — including the one where I have to walk the number back. Pre-registration: github.com/sjh9714/skill-receipts/...