Last week I measured three agent skills twice over: once with Claude Code's built-in claude plugin eval, and once with a runner I maintain. Same skill text, same task prompts, same rubrics.
I expected the interesting part to be which tool scored higher. It wasn't. The interesting part was a result that looked excellent and turned out to be unattributable.
The setup
Three skills from the public addyosmani/agent-skills pack: code review and quality, git workflow and versioning, documentation and ADRs. One case each, every case run twice, once with the skill and once without.
The two tools do not do the same thing, and that matters for everything below.
The built-in eval installs the skill as a real plugin. The model has to discover it and call it through the Skill tool, the task runs with tools in a workspace, and each run is graded pass or fail. Three runs per arm.
My runner puts the SKILL.md text straight into context, so exposure is guaranteed, and scores the answer continuously from 0 to 1 across several draws per arm. It does not test discovery at all.
So one measures discovery plus application plus execution, and the other measures application given guaranteed exposure. Target model and judge were claude-opus-5 in both.
The result that started this
Documentation and ADRs produced the largest apparent benefit in the whole run. Three of three tasks passed with the plugin, one of three without it.
Then I read the traces. The Skill tool was not called in any of those three runs. I added a supplementary run to be sure. It was not called there either. Zero of three, and zero of one.
Something differed between the two arms. The evidence does not say the skill is what differed. With three runs, two failures in the baseline arm are enough to produce a result that reads like a large effect, and nothing in the record ties the passing arm's behaviour to the skill firing.
That is not the built-in eval being wrong. Its output simply cannot support the causal reading, which is a different problem and a more common one.
The opposite failure, one skill down
Code review and quality showed no benefit at all: three of three passing in both arms, a delta of exactly zero.
Scored continuously on the same inputs, the same skill read 0.918 with it against 0.783 without.
Both arms had cleared the pass threshold, so pass/fail had nothing left to report. A zero delta there means the test could not see a difference, not that there wasn't one.
A third way to pass while being wrong
Separately I ran the whole plugin over three tool-using tasks: review a working tree change, make atomic commits, write an ADR in the repository's own convention. Three runs per arm.
Every session passed. Eighteen of eighteen against the native graders, nine with the plugin and nine without, and eighteen of eighteen against post-session verification.
One of the ADRs passed every structural check while asserting a history the fixture never supplied. The file existed, the format was right, the tools had been used, every deterministic check was green, and part of the content was invented.
What this does not prove
One case per skill, so these are effects on those cases and not on the skills in general.
Three runs per arm in the native tool, so a single run moves a pass rate by 33 points.
The two tools measure different things, and I do not compare their numbers with each other.
The ± figures from my runner are the sample standard deviation across draws, a descriptive spread with no coverage probability. Nothing here is a significance claim.
The same model generated and judged, which I did to match the native setup and which carries an obvious self-preference risk.
None of that makes the findings useless. It makes them exploratory, which is what one case per skill can support.
What I check now
Two things, before reading any with-and-without number.
Was the skill invoked at all in the with arm? If activation is not recorded anywhere, the difference is not attributable to the skill, whatever its size.
Did both arms sit at the ceiling? If everything passed either way, the task was not hard enough to answer the question being asked.
Neither check needs my tooling. Both are readable from what the built-in eval already produces.
Someone else hit this from the other side
While I was writing this up, the Superpowers plugin shipped a rebuild of its executing-plans skill after measuring the old one as, in the maintainer's words, "indistinguishable from running with no plugin at all": same tokens, same duration, same shipped bug, every repetition. The measurement is in the pull request. A maintainer publishing a negative result about their own skill is rarer than it should be.
OpenAI's developer post on rethinking skills and prompts for GPT-6 Astra makes a related point from the model side: guidance written for an older model can cost you rather than help you, so a release is a reason to recheck what you already shipped.
The artefacts
Both tools' raw output, every CLI call and the receipt-to-output hash audit are published with the write-up: driftproofhq.com/reports/009
Happy to answer methodology questions.
Disclosure: the continuous runner is Driftproof, which I build and maintain (open source, Apache 2.0). The built-in eval is Anthropic's.
Top comments (2)
The "skill that was never invoked" finding is the sharpest part of this write-up. Documentation and ADRs showed the biggest apparent benefit, but zero Skill tool calls in any of the passing runs. That's not the eval tool being broken — it's a causal attribution problem that's genuinely hard to catch without checking activation traces. Most people would publish the delta and call it a win.
The ceiling problem on code review (three of three passing in both arms, delta of zero) is the other thing that doesn't get enough attention in skill evaluation discussions. Pass/fail with easy cases is almost completely uninformative about whether the skill is doing anything. The continuous 0.918 vs 0.783 result on the same cases at least shows something. The question of which signal is more useful depends on what you're actually trying to answer.
The ADR that passed every structural check while hallucinating history is the hardest failure mode to catch. Deterministic checks (file exists, format correct, tools used) all green, content wrong. This is where rubric design matters more than scoring method — you need a check that actually reads the content, not just validates its shape.
The two checks at the end — was the skill invoked, did both arms sit at the ceiling — are exactly the right failure-mode audit before treating any with/without number as meaningful. These should probably be pre-run requirements rather than post-hoc sanity checks.
The Superpowers rebuild story is a good companion point. Publishing a negative result about your own skill takes something. That kind of measurement culture is still rare.
The pre-run framing is right, and it is where this is going. Both checks work better as preconditions than as habits: refuse to report a delta when the with arm carries no activation record, and refuse when every case in both arms clears the threshold, because at the ceiling the instrument cannot see anything either way. As post-hoc checks they depend on someone remembering. As preconditions they are just a refusal state.
On the ADR case I do not have a good answer yet. A check that actually reads content is a judge, and judges carry their own variance, so you have moved the problem up a layer rather than solved it. That is the honest position and it is why the report stops where it does.