The biggest improvement in my skill evaluation came from a skill that was never invoked

Driftproofhq — Sun, 20 Sep 2026 17:59:55 +0000

Last week I measured three agent skills twice over: once with Claude Code's built-in claude plugin eval, and once with a runner I maintain. Same skill text, same task prompts, same rubrics.

I expected the interesting part to be which tool scored higher. It wasn't. The interesting part was a result that looked excellent and turned out to be unattributable.

The setup

Three skills from the public addyosmani/agent-skills pack: code review and quality, git workflow and versioning, documentation and ADRs. One case each, every case run twice, once with the skill and once without.

The two tools do not do the same thing, and that matters for everything below.

The built-in eval installs the skill as a real plugin. The model has to discover it and call it through the Skill tool, the task runs with tools in a workspace, and each run is graded pass or fail. Three runs per arm.

My runner puts the SKILL.md text straight into context, so exposure is guaranteed, and scores the answer continuously from 0 to 1 across several draws per arm. It does not test discovery at all.

So one measures discovery plus application plus execution, and the other measures application given guaranteed exposure. Target model and judge were claude-opus-5 in both.

The result that started this

Documentation and ADRs produced the largest apparent benefit in the whole run. Three of three tasks passed with the plugin, one of three without it.

Then I read the traces. The Skill tool was not called in any of those three runs. I added a supplementary run to be sure. It was not called there either. Zero of three, and zero of one.

Something differed between the two arms. The evidence does not say the skill is what differed. With three runs, two failures in the baseline arm are enough to produce a result that reads like a large effect, and nothing in the record ties the passing arm's behaviour to the skill firing.

That is not the built-in eval being wrong. Its output simply cannot support the causal reading, which is a different problem and a more common one.

The opposite failure, one skill down

Code review and quality showed no benefit at all: three of three passing in both arms, a delta of exactly zero.

Scored continuously on the same inputs, the same skill read 0.918 with it against 0.783 without.

Both arms had cleared the pass threshold, so pass/fail had nothing left to report. A zero delta there means the test could not see a difference, not that there wasn't one.

A third way to pass while being wrong

Separately I ran the whole plugin over three tool-using tasks: review a working tree change, make atomic commits, write an ADR in the repository's own convention. Three runs per arm.

Every session passed. Eighteen of eighteen against the native graders, nine with the plugin and nine without, and eighteen of eighteen against post-session verification.

One of the ADRs passed every structural check while asserting a history the fixture never supplied. The file existed, the format was right, the tools had been used, every deterministic check was green, and part of the content was invented.

What this does not prove

One case per skill, so these are effects on those cases and not on the skills in general.

Three runs per arm in the native tool, so a single run moves a pass rate by 33 points.

The two tools measure different things, and I do not compare their numbers with each other.

The ± figures from my runner are the sample standard deviation across draws, a descriptive spread with no coverage probability. Nothing here is a significance claim.

The same model generated and judged, which I did to match the native setup and which carries an obvious self-preference risk.

None of that makes the findings useless. It makes them exploratory, which is what one case per skill can support.

What I check now

Two things, before reading any with-and-without number.

Was the skill invoked at all in the with arm? If activation is not recorded anywhere, the difference is not attributable to the skill, whatever its size.

Did both arms sit at the ceiling? If everything passed either way, the task was not hard enough to answer the question being asked.

Neither check needs my tooling. Both are readable from what the built-in eval already produces.

Someone else hit this from the other side

While I was writing this up, the Superpowers plugin shipped a rebuild of its executing-plans skill after measuring the old one as, in the maintainer's words, "indistinguishable from running with no plugin at all": same tokens, same duration, same shipped bug, every repetition. The measurement is in the pull request. A maintainer publishing a negative result about their own skill is rarer than it should be.

OpenAI's developer post on rethinking skills and prompts for GPT-6 Astra makes a related point from the model side: guidance written for an older model can cost you rather than help you, so a release is a reason to recheck what you already shipped.

The artefacts

Both tools' raw output, every CLI call and the receipt-to-output hash audit are published with the write-up: driftproofhq.com/reports/009

Happy to answer methodology questions.

Disclosure: the continuous runner is Driftproof, which I build and maintain (open source, Apache 2.0). The built-in eval is Anthropic's.

DEV Community: Driftproofhq