Anthropic shipped an official eval runner for Claude Code. I made my tool run on it the same day. Field notes.

This week Claude Code 2.1.269 made claude plugin eval public: an official runner for plugin eval suites, with generated starter cases, a no-plugin ablation arm by default, an HTML report and JSON output. I maintain config-drift-checker, a tool that had shipped its own compatible runner while the official one was gated. So release day was migration day, and these are the notes.

The gotchas

1. The frontmatter whitelist rejects, it doesn't warn. My cases carried a custom covers: key for coverage tracking. Under my own lenient parser, fine. Under the official runner, every case carrying it failed to load with "unknown frontmatter key". If you attach custom metadata to cases, put it in a sidecar file next to prompt.md; the runner ignores files it doesn't know.

2. The YAML parser is strict. description: Proves the skill shapes generated Java: envelope, injection dies on the second colon. Two of my six cases failed this way. Quote any description containing ": ".

3. --case matches the name, which defaults to the directory. My case lived in evals/spring-work-triggers-skill/ with a pretty name: Spring work triggers the conventions skill, and filtering by the directory matched nothing. The real fix (credit to a commenter below) is better than any workaround: set no name: at all. It defaults to the folder name, so the filter and the report keys align with your directories. My suite dropped its explicit names the same day.

4. The transcript is ephemeral, but grader verdicts survive it. Tool calls and the full response go to a temp trace file, deleted when the command exits. The JSON you keep has scores, turns, cost, and (another correction from the comments) the per-run verdicts of the cheap graders with their explanations: regex, tool_used and file_exists are computed from the transcript before it vanishes, and lines like "Skill called 1x" persist forever. What you lose is the raw material for anything you didn't write a grader for. If you need that too, run with --keep-temp and copy the trace out before you exit.

What I deleted, and what I didn't

The migration was also a strategy decision. The official runner made my runner a fallback for old versions, and that's fine: the runner was never the product. What the official command deliberately doesn't do (the docs tell you to pin your model so results stay comparable) is remember anything. No stored baseline. No history. No answer to "did this get worse since last month", "is this case just flaky", or "which Claude Code release broke it".

So that's the product: same cases, official format, official runner underneath, and on top a pinned baseline, history across every release, per-case noise bands learned from each case's own past (one of mine naturally swings 0.75, a fixed threshold would either alarm daily or never), a canary track that runs when a Claude Code release actually ships, and a bump PR once a new model proves itself green twice.

Their command answers "does my plugin work right now on my machine". The layer on top answers "did anything stop working since the baseline, across every release, without me watching".

v0.5.0 with all of the above shipped the same day their release did: https://github.com/jameskomo/config-drift-checker/releases/tag/v0.5.0

The repo, with a full comparison table of built-in vs added: https://github.com/jameskomo/config-drift-checker

If you're building your own layer on their JSON instead, I'd genuinely like to compare notes; the noise-band problem in particular is deeper than it looks.

Comments

[Skillselion]

Two refinements from the plugin-evals reference, both of which make your gotchas cheaper for the next person.

On gotcha 3, the documented rule is narrower than it looks: name defaults to the directory name, and --case globs match name. So directory filtering works fine right up until you set an explicit name: that differs from the folder, which is exactly what bit you. Drop the name: key and your directory names become the filter for free, no glob gymnastics. The report keys on name too, so a mismatch costs you the ability to grep your own results by folder.

On gotcha 4, you have more evidence than you think. regex, tool_used, tool_order and file_exists are all computed from the transcript and files and cost nothing, and their per-run verdicts persist in the JSON and the report. So the "zero tool calls versus unknown" ambiguity does not need the transcript, it needs a probe: a tool_used or tool_order grader that is expected to pass on a healthy run. If it reports a real fail, the agent ran and did not call the tool. If the run errored out, you get the error instead. That converts your inference from turn counts into a recorded verdict, which is also the shape your noise bands want, since a free grader adds no cost per arm.

[SmartWord World]

@skillselion Both points are right, thank you. I verified the second against my own results: the tool_used verdict and its explanation ("Skill called 1x") do persist per run in the JSON, so more transcript-derived evidence survives than my gotcha implied. I'm correcting the article.

And I took the first one literally: my example suite dropped its explicit name: keys about an hour after your comment, so folder names are now the filter under both runners, verified with a run where --case 'spring-work-triggers-skill' (the directory) matches under the official runner. Best kind of feedback: the commit.

Are you building on the plugin-evals JSON yourself? Would genuinely like to hear what for.

[Skillselion]

Honest answer: no tooling on it. We run Skillselion, a catalog of skills, so evals reach us as reading material. When we write a deep dive on a skill, we read whatever evals the repo ships next to SKILL.md (some carry an evals/evals.json) to see what the author thinks "working" means. We have not built anything that parses plugin-evals output. And thanks for committing the name: keys. Now that the suite runs under both runners, did any case get a different verdict from the official one than from your own?

[SmartWord World]

Yes, and how it happened is the best argument I have for why the tool exists.

When Claude Code 2.1.287 shipped, my "Vue request should not fire the Spring skill" case flipped from pass to fail under the official runner. Nothing about the agent had changed. The release made tool_used default to min: 1, so my grader with only max: 0 now meant "called between 1 and 0 times" and could never pass. Same transcript, same skill, new verdict.

Run on its own, the official runner shows you a red case and a score. It can't tell you the case was green on the previous release, that the release is what changed, or that the cause is the format rather than the behaviour. That's not a criticism. It's a test runner, and it does that job well. But it stops at "this run, this version, this score."

That gap is what config-drift-checker covers:

• It runs the suite on every Claude Code release, against a pinned baseline, so a flip shows up as "this case regressed between 2.1.286 and 2.1.287" instead of a red score with no context.
• It separates format drift from behaviour drift. A suite doctor loads every case through the installed runner with a $0 cost ceiling before any paid run. A format change like this now surfaces as a load error, not a fake regression. It loaded all six cases cleanly on 2.1.295 this morning.
• It keeps history and noise bands per case, so one bad LLM-judge run doesn't page you, but a consistent drop does.
• It bisects across releases when you need to know exactly which version broke a case.

There was a second disagreement in the other direction: a grader pointed at a file the scaffold writes, which the official runner read and my runner didn't. I fixed that on my side. I treat the official runner as the authority on what a verdict is; my job is to notice when it changes.

After both fixes the suite passes 6 of 6 under both runners. And thanks for explaining how Skillselion reads the evals/ folder: it's a good reason to keep cases readable as documentation, not just as tests.

One note on accuracy: when 2.1.287 broke things, our canary is what caught it, but the suite doctor didn't exist yet. I built it afterwards, and the post says so ("now surfaces"), so we aren't claiming it caught this one.