This week Claude Code 2.1.269 made claude plugin eval public: an official runner for plugin eval suites, with generated starter cases, a no-plugin ablation arm by default, an HTML report and JSON output. I maintain config-drift-checker, a tool that had shipped its own compatible runner while the official one was gated. So release day was migration day, and these are the notes.
The gotchas
1. The frontmatter whitelist rejects, it doesn't warn. My cases carried a custom covers: key for coverage tracking. Under my own lenient parser, fine. Under the official runner, every case carrying it failed to load with "unknown frontmatter key". If you attach custom metadata to cases, put it in a sidecar file next to prompt.md; the runner ignores files it doesn't know.
2. The YAML parser is strict. description: Proves the skill shapes generated Java: envelope, injection dies on the second colon. Two of my six cases failed this way. Quote any description containing ": ".
3. --case matches the name, not the directory. My case lived in evals/spring-work-triggers-skill/ with name: Spring work triggers the conventions skill. Filtering by the directory name matched nothing. Cost me a puzzled ten minutes.
4. The transcript is ephemeral. Tool calls and the full response go to a temp trace file, deleted when the command exits. The JSON you keep has scores, turns and cost, but not what the agent did. Anything that needs the transcript has to run inside the same session, or accept not knowing.
That last one had a real consequence for me: my diff labels likely model refusals (zero tool calls, one turn, short reply) so a guardrail change doesn't masquerade as your setup breaking. With no transcript, "zero tool calls" is indistinguishable from "unknown". The label now requires actual evidence and never fires on official-runner results. If you build analysis on top of this JSON, decide early what "unknown" means in your pipeline.
What I deleted, and what I didn't
The migration was also a strategy decision. The official runner made my runner a fallback for old versions, and that's fine: the runner was never the product. What the official command deliberately doesn't do (the docs tell you to pin your model so results stay comparable) is remember anything. No stored baseline. No history. No answer to "did this get worse since last month", "is this case just flaky", or "which Claude Code release broke it".
So that's the product: same cases, official format, official runner underneath, and on top a pinned baseline, history across every release, per-case noise bands learned from each case's own past (one of mine naturally swings 0.75, a fixed threshold would either alarm daily or never), a canary track that runs when a Claude Code release actually ships, and a bump PR once a new model proves itself green twice.
Their command answers "does my plugin work right now on my machine". The layer on top answers "did anything stop working since the baseline, across every release, without me watching".
v0.5.0 with all of the above shipped the same day their release did: https://github.com/jameskomo/config-drift-checker/releases/tag/v0.5.0
The repo, with a full comparison table of built-in vs added: https://github.com/jameskomo/config-drift-checker
If you're building your own layer on their JSON instead, I'd genuinely like to compare notes; the noise-band problem in particular is deeper than it looks.
Top comments (0)