DEV Community

Cover image for Claude Code plugin eval: gate skills on delta before you merge
Dave Kurian
Dave Kurian

Posted on Originally published at otf-kit.dev

Claude Code plugin eval: gate skills on delta before you merge

#agents #ai #llm #testing

Claude Code now ships a first-party way to prove a plugin or skill actually helps. The command is claude plugin eval. It runs your suite with the plugin loaded and again without it, then reports WITH, W/OUT, and Δ — the lift the plugin contributed. That changes what production teams should do before they merge a skill into a kit repo or pin it in CI.

This is not another “write better prompts” essay. It is a builder workflow for Claude Code v2.1.269+: create cases under evals/, grade outcomes, and fail the build when the score drops. If you already ship agent docs in an owned monorepo, pair this with Claude Code on an OTF kit: the CLAUDE.md and prompts that ship with the repo. For feature-level evals outside plugins, keep using A practical LLM evaluation loop for AI features that need to ship — different layer, different artifact.

What changed for plugin authors

Anthropic’s plugin evals docs describe a dedicated CLI path for plugin and skill authors. The important claims for builders:

  • You need Claude Code v2.1.269 or later (claude --version, then claude update if needed).
  • Cases live under an evals/ directory next to your plugin manifest (plugin.json or .claude-plugin/plugin.json).
  • Each case is a realistic user prompt plus one or more graders (regex, tool_used, tool_order, file_exists, llm, baseline).
  • By default each case runs three times with the plugin and three times without it. Δ is with-arm score minus without-arm score.
  • claude plugin eval init can propose cases and graders interactively; claude plugin eval init --bare <case> writes a blank template.
  • CI can gate on --threshold, pin --model / --judge-model, keep reports local with --no-publish, and cap spend with --max-cost-usd.

That is the DO: stop merging skills because a demo chat looked fine. Measure contribution, then gate.

Dex and Luna compare WITH versus WOUT scores while Nova highlights a positive delta

Why WITH alone is not enough

A case that scores 1.00 with the plugin loaded can still be useless. Claude might already solve the prompt without your skill. The docs call this out: if WITH and W/OUT are both high, Δ near zero means the plugin did not move the outcome.

Production implications:

  1. Trigger failures show up as skill graders failing. A common first finding is Δ ≈ 0 with a tool_used: Skill grader failing — Claude never chose your skill on natural phrasing. Fix the skill description, re-run, compare.
  2. Skill-fired graders are indicators, not free Δ inflation. In two-arm mode, tool_used graders on Skill are excluded from both arms’ scored totals so you do not invent lift by checking something impossible without the plugin.
  3. Judge graders cost money. regex, tool_used, tool_order, and file_exists are free transcript/file checks. llm and baseline call a judge model and add to the run’s list-price estimate.
  4. Non-determinism is assumed. Default three runs per arm exist because one agent run is noise. Cheap iteration uses --runs 1 --ablation none; trust needs the default three.

If you only ever run the with-arm, you are grading absolute behavior. That is fine while drafting graders. Before you call a skill “done,” turn the baseline back on and read Δ.

Minimal suite layout you can commit

From the plugin root (the directory that contains the manifest):

claude --version   # need >= 2.1.269
claude plugin eval init --bare commit-message
Enter fullscreen mode Exit fullscreen mode

You get something shaped like:

evals/commit-message/
├── prompt.md
└── graders/
    └── criteria.md
Enter fullscreen mode Exit fullscreen mode

Edit prompt.md so the body is a request a user would type — do not name the skill in the prompt:

---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
---

Write me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites.
Enter fullscreen mode Exit fullscreen mode

Add a result grader (llm or regex) and a skill-fired grader:

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?your-skill-name"'
---
Enter fullscreen mode Exit fullscreen mode

Then run:

claude plugin eval .
Enter fullscreen mode Exit fullscreen mode

Expect six runs for one case at defaults (3× with, 3× without). Open the HTML report path printed at the end. Iterate on description and graders until Δ is positive for the prompts you care about.

For MCP-backed skills, put mocks under evals/mocks/<server>/<tool>.md so CI does not need the real service. Use --scaffold only for suites you trust — scaffold scripts run as you, outside the agent sandbox.

Gate the suite in CI

The docs’ recommended CI shape is explicit:

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20
Enter fullscreen mode Exit fullscreen mode

Exit codes that matter:

  • 0 — every case met --threshold
  • 1 — a case scored below threshold, files failed to load, or trust was missing without --trust-plugin
  • 2 — partial run (--max-cost-usd hit or auth failed); JSON still written with partial: true

Pin both models so a provider rollout is not mistaken for a plugin regression. Keep every-change suites on free graders when you can; reserve llm judges for short outputs with concrete PASS/FAIL rubrics. Leave partial: true results out of trend charts.

Credentials and install still belong on the runner (ANTHROPIC_API_KEY or your normal Claude Code auth). Without --trust-plugin, a non-interactive job against an untrusted checkout exits 1.

Byte pins a CI gate on plugin eval threshold while Nova holds a report card

How this fits an owned kit repo

OTF’s wedge is not “another sandboxed chat.” It is an owned repo agents keep editing — CLAUDE.md, prompts, and skills that travel with the product. Plugin eval is the missing acceptance gate for those skills:

Layer Question Artifact
Repo conventions Can an agent find the seams? CLAUDE.md / .cursorrules (see the kit docs post above)
Change acceptance Did this PR do what we asked? AI coding agent acceptance checklist
Plugin contribution Did the skill raise the score vs no plugin? claude plugin eval + Δ
Product AI features Does the user-facing model path hold? LLM evaluation loop

Put the suite next to the plugin you ship. Fail CI when Δ collapses after a description tweak or a model pin change. That is how skills stay production assets instead of chat folklore.

Security note from the same docs: pointing claude plugin eval at a plugin is the same trust decision as claude --plugin-dir. Only evaluate plugins you trust. Hooks and real MCP servers you opt into with --allow-real-servers or --mocks off run outside the agent sandbox.

Practical checklist before you merge a skill

  1. Upgrade to Claude Code ≥ 2.1.269.
  2. Add at least one natural-language case that should trigger the skill and one that should not.
  3. Include one outcome grader and one tool_used: Skill indicator.
  4. Run with baseline on; require a positive Δ for the should-trigger cases.
  5. Wire the CI command with --trust-plugin, pinned models, --threshold, and --max-cost-usd.
  6. Mock MCP tools for CI; do not call production services from eval runs.
  7. Treat rate-limit errors mid-suite as invalid scores — re-run after the limit resets before you trust the table.

If you are still choosing between a sandboxed MVP host and a repo you own, start with the kit spine, then add this gate. Skills without Δ are demos. Skills with a CI threshold are release machinery.

Sources

Top comments (2)

Collapse
 
raju_dandigam profile image
Raju Dandigam •

@davekurian, WITH/WITHOUT delta is the right correction for skills that merely ride on top of a model already solving the task. I’d avoid gating only on the aggregate mean: require a per-case floor and retain arm-level outcomes so a positive average cannot hide one important regression. Do you also pin the eval fixture and plugin digest in results.json, so a partial rerun or model change cannot be compared as if it were the same experiment?

Collapse
 
jo-do profile image
Jo Do •

Gating on the delta instead of the absolute score is the key move - a skill that makes the suite greener proves itself; a skill that keeps a green suite green is just along for the ride. Failing the build on a negative delta turns "this prompt felt better" into an artifact with a number. The WITH/W/OUT pair also quietly solves attribution: nobody argues about whether the plugin caused the lift, the harness reruns the question on every merge.