Claude Code 2.1.269 plugin eval: build a cost-bounded, reproducible acceptance suite

Quick answer

Claude Code 2.1.269 adds claude plugin eval, a command that runs a plugin evaluation suite and produces scored JSON plus an HTML report. That is useful release infrastructure, but the command alone does not make a plugin safe or an evaluation reproducible.

Treat it as an acceptance-test runner. Pin the CLI version and package integrity, start with one representative case, keep MCP servers mocked and tools denied by default, run serially under a small cost ceiling, and archive every input alongside both output formats. When a plugin target resolves, the default with-plugin versus without-plugin ablation is a helpful baseline; it is not proof that the result will repeat across models, judge settings, or changing dependencies.

The safest first gate is therefore narrow: one fixed case, one model, one judge model, concurrency 1, no publishing, no real MCP servers, and a written decision tied to the resulting evidence bundle.

Who this is for

This guide is for plugin authors and teams that need a repeatable answer to “did this plugin improve the task enough to release?” It is also useful for maintainers who want to catch prompt, skill, grader, or MCP regressions before distributing an update.

If your concern is credential exposure or workspace trust, first apply the Claude Code 2.1.268 credential safety checklist. Evaluation measures behavior under declared conditions; it does not replace secret rotation, permission review, sandbox analysis, or supply-chain verification.

What changed and why now

The official 2.1.269 release introduces claude plugin eval. The pinned CLI help confirms two case formats: case.yaml, or a directory containing prompt.md plus Markdown graders under graders/. The default directory is evals/, but --eval-dir can make that location explicit.

The command can evaluate a path, an installed plugin name, or a marketplace-qualified plugin. If it resolves a plugin, the default ablation compares runs with and without the plugin. It can write machine-readable results with --json and a reviewable report with --report.

Those capabilities create a practical release gate, but also new operational boundaries. Each run is a full child process that shares account rate limits. Real MCP servers can run as the current user outside an operating-system sandbox. A cost ceiling is checked before new runs start, so already in-flight runs can still create bounded overshoot. These details make a conservative evaluation profile more important than a large test matrix on day one.

A safe, reproducible workflow

1. Freeze the evaluator identity

Record the package name, version, integrity hash, resolved executable, model, judge model, case-tree hash, plugin-tree hash, and operating-system context. For this guide, the verified package is @anthropic-ai/claude-code@2.1.269, published from release commit df52d04a4e65195c1621fe6222e0564bcccb1804.

Do not use a moving latest tag in CI. A result from a different CLI or grader is a different experiment even when the prompt text is unchanged.

2. Scaffold one representative case

The pinned CLI can create a minimal case:

npx -y @anthropic-ai/claude-code@2.1.269 plugin eval init \
  --bare smoke-case \
  --eval-dir evals

The generated prompt.md starts with max_turns and allowed_tools; the generated graders/criteria.md is an LLM grader with a weight. Replace both placeholders with a real user task and observable success conditions. A scaffold is only a starting shape, not a trustworthy test.

3. Begin with least privilege

Keep --mocks record, which is the default, and do not add --allow-real-servers. A missing mock should fail visibly rather than quietly starting infrastructure with your local identity. Grant extra tools only when the case requires them, and record the exact --allow-tools set.

The CLI describes --mocks off and --allow-real-servers as routes that start real servers outside the OS sandbox. Use them only for a trusted plugin, a disposable environment, low-privilege credentials, and an explicit network/side-effect review.

4. Make the first run cheap and serial

Use a narrow case glob, concurrency 1, a small ceiling, and local evidence paths:

npx -y @anthropic-ai/claude-code@2.1.269 plugin eval ./my-plugin \
  --case 'smoke-*' \
  --concurrency 1 \
  --mocks record \
  --max-cost-usd 2 \
  --no-publish \
  --json ./evals/results/smoke.json \
  --report ./evals/results/smoke.html

The ceiling is checked before each run. If it is breached, the CLI can stop with partial results; paid graders may be skipped while free graders still score. Do not interpret partial output as a completed suite. Increasing concurrency can allow several already-started runs to finish after the threshold, so expand only after measuring the serial case.

5. Interpret the baseline, not just the score

For the default ablation, compare task completion, tool calls, side effects, grader reasons, cost, and failure modes between the plugin and baseline arms. A higher aggregate score is insufficient if the plugin widened permissions, contacted an unexpected server, or achieved the result through an unstable workaround.

Rerun important cases at least twice before promotion. LLM outputs and LLM graders are probabilistic; archive every attempt rather than overwriting the first favorable result.

6. Promote through a permission ladder

Move from pure prompt/skill cases to mocked MCP cases, then to isolated real integrations only when prior stages pass. At every stage, add one capability at a time and keep a fail-closed control case: denied tool, missing mock, malformed input, grader failure, cost exhaustion, and interrupted run.

This complements the Qwen-to-Claude delegation identity checklist: both require evidence of which runtime acted, what it was allowed to do, and whether the parent process reached a terminal state.

Release decision record

date / owner / release candidate:
cli_package / version / integrity / release_commit:
plugin_tree_hash / case_tree_hash:
model / judge_model / concurrency:
mocks_mode / real_servers_allowed:
allowed_tools / credential_scope:
max_cost_usd / observed_cost / partial_result:
plugin_arm_score / baseline_arm_score:
unexpected_tools / network / side_effects:
json_path / html_path / log_path:
repeat_run_variance:
decision: hold | revise | limited-release | release

Use hold whenever the evaluator identity is not pinned, a mock is missing, the result is partial, a side effect is unexplained, or the evidence bundle cannot be reproduced from its manifest.

Common mistakes

• Calling one passing LLM-graded case a security review.
• Comparing scores while changing the CLI, model, judge, mocks, or plugin between arms.
• Starting real MCP servers merely because a mock fixture is inconvenient.
• Setting high concurrency before measuring per-run cost and shared rate-limit pressure.
• Ignoring partial results after the cost ceiling or grader failure.
• Publishing the report before checking it for prompts, paths, credentials, or proprietary output.

Building something? Turn your product page into a show people want to watch with PromoFast—hosted, embeddable, and ready to export.

FAQ

Does a passing suite prove that a plugin is safe?

No. It proves only that the tested artifact met the selected graders under the recorded conditions. Security still requires permission, secret, dependency, network, and side-effect review.

Should I enable real MCP servers for integration tests?

Only after mocked cases pass, and only in a disposable environment with low-privilege credentials. The CLI warns that real servers run as your user outside an OS sandbox.

Why start with concurrency 1?

Each case creates a full child process and shares rate limits. Serial execution makes cost, failure ownership, and ceiling behavior easier to understand before parallelism introduces in-flight overshoot.

What should be stored as release evidence?

Store the manifest, pinned versions and hashes, prompt and grader inputs, mocks, tool grants, command line, JSON result, HTML report, sanitized logs, observed cost, repeat runs, and final decision. Never store secrets.

Sources

• Claude Code v2.1.269 official release
• Claude Code tagged changelog
• Official Claude Code 2.1.269 npm package
• Claude Code settings documentation

Originally published on IndieSeek.