Anthropic Shipped Plugin Evals. The Error It Threw Found 5 Agents That Never Existed.

An Anthropic engineer announced plugin evals for Claude Code with a simple message. They heard feedback that it is hard to know whether your skills still work when a new model ships. Plugin evals are here to help. Run claude plugin eval init in your plugin folder.

So I did. And I got an error.

Error: /Users/me/dv/my-project is not a plugin or skill folder

That error is not a bug. It is the first good question my tooling has asked me in a long time: do I actually know what is running in my setup?

The misunderstanding everyone will hit

claude plugin eval init scaffolds a test suite for a plugin or a skill. It wants to run from the root of one of those two things: a folder with a plugin.json, or a folder with a SKILL.md.

An ordinary application repo is neither. The error is therefore instant and correct.

Three ways out:

cd ~/my-skills/my-skill && claude plugin eval init

claude plugin eval init --eval-dir evals

# 3. blank template, no interactive interview
claude plugin eval init --bare my-case --eval-dir evals

The third is the only one an agent can use in non-interactive mode, since the interview needs a real terminal.

The principle behind evals deserves a pause. Every case runs twice by default: with the plugin and without it. Only the gap between the two proves the plugin did anything. That is brutal, and it is the right measure. A skill that changes nothing is not a skill. It is a Markdown file and some hope.

Why I pivoted to an audit

I did not write a single eval that day. The error reminded me that my setup had accumulated ten months of sediment: plugins tried and forgotten, marketplaces added for one test, permission rules piled up session after session.

Here are the five checks I ran, in order. They take ten minutes and replay on any machine.

1. Config file validity.A settings.json with broken JSON is silently ignored. Running python3 -m json.tool on each file is enough to find out.

2. Hooks pointing at scripts that moved.A hook whose script is gone raises no visible error. It simply stops doing anything. I extract the path from every hook command and test that it exists.

3. Frontmatter on every skill.The name in the frontmatter must match the folder name, and description must exist, or the skill will never be surfaced. All twenty-three of mine passed. It was the only check I got right on the first try.

4. Duplicate plugins.This is where it fell apart.

5. MCP server health.claude mcp list connects to each one and reports.

Three real gaps

Subagents that did not exist

For months my global instructions required delegating development to five named subagents: one for narrow lookups, one for multi-file exploration, one for standard implementation, one for high-risk work, one for independent review.

The ~/.claude/agents/ folder did not exist.

None of those five had ever existed. Every delegation following my own doctrine silently fell back to a generic agent, on the default model, without any of the framing I had written. Months of carefully worded rules had been applied to nobody.

The fix is five Markdown files with name, description, tools and model in the frontmatter. The point is not cosmetic. That frontmatter is where you pick the model per role. A narrow lookup agent runs on a small fast model, a review agent on your most capable one. Team rules stop being a wish and become configuration.

The part that matters: verify it actually loads. I ran a headless session that invoked the agent by name, then grepped the session transcript for its identifier. Two hits, so the agent really ran. A plausible answer is not proof.

Seven plugins installed twice

My registry held seven plugins present in both user and local scope, the local entries eight months old and pointing at stale caches.

The cause is mundane: installs run from the home directory, which created a project scope on the home folder itself, then reinstalled properly later without the old entries ever going away.

Clean it through the CLI, never by hand-editing the registry:

claude plugin uninstall <plugin>@<marketplace> --scope local -y

Then I checked two things: zero duplicates left, and every actually-enabled plugin still in place. Unexpected bonus, one of them switched from a local npx launch to a remote HTTP server, which removes a process from every session start.

A security rule that protected nothing

This one is worth knowing.

I had two deny rules meant to stop writes to my secret files:

"deny": [
  "Write(~/infra/secrets/.env)",
  "Write(~/infra/secrets/.env.*)"
]

At startup, Claude Code prints a warning I had never read: only Edit(path) rules are evaluated by file permission checks. A Write(path) rule blocks nothing at all.

My secret files had been writable for months, while I believed the opposite.

"deny": [
  "Edit(~/infra/secrets/.env)",
  "Edit(~/infra/secrets/.env.*)"
]

Edit(path) covers every file-editing tool, creation included. A config file holding an inert rule is worse than one with no rule: it manufactures false confidence.

One nice detail: the agent could not apply that fix itself in auto mode. Editing its own permission rules is classified as self-modification and refused. That is exactly the behavior you want.

The trick that applies to all your projects: scope

A few days later I wanted to install a domain skill bundle, fifteen specialized skills for an automation tool. Out of forty repos, three use it.

The reflex is to install globally. That is a measurable mistake.

This plugin ships a session-start hook that injects roughly 4,300 tokens into every session, and re-injects them after every compaction. Installed globally, that is 4,300 tokens paid in thirty-seven repos that have nothing to do with the subject.

The right command is one word longer:

cd ~/dv/the-repo-that-uses-it
claude plugin install <plugin>@<marketplace> --scope project

Project scope writes a .claude/settings.json inside the repo, which you commit. The plugin follows the repo, reaches your teammates on clone, and exists nowhere else.

Two precautions before committing that folder:

1. Make sure .claude/settings.local.json is in .gitignore. That file holds your personal permissions. In one of my repos it held an API key in plain text, inside the URL of five allow rules. Gitignored by luck, but sitting on disk and reloaded into the context of every session.
2. Check for local state files too, things like a scheduled-tasks lock, which land in the same folder.

One rule follows: what describes the project gets committed, what describes your machine or your person stays local.

Doing it with Codex

I develop with two agents in parallel. Anything I configure for one has to stay usable by the other, or I end up maintaining two diverging setups.

Here are three findings, from simplest to most useful.

Global instructions already share cleanly.One source file sits in a versioned folder, symlinked into both expected locations, with identical content and one truth.

Claude plugins install into Codex too.Codex can add a Claude plugin marketplace and install from it:

codex plugin marketplace add https://github.com/author/skills-repo
codex plugin add <plugin>@<marketplace>

Verified in a real session, the skills show up. But codex plugin add has no scope option. The install is global, and the context cost comes back in every project. My fifteen skill descriptions alone weighed about 2,550 tokens everywhere.

The right method goes through the repo folder.Codex reads <repo>/.codex/skills/. So you can link only the skills you want, for that repo only:

SRC=~/.claude/plugins/marketplaces/<marketplace>/skills
mkdir -p .codex/skills
for s in "$SRC"/*/; do ln -sfn "$SRC/$(basename "$s")" .codex/skills/"$(basename "$s")"; done

Those links hold absolute paths, specific to your machine. So they go in .gitignore, and the recreation command goes in the repo instructions file, the one both agents read.

Queries of real Codex sessions showed fifteen skills visible in the two relevant repos, zero everywhere else.

One limit worth knowing: skills are not tools. The tool's MCP server is declared in a file Claude Code reads and Codex ignores, since Codex keeps its servers in its own config. A Codex session therefore has the knowledge without the tools. That is written plainly in the repo instructions so nobody is surprised.

What I take away

A three-second error showed me that five agents on my org chart did not exist and that a security rule was not being applied.

The lesson is not that my setup was dirty. It is that agent configuration produces no signal when it is wrong. An orphaned hook does not run. A missing agent is silently replaced. A misspelled permission rule lets things through. Everything appears to work.

That is precisely the problem plugin evals attack, one level up. A skill that never triggers does not throw an error either. The only way to know is to run the case with and without, and look at the gap.

So yes, run claude plugin eval init in your plugin folder. And if you get an error, do not close it too fast. It may have more to teach you than the test you were about to write.