I wrote evals for my Claude Code plugin. The first run proved my README wrong.

Every new project I started, I wrote the same Claude Code setup by hand: a CLAUDE.md, a few guard hooks, some reviewer agents. Every template I tried gave me a web-app config with brackets to fill in, which is the wrong shape for a CLI, a Terraform repo or a smart contract.

So I built a plugin that does it for me. You describe the project in one paragraph, it works out the project's traits (language, whether there's a UI, a database, a trust boundary, whether it's published), and it writes the harness that follows from those. A CLI with no database gets no persistence guard. A published library gets a semver reviewer that an app doesn't need.

I tested it by hand on ten very different projects, from a game server to a C++ audio plugin, and fixed what broke each time. It felt done. Then I wrote an eval suite, and it kept telling me things I didn't want to hear.

The setup

Claude Code has a built-in way to test plugins: claude plugin eval. Each eval case is a real Claude Code session. You give it a prompt, it runs, and graders check what came out. I started with eight cases, one for each thing the plugin claims to do:

Case	What it checks
self-triggers-on-bare-description	A plain project description is enough to start it
two-guided-questions	The two setup questions (cost and scope) actually get asked
web-app-auth-db	An app with logins and a database gets a security reviewer, a data-layer guard and a secret scan
published-library-semver	A published library gets a semver reviewer
cli-no-ui-no-persistence	A CLI with no UI or database gets neither kind of reviewer
retrofit-existing-repo	On a repo that already has a setup, it keeps what's there
cost-low-caps-models	Asking to keep cost low means no agent runs on the most expensive model
vague-description	"I want to build an app" doesn't get a made-up stack

Graders come in two kinds. Some are deterministic, like this one, which just checks the skill was invoked:

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?init"'
weight: 1
---

Others are an LLM reading the output against a written rubric, for things a regex can't judge, like "is this agent actually in the roster, or only mentioned as something to add later?"

What the evals caught

1. My README's main promise was false

The README said you could just describe your project and the setup would start. The first eval run said otherwise. Given only a project description, with no mention of Claude Code or setting anything up, the skill never fired.

Its trigger condition only matched requests like "set up Claude Code for this". The description was narrower than the job. I had never noticed, because when I tested by hand I always knew what to type. I widened the trigger, and self-triggers-on-bare-description now makes sure it stays fixed.

2. Two of my "failures" were the graders

• An LLM grader reading a long transcript missed content that was plainly there. The thing it was checking for was a fixed phrase, so I replaced it with a plain text match.
• Another grader flagged the plugin for mentioning an agent it had deliberately not built: a note saying "add a database reviewer later, if you add a database". That was correct behavior. I rewrote the rubric to tell "built now" apart from "noted for later".

A failing eval is a claim about either your code or your grader, and you have to check which.

3. A Claude Code update broke a case that used to pass

In September all eight cases passed. In October, on a newer Claude Code version, vague-description failed twice in a row. Given "I want to build an app", the plugin said it was guessing, then guessed a very specific stack anyway: Next.js, Prisma, Auth.js, Vercel. It also ended half-built, with a CLAUDE.md describing agent files that were never written.

The fix: a description that names no kind of app now gets exactly one question, and before finishing the plugin checks that every file CLAUDE.md mentions actually exists on disk. Back to 8 of 8.

4. The 8 of 8 was one model

This is the one I least wanted to see. Every run so far had used my account's default model. When I added a --model flag to my eval runner and ran the same plugin on the smaller models, version 0.6.1 passed 3 of 8 on Haiku, and 3 of the 6 Sonnet cases that finished before the run was cut short.

The misses had patterns. On both models, the skill didn't trigger for "set up a harness for this existing repo", because its description leaned on describing a new project. Haiku also missed detail in the spec (dev commands, which reviewers to include, keeping CLAUDE.md short), which makes sense: the spec it follows is 55KB.

After fixing the trigger and adding four new cases, version 0.7.0 passes 9 of 12 on Haiku and 9 of 12 on Sonnet. It's not 12 of 12, and the README says so. Most of the remaining misses share one cause: in an eval nobody can answer the two setup questions, and smaller models sometimes ask and then stop without writing anything. Taking the recommended answer when no reply can come is the next change.

All of those are one run per case, which is noisy. I treat a single pass or fail as a signal, not a verdict.

Testing what it generates: an attack suite

Evals have a blind spot here. Eval runs aren't allowed to write into .claude/, so the graders can check CLAUDE.md and what the session tried to write, but never the hooks that actually guard your project.

The hooks are just scripts, though. Claude Code sends each one a JSON description of the tool call it's about to make, and the hook decides whether to block it. So I wrote a test runner that skips the model entirely: it feeds each hook 13 attack payloads and checks that they're blocked.

The 13 are what coding agents actually get hit with:

• piping a download into a shell, in Bash and PowerShell;
• DNS exfiltration, and an inline network call from a script;
• a setup script planted inside a dependency;
• reading .env from the shell;
• a push-guard bypass (git -C . push);
• five ways of disabling the harness itself: editing a guard, overwriting settings, sed -i on a hook, a redirect into settings, switching hooks off;
• an override smuggled into the command text.

Each attack has a harmless twin that must not be blocked, so a guard that blocks everything fails too. Every attack runs from the project root and again from a subfolder. With no model calls, the whole suite runs in seconds.

The result for the seven example harnesses the plugin had generated: 0 of 13 blocked by hooks, 2 of 13 counting permission rules. The guards worked as written; they just didn't cover any of this. The three examples regenerated with 0.7.0 now block 8 or 9 of 13 with hooks, and 13 of 13 counting the permission rules.

A few things I only learned by testing:

• An ask permission rule on Edit also catches shell writes like sed -i, >, cp, tee and rm. It does not catch node -e "require('fs').writeFileSync(...)". So the hook layer only needs to cover script runtimes, not every shell command.
• My own spec set a hook timeout of 60000, assuming milliseconds. Hook timeouts are in seconds, so a hung build gate would have held the session for about 17 hours.
• Claude Code's sandbox runs on macOS, Linux and WSL2, but not on native Windows. Harnesses now turn it on where it exists, and say plainly in CLAUDE.md when it doesn't.

What I still can't measure

• Twelve cases is not many. They cover the promises I make, not the long tail of real projects.
• One run per case is noisy. Results move between runs, especially without the plugin.
• LLM graders are judgment calls. Where a text match works, I switched to one.
• Four of the seven examples still use the old spec. The attack numbers above are for the three that were regenerated.

If you're building a Claude Code plugin

What I'd do from the start next time:

1. Name the model in every result. "8 of 8" meant nothing until I knew which model produced it.
2. Run every case without your plugin too. Without a baseline, a pass tells you nothing.
3. Use deterministic graders wherever a text match will do. Keep LLM rubrics for real judgment, and write them to tell "did it" apart from "mentioned it".
4. Test what your plugin generates directly, without a model. If the output is code, you can feed it inputs. It's faster and cheaper than any eval, and it gives the same answer every time.
5. Record the Claude Code version too. One of my regressions came from an upgrade, not my code.

The plugin

It's open source, MIT licensed, and listed in Anthropic's plugin directory. Install it in Claude Code:

/plugin marketplace add sagaga144/claude-harness-bootstrap
/plugin install harness-bootstrap@harness-bootstrap

Then run /harness-bootstrap:init in any project folder, or just describe your project.

The repo has the eval suite and per-model results, the attack suite and its matrix, a demo GIF, and seven example harnesses for projects from a Rust monorepo to a JUCE audio plugin: github.com/sagaga144/claude-harness-bootstrap

If you try it on a kind of project it hasn't seen, I'd really like to hear what it got wrong. And if it saved you setup time, a star helps other Claude Code users find it.