I built three Claude Code skills for AI product managers (ai-pm-skills). One of them, /eval-plan, exists to stop a specific habit: deciding what "good enough" means after the results come in. A bar set after the numbers can't fail.
So I held the skills to the same rule. Before running a single eval, I committed the pass bar to git. Then I ran them. The first run failed.
The setup
Claude Code has a built-in eval runner, claude plugin eval. Each test case is a prompt plus graders, and it runs every case with the plugin and without it, so you see what the skill actually adds over plain Claude.
I wrote 8 cases across the three skills. Three of them are deliberate "should refuse" cases, because refusing is where AI features quietly fail. Then I committed three gates:
- Every case scores at least 0.8 with the plugin.
- Each skill fires in at least 2 of 3 runs.
- The plugin beats plain Claude on at least some cases, and where it doesn't, the results say so.
Run 1: a real bug in the skill
/build-or-not checks a feature idea against 4–8 real examples before anyone builds it. One test gave it no evidence and no research tools, then demanded a verdict.
It scored 0.00. The skill even wrote that it couldn't run its own check, then said "don't build" anyway, backed by market knowledge it recalled and labeled "public, well-known, not invented." Nobody had checked any of it for this decision.
The skill never said what to do when there's no sample, so the model filled the gap with confidence. The fix was one rule: no sample, no decision. "Can't decide yet" is now an outcome, with the exact sample that would settle it. I didn't touch the grader or the gate. Run 2 passed everything.
Runs 3–6: three bugs in my tests
The third skill, /agent-trust-review, sorts an agent's risks into covered (with evidence), declined on purpose (with a reason), and genuinely missing. It took four runs to measure, and every failure was mine:
- My test cited files that didn't exist in the test workspace. The skill looked, found nothing, and correctly refused to count them. My grader expected it to credit evidence it couldn't see.
- I tested "give two coverage numbers" in a case with nothing declined, where both numbers are equal by definition. The skill said exactly that.
- My fake test files were empty stubs, and the skill caught it. The judge model also marked one correct answer as a fail.
It's tempting to keep re-running until it's green. Instead, I committed a note saying the next run would be final before starting it, and would be reported whatever it showed. It passed.
What the skills add, and what they don't
| Behavior | With the skill | Plain Claude |
|---|---|---|
| States the bar before deciding | 3 of 3 runs | 0 of 3 |
| Refuses a verdict when there's no evidence | 3 of 3 | 0 of 3 |
| Plans a rollback trigger for launch | 3 of 3 | 1 of 3 |
| Separates a reasoned decline from an unexplained gap | 3 of 3 | 2 of 3 |
| Gives two coverage numbers (owned areas vs. all areas) | 3 of 3 | 0 of 3 |
On four other cases plain Claude already did just as well: spotting hits a feature can't reach, pushing back on a bar set after the results, refusing to certify "it's safe" with no evidence, and (in the final run) refusing to credit unsupported claims. The skills aren't what makes those pass, and the README says so.
This is 8 cases, 3 runs each, one model. It's a check of the key behaviors, not a benchmark. Each full run cost about $2.
What I took from it
- Commit the bar first. Every time I was tempted to move it, the commit history made that visible.
- Include cases that should refuse. The only real bug showed up in one.
- When an eval fails, check the test before the model. Three of my four failures were my test setup.
- Report the baseline. "Passes 100%" means little if plain Claude also passes 100%.
The skills, the eval suite, and every failed run are public: github.com/vishalhabib99/ai-pm-skills. Install in Claude Code with:
/plugin marketplace add vishalhabib99/ai-pm-skills
/plugin install ai-pm-skills@ai-pm-skills
If you run one of them on a real decision, I'd like to hear where it was wrong.
Top comments (0)