DEV Community

Dileep Kumar Pandiya
Dileep Kumar Pandiya

Posted on

How to Know If Your Claude SKILL.md Actually Works

#ai #claude #showdev #testing

I spent today shipping a tool I've wanted for months.

If you build with Claude, you've probably written a SKILL.md file. And you've probably shipped it based on gut feel.

That changes today.

The problem nobody talks about
Skills are just system prompt injections. The honest question is: does this skill actually improve Claude's outputs, or does it just feel like it does?

Most teams answer this by eyeballing a few responses. That's not evaluation. That's vibes. Three things make vibes-based skill evaluation dangerous:

Position bias — if you ask Claude to compare its own outputs, it favors whichever it sees first

Silent regression — model updates, skill edits, and context changes can silently make a skill worse

No shared rubric— every engineer scores skills differently, so "this skill is good" means nothing

What I built
skilleval — a CLI that gives you a repeatable, objective score for any SKILL.md in under 2 minutes.

bash
npx @dileeppandiya/skilleval ./my-skill --tasks ./tasks.yaml
Enter fullscreen mode Exit fullscreen mode

Real output from the sample skill in the repo:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
skilleval results - api-design - 2 tasks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Skill effectiveness: +0.3 / 3
Tasks improved: 1 / 2 (50%)
Tasks hurt: 1 / 2 (50%)
Confidence: UNRATED (use --runs 3+ for confidence)
task-003 +2.5 Output A provides more robust API design...
task-004 -2.0 Output A is more comprehensive...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Runner: claude-sonnet-4-6 | Judge: gemini-3.5-flash
Estimated API cost this run: $0.101
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Enter fullscreen mode Exit fullscreen mode

Notice the mixed signal. The skill helped on task-003 but hurt on task-004. skilleval doesn't inflate scores to make skills look good. It reports what the judge actually found.

How it works
Blind A/B testing — each task runs twice concurrently, with the skill injected into the system prompt vs. raw context only.

Randomized judge — a Gemini Flash judge compares outputs. Which output gets labeled A or B is randomized per task with a seeded RNG, eliminating position bias completely.

Margin-based scoring — the judge returns a winner + margin (0–3): margin 3 gives 3.0/0.0, margin 0 gives a genuine tie at 1.5/1.5.

Honest confidence — single runs show UNRATED. One sample tells you nothing about stability. Real confidence (HIGH/MEDIUM/LOW) only appears at --runs 3+.

bash
skilleval ./my-skill --tasks ./tasks.yaml --runs 3
Enter fullscreen mode Exit fullscreen mode

Five things that make it different

  1. Deterministic assertions — not everything should be left to LLM opinion: 
tasks:
  - id: login-endpoint
    prompt: "Design a login endpoint"
    assertions:
      must_contain:
        - "POST"
        - "401"
      must_not_contain:
        - "GET /login"
      min_length: 100
Enter fullscreen mode Exit fullscreen mode

Assertion failures automatically count as hurt tasks, no LLM needed to know "missing POST method" is wrong.

  1. Multi-turn conversation tasks — most real skills operate across turns, not single prompts. The skill injects into the system prompt for the full conversation, and the judge sees complete context when scoring.

  2. Run history + regression detection — every run auto-saves to .skilleval/history/. After two runs:

bash
skilleval diff ./my-skill
Enter fullscreen mode Exit fullscreen mode 
── skilleval diff: api-design ──────────────────
vs previous run: 2026-07-11T14:30:00Z
Effectiveness: +0.3 → +0.8 (+0.5 ↑)
Tasks improved: 1 → 2 (+1 ↑)
Tasks hurt: 1 → 0 (-1 ↓)
Enter fullscreen mode Exit fullscreen mode

This is "skill hell" prevention in practice — you can see the exact moment a skill started regressing.

  1. Skill version comparison— test v1 vs v2 directly: 
bash
skilleval ./skill-v1 --compare ./skill-v2 --tasks ./tasks.yaml
Enter fullscreen mode Exit fullscreen mode

No more "I think v2 is better." Now you know.

  1. One-line CI integration — block PRs that silently break skills: 
on:
  pull_request:
    paths:
      - '**/SKILL.md'

jobs:
  skilleval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dileepkpandiya/skilleval@main
        with:
          skill-path: ./my-skill
          tasks: ./tasks/tasks.yaml
          fail-below: '0.3'
          fail-if-hurt-pct: '50'
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
          gemini-api-key: ${{ secrets.GEMINI_API_KEY }}
Enter fullscreen mode Exit fullscreen mode

plaintext
Exit code 0 = pass, 1 = gate failed, 2 = error.

Cost
Setup Cost
5 tasks, --runs 1, Gemini Flash judge ~$0.10
5 tasks, --runs 3 (real confidence) ~$0.30
10 tasks, --runs 3 ~$0.60

Use --cost to see an estimate before spending anything. Gemini Flash is the default judge, and the free tier handles casual iteration easily.

Quick start

bash

Try it immediately on the built-in sample 

git clone https://github.com/dileepkpandiya/skilleval
cd skilleval
npx @dileeppandiya/skilleval ./samples/api-design \
  --tasks ./tasks/sample-tasks.yaml
Enter fullscreen mode Exit fullscreen mode

Scaffold a new skill 

skilleval --init ./my-new-skill
Enter fullscreen mode Exit fullscreen mode 
## Install globally
npm install -g @dileeppandiya/skilleval
Enter fullscreen mode Exit fullscreen mode

You'll need ANTHROPIC_API_KEY for the Claude runner and GEMINI_API_KEY for the default judge.

What's still missing
Honest gaps in v0.3.0:

Tool-call evaluation — if your skill affects which tools Claude calls, text-output scoring misses that

Visual history dashboard — the diff command is CLI only, no charts yet

Local model judge support — no Ollama/local-model judging for fully offline eval yet

The repo
MIT licensed, open source, TypeScript. 38 unit tests, zero API calls needed to run the test suite, GitHub Action included.

👉 github.com/dileepkpandiya/skilleval

What are you using to evaluate your skills today? I'd genuinely love to know what's broken about this for your use case, you can file an issue or drop a comment below.

Top comments (0)