Your SKILL.md is production config. Test it like one.

Changsu Seong — Fri, 18 Sep 2026 11:06:59 +0000

Your SKILL.md is production config. Test it like one.

You maintain agent skills — the SKILL.md instruction files that steer Claude Code, Cursor, Codex, or any coding agent. Careless edits feel cheap: rewrite one instruction line, done.

The problem is what happens next. That one line silently changes how the agent behaves on the next run. Files you didn't ask it to touch. A tool call sequence that skips a step. A task that used to complete in one shot now stumbles through four attempts.

Usually you find out weeks later. Not from a test — from a user.

Text diffs can't tell you what an agent will do differently

A git diff of SKILL.md shows you which words changed. It can't show you what the agent will do differently, because agent behavior is emergent. The same prompt line can produce completely different tool trajectories depending on context.

Prompt evals miss it too

Offline evals score a final answer against a gold label. They don't see the trajectory: the extra tool hop, the silent regression where the agent still prints the right string but took a forbidden shortcut to get there.

Here's a real example. I dogfooded skilldiff on itself — it tests its own skills. Two versions of a note-taking skill were run against the same scenario:
old skill: 2 tool calls, 1 file(s) changed, 0 command(s) run files: NOTES.md

new skill: 3 tool calls, 2 file(s) changed, 0 command(s) run files: NOTES.md, TODO.md

Assertions: ✓ [files_changed] NOTES.md ✓ [tool_calls] read ✓ [tool_calls] write ✗ [must_not] files_changed does not include TODO.md actual (new): VIOLATED — TODO.md was changed note: this is a REGRESSION — old skill passed, new skill fails

Both versions appended SPIKE RAN OK to NOTES.md — the standard diff would look fine. But the new version also created TODO.md, which the scenario explicitly forbids. A text diff would never have caught that. That's behavioral regression.

The approach: run the skill twice, diff what the agent actually did

skilldiff runs your skill in a real agent harness against a fixture repo — twice:

Old — the skill as it exists on the base branch
New — the skill as changed in your PR

Then it captures what the agent actually did: files changed, commands run, tool calls made. And it asserts on those observations.

Five assertions you can write in plain YAML

Kind	Meaning
`files_changed`	paths the agent modified
`commands_run`	commands the agent executed
`tool_calls`	tools invoked (normalized across harnesses)
`must_not`	forbidden files / commands / tools
`output_contains`	substrings in the final output

A scenario looks like this:

skill: notes-helper
fixture: repo/notes-helper-scratch
expect:
  files_changed: [NOTES.md]
  tool_calls: [read, write]
  must_not:
    files_changed: [TODO.md]
Drop it into your PR flow
npx skilldiff init   # discovers .claude/skills, skills/, .agents/skills
                     # and writes a starter scenario per skill

npx skilldiff run skilldiff/notes-helper.scenario.yaml --live --base origin/main
For CI it replays recorded traces, so it's deterministic and needs no API key or credits. The PR that touched a skill gets a comment with the behavior report; failures gate the merge.

Live runs use whatever harness and login you already have. There's no shared API key shipped with the tool — each contributor runs on their own account (opencode, Claude Code CLI, Cursor, Codex, Freebuff).

Honest limitations (v0.1)
Skills are emergent, so a single run is a sample. Recorded/CI mode is deterministic; live runs vary run to run.
Five assertion kinds is deliberately few. They cover the 80% case: files, commands, tools, forbidden behavior, output markers.
This is not a replacement for full eval harnesses like Inspect or Promptfoo. It answers one question: did this change regress agent behavior I'm relying on?
Contribute
If your harness is missing, an adapter is roughly 40 lines — and the highest-value way to contribute.

Repo: https://github.com/scs0209/skilldiff

If you maintain skills and this doesn't match your workflow, I'd genuinely like to hear how you review skill changes today.

DEV Community: Changsu Seong

Your SKILL.md is production config. Test it like one.

Your SKILL.md is production config. Test it like one.

Text diffs can't tell you what an agent will do differently

Prompt evals miss it too

The approach: run the skill twice, diff what the agent actually did

Five assertions you can write in plain YAML