Your SKILL.md is production config. Test it like one.
You maintain agent skills — the SKILL.md instruction files that steer Claude Code, Cursor, Codex, or any coding agent. Careless edits feel cheap: rewrite one instruction line, done.
The problem is what happens next. That one line silently changes how the agent behaves on the next run. Files you didn't ask it to touch. A tool call sequence that skips a step. A task that used to complete in one shot now stumbles through four attempts.
Usually you find out weeks later. Not from a test — from a user.
Text diffs can't tell you what an agent will do differently
A git diff of SKILL.md shows you which words changed. It can't show you what the agent will do differently, because agent behavior is emergent. The same prompt line can produce completely different tool trajectories depending on context.
Prompt evals miss it too
Offline evals score a final answer against a gold label. They don't see the trajectory: the extra tool hop, the silent regression where the agent still prints the right string but took a forbidden shortcut to get there.
Here's a real example. I dogfooded skilldiff on itself — it tests its own skills. Two versions of a note-taking skill were run against the same scenario:
old skill: 2 tool calls, 1 file(s) changed, 0 command(s) run files: NOTES.md
new skill: 3 tool calls, 2 file(s) changed, 0 command(s) run files: NOTES.md, TODO.md
Assertions: ✓ [files_changed] NOTES.md ✓ [tool_calls] read ✓ [tool_calls] write ✗ [must_not] files_changed does not include TODO.md actual (new): VIOLATED — TODO.md was changed note: this is a REGRESSION — old skill passed, new skill fails
Both versions appended SPIKE RAN OK to NOTES.md — the standard diff would look fine. But the new version also created TODO.md, which the scenario explicitly forbids. A text diff would never have caught that. That's behavioral regression.
The approach: run the skill twice, diff what the agent actually did
skilldiff runs your skill in a real agent harness against a fixture repo — twice:
- Old — the skill as it exists on the base branch
- New — the skill as changed in your PR
Then it captures what the agent actually did: files changed, commands run, tool calls made. And it asserts on those observations.
Five assertions you can write in plain YAML
| Kind | Meaning |
|---|---|
files_changed |
paths the agent modified |
commands_run |
commands the agent executed |
tool_calls |
tools invoked (normalized across harnesses) |
must_not |
forbidden files / commands / tools |
output_contains |
substrings in the final output |
A scenario looks like this:
skill: notes-helper
fixture: repo/notes-helper-scratch
expect:
files_changed: [NOTES.md]
tool_calls: [read, write]
must_not:
files_changed: [TODO.md]
Drop it into your PR flow
npx skilldiff init # discovers .claude/skills, skills/, .agents/skills
# and writes a starter scenario per skill
npx skilldiff run skilldiff/notes-helper.scenario.yaml --live --base origin/main
For CI it replays recorded traces, so it's deterministic and needs no API key or credits. The PR that touched a skill gets a comment with the behavior report; failures gate the merge.
Live runs use whatever harness and login you already have. There's no shared API key shipped with the tool — each contributor runs on their own account (opencode, Claude Code CLI, Cursor, Codex, Freebuff).
Honest limitations (v0.1)
Skills are emergent, so a single run is a sample. Recorded/CI mode is deterministic; live runs vary run to run.
Five assertion kinds is deliberately few. They cover the 80% case: files, commands, tools, forbidden behavior, output markers.
This is not a replacement for full eval harnesses like Inspect or Promptfoo. It answers one question: did this change regress agent behavior I'm relying on?
Contribute
If your harness is missing, an adapter is roughly 40 lines — and the highest-value way to contribute.
Repo: https://github.com/scs0209/skilldiff
If you maintain skills and this doesn't match your workflow, I'd genuinely like to hear how you review skill changes today.
Top comments (3)
Diffing tool trajectories across baseline and candidate runs catches the silent side effects that prompt evals miss, especially unintended file mutations or forbidden CLI flags.
The mechanical friction with behavioral diffing is sample variance across non-deterministic runs. If an edit introduces a 15% probability of taking an unprompted tool branch, comparing a single baseline run against a single candidate run has over a 70% chance of missing the regression entirely. The test harness ends up comparing two random draws from an unmapped distribution.
Treating instruction files like production config eventually requires running these fixture runs as small Monte Carlo batches. Asserting on the empirical failure rate across five or ten runs surfaces tail-risk regressions that clean single-run diffs quietly pass into main.
The "single run is a sample" line is the part worth sitting with. Recorded/CI mode is deterministic, but that's exactly the mode that won't catch a regression only showing up on the live, non-deterministic path. On the review-workflow question — closest thing here to what you're describing is a numeric gate on drafted output instead of trajectory diffing. Score every draft against target features, warn below a threshold, flip warn to block once new output actually holds the line. Same instinct as skilldiff though — don't trust a read-through, make the actual output pass a check before it ships.
That makes sense. I’ll keep these ideas in mind and see how I can take the approach further. The “single run is a sample” point is definitely something worth exploring.