Here's a scenario that's already happened to someone reading this, they just don't know it yet.
You write a Claude Skill. You test it by hand — a few prompts, looks good, ship it. It works for months. Then Anthropic ships a new model. Nothing about your skill changed. But the model reads your instructions slightly differently now: it stops triggering on a phrasing that used to work, or it triggers fine but quietly drops a constraint it used to follow, or it still does the right thing but now costs three times as much to get there.
There's no error. No stack trace. No red CI badge. The skill just… drifts. You find out the way everyone finds out — a workflow breaks downstream, a review comes back wrong, a support ticket shows up, and twenty minutes into debugging you realize: it's not your code, it's the model, and it changed weeks ago.
Every other kind of software would have caught this on day one
If a dependency bump broke your app, you'd have a failing test in CI before it ever reached staging. If a database migration corrupted a query, your integration suite would go red immediately. That's not because those systems are magic — it's because someone decided, a long time ago, that "did this still work after the thing underneath it changed" deserves to be a test, not a vibe.
Skills don't have that yet. Most SKILL.md files in the wild have never been tested against more than one model, because until recently there wasn't a standard way to test them at all. Anthropic clearly sees the gap too — skill-creator now ships with evals, benchmark mode, and A/B testing between skill versions, which is a real step forward for authoring a skill well. But authoring-time testing answers "does this work today." It doesn't answer "does this still work six months from now, on a model that didn't exist when I wrote it" — and that second question is the one that actually burns teams, because it fails silently, in production, on someone else's schedule.
What a safety net for this actually looks like
Not a one-time eval you ran once and forgot about. A safety net that:
Re-runs automatically against every model in your matrix, not just the one you tested on originally
Diffs against last-known-good, so a flaky one-off doesn't cry wolf and a real regression can't hide in the noise
Turns a caught failure into permanent coverage — the exact gap that just bit you gets written down as a test case automatically, so it can never silently regress twice
Tells you which commit broke it, the same way git bisect tells you which commit broke a normal test, instead of leaving you to guess
Treats adversarial input as a real test category, not an afterthought — prompt injection, jailbreak attempts, and instruction leakage are things a skill can regress on too, and most testing setups don't check for that at all
That's the actual shape of the problem. It's not "skills need more eval prompts." It's "skills need the same CI discipline every other piece of software has had for twenty years, and nobody had built it yet."
So I built it
SkillCI is an open-source CLI that treats a Claude Skill like any other piece of software with a regression suite:
skillci check — a local, zero-API-call lint mapped to OWASP's Agentic Skills Top 10 (prompt injection risk, over-privileged access, unpinned dependencies) plus basic bloat checks. Free, no API key, catches the obvious stuff before you even run a model.
skillci regress — runs your eval suite across every model in your config, diffs against the last known-good run, and fails CI only on a new regression, not on every pre-existing flaky case.
The self-growing eval loop — when regress catches a failure with zero prior coverage, it doesn't just print red text. It writes the missing test case for you, so skillci accept turns "we got lucky this time" into permanent, tracked coverage. Catch once, covered forever.
skillci bisect — real git worktree-based binary search over your skill's own commit history, so when something breaks, you get a culprit commit instead of a guessing game.
Redteam plugins — prompt injection, jailbreak attempts (including multi-turn crescendo attacks), PII exfiltration, SSRF-bait, and more, run against your skill the same way a pentest would, with a successful attack becoming a permanent regression case exactly like a caught model drift does.
It's complementary to skill-creator's evals, not a replacement — skill-creator helps you author a good skill today; skillci watches it stay correct over time, across models you haven't tested yet and commits you haven't shipped yet.
Try it in under a minute, no API key needed
brew install kabirnarang39/skillci/skillci
skillci check path/to/your-skill
That's the free, local-only half — no ANTHROPIC_API_KEY, no network calls, just a static scan that'll tell you if your skill is carrying obvious risk. The full regression loop needs a key, but you can see whether this is worth your time before spending a single API call.
It's early — the core CLI is stable, the dashboard is newer, and I'd genuinely rather hear "here's what's missing" than silence. If you've been burned by a skill that quietly stopped working, or you're maintaining enough skills that this is starting to feel like a real problem instead of a hypothetical one, I'd like to know what broke and how you found out. That's the next thing I'm building toward.
kabirnarang39
/
skillci
CI for Claude Skills — lint, eval, and regression-test SKILL.md files across a model matrix, with a self-growing eval loop that turns uncovered regressions into permanent test cases.
SkillCI
Regression testing for Claude Skills. When a model update silently changes how your skill behaves, SkillCI catches it in CI — and turns the failure into a permanent test case, automatically.
A skill fails against a model it's never been tested on → SkillCI doesn't just report red, it writes the missing test case for you (evals/_generated/...) so skillci accept turns it into permanent coverage. That loop — catch once, covered forever — is the whole point.
Why · What's inside · How this compares · Install · Quick start · GitHub Actions · Dashboard · VS Code · Compliance reports · For AI agents · MCP server · Commands · Status
Why
You write a Claude Skill. It works today. Six months from now, Anthropic ships a new model, and nobody tested your skill against it first — because until now, no tool did that automatically…
If this is a problem you've hit, a star helps others find it — and tells me what to build next.
Top comments (0)