I came across a talk by Philipp Schmid from Google DeepMind, titled "Don't Ship Skills Without Evals." His point, boiled down: everyone's writing skills for AI agents right now, almost nobody tests them, and because LLMs are non-deterministic, eyeballing the output during development ("looks right to me") tells you almost nothing about what happens once the thing hits real usage.
That landed uncomfortably close to home. SKILLmama has four skill files, install instructions across four agents, and a scoring formula I'd been trusting on faith since v1.0. I'd never once actually tested whether it did what it claimed. So I built a small eval harness and ran it. Three bugs, first try.
What I Actually Built
Nothing fancy. A markdown file at evals/skillmama-ablation.md: five prompts that should trigger SKILLmama, five that shouldn't, mapped straight to the Trigger and Do-NOT-activate rules already in the skill file. A result log underneath it.
Should trigger:
1. "What vector DB should I use for RAG in my FastAPI app?"
2. "Find me the best job queue for my Node stack, self-hosted only."
3. "/skillmama" (no args)
4. "Scan my project and tell me what I'm missing."
5. "Recommend something for authentication, must have MCP support."
Should NOT trigger:
6. "How do I configure Redis pub/sub in Python?"
7. "Why is my pgvector query returning zero results?"
8. "Can you review this auth middleware I wrote for security issues?"
9. "I've decided to use Stripe for payments, walk me through the webhook setup."
10. "What's the difference between REST and GraphQL?"
First run: 10 for 10. The trigger logic held up. Good sign, but it's the easy part of the test. It only checks whether the skill turns on. The harder question is whether the pipeline is any good once it fires, and answering that meant pointing it at an actual project instead of a checklist.
Bug One: Silent Failure on the Wrong Directory
I ran the vector-DB prompt from inside SKILLmama's own repo, mostly because that's where my terminal happened to be. SKILLmama scanned the current directory for FastAPI files, correctly found none (it's a docs repo, not a FastAPI app), and just... proceeded. Empty stack profile, and it kept searching anyway like nothing was wrong.
That's the kind of failure nobody reports, because nothing looks broken. It just quietly gives worse answers.
The fix: if the user states a stack inline and the scanned directory shows no trace of it, stop and ask which directory to scan instead of completing an empty pass.
I didn't find FastAPI in this directory, this doesn't look like that
project. Want me to scan a different folder, or should I search based
on what you told me with no local verification?
Bug Two: Confident Numbers That Weren't Verified
SKILLmama's Compatibility score already had a rule: verify locally before scoring, don't just infer. Check .env.example for the right env var, check whether a required CLI is on PATH, don't guess.
Maintenance, sitting right next to it in the same formula, had no equivalent rule. When I ran an actual search, two of three candidates got their last-commit date from a search result. The third got a number from general knowledge of how active that org usually is. All three sat in the same table, looking equally solid.
Fix: Maintenance now runs by the same rule Compatibility already had, verify before scoring, don't infer. If a real last-commit date can't be found, the score shows N/A (unverified) instead of a confident-looking guess, and the total renormalizes across whatever factors are left.
Bug Three Needed a Different Kind of Test
The first two bugs came from just running the pipeline and watching it. The third one needed something closer to what Schmid's talk was actually about: comparing the skill against no skill at all, same question, same project, run in parallel. That's its own story, long enough to deserve a separate post rather than a paragraph here. Short version: it exposed a blind spot in how SKILLmama scored a candidate's fit for a specific hosting platform, and fixing it flipped an actual recommendation. That fix ships alongside that post, not with the two below.
What Changed
The first two fixes are live now, v1.4.4 and v1.4.5, across all four adapters (Claude Code, Claude.ai, OpenAI Codex, and Antigravity), kept in sync so none of them quietly drift from the others.
npx skills add Magithar/SKILLmama -a claude-code
If you want to see the full run log, exact prompts, exact outputs, both bugs as they actually surfaced and not cleaned up after the fact, the eval harness lives at evals/skillmama-ablation.md.
github.com/Magithar/SKILLmama, Apache 2.0.
Credit where it's due: this whole detour started with Philipp Schmid's talk and the SkillsBench paper it references (Li et al., arxiv.org/abs/2602.12670). It found that human-written skills lift task performance by around 15 to 16 percentage points on average, and that AI-generated skills often add filler that does nothing but burn tokens. If you maintain a skill and haven't run it against a labeled prompt set yet, that's the whole homework assignment. Took less time than writing this post did.
Top comments (0)