In my last post, I ran an eval harness against SKILLmama and found two bugs. There was a third finding I held back, because it came from a different kind of test, and it's the one that actually changed a real recommendation.
The idea is simple, and it comes straight from the paired-evaluation method in the SkillsBench paper: take one question, run it twice, once with the skill, once without, same project, same everything else. Compare what comes back.
I hadn't actually done this yet. Every eval run up to that point only tested SKILLmama with itself turned on. So I ran the real comparison, on a real external project, not a toy example.
The Setup
Project: nutri-bot, a Telegram bot I built separately, FastAPI backend, Redis for caching, Gemini and Groq for LLM calls, deployed on Render's free tier. No vector database anywhere in it yet.
Question, asked both ways: "What vector DB should I use for RAG in my FastAPI app?"
Skill-off: a plain agent, given read access to the repo, no web search, no structured pipeline, just its own judgment plus whatever it found by reading the files.
Skill-on: SKILLmama, full pipeline, live web search, verified GitHub stars and PyPI downloads, security screening, the whole thing.
What Skill-Off Said
It read requirements.txt, .env.example, and SETUP.md, then recommended pgvector or Upstash Vector, and specifically warned against Chroma or FAISS running in-process:
On Render's free/small web dynos your process can restart or spin down, so an in-memory or local-disk vector index disappears or desyncs. Only reasonable if your corpus is tiny and rebuildable in a few seconds on startup.
It also tied the embedding model choice to the LLM already in use (Gemini's text-embedding-004, since the app already calls Gemini) and asked what the RAG was actually for before locking in a recommendation.
No live verification. It said so itself when I asked it to self-report: five options seriously compared, no star counts or download numbers checked, pure judgment from reading the repo.
What Skill-On Said
SKILLmama's pipeline ran the full search, verified real numbers, and produced a scored table:
| Candidate | Compat | Pop | Maint | Simple | Score |
|---|---|---|---|---|---|
| Chroma | 8 | 10 | 10 | 9 | 9.05 |
| Qdrant | 6 | 10 | 8 | 6 | 7.30 |
| pgvector | 3 | 10 | 7 | 3 | 5.55 |
Chroma came out on top, recommended specifically because it's in-process and needs no extra infrastructure.
That's the exact property skill-off had just flagged as a deployment risk on this platform. SKILLmama's Compatibility check verifies things like whether a required env var exists or whether a CLI is on PATH. It had no check for whether the app's actual hosting target keeps local disk around between restarts. So it confidently scored the one candidate that would silently lose its data on the next deploy.
The Uncomfortable Part
Grounded, verified, live-searched data still lost to a plain agent that just read SETUP.md and thought about it for a second. Not because the plain agent was smarter. Because it happened to look at the one file that mattered for this specific question, and my scoring formula had never told SKILLmama to look there at all.
This is exactly the failure mode Schmid's talk was warning about, and exactly why "run the ablation" isn't optional. If I'd stopped after the first eval run, where I only tested SKILLmama against itself, I would have shipped a skill that confidently recommends the wrong answer for a common deployment shape, and never known.
The Fix
Phase 4 now runs a Deployment Persistence Check before scoring any candidate that stores data locally or in-process. It looks for the same signals the plain agent found by reading a doc file, but does it systematically:
fly.toml -> persistent only if it has a [[mounts]] section
railway.toml -> persistent only if a volume is configured
render.yaml -> persistent only if it has a disk: block
vercel.json -> always ephemeral, serverless by design
docker-compose.yml -> persistent only if volumes: maps the data path
If a deployment target is detected and none of that is configured, Compatibility gets capped and the output carries an explicit warning instead of a clean-looking number.
Re-ran the same question after the fix. Chroma's Compatibility dropped from 8 to 5, with the warning attached. The score gap to Qdrant narrowed to under half a point, which triggered SKILLmama's own tiebreaker rule, and the output now recommends Qdrant Cloud as the safer default if the data is going to accumulate over time, Chroma only if the corpus is small enough to rebuild on every cold start. Same conclusion the plain agent reached, just with verified stars, downloads, and a security pass attached to it.
Then the Same Prompt Found Something Worse
That wasn't the end of it. The old security gate had a rule that just said "has a dependency with a known CVE," with no data source behind it, so in practice it depended on whatever a search happened to surface that day. I replaced it with a real check: query OSV.dev for the exact version being recommended, across npm, PyPI, Go, and crates.io.
Then I re-ran this same eval to make sure the new check actually worked end to end. It did, and it took out the library that had won every single prior run:
curl -X POST https://api.osv.dev/v1/query \
-d '{"package":{"name":"chromadb","ecosystem":"PyPI"},"version":"1.5.9"}'
Chroma's current latest release carries CVE-2026-45829, a CRITICAL pre-authentication code injection vulnerability, and there is no patched version. I checked that directly against the GitHub Advisory API rather than trusting the OSV response alone: first_patched_version: null. Chroma is now BLOCKED before it reaches scoring, and Qdrant wins outright, no narrow tiebreaker needed.
Reading the advisory closer, it only triggers in Chroma's server mode with trust_remote_code set to true, an opt-in flag most embedded, in-process usage never touches, including nutri-bot's. The rule still blocks regardless of that, on purpose. I don't want a discovery skill quietly deciding a CRITICAL CVE doesn't apply to you. But a block with no explanation isn't much better than no block at all, so the same day, I changed the rule to quote the advisory's stated trigger condition straight from OSV into the output. Still an unconditional block, now with the context to judge it yourself instead of taking my word for it.
What Changed
Live in v1.4.6, across all four adapters. The full run, both outputs unedited, is in evals/skill-on-vs-skill-off-comparison.md, and the run-by-run log including this one is in evals/skillmama-ablation.md, if you want the actual transcripts instead of my summary of them.
npx skills add Magithar/SKILLmama -a claude-code
github.com/Magithar/SKILLmama, Apache 2.0.
The lesson isn't "plain agents beat structured pipelines." It's that a pipeline is only as good as the checks someone remembered to put in it, and the fastest way to find the one you forgot is to run the version without it side by side and see where it wins. The second lesson, from finding the CVE, is that "the checks someone remembered to put in" needs re-running every time you change what those checks actually do, not just once when you write them.
Top comments (0)