Someone in your org built a skills/ repo last quarter. Forty-odd markdown files, each one a paragraph of hard-won advice. Verify the path exists before you edit it. Don't chain three tool calls when one will do. If the test fails twice, read the error instead of retrying. Then it got wired into every agent's context — the big model, the small local one, the cheap one running the triage jobs — and everyone nodded and moved on.
That's the myth I want to argue with: that a shared skill library is a uniform team upgrade.
What the paper actually measured
There's a recent paper on harness self-evolution — the loop where an agent rewrites its own skills between runs instead of waiting for a human to hand-tune prompts. The interesting part isn't the loop. It's the ablation.
Across 103 task–skill pairs, the same skill pack scored roughly +31 on the strongest model in the study and roughly −8.8 on the weakest. Same pack. Same tasks. Opposite sign.
I haven't reproduced their harness, so treat that as their result and not mine. But it matches something I'd been quietly avoiding looking at in my own setup: the skills that made my main agent better were noise on the small model, and in two cases actively worse than no skill at all.
If skills were knowledge transfer, that couldn't happen. Facts don't get worse when you're smaller. A skill that says "our API pagination uses cursor tokens, not offsets" is true for every model you run. A skill that says "be more careful" is not a fact. It's a nudge. And a nudge is only useful if the thing you're nudging is in the wrong place to begin with.
Skills are priors, not facts
Read enough of your own skill files and the split becomes obvious. Almost every line is one of two things.
The first is a convention: we use pnpm, this endpoint returns 204 with no body, the migration files live in db/migrations. Those transfer perfectly. They're cheap, they're checkable, and putting them in a shared file is genuinely correct.
The second is a behavioral intervention — a corrective prior. You wrote "verify before editing" because your model, at some checkpoint, was confidently writing to paths that didn't exist. That skill is doing real work: it's pushing the model off a specific failure mode that its weights made likely.
Now drop that same line into a model that never had the failure mode. You've spent context teaching it to defend against a bug it doesn't have. Best case it's dead weight. Worse: the model starts doing an extra verification tool call it didn't need, burning latency on every single turn, and sometimes that extra call is where it gets confused and reads a stale file. You didn't add caution. You added a detour.
That's the −8.8. Not sabotage. Just imported tuning.
The failure mode is structural
Here's the part that bothers me more than the numbers. A shared AGENTS.md that everyone appends to converges on the average of everyone's models. It's a committee document, and committee documents are optimized for not being wrong in the meeting where they're approved.
So you get a file that's slightly useful for the flagship, mildly harmful for the cheap model, and nobody can tell, because nobody's measuring per-model. The file grows. Skills get added after incidents and never removed, because removal requires evidence and addition requires only a bad afternoon.
That's not a knowledge base. That's scar tissue.
What I do instead
Three rules, none of them clever.
Split the file. conventions.md is shared, versioned, reviewed, and kept brutally short — only facts that are true regardless of what's generating the tokens. skills/ is per-model, and it's a manifest, not a library. Each model gets the subset that was measured to help it.
Ablate against your own failures. I keep a set of about thirty tasks my agents used to fail: real ones, pulled from logs, each with a pass/fail check. Running a skill pack with and without, at the same temperature, on the same tasks, is maybe forty minutes of compute for a small model and worth every minute. Pass rate, not vibes. If a skill is neutral-to-negative on your task set, it doesn't ship, no matter how smart the paragraph reads.
Require an incident for every skill. If I can't name the run that motivated it, it doesn't go in. Skills are antibiotics; prescribing them prophylactically is how you get resistance.
The cost math is worth doing once. A skill that adds one extra tool call per turn on a multi-step agent task is not a small tax — round-trips dominate latency in tool-using loops, and you're now paying it on every turn of every task, including the 90% that were fine. Multiply by your daily volume and ask whether the marginal reliability was worth it. Usually the answer is yes for one skill and no for eleven.
Where I might be wrong
If your stack is a single model — one provider, one checkpoint, no local fallback — then this whole post is moot. Your shared skills file is your tuning, and the ablation is just "does it help my one model," which is a much easier question. The problem is specifically the polyglot setup: the moment you run two models, or upgrade one, your skill library silently becomes a bet you didn't know you placed.
And it expires. The skill you wrote to fix checkpoint N's overconfidence might be actively harmful at N+2, because the failure mode got trained away. Skills have a half-life. Most teams have no process for noticing.
So the next time a colleague drops a skills repo into the shared workspace and says "this'll help everyone" — the honest reply is: help me measure it first, per model, on our failures. If it's convention, I'll take it today. If it's a prior, show me the delta on the small one. That's the model that's actually running half the tickets anyway.
Top comments (0)