Installing a skill library on an AI agent fixes tasks it was failing and breaks tasks it was passing - and a new study puts numbers on both sides for the first time. Across 18 skill-library conditions, skills newly solved 553 task conditions while breaking 324 the bare agent had already handled. That is 59% of the gross improvement cancelled, leaving a net gain of 229. Every single library tested broke something.
Key facts
- 553 fixed, 324 broken across 18 skill-library conditions; net improvement 229.
- 5,832 paired task-condition runs covering two office-work benchmarks, three model-and-harness stacks and three skill libraries.
- Posted 24 July 2026 as arXiv preprint 2607.22520, The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents.
- Primary source: the full paper text.
A "skill" here is a reusable procedural unit: a short triggering description, a natural-language instruction body, and sometimes code. The crucial design detail is that the description stays resident in the agent's system context on every single step, while the body only loads once a skill is actually invoked. That split is what makes the study possible - it lets the authors isolate harm caused by a skill that is installed but never called. Our earlier coverage of agent skills becoming a package format tracks how quickly this pattern spread.
Where the damage comes from
The paper names three mechanisms.
Description osmosis is the strangest. The one-line description alone shifts the agent's behavior even when no skill body is ever loaded. The worked example has an agent move from a passing answer to the same failing answer under all three libraries, having read no instructions at all - the authors trace the shift to words sitting in the resident descriptions. Think of a mechanic who glances at a shelf of manuals labelled "brake service" and starts thinking about brakes before opening one.
Grounding displacement is the biggest single cause. An invoked procedure overrides a correct reading of the task: the agent follows the skill's assumptions about which table, range, definition, year or entity to use instead of what the question actually said. Of 81 regressions on one office-work benchmark, 59 were classified this way.
Verification displacement is the third: the procedure suppresses or replaces the agent's final answer check. It was uncommon in isolation on the question-answering benchmark, but the spreadsheet results point at verification as a major failure point.
Why it changes how you read a leaderboard
On one setup, the library that fixed the most tasks also broke enough previously-working ones that a different library - with fewer new wins - produced the better net result. Average pass rate hides that entirely. The paper's argument is that a skill library needs to be measured for preserved competence as well as new competence, which is a different quantity from the one every leaderboard reports. Our lesson on how AI gets benchmarked covers why aggregate scores conceal this kind of trade.
There is also a bonus finding that has nothing to do with agents. One spreadsheet benchmark's grader could not recalculate some valid modern formulas. Re-grading with a full spreadsheet engine recovered 226 outputs that had been marked wrong. Some of what looked like agent failure was evaluation failure.
The companion paper is not the rebuttal
A second skills paper landed the same day: Skill Self-Play, arXiv 2607.22529, which reached second place on Hugging Face's daily papers with 30 upvotes. It looks like a contradiction and is not.
Skill Self-Play uses skills at a different lifecycle stage. Its packages contain routing metadata, rules, generation hints, examples, an executable validator and usage statistics. A router samples one package to condition a task-generating proposer; the solver then trains on validated tasks. The finished solver is prompt-only at inference - these are not permanently installed end-user skills. The library grows by roughly 20 packages per round across five iterations, reaching 86 active skills in one run, while stale ones are pruned.
That is not evidence against the regression tax. It is an attempted answer to it: do not dump a growing library into every inference context, route to one package at a time, give each an executable validator, and retire the weak ones. Tellingly, its own ablation finds that skill-only curriculum generation hurts generalization, and that frozen skills underperform the dynamic system. Its code is publicly released under Apache-2.0.
The one-line synthesis: skills work best as governed infrastructure - routed to one job, tested against regressions, and eventually removed - not as a pile of instructions bolted onto an agent.
The honest caveat
The statistical case is narrower than the headline feels. There is no seed replication, model and harness are confounded, and after correcting for 18 comparisons only three effects remain significant - all on a single spreadsheet setup. The mechanism taxonomy was assigned by one author. And the paper does not establish that the tax scales with library size: its libraries range from 3 to 23 skills, but they are not length-matched and the authors say plainly this is not a controlled dose study. Skill Self-Play, for its part, never measures paired pass-to-fail regressions at all, so it cannot yet claim to have solved the problem it appears to answer.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)