Your Claude Code Plugins Are Probably Costing You Money
📖 Read the full version with charts and embedded sources on AgentConn →
GitHub is in the middle of a middleware gold rush. Of the fifteen repositories trending this week, thirteen are agent frameworks, skill packs, or plugin bundles — Orca at 127K stars, ECC at 305K, Superpowers at 319K, Graphify at 149K. The ecosystem is telling you that the scaffolding around your coding agent matters as much as the model inside it.
There is just one problem: the benchmarks say the opposite.
A Chase AI benchmark tested five of the most popular Claude Code plugins — Superpowers, ECC, Graphify, Caveman, and Claude Mem — head to head against plain Claude Code with no plugins. Every single one cost more, ran slower, or both for the same output quality. Caveman, the plugin that markets itself on token savings, came out 8% more expensive than the baseline. A one-line instruction in CLAUDE.md ("answer concisely") beat it by 23% on token usage.
The data is not ambiguous. The majority of 2026-era Claude Code plugins were built to compensate for weaker models. Frontier models no longer need most of that compensation. And you are paying for it on every request.
The Overhead Tax You Are Not Seeing
Every Claude Code plugin loads its skill descriptions, agent definitions, and hook registrations into your session context. These tokens are billed as input on every single request — whether you use the plugin's features or not.
DeveloperToolkit.ai measured this directly across five popular plugins in September 2026:
| Plugin | Skills | Agents | Always-On Tokens |
|---|---|---|---|
| Superpowers | 15 | 0 | ~838 |
| mattpocock-skills | 25 | 0 | ~1,609 |
| agent-skills | 34 | 4 | ~3,620 |
| gsd-core | 144 | 64 | ~10,700 |
| ECC | 386 | 68 | ~41,515 |
Read the full DeveloperToolkit analysis →
ECC — the most popular harness bundle on GitHub with 305K stars — injects approximately 41,515 tokens of always-on overhead into every session. That is 50 times the overhead of Superpowers. Add the SessionStart hook injection (capped at 8,000 characters by default), and you are burning nearly 50,000 tokens before your first prompt reaches the model.
At Opus pricing ($15 per million input tokens), that overhead alone costs roughly $0.75 per session. Run 20 sessions a day, and you are spending $450 per month on plugin metadata the model never uses.
💡 The always-on tax explained: Claude Code loads every installed plugin's skill roster, agent descriptions, and MCP server definitions into the system prompt. This context persists across every exchange in a session. Even if you never invoke a plugin's features, you pay for its presence in every API call.
A separate practitioner test confirmed the pattern at smaller scale: running 10 plugins simultaneously added roughly 20,000 extra input tokens per exchange and burned through credits "twice as fast as expected."
The Caveman Paradox: When Token Savings Cost You More
Caveman is the most instructive case study. It explicitly markets itself as a token-saving tool — it teaches the model to respond in compressed, abbreviated language. The README claims 65% fewer output tokens.
Independent benchmarks tell a different story.
JetBrains ran an A/B test on SkillsBench and measured 8.5% savings — not 65%. That is a 7.6x gap between the advertisement and the measurement.
A rigorous 900-trial evaluation by an independent researcher went deeper. Across 60 tasks (SWE-Bench, USACO, Terminal Bench), three runs per configuration:
- Caveman reduced total tokens by 18.4% vs plain Claude Code on Sonnet 5
- But the savings came from fewer steps (17%) and tool calls (20%), not shorter prose
- Visible prose is only 0.05% of session tokens — compressing it barely moves the bill
- Setting Claude Code to medium reasoning effort matched Caveman's Pass@3 score at lower cost
- One researcher called the failure to test against a simple "be brief" instruction the "Terse Control Fallacy"
Read the full Caveman benchmark →
⚠️ The Terse Control Fallacy: Caveman advertises 65% token savings. JetBrains measured 8.5%. A 900-trial study found that setting reasoning effort to medium achieved the same accuracy at lower cost. The token savings are real but small, and a simpler intervention achieves the same result.
The bottom line: Caveman's behavioral changes (fewer tool calls, shorter reasoning chains) do save tokens. But a one-line CLAUDE.md instruction or a reasoning-effort toggle achieves the same effect without a plugin's overhead, maintenance burden, or update risk.
The Harness Cost Multiplier: What the Research Says
The plugin-level benchmarks are part of a larger pattern. Three independent studies in 2026 converge on the same finding: the infrastructure around a model can multiply cost without improving results.
Study 1: FrontierHarness Eval (27,280 runs)
The FrontierHarness Eval v1.1 is the largest independent harness benchmark to date. It ran 22 harness configurations across 5 models on 62 terminal tasks — 27,280 total runs consuming 83 billion tokens.
The headline finding: 64.5% of all spending went to trials that did not pass verifiers. Of that, 52.7% went to outright failures and 11.8% to timeouts. For every dollar that produced a working solution, nearly two dollars went to attempts that produced nothing.
The HN discussion (223 points, 87 comments) captured the tension. The top comment noted the framing problem: "64% of the spending was on tasks that didn't pass terminal bench verifiers. It's unclear whether that's supposed to be surprising." Another commenter pushed back: "For a given model, the best harness depends on the problem."
But the cost data was harder to argue with. In v1.0, cost per pass varied 17x across nine harnesses on the same model (Kimi K3). In v1.1, the spread narrowed to 1.9x-4.9x within each model — still significant. And model choice explained 92.8% of cost variance, while harness choice explained only 2.3%.
That last number is the one that matters for plugin buyers: the model you pick explains 40 times more of your bill than the harness around it.
Study 2: Shandong University (492 tasks)
Shandong University researchers (arXiv:2610.04433) tested three harnesses — Claude Code, OpenCode, and mini-SWE-agent (a minimal 100-line Python scaffold) — across five models on 492 SWE-bench Verified tasks.
The findings:
- Claude Code and mini-SWE-agent scored within 2 percentage points of each other on accuracy
- Cost per task varied by up to 3x across harnesses with the same model
- Claude Code's initial fixed context: 16,581 tokens. Mini-SWE-agent: 829 tokens
- Switching to a newer Claude Code release improved task pass rate by one task (34 to 35-36) but doubled the cost ($0.73 to $1.65 per trial)
The conclusion was blunt: more elaborate harnesses did not reliably improve accuracy but substantially raised costs.
The pattern extends beyond coding tasks. A separate HN thread on computer-use agents reached the same conclusion: the harness around the model matters as much as the model — and costs more when it gets it wrong.
Study 3: TestDino Overhead Benchmark (11 agents)
TestDino measured first-call overhead across 11 coding agents in a clean Playwright repo. The spread was 16.8x between the lightest and heaviest agent:
| Agent | First-Call Tokens |
|---|---|
| Pi 0.73 | 1,894 |
| Aider 0.86 | ~2,500 |
| Cline CLI 2.11 | 5,382 |
| Gemini CLI 0.59 | 9,290 |
| Claude Code 2.1.293 | 31,480 |
| Kiro CLI 2.28 | 31,798 |
Read the full TestDino benchmark →
Claude Code already sits near the top of the overhead chart before plugins. Every plugin you add makes it heavier. The harness resends this prefix on every step, so the cost multiplies over a task.
💡 The compounding effect: A 31,480-token base overhead, multiplied by 15 steps in a typical task, means Claude Code sends ~472K tokens of harness context before your code even enters the picture. Add a heavy plugin and that figure crosses 700K.
The GitHub Gold Rush vs. the YouTube Reality Check
The disconnect is striking. GitHub's trending page tells one story: install more tools, add more scaffolding, the harness is the product. YouTube's benchmarks tell another: most of that scaffolding is dead weight on a frontier model.
Both are right — for different eras. An earlier HN thread on a talk titled "Why the Harness Matters More Than the Model" drew the same battle lines — practitioners split on whether harness investment pays off or just burns tokens.
The agent framework explosion on GitHub reflects genuine demand from 2025 and early 2026, when models were weaker and needed more guardrails. Superpowers was built when Claude needed to be reminded to write tests. ECC was built when agents needed 68 subagents to handle tasks they can now do natively. Caveman was built when output verbosity was a real cost driver.
But models improved faster than the ecosystem adapted. Nate Herk's analysis captures this shift: Anthropic's own guidance now suggests re-examining your harness when a new model lands and stripping away pieces that are no longer load-bearing.
The 319K stars on Superpowers are not evidence that it makes you more productive. They are evidence that 319K developers installed it. Installation counts measure adoption, not value. And adoption follows marketing, recommendations, and "best plugins" listicles — not benchmarks.
The Contrarian Case: When Plugins Earn Their Keep
Not every plugin is overhead. The argument for plugins has three defensible legs:
1. Structural discipline. Superpowers at 838 tokens of overhead is negligible. If its methodology (plan-then-execute, test-then-commit) prevents one wrong-turn refactor per week, it pays for itself many times over. The cost argument is about heavy plugins, not all plugins.
2. Domain-specific tools. A Playwright MCP server, a database schema loader, or a codebase indexer like Graphify provide capabilities the model genuinely lacks. These are tools, not scaffolding — and tools earn their context cost when they provide information the model cannot derive from the codebase alone.
3. Team standardization. In a 20-person team, a shared plugin that enforces commit conventions, test coverage gates, and review checklists has organizational value beyond token economics. The overhead is a team tax, not waste.
The distinction is between tools (which give the model new capabilities) and scaffolding (which compensates for model weaknesses). Tools retain value as models improve. Scaffolding becomes dead weight.
What to Do Instead: A Measurement Protocol
The actionable thesis is simple: measure your own stack before installing anything.
Step 1: Establish your baseline.
Run /cost in a fresh Claude Code session. Note your base overhead. Run a representative task. Record total tokens, cost, and time.
Step 2: Add one plugin at a time.
Install one plugin. Run the same task. Compare. If the plugin did not measurably improve your outcome (fewer bugs, faster completion, lower cost), uninstall it.
Step 3: Try the free alternative first.
Before installing Caveman, add Answer concisely. Skip explanations unless asked. to your CLAUDE.md. Before installing a planning plugin, try setting reasoning effort to medium. Before installing a test-enforcement plugin, add a pre-commit hook.
Step 4: Audit quarterly.
Models improve. Plugins that were load-bearing six months ago may be dead weight today. Re-run your measurement protocol after every major model release.
💡 The one-line test: Before installing any token-saving plugin, add
Answer concisely. Avoid verbose explanations.to your CLAUDE.md and measure the result. If a free instruction achieves 80% of a plugin's benefit at zero overhead, the plugin is not worth its context cost.
For teams managing agent spend at scale, the FrontierHarness Eval methodology provides a framework: measure cost per passed task, not just pass rate. A harness that costs 3x more per task but passes 2% more tasks is a bad trade.
The Bigger Picture: Scaffolding Debt
The plugin ecosystem is accumulating what we might call scaffolding debt — infrastructure built to compensate for yesterday's model limitations that now imposes an ongoing cost on today's workflows.
This is the agent-era equivalent of technical debt. It compounds silently. Every session loads the overhead. Every team member pays the tax. And unlike technical debt in your own codebase, plugin debt is controlled by external maintainers with their own incentives and release schedules.
The Shandong University researchers put it most precisely: "The numbers to carry between vendors are the first call, the growth per step, and the step count." That is the measurement framework. First call overhead (how much the harness injects before you say anything), per-step growth (how much it adds on each exchange), and step count (how many exchanges the harness induces to solve a task).
For related analysis, see our coverage of how the harness is becoming the real moat, why Big Tech is open-sourcing agent harnesses, and the system prompt reduction audit that showed how stripping unnecessary context improved both speed and accuracy.
One HN discussion demonstrated the inverse case most starkly: by changing only the harness — same models, same tasks — researchers improved 15 LLMs' coding scores in a single afternoon. The harness is load-bearing infrastructure, not decoration. But when it is built for a model that no longer exists, it is also dead weight.
The middleware gold rush will continue. Stars will accumulate. "Best plugins" listicles will proliferate. But the benchmarks are clear: for most developers, in most workflows, plain Claude Code with a well-written CLAUDE.md outperforms the scaffolding built for weaker models. Measure before you install. And measure again after every model upgrade.
Originally published at AgentConn







Top comments (0)