JetBrains spent about $320 of API credit in July to test rtk, the shell proxy tens of thousands of us installed to save tokens in Claude Code. Their sessions came out 7.6% more expensive per task. Two weeks earlier the same team measured caveman, the skill that advertises a 65% cut, and got 8.5%.
I read both posts with rtk gain open in another tab. It said 11.6 million tokens saved over 25,599 commands.
So who is wrong? Nobody. The two numbers count different things, and once you see which slice of the bill each tool touches, the four tools I run sort themselves into a ranking. That ranking is the useful part, so here it is.
What I actually tested
Four tools, all free, 575,612 GitHub stars between them on September 2: graphify, rtk, Superpowers and caveman. For each one I took the number it advertises, the number JetBrains measured in their paired benchmarks (claude-sonnet-5, 82 to 86 agentic tasks, run twice), and what my own machine shows after months of daily Claude Code use.
The frame that made it click comes from rtk's own docs. A Claude Code bill has two sides. Input tokens are everything the model reads: shell output, your prompt, the system prompt, and the whole conversation replayed on every call. Output tokens are everything it writes. JetBrains replayed 83 sessions and split the reading side:
1.9M characters of tool output, by origin
19.7% shell output rtk can compress
46.3% shell output rtk has no rule for
34.0% Read / Grep / Glob tools that never touch the shell
Only a fifth of what the model reads can pass through a shell proxy at all. Keep that table in mind for the rest.
rtk shrinks shell bytes, and shell bytes are not the bill
rtk sits between Claude Code and your shell. A PreToolUse hook rewrites git status into rtk git status, and the compact output comes back instead of the noisy one. My dashboard after 25,599 commands:
rtk find 354 calls 2.2M saved 46.6%
rtk read 3,504 calls 2.2M saved 10.4%
rtk grep 2,760 calls 1.9M saved 47.7%
rtk ps aux 24 calls 1.1M saved 98.0%
total 25,599 calls 11.6M saved 41.6%
That 11.6M is bytes divided by four, counted on the commands that had a rule. JetBrains found a rule for 349 of 1,056 shell commands, one in three, and combined with the table above that puts the ceiling around 3% of a session's cost. Their measured result was +7.6% per task at low reasoning effort (p=0.004), with 13.8% more turns, and no difference at high effort. Quality unchanged.
The rtk README now says it in one line: up to 90% of the bash output, which is not the same as cutting your bill by 90%. I still run it. One-line test output is nicer to read and it costs nothing. I just stopped expecting it to move the invoice.
caveman trims the smallest slice
caveman makes the agent talk like a caveman: drop articles, filler, pleasantries. Code, paths and error messages are never touched. Its own table shows 1,214 output tokens per answer without and 294 with, the 65% figure.
JetBrains ran it on 82 paired agentic tasks and got 8.5% of output tokens, 592k down to 542k, with no detectable quality change (sign test p=0.82). The reason is structural. An agent's output is mostly code and tool calls, which caveman leaves alone on purpose, so only the prose around them shrinks. Meanwhile the skill's rules cost about 1 to 1.5k input tokens on every turn.
Claude Code has shipped a built-in equivalent since v2.1.237, the Concise output style under /config. Same effect, no plugin. Both apply to the main conversation only; subagents keep their own prompt.
graphify moved the most tokens on a large repo
graphify parses the project with tree-sitter, locally, and builds a knowledge graph the agent queries instead of grepping files. Zero model calls for the code pass. I ran its built-in benchmark on one of my projects:
nodes 34,031
edges 56,865
naive full read 2,268,733 tokens
average graph query 24,702 tokens
91.8x fewer per query
Two caveats that the number hides. The comparison is against reading everything, and a grep-driven session was never that expensive, so the real gain is smaller. And the graph goes stale; you rebuild it with graphify update or the git hooks, and the semantic pass over docs and PDFs does spend tokens. On a small repo the gap shrinks to nothing.
Install: uv tool install graphifyy (double y), then graphify install.
Superpowers moved the bill, without compressing anything
Superpowers is Jesse Vincent's plugin, 280,792 stars, fourteen skills. Nothing in it compresses output. It changes what gets read and which model reads it.
A brainstorming gate blocks any code until the intent is approved. The plan skill cuts work into steps of two to five minutes: failing test, minimal code, tests green, commit. Each step then runs in a fresh subagent whose context holds only that task, never the session history. And the rule that pays for everything, quoted from the skill file: "Use the least powerful model that can handle each role to conserve cost."
mechanical, well specified -> small model
multi-file, debugging -> standard model
architecture, final review -> most capable model
That is what makes Opus and Fable usable on a $20 Pro plan. The expensive model touches a handful of tasks instead of the whole session. The cost is ceremony: the gate fires on a one-line fix too, and that is tokens as well.
What this doesn't prove
My graphify and rtk numbers come from my own projects, and repo shape changes them a lot. JetBrains measured on Sonnet 5 with 82 to 86 tasks; a different model at a different effort level can move the rtk result in either direction, as their own high-effort arm shows. I did not benchmark Superpowers with a paired test; its effect is on model selection, which is visible on the invoice but harder to isolate per task.
The full run
The video walks through the JetBrains tables and my dashboards on screen, with the four tools ranked at the end: Superpowers, graphify, rtk, caveman.
What do you run to save tokens, and did you ever check it against the bill? A) a shell proxy B) a code graph C) task and model discipline D) never measured, just installed
English isn't my first language; I used an editing tool to polish the wording. The measurements and opinions are mine.
Top comments (1)
The 13.8% increase in turns explains why micro-optimizations backfire on agent invoices. When you compress tool output, you strip out ambient context that the model uses to decide its next action. If that missing context triggers even one extra exploratory turn, the full conversation replay on that turn wipes out thousands of saved shell bytes. In multi-step agent loops, turn count is the primary cost driver because context replay scales quadratically with session depth.