Originally published on the Cloud ex Machina engineering blog.
rtk is one of the tools we'd least like to give up. It sits between our coding agents and the shell and trims command output before the agent reads it, and on one laptop alone it has kept 6.7 million tokens out of agent context. This week we read every note our agents had saved for themselves, all 802 of them, and seven described rtk quietly mangling their output. Every one ended with the same workaround, and none of them had reached a person. By the time we read them, the fix had been public for 47 days. We upgraded the next morning.
Why we run rtk on every shell command
AI coding agents live in the shell, ours included, and every line of output they read lands in the model's context and stays there until the session ends or gets compacted. Most of it is noise, like the hundreds of lines a passing test suite prints when the agent needs one.
rtk is an open-source proxy, written in Rust, that cuts that noise. A Claude Code hook rewrites git diff into rtk git diff, and rtk runs the real command and returns a version filtered for that tool, so test runs come back as their failures and git output comes back compacted, with nothing extra for the agent to do.
On the author's laptop alone, rtk has kept 6.7 million tokens out of agent context, 13.8% of everything the commands it handled printed. We'd install it again tomorrow.
So this isn't a post about a bad tool. Compressing output is the right trade when the agent only reads it, and it goes wrong when the output feeds something else, like a patch file, a count, or another agent, which is where our story starts.
Seven agent memories, one workaround
Our agents keep private notes between sessions, so that a lesson learned on Monday is still there on Tuesday. When we triaged all 802 of them, seven came from the agents of two developers and described rtk failures.
| Date | What the agent recorded |
|---|---|
| Aug 12 |
gh pr edit and gh pr comment lose their subcommand, so gh prints its usage instead |
| Aug 28 | Search flags get mangled, and a commit reports success but creates nothing |
| Sep 18 |
git diff > patch.diff writes a summary instead of a patch, so git apply refuses it |
| Sep 23 | A search returns zero hits for a pattern that matches |
| Sep 28 |
git log --format='%s' > file drops most of the subjects |
| Sep 28 |
gh pr diff N > file drops about half of the diff |
| Sep 29 | A lint run reports exit 2 although the lint passed |
Each memory ends by telling the agent to run the command through rtk proxy, which skips the filter, so each agent found the bug, found a way around it, and wrote both down in a memory file that only that developer's agents load, where none of us would think to look.
One of these failures reached a release note. Our deployment agent writes the note from the commit subjects of the release. For the October 7 release, the first version of the note listed fewer than half of the pull requests that shipped, and a person spotted the gap and fixed it 44 minutes after the deploy. The code that shipped was right, and only its internal description was wrong. Still, no command had failed, and nothing had warned anyone.
Our reviews held up better. The review skill fetches each pull request's diff inside a script, and the hook only rewrites commands an agent types itself. The truncated diff in the September 28 memory came from an agent that fetched one by hand.
The workaround also turned into a habit, and a busy one. In a sample of 206,965 shell commands we pulled for this post, 7,128 went through rtk proxy. Agents of five of our six developers do it, and in the same sample only 33 of 5,952 failed shell commands mention rtk, because a failure that doesn't look like an error doesn't get looked at.
In other words, our agents had built a workflow around the bug.
xkcd 1172: Workflow by Randall Munroe, CC BY-NC 2.5.
Reproducing it
A memory is only the agent's own account, so on October 28 we re-ran the commands on the setup the agents had, rtk 0.37.2 and our hook script.
| Command, output written to a file | Through the hook | Unfiltered |
|---|---|---|
gh pr diff on a 25-file PR |
23,379 chars, 0 file headers | 58,840 chars, 25 file headers |
git log --format='%s' over a long commit range |
1% of the subjects | All of them |
git diff over 20 commits |
580 lines, 0 file headers | 2,485 lines, 62 file headers |
Two memories didn't reproduce, the empty searches and a push failure whose cause went missing, so we left them out.
Upstream had already fixed it
We were on rtk 0.37.2, released April 20, and the latest release when we checked was 1.1.0, from October 25. Our hook was older still, installed in February when rtk's hook was a shell script that did its own command matching. Current rtk ships a rtk rewrite command that decides which commands are safe to rewrite, but upgrading the binary doesn't replace a hook that's already installed, so our old script never called it. Neither side noticed. rtk init --show printed [ok] Hook: rtk hook claude (native binary command) while our settings still ran the February script.
On 1.1.0, rtk rewrite leaves alone every command we tested whose output goes to a file or into wc or head. The changelog puts that work in 0.49.0, on September 11, and five of the seven memories are from after that date.
One rough edge remained in 1.1.0 when we checked. When an agent reads gh pr diff directly, it sees 14 of the 25 files, and the output ends with ... (more changes truncated) and a hint to run rtk git diff --no-compact. That hint names the wrong command. In a clean checkout it prints nothing, so an agent that follows it decides there's nothing else to see, while rtk gh pr diff --no-compact prints all 25 files. The same wrong hint on git show is open as rtk#4817, and both commands share the code that prints it. Related open issues cover silent loss on large diffs, swallowed exit codes and rewritten pipe producers.
Why coding agents don't report bugs
rtk had bugs, and its maintainers fixed most of them while we weren't looking. For 47 days the fix was public and our agents held the evidence that we needed it, and the two never met.
A developer who hits the same tool failure twice gets annoyed, and annoyance is what makes people check for an update or file an issue. An agent doesn't get annoyed. Its job ends with the task, so it writes down the workaround and moves on, and since the workaround works, nobody looks again. Seven agent sessions knew about the bug and none of us did, until we sat down and read what they had written. That reading is the part we're keeping.
What we're changing
Anything a machine reads, like a patch or a diff we hand to a review agent, now goes through rtk proxy or a script the hook can't reach.
We upgraded to rtk 1.1.0 and regenerated the hook with rtk init -g, and the git log that kept 1% of the subjects now keeps all of them. We also pinned rtk, so it now gets updated with our other dependencies, hook included. It sees every shell command our agents run, which is reason enough to give it the same care as anything in our lockfile. We'd been treating it as part of the furniture.
The bigger change is that a workaround in a memory now counts as a bug report, and the weekly memory review we're building will send each one to a ticket rather than to an instruction file. rtk was the easy case, because its maintainers had already done the fixing and all we had to do was upgrade. Across all 802 memories, 93 (12%) record a workaround for a defect, and most of those defects live in our own code and tools, where nobody upstream is going to fix anything for us. Those are the subject of our next post.
Top comments (0)