Don’t Break the Agent: Lessons in Token Optimization

By Shay Dahan, JFrog Boost Co-Founder Yahav Ohana, JFrog Boost Co-Founder August 20, 2026

This one is for the curious souls who wonder how somebody actually builds a harness optimizer — and, more to the point, how they know it works.

When we launched JFrog Boost into public preview, we told the story of the bill that broke us and the 100 billion tokens we clawed back across JFrog R&D. What that post didn’t cover is the question that consumed most of our engineering time: how do you measure any of this?

Because it turns out that “we cut 84% of the output” and “we made your session better” are two completely different claims, and only one of them is easy to put on a dashboard.

This is the first post in a series that opens up how JFrog Boost measures itself, then benchmarks it against the token-saving tools teams are already running. The next post will dig into RTK specifically: where its approach gets the category right, and where its measurement falls short. Everything below is the methodology those benchmarks are built on.

Join Token Saving Slack Community

The Manifest: Why Boost Exists

The gap we set out to close. Input context is now the dominant line item in an agentic coding bill — by Cursor’s own accountingExternal link., roughly 70% of it. Your agent isn’t paying to think. It’s paying to re-read a pytest log it already understood four turns ago. The waste is real, it’s enormous, and it’s almost entirely invisible from inside the session.

Why the obvious fix doesn’t work. Once you start looking, you find a shelf full of tools promising 60–90% reduction. We tried them, we borrowed from them, and we kept running into the same three gaps:

1. They measure the wrong boundary. Compression measured at the wrong point in the pipeline flatters itself. More on this below — it’s the most subtle problem in the space.
2. They optimize silently. The agent doesn’t know it’s been handed a summary. So when it needs the part you dropped, it can’t ask for it. It guesses, or it re-runs the command, or it spirals.
3. They have no runtime feedback loop. A benchmark tells you a filter was safe on the tasks you thought to test. It tells you nothing about the 400th session on Tuesday afternoon in a repo nobody on your team has ever opened.

Our approach, in four commitments:

• Accuracy is the constraint; savings is the objective. The goal was never minimal tokens. It was accurate agents that use fewer tokens. If a filter saves 90% and costs one correct answer, it’s a bad filter.
• Optimize last. Boost is the final stage before output enters the context window — never an earlier one. This is a measurement decision as much as an architectural one.
• Tell the agent what happened. Every optimized output carries a marker and a way back. The agent understands the optimization layer instead of being quietly manipulated by it.
• Treat every recovery as data. When an agent asks for the original, that’s not a failure to hide. It’s the highest-signal event in the system.

Everything below is what those commitments cost us to actually implement.

Why We Started Measuring in the First Place

We didn’t set out to build a measurement stack. We set out to build filters.

What sent us down this road was a period where the numbers stopped making sense. The savings percentages coming off our own dashboards looked fantastic. Meanwhile, sessions didn’t feel proportionally better. Developers weren’t reporting that they’d stopped hitting compaction. Some of the biggest reported wins came from commands where, when we sat down and read the actual output, there hadn’t been that much to cut.

Something was off, and “the graph is up and to the right” was not an acceptable answer for a tool we were about to push to a thousand engineers.

Pulling on that thread, we found two separate questions tangled together under one metric:

1. Am I breaking the agent?
2. Am I actually saving tokens?

They need completely different instruments.

Question 1: Am I Breaking the Agent?

The first rule is that no external intervention of ours should make the agent worse — slower, dumber, or stuck. Which raises an awkward problem: how do you detect a confused agent, at runtime, across thousands of live sessions a day?

You can’t A/B test a developer’s real workday. You can’t read the conversation — Boost collects only high-level telemetry like token counts and macro errors; we never see, store, or transmit code or conversations, and we weren’t about to start. And confusion doesn’t throw an exception. It looks like an agent quietly running ls six times to re-establish where it is.

So we needed the agent itself to tell us. In band.

The Boost suffix. Every optimized output ends with a short marker:

[Boost filtered 84% of the original output, get original content with boost retrieve 123]

The awareness rule. That marker is useless if the agent doesn’t know what to do with it. So Boost ships a bundled skill — a minimal system prompt teaching the agent what the optimization layer is and that the full output is one command away.

The signal. Now the interesting part. When an agent calls boost retrieve, it is telling us, unprompted and in production, that the filter we applied removed something it needed. That’s a filter which cost the agent focus instead of buying it focus.

We emit telemetry on every retrieve. Aggregated across sessions, it becomes a ranked list of our own worst filters — by command, by repo type, by language. A retrieve rate that spikes after a filter change is a regression, and we treat it like one.

This is the flywheel: the system surfaces its own failures, ranks them by frequency, and proposes where to loosen. Close to self-improving — with a human reviewing every change before it ships. We’re optimizing other people’s context windows. That’s not a place for full autopilot.

The offline half. Runtime signal tells us where we hurt real users; it can’t tell us whether we’ve drifted on quality overall. For that we run Terminal-Bench 2.0: identical task pass rate, roughly 12% lower cost. Same answers, less money. That’s the bar — a savings number that comes with a pass rate attached, or it doesn’t ship.

Question 2: Am I Actually Saving Tokens?

Much of our CLI and MCP noise filtering was inspired by RTK, a genuinely clever open-source project that deserves credit for putting this problem on the map. But building on that idea surfaced a measurement trap that we think is underappreciated across the whole category.

Where you stand in the pipeline determines what you can honestly claim.

A prefix-based optimizer wraps the command:

rtk git log | grep ABC-123

Read the shell carefully. The optimizer compresses git log‘s output — and then grep runs. Whatever grep was about to discard, the optimizer already claimed credit for discarding. The tokens it reports saving were never going to reach the context window in the first place. If you intercept output before a grep filter and take credit for what grep would have cut anyway, that isn’t savings. It’s arithmetic.

Standing early has a second cost: you’re rewriting output before the pipeline that the developer actually designed has finished with it, which is exactly when you’re most likely to cut something downstream needed.

So we chose the other end:

(git log | grep ABC-123) | boost

Boost runs last. What it sees is precisely what would have landed in the context window, byte for byte. Every token it reports is a token that was genuinely on its way in.

Then multiply by turns. Here’s the part most savings estimates miss entirely.

Every time a user sends a new message in an existing session, all the previous tokens are sent again. Until the session hits compaction, every CLI command, every output, every MCP request and response stays in the window — and you’re billed for it on every single turn.

So the value of trimming a command output isn’t the tokens you cut. It’s the tokens you cut, multiplied by the number of turns that output would have sat in context. A 3,000-token log removed at turn 4 of a 40-turn session isn’t a 3,000-token saving. Our per-session estimate accounts for this residency, which is why our numbers are calculated rather than extrapolated from a compression ratio.

And it’s why the savings compound in both directions: fewer tokens per turn means a cheaper turn, and more turns before you hit compaction, and fewer compactions — each of which costs you real context fidelity.

If You’re Building One of These

Four things we’d hand to anyone starting down this road:

• Pick your measurement boundary before you pick your filters. It determines whether every number you produce afterward is real.
• Give the agent a way to disagree with you. An optimizer the agent can’t override is an optimizer you can’t debug.
• Instrument the recovery path, not just the savings path. Your retrieve rate is a more honest quality metric than your compression ratio will ever be.
• Never report a savings number without a correctness number next to it. Compression without a pass rate is a claim, not a result. Boost is free and in public preview. Try it at boost.jfrog.com. — and if you find a filter that makes your agent stupider, boost retrieve will tell us before you have to.

This is the first post in a series benchmarking the approach behind JFrog Boost against the token-saving tools teams already run in production. Next up: a closer look at RTK, where its measurement boundary breaks down under real agentic sessions, benchmarked head-to-head against JFrog Boost.

Try Boost today