DEV Community: Yahav Ohana

Don’t Break the Agent: Lessons in Token Optimization

Yahav Ohana — Thu, 20 Aug 2026 17:26:38 +0000

By Shay Dahan, JFrog Boost Co-Founder Yahav Ohana, JFrog Boost Co-Founder August 20, 2026

This one is for the curious souls who wonder how somebody actually builds a harness optimizer — and, more to the point, how they know it works.

When we launched JFrog Boost into public preview, we told the story of the bill that broke us and the 100 billion tokens we clawed back across JFrog R&D. What that post didn’t cover is the question that consumed most of our engineering time: how do you measure any of this?

Because it turns out that “we cut 84% of the output” and “we made your session better” are two completely different claims, and only one of them is easy to put on a dashboard.

This is the first post in a series that opens up how JFrog Boost measures itself, then benchmarks it against the token-saving tools teams are already running. The next post will dig into RTK specifically: where its approach gets the category right, and where its measurement falls short. Everything below is the methodology those benchmarks are built on.

Join Token Saving Slack Community

The Manifest: Why Boost Exists

The gap we set out to close. Input context is now the dominant line item in an agentic coding bill — by Cursor’s own accountingExternal link., roughly 70% of it. Your agent isn’t paying to think. It’s paying to re-read a pytest log it already understood four turns ago. The waste is real, it’s enormous, and it’s almost entirely invisible from inside the session.

Why the obvious fix doesn’t work. Once you start looking, you find a shelf full of tools promising 60–90% reduction. We tried them, we borrowed from them, and we kept running into the same three gaps:

They measure the wrong boundary. Compression measured at the wrong point in the pipeline flatters itself. More on this below — it’s the most subtle problem in the space.
They optimize silently. The agent doesn’t know it’s been handed a summary. So when it needs the part you dropped, it can’t ask for it. It guesses, or it re-runs the command, or it spirals.
They have no runtime feedback loop. A benchmark tells you a filter was safe on the tasks you thought to test. It tells you nothing about the 400th session on Tuesday afternoon in a repo nobody on your team has ever opened.

Our approach, in four commitments:

Accuracy is the constraint; savings is the objective. The goal was never minimal tokens. It was accurate agents that use fewer tokens. If a filter saves 90% and costs one correct answer, it’s a bad filter.
Optimize last. Boost is the final stage before output enters the context window — never an earlier one. This is a measurement decision as much as an architectural one.
Tell the agent what happened. Every optimized output carries a marker and a way back. The agent understands the optimization layer instead of being quietly manipulated by it.
Treat every recovery as data. When an agent asks for the original, that’s not a failure to hide. It’s the highest-signal event in the system.

Everything below is what those commitments cost us to actually implement.

Why We Started Measuring in the First Place

We didn’t set out to build a measurement stack. We set out to build filters.

What sent us down this road was a period where the numbers stopped making sense. The savings percentages coming off our own dashboards looked fantastic. Meanwhile, sessions didn’t feel proportionally better. Developers weren’t reporting that they’d stopped hitting compaction. Some of the biggest reported wins came from commands where, when we sat down and read the actual output, there hadn’t been that much to cut.

Something was off, and “the graph is up and to the right” was not an acceptable answer for a tool we were about to push to a thousand engineers.

Pulling on that thread, we found two separate questions tangled together under one metric:

Am I breaking the agent?
Am I actually saving tokens?

They need completely different instruments.

Question 1: Am I Breaking the Agent?

The first rule is that no external intervention of ours should make the agent worse — slower, dumber, or stuck. Which raises an awkward problem: how do you detect a confused agent, at runtime, across thousands of live sessions a day?

You can’t A/B test a developer’s real workday. You can’t read the conversation — Boost collects only high-level telemetry like token counts and macro errors; we never see, store, or transmit code or conversations, and we weren’t about to start. And confusion doesn’t throw an exception. It looks like an agent quietly running ls six times to re-establish where it is.

So we needed the agent itself to tell us. In band.

The Boost suffix. Every optimized output ends with a short marker:

[Boost filtered 84% of the original output, get original content with boost retrieve 123]

The awareness rule. That marker is useless if the agent doesn’t know what to do with it. So Boost ships a bundled skill — a minimal system prompt teaching the agent what the optimization layer is and that the full output is one command away.

The signal. Now the interesting part. When an agent calls boost retrieve, it is telling us, unprompted and in production, that the filter we applied removed something it needed. That’s a filter which cost the agent focus instead of buying it focus.

We emit telemetry on every retrieve. Aggregated across sessions, it becomes a ranked list of our own worst filters — by command, by repo type, by language. A retrieve rate that spikes after a filter change is a regression, and we treat it like one.

This is the flywheel: the system surfaces its own failures, ranks them by frequency, and proposes where to loosen. Close to self-improving — with a human reviewing every change before it ships. We’re optimizing other people’s context windows. That’s not a place for full autopilot.

The offline half. Runtime signal tells us where we hurt real users; it can’t tell us whether we’ve drifted on quality overall. For that we run Terminal-Bench 2.0: identical task pass rate, roughly 12% lower cost. Same answers, less money. That’s the bar — a savings number that comes with a pass rate attached, or it doesn’t ship.

Question 2: Am I Actually Saving Tokens?

Much of our CLI and MCP noise filtering was inspired by RTK, a genuinely clever open-source project that deserves credit for putting this problem on the map. But building on that idea surfaced a measurement trap that we think is underappreciated across the whole category.

Where you stand in the pipeline determines what you can honestly claim.

A prefix-based optimizer wraps the command:

rtk git log | grep ABC-123

Read the shell carefully. The optimizer compresses git log‘s output — and then grep runs. Whatever grep was about to discard, the optimizer already claimed credit for discarding. The tokens it reports saving were never going to reach the context window in the first place. If you intercept output before a grep filter and take credit for what grep would have cut anyway, that isn’t savings. It’s arithmetic.

Standing early has a second cost: you’re rewriting output before the pipeline that the developer actually designed has finished with it, which is exactly when you’re most likely to cut something downstream needed.

So we chose the other end:

(git log | grep ABC-123) | boost

Boost runs last. What it sees is precisely what would have landed in the context window, byte for byte. Every token it reports is a token that was genuinely on its way in.

Then multiply by turns. Here’s the part most savings estimates miss entirely.

Every time a user sends a new message in an existing session, all the previous tokens are sent again. Until the session hits compaction, every CLI command, every output, every MCP request and response stays in the window — and you’re billed for it on every single turn.

So the value of trimming a command output isn’t the tokens you cut. It’s the tokens you cut, multiplied by the number of turns that output would have sat in context. A 3,000-token log removed at turn 4 of a 40-turn session isn’t a 3,000-token saving. Our per-session estimate accounts for this residency, which is why our numbers are calculated rather than extrapolated from a compression ratio.

And it’s why the savings compound in both directions: fewer tokens per turn means a cheaper turn, and more turns before you hit compaction, and fewer compactions — each of which costs you real context fidelity.

If You’re Building One of These

Four things we’d hand to anyone starting down this road:

Pick your measurement boundary before you pick your filters. It determines whether every number you produce afterward is real.
Give the agent a way to disagree with you. An optimizer the agent can’t override is an optimizer you can’t debug.
Instrument the recovery path, not just the savings path. Your retrieve rate is a more honest quality metric than your compression ratio will ever be.
Never report a savings number without a correctness number next to it. Compression without a pass rate is a claim, not a result. Boost is free and in public preview. Try it at boost.jfrog.com. — and if you find a filter that makes your agent stupider, boost retrieve will tell us before you have to.

This is the first post in a series benchmarking the approach behind JFrog Boost against the token-saving tools teams already run in production. Next up: a closer look at RTK, where its measurement boundary breaks down under real agentic sessions, benchmarked head-to-head against JFrog Boost.

Try Boost today

Why Model Routing Backfires and How to Build Agents That Don’t Burn Your Budget

Yahav Ohana — Thu, 23 Jul 2026 21:20:34 +0000

By Shay Dahan, JFrog Boost Co-Founder Yahav Ohana, JFrog Boost Co-Founder | July 23, 2026

Model routing promises to cut your AI agent spend by offloading routine tasks to cheaper models like Claude Haiku while reserving frontier models like Claude Sonnet for complex reasoning. In the right configuration, routing strategies can reduce inference costs by 40–85%. But if you implement routing incorrectly in a multi-turn agent, you can end up paying more than if you’d never routed at all. Here’s why and how to fix it.

How Token Costs Accumulate Across a Session

Understanding why routing backfires requires understanding how agentic sessions accumulate costs. Each turn in a conversation compounds the token count, and that compounding is the engine behind both your savings and your risk.

Every turn in an agent session carries a growing token payload. The table below shows how the distribution shifts across a typical three-turn session:

By Turn 3, cached tokens make up the vast majority of your token payload. This is the efficiency you’re trying to protect. Switching models mid-session destroys that protection entirely.

Why Does Switching Models Mid-Session Increase Your Costs?

Switching LLM models mid-session wipes out accumulated prompt cache savings because every provider’s cache is model-specific — the new model has no access to the previous model’s stored history and must re-read the entire conversation from scratch at standard input token rates.

Prompt caches operate on prefix matching. The LLM provider stores a hash of your message prefix. As long as the prefix stays stable and the model stays the same, every subsequent turn benefits from heavily discounted cached token pricing. Anthropic, for example, offers up to a 90% discount on cached input tokens. The moment you switch models, the new model starts with a cold cache. It processes the entire conversation history at full input token cost, as if Turn 1 never happened. For short sessions, this is a minor annoyance. For agentic coding workflows where context windows can stretch to 50,000 tokens or more, a single mid-session model switch can spike costs enough to eliminate all the savings routing was supposed to deliver.

What You Should Not Do

Several common routing patterns look like optimizations but actively hurt your cost profile. These are the most frequent causes of unexpected cost increases in production agent systems:

Switching models in the middle of a long conversation without summarization — This forces the new model to reprocess the entire conversation history at full input token cost. The cache miss penalty for a 20,000-token session can cost more than the savings the switch was intended to generate.
Enabling auto-routing mid-session in LLM gateways — LLM gateway tools like LiteLLM, OpenRouter, and Portkey support automatic routing based on query complexity. This is powerful for single-turn queries, but enabling it mid-session in an agentic loop invalidates your KV cache at every routing boundary.
Modifying tools or system prompts mid-session — Tool definitions sit inside the cached prefix. Adding or removing a tool during a session invalidates the entire cache downstream from that change. Treat session tool configuration as immutable once started.
Using compression calls with a separate model — Many agents compress long history by spawning a summarization call with a cheap model. If that call uses a different model or a different system prompt, it generates its own cache miss and resets the main session cache on return.

Avoiding these patterns requires intentional session design, not just correct routing logic. Your routing strategy is only as good as your session architecture.

What to Do Instead

Effective model routing in agentic systems is a session-level concern, not a query-level one. The following practices let you capture the cost benefits of routing without sacrificing the prompt cache savings that make agentic AI economically sustainable:

Minimize turns per task and delegate early — Keep individual agent sessions as short as possible. When you know a task requires extensive context (e.g., a complex multi-file refactor, a long-running compliance check) delegate it to a subagent at the start, before the session accumulates history. Short sessions mean small caches, which means a model switch costs less.
Use the subagent pattern for model diversity — Instead of switching models within a single session, build an orchestrating agent that delegates subtasks to purpose-built subagents. Each subagent starts fresh with only the context it needs and runs on whatever model is best for its specific task. No cache invalidation. No accumulated history penalty. This pattern (sometimes called the orchestrator-subagent architecture) is the cleanest way to get cost-based model diversity without paying the mid-session switch penalty.
Summarize deliberately before any required model switch — If a model switch mid-session is unavoidable, create a structured handoff: explicitly summarize the active context, the current task state, and any open decisions into a compact message. Start a fresh session with the new model using only that summary as input. You pay standard input costs on a small summary instead of cached costs on a 20,000-token history replay.
Lock your session configuration — System prompts, tool definitions, and model selection should be fixed at session start. Treat them like a schema — changing them mid-session is a migration event, not a configuration tweak. If your agent framework allows dynamic tool loading, audit where and when those loads happen relative to your cache prefix boundary.

How JFrog Boost Protects Your Cache Efficiency Automatically

While managing model routing logic is critical for agent sustainability, context bloat acts as a massive force multiplier for token waste. Every time an agent runs a build, executes a test suite, or pulls logs, hundreds of lines of repetitive terminal noise flood the conversation history.

In a multi-turn session, this noise compounds rapidly. It turns what should have been a lean history into a 50,000-token payload within a few turns. This directly breaks your cost engineering: if your session is bloated with terminal noise, the “cache miss penalty” we discussed earlier becomes exponentially more expensive the moment a mid-session model switch occurs.

When utilizing coding agents, developers shouldn’t have to manually curate terminal output just to prevent a noisy build step from inflating their LLM bill.

JFrog Boost is designed to solve this context bloat directly within the developer workflow. Boost is a lightweight CLI tool that intelligently compacts terminal output during agentic loops, stripping out repetitive boilerplate while perfectly preserving the critical signal, like exact error stack traces. Instead of sending thousands of tokens of raw build logs to your model, Boost trims the payload before it ever hits the agent.

By keeping your session history lean, Boost drastically reduces the baseline tokens processed per turn. This minimizes the financial penalty of unavoidable model routing switches, extends the life of your context window, and maximizes the economic efficiency of your prompt caches.

Designing for Sustainable Agent Economics

Model routing delivers real cost savings, but only when it respects the underlying mechanics of provider prompt caches. To build sustainable agent workflows, treat your conversation history as a highly optimized asset: Keep your primary sessions lean, delegate heavy context to short-lived subagents, and ensure any necessary model transition is handled through a structured state summary rather than a raw history replay.

Ultimately, engineering sustainable agentic systems requires treating token efficiency as a core constraint alongside speed and accuracy.

To implement these session-level optimizations automatically and eliminate context waste in your development workflows, you can install Boost and start saving tokens today.

Connect with us

We would love to hear about your experiences, architectural patterns, and challenges with model routing. Feel free to reach out or follow our ongoing research into coding agents:
→ X: https://x.com/ShayFrektman, https://x.com/yahav_ohana
→ LinkedIn: https://www.linkedin.com/in/shay-dahan, https://www.linkedin.com/in/yahav-ohana
→ Substack: https://substack.com/@yahavohana

Published originally on: https://jfrog.com/blog/why-model-routing-backfires/

Stop Wasting AI Tokens: How to Slash LLM Costs with JFrog Boost [free --forever] 🚀

Yahav Ohana — Thu, 09 Jul 2026 20:11:18 +0000

🚀 TL;DR: AI coding assistants (like Cursor, Claude Code, and Copilot) burn through tokens by sending massive, bloated context windows. We built JFrog Boost to intellectually trim context noise and slash your LLM costs without sacrificing code quality. Try it out, read our launch deep-dive, and star us on GitHub if you like saving budget!

AI coding tools are an absolute game-changer—until you see the token bill.

The reality is that context windows get bloated fast. Your LLM doesn't need to swallow half your repository just to fix a simple bug or refactor a function. Massive prompts don't just cost more; they introduce context drift, slow down response times, and occasionally make the AI hallucinate.

To solve this, we built JFrog Boost.

It’s an open-source developer tool designed to optimize your prompt mechanics, strip out the noise, and drastically lower your token footprint.

🛠️ What it does under the hood:

Token Trimming: Minimizes context bloat before it hits the LLM.
Cost Efficiency: Maximizes the performance-to-cost ratio of your AI engineering workflows.
Faster Iterations: Smaller, tighter contexts mean faster response times from your coding agents.

We just dropped a detailed launch blog post breaking down the benchmarks, the architecture, and exactly how much token overhead you can save starting today.

🌟 Support the Project

We are fully open-source! If you find the tool useful or want to contribute, please give us a star on GitHub.

Give the tool a spin at boost.jfrog.com and let us know what you think in the comments! How are you currently managing your AI token budgets?