The Problem Nobody Talks About
Most teams building agents watch their token costs climb and shrug. "That's the price of intelligence." I did the same thing for months. Then I instrumented a single multi-step agent workflow and found that 40% of my tokens were going to repeated context, another 15% to model calls that didn't need a frontier model, and about 10% to prompts stuffed with filler the model never used.
The bill wasn't a tax on intelligence. It was a tax on laziness.
So I rebuilt the infrastructure layer under my agents. Not the agents themselves — the plumbing. Four changes, applied in order, compounding on each other. The result was a 60% reduction in token spend with no measurable degradation in output quality on my evals.
Here's every layer.
Layer 1: Per-Span Measurement
You cannot optimize what you do not measure. And "total tokens per request" is not measurement — it's a bar tab.
I instrument every agent call as a span. Each span records:
- Model used (exact model ID, not just "GPT-4" or "Claude")
- Input tokens (prompt + system + tool definitions)
- Output tokens (completion + tool calls)
- Cache status (hit, miss, semantic-near)
- Latency (wall clock, not just model time)
- Task complexity score (more on this below)
The spans nest. A top-level "research agent" spawns sub-agents for search, extraction, and synthesis. Each sub-agent gets its own span, parented to the root. I export these to a lightweight JSONL log and run a daily rollup.
What this revealed immediately: my extraction sub-agent was consuming 3x the tokens of search and synthesis combined, because it was receiving the full parent context on every call. The fix took ten minutes — scope the context window to only what extraction needs. That single change was worth 12% of my total spend.
The instrumentation itself costs almost nothing. A few metadata fields per call. But without it, you're guessing.
Layer 2: 3-Tier Cache
Caching LLM responses sounds obvious until you try it. Exact-match caching has a terrible hit rate because prompts vary by a few tokens constantly. Pure semantic caching is slow and unreliable. So I run three tiers:
Tier 1 — Exact match (Redis). Hash the full prompt + model + temperature. Hit rate is low (~8-12%) but lookup is sub-millisecond. This catches retries, polling loops, and duplicate tool calls.
Tier 2 — Semantic near-match (vector store). Embed the prompt, search for neighbors within a tight cosine threshold. I keep the threshold aggressive (0.97+) because a loose match on an LLM call can silently corrupt downstream reasoning. Hit rate sits around 15-20% depending on workload repetitiveness.
Tier 3 — LRU in-process. A simple bounded map keyed on a normalized prompt hash. This catches the "same agent called three times in a row during a retry loop" pattern, which happens more than you'd think in agentic systems.
Combined hit rate across the three tiers: 30-35% on a typical workload. That's 30-35% of calls that never touch a model endpoint. The cache layers cost pennies to run compared to the model calls they eliminate.
One critical detail: cache entries expire based on the task type. Factual lookups get a long TTL. Creative generation gets no cache at all. Classification tasks sit in between. If you cache everything with the same TTL, you'll serve stale reasoning and wonder why your agent started hallucinating.
Layer 3: Prompt Compression
Most prompts I write — and most prompts I see others write — are bloated. Not because the instructions are wrong, but because they carry context the model doesn't need for that specific call.
My compression pipeline runs before every model call:
- Strip redundant system instructions. If the model already knows it's a JSON extractor (from the system prompt), don't repeat it in the user message.
- Truncate tool definitions. Most tool schemas include descriptions, examples, and nested type definitions. For calls where the agent has already selected the tool, I send only the parameter schema, not the full definition block.
- Summarize prior conversation turns. For multi-turn agent loops, I replace older turns with compressed summaries. The current turn and the immediately previous turn stay verbatim; everything else gets a one-line summary.
- Remove whitespace and formatting padding. JSON tool results get minified. Markdown gets stripped to plain text when the model doesn't need to preserve formatting.
This consistently saves 20-30% of input tokens per call. On long-running agent chains where context accumulates, the savings compound — later calls in the chain benefit more because there's more prior context to compress.
I keep the compression deterministic and log the before/after token counts per span so I can verify nothing important got dropped.
Layer 4: Complexity-Based Model Routing
This is where the biggest single savings come from, and where most teams leave money on the table.
Not every agent call needs a frontier model. Classification? A small model handles it. Extraction from structured data? Small model. Summarization of a known-good source? Small model. The only calls that need the expensive model are novel reasoning, complex multi-step planning, and nuanced generation.
I score every call's complexity before routing:
score = (prompt_length / 900) + (tool_count * 0.3) + (reasoning_depth * 0.5)
- Score < 1.2 routes to a fast, cheap model (8B-70B class)
- Score 1.2-2.5 routes to a mid-tier model
- Score > 2.5 routes to frontier
The scoring isn't magic. It's a heuristic I tuned over a few weeks by comparing outputs across model tiers on my actual workloads. About 60% of my agent calls score below 1.2. Those calls cost roughly 1/20th of what they'd cost on a frontier model, with equivalent output quality for their task type.
I validate this continuously. Every routed call gets a quality flag in the span log. If a cheap-routed call produces output that fails downstream (a tool call that errors, a classification that gets corrected in a later step), I bump that call pattern's complexity score. The router self-corrects over time.
What I'd Change
If I were rebuilding this from scratch, I'd add one more layer: speculative execution. For agent steps where I can predict the likely output (classification with a small label set, boolean decisions), I'd run the cheap model speculatively and only fall back to the expensive model if confidence is below a threshold. That would push savings past 70%.
I'd also invest more in the semantic cache tier. The vector similarity approach works, but a learned similarity function trained on my specific prompt distribution would improve hit rates significantly.
The 60% number isn't theoretical. It's the delta between my January and April bills running the same agent workloads at the same volume. The agents didn't get dumber. The infrastructure got smarter.
Measure every span. Cache aggressively but carefully. Compress what the model doesn't need. Route by complexity, not by habit. That's the whole thing.
Top comments (1)
"The bill is a tax on laziness, not intelligence" is the right frame, and per-span measurement is the most underrated line in here — most teams watch total tokens per request, which is a bar tab, not a diagnosis. Our single biggest win was identical to yours: a sub-agent receiving the full parent context on every call, fixed in about ten minutes once the spans made it visible.
Where I'd add a caution is Tier 2, the semantic near-match cache. A 0.97 cosine threshold is tight, but semantic caching on generative calls still makes me nervous — two prompts that are 0.97 similar can want genuinely different answers, and a near-match hit returns a confidently wrong cached response with no error. How do you guard that — short TTLs, validating the hit against the actual request, or restricting semantic caching to read-only/lookup-shaped calls? Separately: does your Tier 1 exact-match cache stack with provider-side prefix caching, or does one make the other redundant? Curious whether the 60% already bakes in prompt-caching or if that's a fifth lever still on the table.