The hidden multiplier nobody budgets for
When we moved from single-turn chatbots to agentic workflows in early 2026, the first thing that broke wasn't the code — it was the budget spreadsheet.
A simple chat completion costs one API call. An agent that plans, selects tools, executes them, evaluates the results, and synthesizes a final answer? That same user request now triggers 5 to 20 LLM calls. Sometimes more.
I ran an experiment last month with a production agent doing research tasks — web search, summarization, multi-hop reasoning. A single user prompt averaged 14 LLM round-trips across GPT-5 and Claude 4.6 Opus. At GPT-5's input/output pricing, that one "simple question" cost $0.47. Multiply by 1,000 daily active users and you're looking at $470/day you never planned for.
Where the cost actually hides
After instrumenting our gateway logs for two weeks, here's what I found:
1. Planning overhead
Every agent loop starts with a planning step. The model reads the full conversation history, decides what tool to call, and outputs a structured action. This step alone can consume 800–2,000 tokens of input per iteration — and it happens on every single loop.
With Claude 4.6 Opus at $15/M input tokens, a 5-iteration agent spends $0.06 just on planning. That's before it does anything useful.
2. Context window bloat
Agents accumulate context. By iteration 4, the prompt includes the original question, all prior tool outputs, all prior reasoning traces, and the full system prompt. I measured prompts growing from 1,200 tokens at iteration 1 to 18,000+ tokens by iteration 6.
This is the insidious part: each iteration's cost is superlinear because the context grows with every step.
3. Tool call redundancy
Agents are surprisingly bad at knowing when to stop. In our logs, 23% of agent runs made at least one redundant tool call — re-searching something it already found, or re-reading a document it already summarized. Each redundant call is a full LLM round-trip with the bloated context.
4. Fallback cascade failures
When a primary model returns a 429 rate limit or 503 timeout, the agent retries — often with a different model. But the retry replays the entire context from scratch. One rate limit event can triple the cost of a single agent turn.
5. Token amplification in multi-model setups
When your agent routes between GPT-5, Claude 4.6, and DeepSeek V3 for different subtasks (common in 2026 production setups), each model has different tokenizers. The same prompt tokenizes differently across models — I measured up to 15% variance in token counts for identical text between OpenAI and Anthropic tokenizers. Your cost estimates based on one tokenizer are wrong for the others.
What actually works for cost control
After burning through more budget than I'd like to admit, here's what we implemented:
Gateway-level token accounting
Stop relying on application-level logging to track costs. Application code sees the request before it's sent; the gateway sees the actual token counts in the response. We moved all cost tracking to the API gateway layer, which gives us:
- Per-request input/output token counts (actual, not estimated)
- Per-model cost breakdown
- Per-user cost attribution
- Real-time spend alerts
Iteration budgets with hard caps
We enforce a maximum of 8 iterations per agent run at the gateway level, not the application level. Application-level caps get bypassed when the agent framework has retry logic. Gateway-level caps are absolute.
Context compression checkpoints
Every 3 iterations, the agent must summarize its context into a compressed form before continuing. This cuts the context window growth from superlinear to roughly linear. We implemented this as a gateway middleware that intercepts the agent's requests and injects a compression instruction when the context exceeds a token threshold.
Per-user daily spend limits
The gateway tracks cumulative spend per API key per day. When a user hits their limit, subsequent requests get a clear 429 with a message explaining the cap. This prevents the "one rogue agent run costs $50" scenario.
Model routing based on task complexity
Not every agent step needs Claude 4.6 Opus. We route simple tool-selection steps to cheaper models (DeepSeek V3 at $0.27/M input tokens) and reserve Opus for complex reasoning. The gateway makes this routing decision based on the request characteristics, not application code.
The architecture that scales
Here's the gateway configuration pattern that's worked for us in production:
User Request
→ Gateway (token budget check, model routing)
→ Agent Planning Step (cheaper model)
→ Tool Selection (cheaper model)
→ Tool Execution (no LLM call)
→ Result Evaluation (flagship model)
→ Synthesis (flagship model)
→ Gateway (token accounting, cost attribution)
→ Response to User
The gateway sits at both ends of the pipeline. It controls what goes in (budget checks, model selection) and measures what comes out (actual token counts, cost attribution).
The real lesson
The agent cost problem isn't a model pricing problem — it's an observability problem. You can't optimize what you can't measure. And application-level instrumentation consistently undercounts because it misses retries, context bloat, and tokenizer variance.
If you're running agents in production in 2026, your first investment should be gateway-level token accounting. Not a better model, not a cheaper provider — just visibility into where your tokens actually go.
The teams that figure this out early will be the ones who can afford to scale their agent deployments. The rest will hit a budget wall and wonder what happened.
What patterns are you using to control agent costs in production? I'm curious whether others are seeing the same 5–20x multiplier, or if different architectures fare better.
Top comments (0)