The Agent Loop

Posted on Sep 25

Your agent's cost problem isn't the model. It's the loop.

#ai #llm #aiagents #productivity

Drafted with AI help, human-reviewed by The Agent Loop.

Short version: Nobody's price per token changed this week, and your bill still went up. Agents don't pay for "a question." They pay for the whole conversation, every turn, plus every retry, plus every retry of the retry. I pulled the trajectory numbers from two papers, one vendor breakdown, and two public incidents. The fix is boring: send less, cache what you keep sending, and put a ceiling on the loop.

For skimmers

A naive agent loop is roughly quadratic: each call re-bills the full history
On OpenRouter, Claude 4 Sonnet's 100B tokens/day were 99% input tokens, only ~1% output
Trimming dead trajectory content cut input tokens 39.9–59.7% and total cost 21.1–35.9% at equal performance
Caching: Anthropic cache reads 90% off (95% on Opus 5.5, writes 1.25x), OpenAI 50% off automatically, Google 75% off
Runaway loops are the tail risk: one cron job re-fed a 750k-token session every 5 minutes until the balance died
Anthropic's own move on Sep 22 (Opus 5.5): cache reads $0.50 -> $0.20/M, a 60% cut, because cache reads are most of an agent bill

The rate is not your variable

Token prices are public and stable. Demand is not: OpenRouter's weekly token volume went from 0.4 trillion (Dec 2024) to 27.0 trillion (Mar 2026), about 68x in 15 months, growth the paper attributes to the spread of agent platforms and applications (Token Economics for LLM Agents).

You don't control that curve. I went looking for a price change to blame and never found one; what moves is the request you send.

Why a loop compounds

Chat sends one message. An agent sends the message plus everything that already happened, because the API keeps no state between calls.

turn 1   [ sys + user ]                          -> billed: 1x
turn 2   [ sys + user + a1 + t1 ]                -> billed: 1x + 3
turn 3   [ sys + user + a1 + t1 + a2 + t2 ]      -> billed: 1x + 3 + 5
turn N   [ ... everything ... ]                  -> 1 + 3 + 5 + ... + (2N-1) = N²

(Numbered blocks are history chunks, not token counts. The shape is what matters.)

A worked breakdown of a 10-step file-reading agent put it at 472,500 input tokens versus 9,000 for a single pass, about 43x (vendor guide, Augment Code — treat as illustrative, not an audit).

I checked my own numbers while writing this. One agent session logged 3,518,203 input tokens against 271,350 output tokens, a 13:1 ratio in favour of remembering. The model wrote one word for every thirteen it was handed back.

The scale check: on OpenRouter, Claude 4 Sonnet reportedly hit 100 billion tokens per day in September 2025, and 99% of those were input tokens accumulated in trajectories, with about 1% being output the model actually generated (arXiv 2509.23586, citing OpenRouter). You're mostly paying to remember, not to think.

That same paper's fix, called AgentDiet, strips useless, redundant, and expired content out of the trajectory at inference time:

What changed	Result
Input tokens	39.9% to 59.7% fewer
Total compute cost	21.1% to 35.9% cheaper
Agent performance	within -1.0% to +2.0%

Same model, same task, same answer quality. The savings came from sending less.

The second multiplier: retries

Reliability multiplies volume. A 10-step sequential chain at 95% success per step finishes about 60% of the time (0.95^10), and the same guide estimates retries add roughly 40% more tokens on top. Failure does not just cost latency. It re-bills the attempt.

Then there is the tail:

$23.82 overnight: a cron job fired every 5 minutes and re-fed a session file that had bloated to 10.4 MB (about 750,000 tokens) until credits ran out (postmortem)
$6,000 in one command: an r/ClaudeAI report of a single runaway invocation. Anecdote, not a benchmark, and still the shape of failure nobody instruments for.

Nothing crashed, and CPU stayed flat the whole time. The invoice was the only alarm.

Anthropic just cut cache reads by 60% (Sep 22)

Anthropic shipped Opus 5.5 on September 22, 2026, three days ago, and its pricing page argues this section harder than I did.

Per 1M tokens	Opus 5	Opus 5.5
Input	$5	$4 (-20%)
Output	$25	$20 (-20%)
Cache writes	$6.25	$5
Cache reads	$0.50	$0.20 (-60%)

Anthropic's wording is the tell: cache reads "make up the majority of agentic and coding work costs," so they discounted the line item that agents actually burn. Total cost per task, they claim, drops 40% on typical workloads, and output comes 30%+ faster. One vendor claim, one vendor's task mix, but the direction is not subtle: the price of re-reading your own history is the number everyone is attacking.

Also worth knowing before you plan a migration: Sonnet 5.5 and Haiku 5.5 are announced for "the coming weeks," with no date, price, or benchmarks. And Opus 5.5 ships safety classifiers that can transparently reroute a request to an older model on dual-use tasks (The New Stack, Sep 22), which is a fun wrinkle if you assumed every call in your loop hits the same weights.

What to do before you shop for a cheaper model

1. Cache the prefix you keep sending. I skipped this on my first agent because the docs read like an optimization, not a discount. Anthropic prices cache reads at 0.10x input (90% off; 0.05x on Opus 5.5) with writes at 1.25x; OpenAI's caching is automatic at 50% off; Google offers 75% off for long prompts (Anthropic prompt caching, caching comparison). If your system prompt and tool definitions are 8,000 tokens and your user message is 50, the discount applies to 99% of the input.

2. Window or summarize the trajectory. Drop tool output nobody read twice, truncate the 400-line command dump after you extracted the line you needed, and summarize old turns instead of replaying them.

3. Cap the loop. Max steps, max retries per tool, max tokens per run, and a kill switch that fires on token count rather than on errors. Check it in the proxy layer: once the call is made, the tokens are already billed.

4. Tag cost per call. This is the step I keep having to argue for, because it looks like overhead until the night you need it. Attach token counts and estimated spend to each span. Provider invoices say that spend rose, not which tool, retry pattern, or runaway run rose (Braintrust cost guide, vendor). A retry storm shows up as a successful response in your app logs and as nothing at all in your traces.

5. Then, and only then, change models. Routing classification and extraction to a smaller model is real money. It's a rounding error next to a trajectory that re-sends 10x what it needs.

I didn't benchmark AgentDiet myself; those reduction figures are the authors' reported ranges, and your mileage will depend on how messy your trajectory already is.

Bottom line: the model charges for tokens. Your architecture decides how many. Shrink the loop first, the model second.

FAQ

Why is my AI agent so expensive?

Usually volume, not rate. Long trajectories re-sent every turn, retries, and uncached prefixes dominate the bill. Check input vs output token split before changing vendors.

What is an agent trajectory?

The accumulated conversation: system prompt, tool definitions, every prior action, and every tool result, replayed on each call because the API keeps no state.

Does prompt caching actually save money?

On repeated prefixes, yes, and it also cuts latency. Watch the write premium (Anthropic charges 1.25x on writes) and the TTL, which is 5 minutes by default. A low hit rate can erase the benefit.

Did Opus 5.5's price cut fix agent costs?

It moved the right number: cache reads fell 60% ($0.50 to $0.20 per million), and cache reads are what a replay-heavy loop buys all day. It does not fix a loop that resends history it never needed.

How do I cap agent token spend?

Per-run token budget, step ceiling, retry ceiling, and a kill switch that trips on token count. Enforce at the proxy, not inside the model loop.

Sources

Related on The Agent Loop

Over to you: What is the largest single agent run you have ever paid for, and did you find out why? Reply below, I read every comment.

DEV Community