DEV Community

Shalendra gaur
Shalendra gaur

Posted on Edited on Fully Autonomous

Your LLM agent re-reads its history every step. We measured what that costs

This article was created with the help of AI. The AXI architecture is by Rajnish Choubey. The tests were built and run by Shalendra Gaur.

At turn 223, a Gemini 3.8 Flash chat started making mistakes. By turn 287 it was wrong. The same model, on the same job, made none in 1,500 steps when the host held the state.

The difference was not the model. It was what the model was handed at each step.

The cost nobody prices in

Many LLM agents work like a chat: every step, the whole history goes back into the prompt. Step k sends about k units of history, so a job of N steps sends about N²/2 units in total. Double the job and the cost of the history quadruples. Faster chips make each token cheaper by a constant factor. They don't change that curve.

We measured it. Over its 457 turns (it stopped there, at its context limit) the kept chat's mean prompt was 476,835 tokens, peaking at 949,591. The alternative held the exact state of the job on the host and sent a fresh prompt of 2,824 tokens at every step, for 1,500 steps. That is 169 times smaller on average.

What that did

  • Exactness: 1,500 of 1,500 steps exact with the host-held state. The kept chat gave its first inexact reply at turn 223, and from turn 287 to its last turn (457) the state it held was wrong.
  • Latency: at a prompt of 500k tokens or more, a call took 10.66 s on average against 3.07 s for an AXI step, and the median time to first token was 9.13 s against 2.27 s.
  • Throughput: 1,280 exact steps per hour against 488 (2.6x), over the first 457 steps of each, counting the seconds of the calls only.
  • Cost per correct step, in a separate 400-request test at the prices typed in: $0.0012 with AXI against $0.0052 when replies were applied as written (4.5x), and 9.3x on a second seed. This is arithmetic on the saved files, added after the results.

The error problem is worse than the cost problem

In the first 400 requests of a separate 4,000-request run, 393 of 400 replies were right, 98%. Applied exactly as the model wrote them, only 85 of 400 steps ended in the correct state, 21%. A small error rate per reply becomes a large error rate per job, because each slip is carried into everything after it.

In a fuzzy-input test, one slip applied as written at turn 191 stayed in the held list for 140 turns, although 398 of 400 replies (99.5%) were exact.

On a small open model (Qwen2.5-7B, 30 requests in a row), applying the replies as written corrupted the state in 12 of 12 episodes. The host's gate corrupted it in 0 of 360 steps.

How it works

The architecture, AXI, is by Rajnish Choubey. The exact state lives on the host. At each step the model gets a short prompt built from it, and proposes the next move. A deterministic gate checks the proposal in about 2 microseconds (measured in Python), a rule-based repair fixes a slip, and the host updates the state.

There is no training and no change to the model, and it works from the first step, on short jobs and long ones. The cost grows linearly with steps, not with their square.

How we tested

Every test page was locked with a SHA-256 hash before the run, with probabilities attached to each prediction. In the 4,000-request test, nine headline rules were written first with a probability on each. All nine held, and one other rule we had set failed; we report it as failed. A test in the repo recomputes every figure from the saved results.

Limits

These are specific jobs on Gemini 3.8 Flash and Qwen, one sampled run each. On a TPU v6e-1, memory headroom allowed 927 sessions with host-held state against 5 with a kept chat, but that is for a 0.5B cache shape and is not end-to-end throughput.

Tests built and run by Shalendra Gaur, published with Rajnish Choubey's permission.

Try it yourself

Top comments (5)

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@shalendra_gaur_ecd0567a49 On the benchmark, the check runs once at the end of each run: a verifier scores the outcome against ground truth the agent never sees, and no LLM judges it. Per-turn correctness is the part we haven't measured there, so we don't claim it. Method: pi-infected.github.io/token-harnes...

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

The first wrong answers arrive long before the run ends, so a run can be expensive and wrong at once. A quality check per turn next to the token count is what shows whether a fix helped. (I built it: Tokenade, MIT, trims what the agent keeps between turns.)

Collapse
 
shalendra_gaur_ecd0567a49 profile image
Shalendra gaur •

Thanks, Paul. Agreed: a quality check per turn next to the token count is the only way to tell whether a fix helped, and that's how we measured it (exactness per step against a program-computed true state, alongside prompt size). Architecture is by Rajnish Choubey. I'm curious how Tokenade decides what to trim: do you measure per-turn correctness against a ground truth, or does the check come from the model's own output?

Collapse
 
kolmopdf profile image
KolmoPDF •

That quadratic token growth from replaying full conversation history matches what shows up in multi-step document workflows too.

When an agent processes a 100-page specification or regulatory filing across dozens of tool calls, stuffing every prior extracted section and intermediate verification output back into the prompt pushes context past 400k tokens fast. Once the window fills with stale tool payloads, accuracy degrades long before the hard context limit hits, much like your Gemini 3.8 Flash run slipping around turn 223.

Keeping canonical document state on the host and passing only the active schema slice plus a deterministic gate on each step keeps the prompt flat and linear. It also makes failures reproducible, because you can inspect the exact host state transition that broke rather than digging through half a million tokens of chat history.

Collapse
 
shalendra_gaur_ecd0567a49 profile image
Shalendra gaur •

Thanks, that's a good example, and the reproducibility point is exactly why we keep the state on the host: you can replay the transition that broke instead of reading a long history. How do you define the canonical state for a document workflow, as a schema with typed fields or something looser? If you ever try the Colab notebook in the repo, I'd like to hear how it compares on your jobs.