DEV Community

Shalendra gaur
Shalendra gaur

Posted on Fully Autonomous

A chat that keeps its whole history: 949,591-token prompts, calls 3.5x slower, state wrong from turn 287

This article was created with the help of AI. The AXI architecture is by Rajnish Choubey. The tests were built and run by Shalendra Gaur.

Long context windows make it tempting to keep the whole history and let the model sort it out. I ran that on a real job and measured what it did to speed and to the state.

The test

The same job and the same model (Gemini 3.8 Flash), run on 22 October. One arm kept the whole chat. The other, AXI, rebuilt a short prompt from the host-held state at every step. The plan was written and hash-locked before the run. Every request came with about 40 lines of routine operations log, so every turn was heavy.

What happened

Chat that keeps its history AXI (state held on the host)
Steps run 457 (it stopped at its context limit) 1,500
Mean prompt, tokens a step 476,835 2,824
Largest prompt 949,591 3,112
Held state exactly right 286 of 457 turns 1,500 of 1,500 steps
Time of a call 10.66 s on average over the 217 calls at 500,000 tokens or more 3.07 s a step, over all steps
Median time to first token 9.13 s (same 217 calls) 2.27 s (all steps)
  • The slowest call of the kept chat took 55.6 seconds.
  • 99.1% of its prompt tokens were served from the cache, so caching was working.
  • Its replies were not exactly right 15 times, the first at turn 223. Applied as each reply came, the held state was exactly right on 286 turns and wrong on every turn from 287 on, with a corrupted state on 146 turns.
  • With AXI the prompt grew 8% from the first tenth of the job to the last (2,499 to 2,698 tokens).

Why

The chat re-reads everything at every turn, so each turn is slower than the last, and a slip stays in the history for good. AXI does not re-read the history. It rebuilds the prompt from the exact state, so the prompt does not grow with the job.

The seconds are the seconds of the calls, not the waits that the key's per-minute limit added. The time to first token includes the network from the notebook's machine.

What this does not show

One model, one job, one sampled run. The kept chat ran with the service's own caching on, and it was never shortened. I have not compared other ways of managing a chat in a result you can replay. The prompt averages are over the 456 turns after the first.

For the cost side of the same test, see my earlier post Your LLM agent re-reads its history every step. To recompute the figures without a key, see How to make an LLM agent test replayable in 3 seconds.

AXI is a layer on the host side by Rajnish Choubey: the model is unchanged and the claims are about the state the host holds. The test page is docs/GEMINI_LONG_TEST.md in the repository: https://github.com/gaurshalendra2412/axi-runtime

White paper, "AXI: The Cure for the Cost Crisis in AI" (Rajnish Choubey and Shalendra Gaur): https://doi.org/10.5281/zenodo.23958737

Top comments (0)