This article was created with the help of AI. The AXI architecture is by Rajnish Choubey. The tests were built and run by Shalendra Gaur.
Long context windows make it tempting to keep the whole history and let the model sort it out. I ran that on a real job and measured what it did to speed and to the state.
The test
The same job and the same model (Gemini 3.8 Flash), run on 22 October. One arm kept the whole chat. The other, AXI, rebuilt a short prompt from the host-held state at every step. The plan was written and hash-locked before the run. Every request came with about 40 lines of routine operations log, so every turn was heavy.
What happened
| Chat that keeps its history | AXI (state held on the host) | |
|---|---|---|
| Steps run | 457 (it stopped at its context limit) | 1,500 |
| Mean prompt, tokens a step | 476,835 | 2,824 |
| Largest prompt | 949,591 | 3,112 |
| Held state exactly right | 286 of 457 turns | 1,500 of 1,500 steps |
| Time of a call | 10.66 s on average over the 217 calls at 500,000 tokens or more | 3.07 s a step, over all steps |
| Median time to first token | 9.13 s (same 217 calls) | 2.27 s (all steps) |
- The slowest call of the kept chat took 55.6 seconds.
- 99.1% of its prompt tokens were served from the cache, so caching was working.
- Its replies were not exactly right 15 times, the first at turn 223. Applied as each reply came, the held state was exactly right on 286 turns and wrong on every turn from 287 on, with a corrupted state on 146 turns.
- With AXI the prompt grew 8% from the first tenth of the job to the last (2,499 to 2,698 tokens).
Why
The chat re-reads everything at every turn, so each turn is slower than the last, and a slip stays in the history for good. AXI does not re-read the history. It rebuilds the prompt from the exact state, so the prompt does not grow with the job.
The seconds are the seconds of the calls, not the waits that the key's per-minute limit added. The time to first token includes the network from the notebook's machine.
What this does not show
One model, one job, one sampled run. The kept chat ran with the service's own caching on, and it was never shortened. I have not compared other ways of managing a chat in a result you can replay. The prompt averages are over the 456 turns after the first.
For the cost side of the same test, see my earlier post Your LLM agent re-reads its history every step. To recompute the figures without a key, see How to make an LLM agent test replayable in 3 seconds.
AXI is a layer on the host side by Rajnish Choubey: the model is unchanged and the claims are about the state the host holds. The test page is docs/GEMINI_LONG_TEST.md in the repository: https://github.com/gaurshalendra2412/axi-runtime
White paper, "AXI: The Cure for the Cost Crisis in AI" (Rajnish Choubey and Shalendra Gaur): https://doi.org/10.5281/zenodo.23958737
Top comments (0)