A kept chat grows every step, so it runs into the context window. Measured on a 7B model: with the chat kept, the job ended after 108 to 235 steps in a 32,768-token window and after 337 steps in a 131,072-token window, where a step slowed from 0.35 s to 1.36 s. With the host holding the state, the same kind of job ran 1,500 of 1,500 steps in every run at a flat 0.28 to 0.31 s, and the prompt never passed 660 tokens.
Naming note: "AXI" here is the AXI host-side runtime for LLM agents, an architecture by Rajnish Choubey. It is not the ARM AMBA AXI chip-bus protocol.
What happens to an agent when the context window fills?
The prompt is the whole chat so far, so it gets one step longer every step. The model reads more before it answers, each step slows, and at the window's limit the job simply stops. A cache removes part of the reading cost (98 to 99 percent of the kept chat's prompt tokens came from the cache here) but not the growth.
What was measured?
Test 42: Qwen2.5-7B-Instruct (4-bit), vLLM 0.31.0, one NVIDIA L4, prefix caching on, greedy decoding. The kept chat shows the whole current state in every message (the heaviest honest version) and stops when its next prompt would pass 32,000 tokens (32,768 window) or 130,000 (a 131,072 window using YaRN scaling). The exact-state arm shows the host's state fresh each step.
| Run | Arm | Steps run | Steps with the state exact | Ended by | Seconds a step | Longest prompt (tokens) |
|---|---|---|---|---|---|---|
| 32k window, job 1 | kept chat | 108 | 108 (100 %) | context limit | 0.4065 | 31,776 |
| 32k window, job 2 | kept chat | 208 | 208 (100 %) | context limit | 0.3449 | 31,885 |
| 32k window, job 3 | kept chat | 228 | 228 (100 %) | context limit | 0.3282 | 31,968 |
| 32k window, job 4 | kept chat | 235 | 235 (100 %) | context limit | 0.3403 | 31,910 |
| 131k window | kept chat | 337 | 337 (100 %) | context limit | 0.8104 | 129,759 |
| 32k window | exact state | 1,500 | 1,500 (100 %) | all steps run | 0.2998 | 660 |
| 32k window, four jobs side by side | exact state | 6,000 | 6,000 (100 %) | all steps run | 0.116 per step | 658 |
| 131k window | exact state | 1,500 | 1,500 (100 %) | all steps run | 0.2985 | 660 |
Corrupted steps: 0 in every cell. The kept chat was exact on every step it ran; what it lost was time, tokens and its end.
How much slower does a kept chat get?
In the 131k run, seconds a step by blocks of 50 steps were 0.35, 0.43, 0.66, 0.84, 1.01, 1.17 and 1.36, while the prompt grew from 28,807 tokens at step 100 to 71,227 at step 200, 113,855 at step 300 and 129,759 at its last step. Over the same steps the exact-state arm computed fewer tokens and took less time:
| Pair | Computed prompt tokens a step | Seconds a step |
|---|---|---|
| 32k kept chat vs exact state, steps 1 to 108 | 290.7 vs 163.5 (1.78 times) | 0.4065 vs 0.2692 (1.51 times) |
| 131k kept chat vs exact state, 337 steps | 382.5 vs 258.1 (1.48 times) | 0.8104 vs 0.2932 (2.76 times) |
Does the host-held state also fill a window?
It follows the size of the state, not the length of the history. In the four-job run the exact-state prompt rose from 317.9 tokens over steps 1 to 100 to 526.4 over steps 1,401 to 1,500 and never passed 660 tokens, because the held state grew a little. That reading is the author's and was not tested by a separate run.
A large state is a different test (Test 40, a CPU run on the Python reference model, characters counted with 3 characters per token assumed). At 20,001 items the whole state is 378,399 characters on an even graph, about 124,000 to 126,000 tokens, which nearly fills a 128,000-token window. The focused view the model is shown stayed at a median of 207 to 323 characters from 20 to 20,000 items; a request that names the biggest hub shows 355 items (5,553 characters, 1.8 percent of the state).
One reading went against the framework: the host's own check is not flat in the size of the state. It took 3.9 microseconds at 201 items and 257 at 20,001 (66 times). That is 0.3 ms at about 6,700 services, small beside a model's reply time, and the test page gives the likely cause. Not shown: that a model is as exact when it reads the focused view as when it reads the whole state.
What is not shown?
One card, one 7B quantised model, one engine and version, synthetic service graphs, four kept chats on four jobs of one family. The kept chat's stop is set by the window and the test's stop rule: it is the arithmetic of a growing chat, not a saving of the exact-state arm. The saving this test can claim is in computed tokens and seconds over the same steps.
Check it
The test pages are docs/KEPT_CHAT_LONG_TEST.md (Test 42) and docs/STATE_SIZE_SCALING_TEST.md (Test 40) in the repository. Test 40 runs on any machine with Python in about 3 seconds:
git clone https://github.com/gaurshalendra2412/axi-runtime
cd axi-runtime/python
mkdir -p /tmp/axi_out
python3 -m axi.experiments.state_size_scaling /tmp/axi_out
Repository: github.com/gaurshalendra2412/axi-runtime. White paper: doi.org/10.5281/zenodo.23958737. Ten open-model tests: doi.org/10.5281/zenodo.24319561.
Architecture by Rajnish Choubey. Tests built and run by Shalendra Gaur.
Top comments (0)