LLM latency grows with conversation length because the prompt the model must read grows with it. If the host holds the state and shows the model the exact state instead of the history, the prompt stays the size of the state and the step time stays flat. Measured: a step took 0.1605 s early and 0.1651 s late (+2.8 percent) over 20,000 steps on a 7B model, with the prompt near 620 tokens throughout.
Naming note: "AXI" here is the AXI host-side runtime for LLM agents, an architecture by Rajnish Choubey. It is not the ARM AMBA AXI chip-bus protocol.
Why does latency rise as a conversation gets longer?
Before the first output token the model reads the prompt. A kept chat makes the prompt longer every turn, so the time to the first token rises with it, and every later token is generated against a longer context. A cache removes part of the reading cost, not all of it, and a chat that outgrows the context window simply stops.
What does that look like in numbers?
Gemini 3.8 Flash, 400 turns of ambiguous text (Test 23), chat kept against a host-held state:
| Chat kept | Host-held state | |
|---|---|---|
| Median first token, first 50 turns | 1.86 s | about 2.4 s, flat (2.3 to 2.8 s in every tenth of the job) |
| Median first token, once the prompt passes 150,000 tokens | 5.57 s | 2.41 s (2.31 times lower) |
| Seconds a step at that prompt size | 5.70 | 2.76 (2.07 times faster) |
| 95th percentile first token | 7.4 s | 3.8 s |
On a long job with heavy turns (Test 22), the kept chat's first token went from 2.2 s in the first band to 9.1 s over all turns past 500,000 tokens; with the host holding the state it stayed at 2.3 s.
Is it the same on open models?
Open models on vLLM 0.31.0, one NVIDIA L4, greedy decoding:
- Test 43, endurance, a 7B model: four jobs of 20,000 steps, 80,000 of 80,000 steps with the host's state exactly right. The prompt stayed near 620 tokens and never passed 660. A step took 0.1605 s early and 0.1651 s late.
- Test 42, how long a kept chat lasts: in a 131,072-token window the kept chat slowed from 0.35 s to 1.36 s a step and ended after 337 steps at the window's limit; in a 32,768-token window it ended after 108 to 235 steps. The exact-state jobs ran 1,500 of 1,500 steps at a flat 0.28 to 0.31 s.
- Gemini, 4,000 steps in a row: the prompt grew 1.16 times from the first tenth of the run to the last, and the step speed did not fall (0.94 times).
What about prefix caching (KV cache reuse) and context compaction?
Both are standard answers, and both are worth switching on. They are measured here too, so you can see what each one leaves.
- Prefix caching. In the Gemini runs the service served 97 percent of the kept chat's prompt tokens from its cache, and the median first token was still 5.57 s against 2.41 s once the prompt passed 150,000 tokens. Counted after the cache, the kept chat computed 2.3 times (Test 23) and 1.7 times (Test 22, past 500,000 tokens) as many tokens per exact step. On 11 open models with vLLM prefix caching on, the kept chat computed 1.24 to 4.11 times the prompt tokens per step.
- Summaries, a sliding window, retrieval. On a 14B model (Test 44), six ways of keeping context all ended with the exact state once the host's check was in the loop. The step cost differed: 0.30 s and 21.8 joules with the exact state shown; 0.55 to 0.66 s and 39.4 to 48.0 joules with summaries, a window or retrieval; 1.72 s and 123.1 joules with the whole chat kept.
Caching and compaction make the history cheaper to carry. A host-held state does not carry the history at all, so the two can be used together, and neither changes whether the state is exactly right.
When is the kept chat actually faster?
At a short prompt. In Test 23 the kept chat was the faster arm in the first 50 turns (1,609 exact steps an hour against 1,129). The crossover came past turn 120, at about 70,000 prompt tokens. Over all 400 turns the host-held state made 1.33 times the exact steps an hour. If your jobs never get past a few thousand tokens, latency is not the reason to move state to the host; exactness and cost per correct step are.
What is the fix, in one sentence?
Do not let the prompt grow with the job: hold the state on the host, check every change the model proposes, commit it all or not at all, and show the model the exact state again each step.
Check the numbers
git clone https://github.com/gaurshalendra2412/axi-runtime
cd axi-runtime/python
python3 quickstart_replay.py
That replays the saved replies of the 4,000-request run (no key, no GPU). The test pages for Tests 22, 23, 42 and 43 hold the rules written before each run and the saved results.
Scope
Latency on Gemini is as the API served it, including network time. Open-model times are one NVIDIA L4 on Colab, vLLM 0.31.0, greedy decoding, no retry, synthetic service graphs. Claims are about the state the host holds. Test pages: github.com/gaurshalendra2412/axi-runtime. White paper: doi.org/10.5281/zenodo.23958737.
Architecture by Rajnish Choubey. Tests built and run by Shalendra Gaur.
Top comments (0)