DEV Community

Shalendra gaur
Shalendra gaur

Posted on AI-assisted

How many LLM agent sessions can one GPU serve? Measured: 8 to 256 sessions on one NVIDIA L4, every step exact

Concurrent LLM agent sessions per GPU depend on how much prompt each session sends and how much KV cache it holds. If the host holds the state and shows the model only the exact state, each session's prompt stays the size of the state and does not grow with the steps, so many sessions fit in one cache. Measured on one NVIDIA L4 with a 7B model: 256 sessions at once, 25,557 of 25,557 steps with the host's state exactly right, 105,230 exact steps per GPU hour, 2.458 joules per exact step. The card was full at about 128 sessions.

Naming note: "AXI" here is the AXI host-side runtime for LLM agents, an architecture by Rajnish Choubey. It is not the ARM AMBA AXI chip-bus protocol.

What limits the number of agent sessions on a GPU?

For multi-turn agents, GPU memory for the KV cache is usually the first limit, and prefill work is the second. A session that keeps its whole chat holds a longer cache and sends a longer prompt every step. A session whose host holds the state sends a prompt the size of the state, so the cache per session is small and does not grow with the number of steps. In the 20,000-step runs on the same 7B model (Test 43) the prompt stayed near 620 tokens a step and never passed 660. In the session test below the cache was set to 160,000 tokens so that all 256 sessions' prefixes could be held, and the cache share stayed near 71 percent from 8 to 256 sessions, so they were.

What was measured?

Test 38: Qwen2.5-7B-Instruct (AWQ 4-bit) on vLLM 0.31.0, one NVIDIA L4 on Colab, prefix caching on, greedy decoding, no retry, 100 steps a session, all sessions advancing in lock-step rounds.

Sessions at once Steps Steps with the host's state exactly right Exact steps per GPU hour Median round (every session gets one step) Cache share of prompt tokens Joules per exact step
8 800 800 (100.0 %) 51,134 0.57 s 69.7 % 5.035
32 3,200 3,200 (100.0 %) 88,635 1.28 s 71.1 % 2.923
128 12,800 12,800 (100.0 %) 105,043 4.27 s 71.5 % 2.464
256 25,557 25,557 (100.0 %) 105,230 8.49 s 71.5 % 2.458

Zero corrupted steps in every cell. Computed prompt tokens per step were 94 to 105 at every size. (The cell of 256 holds 25,557 steps, not 25,600, because one job's script had 57 steps.)

What does that say about sessions per GPU?

  • Exactness held at every size. The host's state was exactly right on every step up to 256 sessions on one card.
  • Throughput rose, then stopped. Exact steps per hour went up 1.73 times from 8 to 32 sessions and 1.19 times from 32 to 128, and did not rise from 128 to 256 (1.002 times). On this card with this model and engine the chip was full at about 128 sessions; past that every session waits longer without the chip doing more. The test also read NO on one of its own predictions (at least 1.5 times from 32 to 256: measured 1.19 times), and the page prints it as NO.
  • Energy per exact step fell from 5.035 joules at 8 sessions to 2.458 joules at 256, as the card filled.
  • The cache held every session's prefix. The cache share stayed near 71 percent from 8 to 256 sessions.

What does a step cost in energy against other ways of remembering?

On a 14B model (Test 44), six ways of keeping context all ended with the exact state once the host's check was in the loop. A step took 0.30 s and 21.8 joules with the exact state shown; 0.55 to 0.66 s and 39.4 to 48.0 joules with summaries, a sliding window or retrieval; 1.72 s and 123.1 joules with the whole chat kept.

What is not shown?

Arrivals at random times (these rounds are lock-step), other cards, models bigger or smaller than 7B (a ladder of 11 open models, 1.5B to 32B, is in the repository), and sessions longer than 100 steps in this test (separate 20,000-step jobs are reported in Test 43). The flat line from 128 to 256 is this card's limit with this engine and setting, not a statement about the framework.

Check it

git clone https://github.com/gaurshalendra2412/axi-runtime
cd axi-runtime/python
python3 quickstart_replay.py
Enter fullscreen mode Exit fullscreen mode

That replays the saved replies of the 4,000-request Gemini run (no key, no GPU). The Test 38 page, with its predictions written before the run and its result below a line, is docs/SESSIONS_PER_CHIP_TEST.md in the repository: github.com/gaurshalendra2412/axi-runtime. White paper: doi.org/10.5281/zenodo.23958737. Ten open-model tests: doi.org/10.5281/zenodo.24319561.

Architecture by Rajnish Choubey. Tests built and run by Shalendra Gaur.

Top comments (0)