If several people share one local model, how much does total throughput go up as you add streams, and what does each person give up? I tested a dense 27B and a mixture-of-experts 35B-A3B on a single 16 GB Radeon RX 6900 XT with 1 to 8 parallel streams and context from empty to 16k tokens. It is a 2020 RDNA2 card running through Vulkan, so treat these as conservative, entry-level numbers: a baseline that newer hardware and newer kernels should have room to improve on. This includes the part where my first results were wrong.
Raw data, scripts and the full write-up are on GitHub: local-llm-benchmarks.
The short version
Four users on the MoE model get about 240 tokens/s in total, around 60 each, on a 2020 card. Here is how total and per-stream speed change with the number of streams (empty context, tokens/s):
| streams | dense total | dense per stream | MoE total | MoE per stream |
|---|---|---|---|---|
| 1 | 31.9 | 31.9 | 108.2 | 108.2 |
| 2 | 53.9 | 26.9 | 171.9 | 86.0 |
| 4 | 62.9 | 15.7 | 239.8 | 59.9 |
| 8 | 68.4 | 8.5 | 264.6 | 33.1 |
- Run more streams at once and total throughput goes up, but nowhere near linearly. Dense topped out around 2.3x one stream; the MoE around 2.5x. Most of the gain arrives by 3 to 4 streams.
- The MoE model generates 3.4x faster than dense for one user and 3.8x faster for four. Eight users on the MoE each still get roughly the speed one user gets on the dense model (32 vs 31 tokens/s).
- Long context is cheap for generation but expensive for the first token, and there is a cliff when the GPU runs out of memory.
The setup
- GPU: one Radeon RX 6900 XT (16 GB, RDNA2, released in 2020), Windows 11, Ryzen 9 7900X, 64 GB DDR5.
-
Software: llama.cpp
llama-serverwith N slots, Vulkan backend (build 11624). -
Models: Qwen3.6-27B (dense) and Qwen3.6-35B-A3B (MoE, about 3B parameters active per token), both the same
IQ3_XXSquantization so they fit in VRAM (11.2 and 12.3 GB). - Method: for each combination of 1 to 8 simultaneous streams and 0 to 16k tokens of context per stream, I started a fresh server and measured how fast all streams generate while they are all running.
This is the card I have. It is also the kind of hardware plenty of people already own: lots of VRAM, not the newest silicon, and not what most inference work is tuned on first. If it handles a small group of users, a newer card should be at least as comfortable, though I have not tested that.
Parallel streams: more total, less each
For the dense model a single stream runs at 32 tokens/s. Two streams give 54 total (1.7x), four give 63, and the best I saw was 73 at 7 streams. Each stream gets slower as you add people (16 tokens/s each at four, 9 at eight), and past about four streams you are mostly trading speed per user for a small total gain.
The MoE starts at 108 tokens/s for one stream and reaches about 270 total at 7 streams. That is a better multiple than dense, and it means eight users each get about 32 tokens/s, which is already comfortably faster than reading speed.
Why is scaling so far from linear? In theory decoding is limited by reading the weights once per step, so a bigger batch should be almost free. I did not isolate the cause, but in a quick comparison an older ROCm/HIP build of llama.cpp scaled the dense model further on this same card (97 vs 68 tokens/s at 8 streams). That points at software kernels more than at the hardware ceiling, and it is why I read these Vulkan numbers as a conservative baseline. It is one comparison with a different llama.cpp version, so I would not draw more from it than that.
Context: cheap to generate with, costly to start
With everything in memory, depth costs only a little: a single stream loses about 10% (dense) or 12% (MoE) of its speed at 16k tokens of context compared with an empty one.
The painful part is prefill, reading the prompt. Dense processes about 326 tokens/s and the MoE about 1280 at 1k, and both slow down as prompts get longer. At 16k the dense model needs 2.2 minutes for one prompt and the MoE 42 seconds. Prefill speed barely changes with the number of streams, so four users with 4k prompts each means the last one waits 66 s (dense) or 18 s (MoE) for a first token.
The cliff
Some of my cells fell off a cliff (hollow dots in the chart): for example dense with two streams at 16k context, or the MoE with six streams at 4k each, where throughput roughly halves. I measured GPU memory during these runs: in every slow case the GPU started using shared (system) memory, and in every normal case it did not. Adding up weights, KV cache, and the per-slot recurrent state these models carry, the slow cases all sit just past roughly 13 GiB of allocations (13.1 for dense, 12.9 for the MoE) on a 16 GiB card where Windows already holds about 1.7 GiB.
That is a correlation with a plausible mechanism, not a proof, but the practical lesson is clear: if you plan for N users, budget VRAM for N slots (KV plus state), and leave headroom. A smaller quant, quantized KV cache or fewer slots would likely move the cliff. I did not test those.
I got it wrong the first time
My first sweep sized each server differently per row and reused servers across cells. It looked like parallelism behaved erratically (weird dips at three and six streams), and deep-context numbers were 20 to 40% too low. A repeat run of one cell on a smaller server gave 221 tokens/s versus 129 in the original sweep. I redid everything with a fresh, size-matched server per cell, repeated a few cells (they agreed within 1%), and those dips disappeared. If you benchmark parallel serving, hold the KV allocation constant, or match it to the cell, and re-run something to see your noise floor.
Caveats
One machine, one quant, two models. Streams all start at once with different prompts (real traffic is messier). 128 generated tokens per stream. Single run per cell. I only sanity-checked that both models answer simple prompts correctly; this is a speed test, not a quality comparison. I only tested this one card, so how much newer hardware improves on these numbers is my expectation, not a measurement.
Full numbers, method and raw data: WRITEUP.md. All scripts, CSVs and notes: github.com/amurocje-cpu/local-llm-benchmarks


Top comments (0)