A while back I benchmarked llama.cpp on a 96GB GPU setup and found its concurrency behavior underwhelming.
That isn't llama.cpp's fault. From the start it was built to solve the capacity problem — "this model won't fit in one machine's VRAM" — not the throughput problem of many users hitting the server at once. If you're running a 70B or even 104B model at home for personal research or solo testing, "it runs at all" is the top priority. Under concurrency, when 10 users send requests at the same time, the engine should combine those 10 into a single Batch to keep the GPU busy. But the moment you batch them, the tensor volume moving between host A and host B blows up by orders of magnitude — from a few MB to tens or even hundreds of MB.
vLLM's whole architecture exists to solve high concurrency head-on. It slices the KV cache into fixed-size "pages," so VRAM doesn't need contiguous allocation: a request claims pages as it needs them and frees them the instant it finishes. On the communication side it uses NCCL directly, the CUDA-native library for GPUs. I have no NVLink here, but NCCL falls back gracefully and supports multiple links.
Personally: people say vLLM plus tensor parallelism over a single LAN cable at home has all sorts of problems, and I do have to throw away 32GB of VRAM. Even so, a model that fits in 32GB of memory is still a step below one that fits in 64GB of VRAM. And if several tasks can share one larger model, I expect the results to be better overall.
Anyway — let's measure it.
1. First test: a head-to-head with llama.cpp
The parameters were slightly off: llama.cpp's KV was q8_0 while vLLM defaulted to fp16, so llama.cpp had the advantage.
Single-stream decode, tok/s:
| Context | vLLM (TP4) | llama.cpp | Delta |
|---|---|---|---|
| short | 73.3±3.1 | 73.1±0.1 | intervals overlap = no difference |
| 5K | 75.0±0.1 | 70.5±0.2 | +6% |
| 32K | 71.4±0.5 | 63.1±0.1 | +13% |
| 52K | 69.3±0.1 | 58.0±0.1 | +20% |
| 100K | 63.7±0.7 | 48.2±0 | +32% |
Concurrency (same N=4, end-to-end tok/s):
| Context | vLLM | llama.cpp | Ratio |
|---|---|---|---|
| short×4 | 188.1 | 110.1 | 1.7× |
| 5K×4 | 31.0 | 29.9 | tie |
| 32K×4 | 5.6 | 1.9 | 2.9× |
| 52K×4 | 3.4 | 0.7 | 4.9× |
llama.cpp doesn't respond well to added concurrency: 32K single-stream is 5.4 and drops to 1.9 at N=4. The overlap at 100K×4 is −0.2, worse than running the requests one after another. Above 32K, 4-way crashes outright and the most you can run is 2.
The one thing llama.cpp wins is capacity: n_ctx 262144 vs 102400. With TP=4 the KV block count takes the minimum across all cards, so the 16G card sets the ceiling and roughly 15G of the 5090's 32G goes unused.
2. The actual test matrix
4 models × 3 links × 5 context sizes × concurrency pushed until the KV pool is blown, over 200 cells in total.
| key | Weights | Active | KV pool (tokens) | ctx ceiling |
|---|---|---|---|---|
| qwen36fp8 | 35B-A3B MoE / FP8 | ~3B | 635,712 | ≥100K |
| qwen3next | 80B-A3B MoE / AWQ | ~3B | 230,656 | ≥100K |
| llama33awq | 70B dense / AWQ | 70B | 92,416 | 100K rejected |
| qwen25awq | 72B dense / AWQ | 72B | 78,912 | rejected from 52K up |
| Link | Hardware | Link-layer measurement |
|---|---|---|
| 5g | RTL8126A / r8169 | 4.71 Gb/s one-way, NCCL 16KB median 0.146 ms |
| usb10g | RTL8159 / r8152 | 7.28 Gb/s one-way, NCCL 16KB median 0.231 ms |
| bond | the two above, balance-xor + layer3+4 | iperf3 multi-stream 11.9 Gb/s |
Parameters: GEN=128, seed=1234, temp=0, 2 rounds per cell. Concurrency is derived adaptively from concurrency × (context + GEN) ≤ KV pool, and once the pool is saturated I deliberately push 1.5× beyond it (marked 超KV in the tables). The goal is to find OOM, not to pick a production configuration.
3. Results
3.1 Links: more bandwidth changes nothing, more links helps a lot
At the link layer USB 10GbE beats 5GbE by 55% in bandwidth, 29% in large-message NCCL bandwidth, and has lower raw RTT. End to end, the difference is zero:
| Observation | 5g | usb10g | Delta |
|---|---|---|---|
| llama33awq / 5K / prefill | 148.5 | 148.6 | +0.07% |
| qwen3next / 5K / prefill | 1185.3 | 1184.2 | −0.09% |
| qwen36fp8 / 100K / prefill | 1392.8 | 1390.2 | −0.19% |
| qwen36fp8 / 50 / c256 end-to-end | 535.6 | 537.0 | +0.26% |
| qwen25awq / 50 / c443 end-to-end | 98.7 | 98.8 | +0.10% |
Across 4 models, 5 context sizes and every concurrency cell: direction random, magnitude ≤1%.
Bond the two links together and prefill rises across the board:
| Model | 5K | 32K | 52K | 100K |
|---|---|---|---|---|
| llama33awq | +80.0% | +73.1% | +69.1% | — |
| qwen3next | +64.4% | +59.7% | +57.8% | +54.0% |
| qwen25awq | +51.4% | +46.2% | — | — |
| qwen36fp8 | +15.5% | +14.0% | +12.1% | +11.2% |
For long contexts, look at total elapsed time:
| Observation | 5g | bond | Speedup |
|---|---|---|---|
| llama33awq / 32K / c1 | 230.8 s | 136.1 s | 1.70× |
| llama33awq / 52K / c1 | 375.8 s | 224.9 s | 1.67× |
| qwen25awq / 32K / c1 | 231.9 s | 161.0 s | 1.44× |
| qwen3next / 100K / c1 | 90.4 s | 60.1 s | 1.50× |
| qwen36fp8 / 100K / c1 | 73.8 s | 67.2 s | 1.10× |
The concurrency crossover point is low (short context, ctx=50, bond relative to 5g):
| Model | c1 | c4 | c16 | c64 | c256 |
|---|---|---|---|---|---|
| llama33awq | +4.9% | +22.0% | +33.5% | +69.4% | +76.1% |
| qwen25awq | −2.2% | +19.0% | +13.9% | +43.8% | +48.0% |
| qwen3next | −30.9% | −17.8% | +17.3% | +27.4% | +27.7% |
| qwen36fp8 | −32.0% | −15.0% | +32.5% | +44.7% | +70.3% |
Bond only loses on short prompts at low concurrency with MoE models, and the cost lands on decode:
| decode @5k (tok/s) | 5g | usb10g | bond |
|---|---|---|---|
| llama33awq | 18.8 | 19.1 | 19.4 |
| qwen25awq | 18.7 | 18.6 | 18.0 |
| qwen3next | 58.6 | 59.0 | 40.4 (−31%) |
| qwen36fp8 | 72.3 | 70.6 | 48.9 (−32%) |
Once the prompt reaches 5K, bond already wins on three of the four models at concurrency 1 (+13.6% to +59%).
⇒ Widening a single link barely moves the needle; combining two links does.
3.2 Models: MoE wins across the board
Same vLLM, same link (5g), same parameters — only the model changes:
| qwen36fp8 (35B-A3B) | qwen3next (80B-A3B) | llama33awq (70B dense) | qwen25awq (72B dense) | |
|---|---|---|---|---|
| prefill @5k | 1448.7 | 1185.3 | 148.5 | 148.0 |
| decode @5k | 72.3 | 58.6 | 18.8 | 18.7 |
| TTFT @5k | 3.45 s | 4.21 s | 33.68 s | 33.79 s |
| KV pool | 635,712 | 230,656 | 92,416 | 78,912 |
| Short-context throughput peak | 535.6 | 388.3 | 96.0 | 99.4 |
| Short-context c1→saturation | 7.5× | 6.7× | 5.2× | 5.5× |
| 5K c1→saturation | +49% | +49% | +9% | +9% |
- prefill differs by 9.8×, TTFT by 9.8×, KV pool by 8×.
- At 5K, dense gains only 9% from added concurrency; MoE gains 49%.
- Context ceiling: both MoE models reach 100K; the dense pair stops at 92160 and 32768.
- prefill is nearly flat against context length (only −3.9% from 5K to 100K), and decode decay is comparable across all four models (−5% to −8%).
- Dense AWQ requires
--enforce-eager; without it the Marlin kernel crashes — and it crashes only after running for over ten minutes.
3.3 What vLLM is good at
Blowing past the KV pool doesn't crash anything; requests simply queue. Every cell marked 超KV came back with preemptions = 0, failures = 0, constant throughput, and total elapsed time strictly linear in concurrency.
| qwen36fp8 / 5g / 32K | KV demand / pool | tok/s | Total time |
|---|---|---|---|
| concurrency 19 | 610,432 / 635,712 | 5.9 | 415.1 s |
| concurrency 29 (1.5× over) | 931,712 / 635,712 | 5.9 | 632.2 s |
Elapsed-time ratio 1.523×, concurrency ratio 1.526×. What does crash is the VRAM allocation at startup: util 0.88 + max-num-seqs 256 OOMs immediately; 0.85 + 128 comes up fine.
Long-context aggregate throughput = prefill × 128 / (ctx + 128).
| Link | Context | Predicted | Measured peak |
|---|---|---|---|
| 5g | 5K | 36.3 | 36.7 |
| 5g | 32K | 5.87 | 5.9 |
| 5g | 52K | 3.58 | 3.6 |
| 5g | 100K | 1.78 | 1.8 |
| bond | 32K | 6.69 | 6.6 |
| bond | 100K | 1.98 | 2.0 |
In long-context work neither concurrency nor batch size changes the order of magnitude — only prefill matters.
4. Conclusions
4.1 vLLM runs fine at home, and the results aren't bad
Two home machines, four consumer GPUs (5090 + 5080 + 5060Ti ×2), one ordinary LAN cable: all four 70B–80B class models came up, and 100K context works. The claim that cross-machine tensor parallelism "just dies" does not hold.
Numbers you actually feel day to day (qwen36fp8):
| Value | |
|---|---|
| Time to first token, 5K prompt | 3.45 s |
| Single-stream output | 72 tok/s |
| Single-stream output at 100K | 66.8 tok/s |
| Short-request throughput peak | 535.6 tok/s (912 on bond) |
Against llama.cpp single-stream you lose nothing: tied on short prompts, +6% at 5K, +13% at 32K, +20% at 52K, +32% at 100K. The only thing you give up is context capacity, 102400 vs 262144.
The links need no extra spend. Bonding two cables you already own makes long contexts 1.1–1.7× faster and high concurrency 70% faster.
Pick an MoE model. On the same hardware a dense 70B takes 33 seconds to first token on a 5K prompt and gains only 9% from added concurrency.
4.2 The value is in parallelism — it suits interleaved multi-tasking
The biggest gap is simply how many concurrent streams you can open at long context. On llama.cpp (5 cards, 96GB, KV q8_0), 4-way crashes above 32K and the most you can run is 2:
| Context | llama.cpp max concurrency | vLLM max concurrency | vLLM pushed 1.5× further |
|---|---|---|---|
| 32K | 2 (4-way crashes) | 19 | 29, zero failures |
| 52K | 2 (4-way crashes or overflow error) | 12 | 18, zero failures |
| 100K | 2 (4-way crashes); dense only 1 | 6 | 9, zero failures |
And llama.cpp's 2-way is fake concurrency: measured overlap is 0.0, meaning the two streams run fully serialized. At 4-way the overlap is −0.2, slower than processing them one at a time.
Note that vLLM had less VRAM to work with — 4 cards at 64GB versus 5 cards at 96GB.
Throughput points the same way. At 32K both start from 5.4 tok/s single-stream; at 4-way:
| Single | N=4 | Change | |
|---|---|---|---|
| llama.cpp | 5.4 | 1.9 | −65% |
| vLLM | 5.4 | 5.8 | +7% |
On short contexts, going from concurrency 1 to 256 takes qwen36fp8 from 71.0 to 535.6 — 7.5×.
What this means for actual use at home:
- Leave a long-document translation running (32K prompt, several minutes) and ask a quick coding question on the side — neither interferes with the other.
- Dump dozens of batch jobs in without writing your own queue or rate limiter.
- No need to worry about opening too many at once and hitting OOM; blowing past the KV pool at runtime is not a failure mode.
If VRAM at home isn't enough and you need to borrow system memory, go with llama.cpp.
If VRAM is sufficient and the GPUs share an architecture so they can be grouped together, go with vLLM.
Yeah — for a home setup, this looks pretty good.
Top comments (0)