This post is a follow-up to my previous benchmark. It tests six models across three links — 5GbE, "fake TB5," and USB 10GbE — comparing how link speed affects performance.
I'd originally planned to test only TB5, then move on to other things. Buy a cable, plug it in, measure, done — a few hours, tops.
But TB5 failed to establish end-to-end connectivity on the ASMedia4242 chip. I wasn't convinced at first, but eventually gave up trying. It looks like ASM870/870E can't do DMA-based end-to-end transfer unless you swap out the ASMedia4242 module itself.
So I bought USB 10GBps afterward — except TB5 still doesn't support USB10GBps, so it only works over USB-C 10 (roughly 7.2Gbps measured). Manually installing Realtek and tuning the protocol turned out to be the easier path in comparison. I might write up the TB5/USB10gbps debugging process separately at some point…
Given all this mess, "deploy" felt like too heavy a word for a home setup, but I used it anyway.
Enough preamble — on to the actual results.
This round covers speed and concurrency tests across six models. Still no optimization work — that's for later. The whole thing took a week of tuning and 72 hours of actual test runtime, with ambient temperature around 35°C.
Each (model × link) combination only got a single launch. I believe temperature drift is the main confound here — the day/night temperature swing was noticeable. Precision notes are at the end of the post.
Still no additional optimization this round, but I fixed two bugs from last time:
- Fixed a script bug from the previous version where input length would drift depending on the model
- Dynamically allocate ctx, aiming for a larger KV budget
Links
| Link | Actual protocol | A→B | B→A | Bidirectional total | Idle ping |
|---|---|---|---|---|---|
| Fake TB5 | TB5 cable, but negotiation falls back to USB4 40Gbps. The motherboard's integrated ASMedia ASM4242 only reaches USB4 40G, not true TB5 80G; link-layer negotiates 40Gb/s = 20Gb/s × 2 lanes | 4.38 | 4.76 | 6.64 | 0.224 ms |
| 5GbE | Direct CAT6A connection | 4.71 | 4.71 | 9.38 | 0.084 ms |
| USB 10GbE | TB5 doesn't support it, so USB-C 10G only. Realtek RTL8159 10GbE, direct Cat6a | 7.28 | 7.28 | 11.93 | 0.136 ms |
(bandwidth in Gbps)
Fake TB5's unidirectional bandwidth is basically on par with 5GbE, but the bidirectional total is 29% lower — thunderbolt-net 's single-ring DMA engine caps full-duplex throughput.
I separately measured a 16KB request-response round trip on it: p50 856µs, of which 326µs is fixed overhead.
1. Link Comparison Summary
prefill shown as relative improvement, decode as per-token latency delta. Baseline = 5GbE.
| Model | prefill (short / 5K / 32K / 52K / 100K) | decode Δms/token |
|---|---|---|
| Qwen3.6-35B-A3B Q8_0 | Fake TB5 −2% / −8% / −5% / −3% / −2% USB10G +28% / +8% / +13% / +15% / +17% |
Fake TB5 +1.59 / +1.71 / +1.63 / +1.77 / +2.14 USB10G −0.86 / −0.82 / −0.97 / −1.01 / −1.06 |
| gpt-oss-120b MXFP4 | Fake TB5 −30% / −8% / −3% / −3% / −0% USB10G −9% / +8% / +13% / +15% / +16% |
Fake TB5 +1.46 / +1.71 / +1.63 / +1.65 / +2.13 USB10G −0.48 / −0.33 / −0.17 / −0.51 / −0.66 |
| GLM-4.5-Air Q4_K_M | Fake TB5 −18% / −4% / −2% / −1% / −0% USB10G −3% / +5% / +6% / +5% / +5% |
Fake TB5 +1.77 / +1.78 / +1.71 / +1.83 / — USB10G −0.27 / −0.18 / — / — / — |
| Qwen3-235B-A22B Q2_K_XL | Fake TB5 −12% / −6% / −3% / — / — USB10G −5% / +2% / +4% / — / — |
Fake TB5 +2.23 / +2.93 / +2.56 / — / — USB10G +0.21 / +0.44 / — / — / — |
| Llama-3.3-70B Q6_K | Fake TB5 −5% / −1% / −1% / −0% / +0% USB10G +1% / +4% / +5% / +5% / +5% |
Below measurement resolution |
| Qwen2.5-72B Q6_K | Fake TB5 −6% / −1% / −1% / — / — USB10G +0% / +4% / +5% / — / — |
Below measurement resolution |
The "—" cells in the decode column are where the difference falls within a single quantization step and can't be read out (the script only logs one decimal place; at 9 tok/s, one step already equals 1.2ms). Both dense models show this across the entire row — but adding the 1.7ms back and rounding to one decimal, 5 of 8 cells land exactly on the measured value, versus only 1 under the "no effect" assumption. The same overhead, spread across 97–204ms, works out to only 1–2% — right at the floor of the recording precision.
All single-stream runs completed cleanly with no errors, and the decay curve is smooth (Llama-3.3, short→100K: 100%→94%→74%→63%→48%), tracking KV growth. OOMs only showed up in the dense models' 52K×4 and 100K×2 concurrency cells.
What's readable: fake TB5 trails across the entire table, and the gap is independent of model speed or context length, holding steady at 1.5–2.9ms/token. USB10G is mostly slightly faster, except Qwen3-235B, which goes the other way.
2. Single Stream
USB10G only shown here; see the table above for the relative relationship across all three links.
decode (tok/s)
| Model | Short | 5K | 32K | 52K | 100K | 100K/Short |
|---|---|---|---|---|---|---|
| gpt-oss-120b MXFP4 | 85.5 | 78.3 | 60.3 | 53.3 | 39.4 | 46% |
| Qwen3.6-35B-A3B Q8_0 | 78.0 | 74.8 | 67.2 | 61.6 | 50.8 | 65% |
| GLM-4.5-Air Q4_K_M | 51.5 | 41.4 | 20.7 | 15.1 | 9.1 | 18% |
| Qwen3-235B-A22B Q2_K_XL | 37.9 | 30.1 | 15.5 | ❌ 400 | ❌ 400 | — |
| Llama-3.3-70B Q6_K | 10.3 | 9.7 | 7.6 | 6.6 | 4.9 | 48% |
| Qwen2.5-72B Q6_K | 9.3 | 8.9 | 7.1 | ❌ 400 | ❌ 400 | — |
prefill (tok/s)
| Model | Short | 5K | 32K | 52K | 100K |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B Q8_0 | 300 | 2528 | 1756 | 1332 | 844 |
| gpt-oss-120b MXFP4 | 400 | 2221 | 1536 | 1134 | 693 |
| GLM-4.5-Air Q4_K_M | 295 | 1016 | 586 | 394 | 214 |
| Qwen3-235B-A22B Q2_K_XL | 146 | 676 | 494 | — | — |
| Llama-3.3-70B Q6_K | 293 | 419 | 301 | 243 | 165 |
| Qwen2.5-72B Q6_K | 248 | 425 | 305 | — | — |
TTFT isn't listed as its own column, since TTFT ≈ prompt_tok ÷ prefill tok/s. For Qwen3.6 at 100K, for example, 100000 ÷ 844 ≈ 118s — measured value was 120s.
3. Concurrency
Usable boundary (all three links gave the same result, so shown once)
Single-stream decode below 1 tok/s is counted as a crash — this is not gradual degradation, it's the catastrophic collapse that follows VRAM overflow. 400 = context wall; 500 = overflow error.
| Model | Short | 5K | 32K | 52K | 100K |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B Q8_0 | 4 | 4 | 2 (crashes at 4) | 2 (crashes at 4) | 2 (crashes at 4) |
| gpt-oss-120b MXFP4 | 4 | 4 | 2 (crashes at 4) | 2 (crashes at 4) | 2 (crashes at 4) |
| GLM-4.5-Air Q4_K_M | 4 | 4 | 2 (crashes at 4) | 2 (500 at 4) | 1 (fails at 2) |
| Qwen3-235B-A22B Q2_K_XL | 4 | 4 | 1 (fails at 2) | ❌ 400 | ❌ 400 |
| Llama-3.3-70B Q6_K | 4 | 4 | 2 (crashes at 4) | 2 (500 at 4) | 1 (fails at 2) |
| Qwen2.5-72B Q6_K | 4 | 4 | 2 (crashes at 4) | ❌ 400 | ❌ 400 |
True concurrency ratio (overlap) (5GbE, 2-way / 4-way)
| Model | Short | 5K | 32K | 52K | 100K |
|---|---|---|---|---|---|
| gpt-oss-120b | 1.0 / 1.0 | 1.0 / 0.5 | 0.1 / 0.0 | 0.0 / 0.0 | 0.0 / −0.2 |
| Qwen3.6-35B-A3B | 1.0 / 1.0 | 0.8 / 0.3 | 0.1 / 0.0 | 0.0 / 0.0 | 0.0 / −0.2 |
| Llama-3.3-70B | 1.0 / 1.0 | 0.6 / 0.3 | 0.1 / 0.0 | 0.1 / — | — |
| Qwen2.5-72B | 1.0 / 1.0 | 0.6 / 0.3 | 0.1 / 0.0 | — | — |
| GLM-4.5-Air | 1.0 / 1.0 | 0.5 / 0.3 | 0.1 / 0.0 | 0.0 / — | 0.0 / — |
| Qwen3-235B | 1.0 / 1.0 | 0.5 / 0.2 | — (fails at 2-way) | — | — |
The client side is genuinely concurrent : concurrency_once fires N requests simultaneously from N threads, placing every token on a single absolute clock to compute overlap (the fraction of time N streams are decoding at the same time ). Whether the server can actually sustain concurrent processing is answered quantitatively by this table.
overlap = 1.0 means fully genuine concurrency; = 0 means fully serial (fake concurrency); < 0 means worse than serial (repeated KV eviction and recomputation due to insufficient KV space).
Mechanism: llama.cpp's prefill (prompt processing) handles only one slot at a time (serial), while decode is batched and processed in parallel. For short prompts, prefill is negligible, so all four streams quickly enter batch-decode — genuine concurrency. For long prompts, serial prefill eats up most of the time budget, so by the time parallel decode gets its turn there's barely any window left — fake concurrency.
The 5K column quantifies this mechanism nicely: gpt-oss, the fastest at prefill, hits 1.0/0.5, while Qwen3-235B, the slowest, only manages 0.5/0.2 — the ranking matches the prefill leaderboard exactly. Past 32K, everything collapses to 0.
Conclusions
1. prefill scales with unidirectional bandwidth, but returns diminish fast past 5GbE
Raising unidirectional bandwidth from 5GbE's 4.71 Gbps to USB 10GbE's 7.28 Gbps (+55%) only buys prefill gains of +5% (GLM / Llama-3.3 / Qwen2.5 / Qwen3-235B) to +17% (Qwen3.6 and gpt-oss at long context).
Bandwidth sensitivity rises with two things: the faster a model's prefill is , the more data crosses the link boundary per unit time — the two MoEs running 2000+ tok/s see +13–17%, while the four models in the 300–500 tok/s range all sit at a flat +4–6% regardless of context length. The longer the context , the larger the hidden state passed at each layer boundary — Qwen3.6 climbs monotonically from +8% at 5K to +17% at 100K.
The takeaway: prefill is primarily compute-bound, and link speed is a secondary factor. At 5GbE, this setup's prefill is no longer cable-limited.
2. decode pays for per-step transaction overhead, not bandwidth
This is the most substantive finding of this round, and it's completely invisible if you look at tok/s.
Looked at in tok/s, fake TB5 trails by roughly 10% on fast models and barely shows on slow ones, which makes it look like faster models are more network-sensitive. Converted to per-token time, the pattern flips: fake TB5 trails 5GbE by a constant 1.5–2.9ms/token, independent of model speed and independent of context length.
The cleanest evidence comes from GLM-4.5-Air: within a single launch, per-token time rises from 19.7ms (short) to 66.7ms (52K) — a 3.4× increase — while the link-attributable delta stays put at 1.71–1.83ms throughout. Per-token time more than triples; the delta doesn't move at all. That's the signature of a fixed cost, not a proportional one.
The 16KB request-response measurement answers what that fixed cost is: fake TB5's p50 is 856µs, of which 326µs is fixed overhead. At 4.38 Gbps, the wire time for a 16KB round trip is only 59.9µs — just 7% of the total round trip. The remaining 93% is spent in the protocol stack, DMA scheduling, and context switches. The payload that decode moves across the boundary at each step is a single token's activation — a few KB — landing squarely in that same range.
Working backward from this, each decode step involves at least 3 cross-machine transactions (N=2 would require 5GbE's 16KB RR to be just 6µs, below the physical floor of 84µs ping + 56µs wire time). Under the tensor-split topology, one token should theoretically need only one round trip — 2 transactions. The extra transactions represent real optimization headroom, and it's worth more than swapping cables.
3. Where fake TB5 loses
Its unidirectional bandwidth is nearly identical to 5GbE's (4.38–4.76 vs 4.71), but bidirectional total is 29% lower (6.64 vs 9.38), and the 16KB round-trip fixed overhead is 326µs. These same three numbers show up very differently depending on the workload:
- prefill (large one-way transfers): worst at short context, gradually catching up to 5GbE as context grows — once transfer becomes bandwidth-bound, the two links' unidirectional bandwidth is essentially the same to begin with.
- decode (a few KB, request-response): a constant ~1.7ms/token deficit from start to finish, never converging — this payload size sits in the transaction-overhead-bound regime, where fake TB5 stays at a permanent disadvantage.
The root cause is probably the same one: thunderbolt-net 's single-ring DMA serializes send and receive on one engine, and "lower bidirectional throughput" and "higher transaction overhead" are likely two faces of the same bottleneck.
4. The collapse at very long context is a VRAM problem, not a link problem
Comparing 100K to 32K, prefill drops sharply (Qwen3.6 goes 1756 → 844 tok/s, gpt-oss 1536 → 693 tok/s), but all three links show the same collapse , with matching magnitude.
So this isn't a transfer problem — under the pressure of massive computation, KV read/write is hitting a VRAM bandwidth ceiling. The concurrency data corroborates this: past 32K, overlap collapses to 0 across the board, and 4-way concurrency either crashes outright or dies with an error — both symptoms of VRAM pressure, unrelated to the link.
5. Differences between models
Single-stream decode ranking : gpt-oss starts fastest (short 85.5 → 100K 39.4), Qwen3.6 next (78.0 → 50.8), Qwen3-235B mid-pack (37.9 → 15.5 @32k), GLM decays steepest (51.5 → 9.1), and the two dense models trail (Llama-3.3 10.3 → 4.9, Qwen2.5 9.3 → 7.1).
But gpt-oss retains only 46% of its starting throughput, versus 65% for Qwen3.6 — the two cross over around 32K. If your workload leans toward long context, the ranking flips.
Robustness under high concurrency : only gpt-oss and Qwen3.6 survive 52K×4 / 100K×4 — degraded, but not crashed. GLM and Llama-3.3 hit HTTP 500 the moment they overflow. Qwen2.5 and Qwen3-235B slam straight into the context wall (400 rejection). None of this depends on the link.
6. Implications for configuration decisions
"600MB/s of layer splitting is enough" — this judgment holds, but for a different reason than originally assumed. I'd assumed it was because "bandwidth is sufficient." It's actually because prefill has already escaped bandwidth-bound territory at 5GbE, and decode was never bandwidth-bound to begin with.
So:
- Adding VRAM and switching to a faster MoE model beats spending on network speed. Every crash, context wall, and concurrency collapse happened on the VRAM side.
- Adding bandwidth doesn't help decode at all — what needs cutting is the number of cross-machine transactions per step. N≥3 against a topological floor of 2 means there's real headroom here; swapping cables only optimizes that 7% wire time.
- The ceiling on cross-machine RPC isn't cable bandwidth.
Final thoughts
There's a big gap between a model "running" and a model being "usable." A lot of home-lab/personal-scale models underperform, and it's not just VRAM — engineering matters just as much. For example, an unadapted continue can't recognize skills, but copilot running under ollama can. The role of context and context caching, the computation and configuration of KV — these are all genuinely interesting, high-value directions to keep digging into.
Looking back at the conclusions... did I just waste money on USB10G? Maybe I should try vLLM tensor parallelism next.
Hoping the bubble problem and juggling 5 GPUs won't be too painful to sort out. Getting sucked into parameter tuning and forgetting the day job...
Oh, and in the first few days I also bought a 3-meter "thunder bot5" cable (yes, that's literally what their marketing called it).
If you've ever bricked your ASMedia4242, I do have a UEFI shell guide for recovery. Can't share the resource itself due to copyright, but I learned a lot going through it. I also submitted a patch to the Linux kernel folks regarding the ASMedia4242/TB5 handshake — might not get merged. But hello world, anyway.
Top comments (0)