Pushing the Limits: Extreme Inference Speedup of Qwen 3.8 27B on NVIDIA B300 (100 to 10k+ tok/s)

Deploying a 27-billion parameter reasoning model like Qwen 3.8 27B on modern hardware presents a stark paradox. If you boot a default configuration on an NVIDIA B300 SXM6 GPU and send a solitary stream, you will measure roughly 104 tokens per second. The GPU sits largely cold, constrained by memory bus round-trips for every single token it emits.

Yet the same model on the same B300 GPUs reached 9,825 tokens per second on our custom stack with eight GPUs and 256 concurrent streams, and a dedicated Fireworks AI deployment on the same eight GPUs reached 16,931 tokens per second. These are aggregate numbers across many streams, not the speed of one stream.

Getting there is not a matter of flipping a single “turbo” flag. It requires systematically dismantling five consecutive systems bottlenecks: memory bandwidth latency, token serialization, redundant prompt prefill, repetitive structure generation, and multi-GPU interconnect overhead.

Over the past several days on our research cluster, we ran over 1,000 controlled benchmark trials across 2×, 4×, and 8× NVIDIA B300 GPUs, exploring tensor parallelism, data parallelism, multi-token speculation (MTP4 vs. MTP8), prompt prefix caching, and suffix reuse. Below is the verified engineering telemetry and the practical playbook for scaling Qwen 3.8 27B in production.

Start here

Explore how Qwen 3.8 27B scales from 104 tok/s to over 10,000 tok/s on NVIDIA B300. Discover why Tensor Parallelism rules single-stream latency while Data Parallelism dominates batch serving, why MTP4 beats MTP8 on reasoning workloads, how prefix caching plus suffix decoding delivers 99.7% speculation accuracy, and why doubling from 4 to 8 GPUs only yielded a 10% throughput bump.

• Autoregressive Decode Bound: In single-stream generation, the entire 27B parameter weight matrix must be read from HBM for every individual token, bottlenecking throughput on memory bandwidth rather than compute.
• Tensor Parallelism (TP): Sharding individual matrix multiplications across GPUs over high-speed NVLink to cut per-device memory-fetch latency for a single sequence.
• Data Parallelism (DP): Running multiple independent model replicas on dedicated GPUs, serving concurrent requests in parallel with zero inter-GPU sync.
• Multi-Token Prediction (MTP): Speculative decoding with a small prediction module trained with the model; it drafts k future tokens that the main model then verifies in one pass.
• Prefix Caching: Reusing computed KV cache blocks for identical prompt segments across independent requests to bypass prefill compute.
• Suffix Decoding: Speculating entire candidate phrases based on previously observed prompt or output n-gram histories.

1. The 104 tok/s Baseline: Why Raw Models Crawl

To understand how to make an LLM run at thousands of tokens per second, you must first understand why it runs at 100 tokens per second out of the box.

During the autoregressive decode phase, an LLM generates one token at a time. To emit a single token for a 27B parameter model in FP8 precision (~27 gigabytes of weights), the GPU must read those 27 GB from high-bandwidth memory (HBM) for every decode step. That sets a floor on the time per step; kernel launches, attention over the KV cache and scheduling add more. Without speculation, one stream measured 104 tok/s on a single-GPU replica and 135 tok/s with the model split across two GPUs.

time per decode step ≈ model weight bytes ÷ HBM bandwidth + kernel and scheduling overhead

At concurrency 1, most of the GPU’s compute sits idle: the execution units wait on memory loads. Therefore, scaling inference throughput requires solving two orthogonal engineering goals:

1. For Low Concurrency (Interactive Chat & Agent Tooling): How do we minimize latency per token so a human or autonomous agent experiences instant responses?
2. For High Concurrency (Batch APIs & Synthetic Data Generation): How do we pack requests together so that each memory-load pass over the weights computes tokens for dozens of requests simultaneously?

2. The Speedup Journey: From 100 to 16,900+ tok/s

We took a disciplined, step-by-step approach. Each experiment introduced a single architectural intervention: slicing layers across NVLink, deploying speculative draft heads, caching repeated prompt prefixes, mining repetitive output structures, and scaling independent data-parallel replicas.

The chart below shows the main steps of our tests on Qwen 3.8 27B across 2×, 4×, and 8× NVIDIA B300 GPUs:

The main steps, one change at a time (log scale). Going from 1 to 64 concurrent streams moved throughput the most. Our stack peaked at 9,825 tok/s; the 16,931 bar is Fireworks AI on the same eight GPUs.

Notice the distinct operational leaps along this curve:

• The Latency Leap (104 → 305 tok/s): Tensor Parallelism (TP2 and TP4) combined with MTP4 speculative decoding cut inter-token latency by nearly 3×, pushing solitary stream speed to 305 tok/s.
• The Concurrency Multiplier (305 → 4,464 tok/s): Bringing concurrency to 64 streams allowed continuous batching on 2 B300 GPUs to reach 4,464 tok/s with MTP4.
• The Caching & Suffix Breakthrough (4,464 → 7,012 tok/s): On structured catalogs, prefix caching hit on 85% of prompt tokens, while suffix history predicted output sequences with 99.7% acceptance, breaching 7k tok/s on just two cards.
• The Multi-GPU Expansion (7,012 → 9,825 tok/s): Scaling to 4 and 8 B300 GPUs with a balanced hybrid topology (DP4×TP2) pushed our custom stack to 9,825 tok/s.
• A different stack on the same hardware (16,931 tok/s): A dedicated Fireworks AI deployment on the same 8 GPUs reached 16,931 tok/s at concurrency 256 on short prompts. It is a comparison point, not a further step of our stack.

3. Topology Dynamics: Tensor Parallelism vs. Data Parallelism

A frequent misconception in inference engineering is the dogma that “Data Parallelism is always best for throughput, and Tensor Parallelism is only for fitting big models in VRAM.”

With Qwen 3.8 27B FP8, the model easily fits onto a single B300 (288 GB of HBM). Yet when we benchmarked different allocations across 2, 4 and 8 GPUs, the reality proved far more nuanced. The table shows aggregate output tok/s on short prompts; the 2× rows are vanilla FP8 without speculation, the 4× and 8× rows use MTP4:

Hardware	Topology	Speculation	c = 1	c = 8	c = 64	c = 256
2× B300	DP2×TP1	Vanilla FP8	103.97	654.56	2,490.11	—
2× B300	DP1×TP2	Vanilla FP8	135.42	792.91	3,427.21	—
4× B300	DP1×TP4	MTP4	304.80	1,133.68	4,453.83	7,092.10
4× B300	DP2×TP2	MTP4	281.55	1,404.05	4,868.94	7,352.23
4× B300	DP4×TP1	MTP4	243.10	1,461.10	6,048.40	7,230.67
8× B300	DP1×TP8	MTP4	282.61	1,050.29	4,252.37	7,136.17
8× B300	DP2×TP4	MTP4	288.12	1,406.05	5,932.80	8,075.58
8× B300	DP4×TP2	MTP4	277.34	1,636.08	6,360.69	— (gateway timeout)
8× B300	DP8×TP1	MTP4	241.93	1,329.37	7,153.35	8,939.26

The Counter-Intuitive Vanilla Finding: On 2× B300 running Vanilla FP8 (without MTP), TP2 beat DP2 at every single concurrency level, including c=64 (3,427 tok/s for TP2 vs 2,490 tok/s for DP2). A likely reason: without speculation, sharding across NVLink shortened each forward pass enough for batches to cycle faster than on two slower independent workers. Once MTP4 was enabled, DP2 pulled ahead at c=64 (4,464 vs. 3,542 tok/s for TP2).

Key architectural principles from the topology sweeps:

• Low Concurrency (c = 1 to 4): Tensor Parallelism is mandatory if user-facing latency matters. DP1×TP4 delivered 304.8 tok/s at c1 on 4× B300, compared to 243.1 tok/s for DP4×TP1. Every token decode step is shared over high-speed NVLink.
• High Concurrency (c = 64 to 256): Data Parallelism dominates. At c=64 on 4 GPUs, DP4×TP1 delivered 6,048 tok/s, beating DP1×TP4 (4,454 tok/s) by +35.8%. Under heavy concurrency, TP’s AllReduce synchronization across GPUs on every layer is the likely cost.
• The 8-GPU Limit (TP8): Never run TP8 across a full 8-GPU node for throughput. DP1×TP8 at c=64 delivered only 4,252 tok/s—slower than a 4-GPU system. The cost of 8-way AllReduce likely outweighed the extra compute.
• The Long-Context Sweet Spot (DP4×TP2): On long prompts (2.8k tokens), the balanced hybrid topology—four replicas of TP2 pairs (DP4×TP2)—gave the best complete result at c=256, 9,825 tok/s, and also led eight single-GPU replicas (DP8×TP1) at c=1, 8 and 64. DP8×TP1 posted 11.2k and 11.6k tok/s in two of three repeats at c=256, but the third hit gateway timeouts, so we report no mean for it.

4× B300 with MTP4 on short prompts: one replica split over four GPUs gave the fastest single stream, four independent replicas gave the most total throughput at 64 streams.

4. Speculative Decoding: MTP4, MTP8, and the Hybrid Myth

Qwen 3.8 natively incorporates Multi-Token Prediction (MTP). Unlike traditional speculative decoding, which requires loading a separate smaller draft model (and paying memory bandwidth to read two sets of weights), MTP uses a lightweight prediction module trained together with the main model.

The MTP module drafts k future tokens, one draft step at a time, and the next forward pass of the main model checks all k at once. If each draft token is accepted with probability α, the expected number of tokens per verification pass is:

expected tokens per verification pass = (1 − α^(k+1)) ÷ (1 − α)

This is the standard estimate from Leviathan et al., 2023; the real speedup is lower because drafting and verification cost time.

We tested MTP with 4 draft tokens (MTP4), 8 draft tokens (MTP8), and a “Hybrid” engine that first attempts suffix matching before falling back to MTP. All rows ran on 2× B300 with DP2. Draft acceptance is accepted ÷ proposed draft tokens over the whole run; the c = 256 values come from a separate high-load run:

Speculation Mode	Workload Type	Draft Acceptance	c = 1 tok/s	c = 8 tok/s	c = 64 tok/s	c = 256 tok/s
MTP4 (Optimized)	Short Reasoning	59.09%	242.37	1,287.05	4,362.11	5,811.65
MTP8 (Optimized)	Short Reasoning	38.42%	240.87	1,272.54	4,122.65	—
Hybrid (Suffix → MTP4)	Short Reasoning	70.85%	199.58	1,064.13	3,784.48	—
Prefix + Suffix4	Long Structured	99.69%	325.53	1,936.26	5,704.54	7,011.98
MTP4 (Optimized)	Long Structured	99.99%	288.22	1,317.17	2,165.26	—
MTP8 (Optimized)	Long Structured	99.89%	368.55	1,520.00	2,227.49	—

The Lessons of Speculative Tuning

1. Why MTP4 beats MTP8 on reasoning: Reasoning workloads have high branch entropy. The model often changes course, backtracking or exploring logic. Under these dynamics, predicting 4 tokens ahead hit a healthy 59.1% acceptance rate. Pushing to 8 tokens (MTP8) caused acceptance to collapse to 38.4%. The compute spent generating and verifying rejected draft tokens 5 through 8 degraded throughput by over 200 tok/s at c=64.
2. When MTP8 wins: When sequences are highly structured or predictable (or when deep queue batching at c=256 keeps all workers saturated), MTP8 pulled ahead on 4 GPUs (8,239 tok/s vs 7,231 tok/s for MTP4, both DP4×TP1 at c=256).
3. The Hybrid result is mixed: In architecture discussions, engineers often propose: “Let us try suffix history first, and if that misses, fall back to MTP.” We built and measured this runtime. On TP2 it beat plain MTP4 at low concurrency (+7% on short and +12% on long prompts at c=1), but it lost in every c=64 cell, where throughput matters most. For batch serving, a single speculative method was the better choice.

5. Caching Strategies: Prefix Caching vs. Suffix Decoding vs. N-gram

In real-world applications, models rarely process isolated prompts. Enterprise workloads fall into two broad categories:

• Ad-hoc High-Variance Prompts: Unique user queries, diverse reasoning tasks, or creative writing.
• Structured Enterprise Prompts: System prompts, few-shot examples, JSON schemas, catalog extraction, and multi-turn agent tool loops.

We tested our dual-B300 stack with identical models under three caching regimes:

Workload	Cache Strategy	Prefix Hit %	Draft Accept %	c = 1	c = 8	c = 64
Short (564 in)	Cache Off	—	—	98.71	681.68	2,925.91
Short (564 in)	Prefix Cache Only	0.0%	—	95.71	656.24	2,871.04
Short (564 in)	Prefix + N-gram4	0.0%	78.98%	146.72	454.22	1,944.88
Short (564 in)	MTP4 Optimized	0.0%	59.09%	242.37	1,287.05	4,362.11
Long (2,870 in)	Cache Off	—	—	94.73	576.81	1,599.46
Long (2,870 in)	Prefix Cache Only	81.75%	—	95.98	659.48	2,851.20
Long (2,870 in)	Prefix + N-gram4	85.08%	90.17%	307.43	1,784.28	5,668.86
Long (2,870 in)	Prefix + Suffix4	85.08%	99.69%	325.53	1,936.26	5,704.54

Key Architectural Findings

• Prefix Caching on Short Prompts is a Trap: If your prompts do not share a common prefix, enabling prefix caching provides a 0.0% hit rate while adding hashing and cache-lookup work. Throughput dropped from 2,925 to 2,871 tok/s at c=64. Do not turn prefix caching on blindly for randomized workloads.
• The 1.78× Prefill Unlock: On long documents sharing a common system prompt or catalog base (2,820 shared tokens), prefix caching was transformative. Throughput at c=64 jumped from 1,599 to 2,851 tok/s (+78.3%), while p95 time to first token fell 63% (3,286 → 1,224 ms).
• The Suffix 99.7% Acceptance Wonder: In repetitive workflows like catalog formatting or code generation, the model repeatedly outputs structured syntax. Suffix decoding stores and speculates matching n-grams from prompt and generation history. It achieved an astonishing 99.69% draft acceptance, catapulting long-context throughput to 5,705 tok/s at c=64 and 7,012 tok/s at c=256.

Prefix caching on 2× B300 at 64 streams: it never hit on short prompts with nothing shared, and added 78% on long prompts with a shared 2,820-token prefix. Suffix drafts on top more than tripled throughput.

6. Scaling 4 vs. 8 B300 GPUs: Where Efficiency Bends

One of the most consequential findings for platform teams is the scaling relationship between 4 and 8 B300 GPUs. When you double your hardware budget from 4× to 8× B300 SXM6, do you get a 2× throughput increase?

The hard telemetry answer is no. Under identical workloads, software stacks, and fixed concurrency, the scaling ratio was surprisingly modest:

Workload	Concurrency	Best 4-GPU Configuration	4-GPU tok/s	Best 8-GPU Configuration	8-GPU tok/s	Scaling Gain
Short Context	c = 1	MTP4 · DP1×TP4	304.80	Suffix4 · DP1×TP8	291.08	−4.5% (Regression)
Short Context	c = 8	MTP4 · DP4×TP1	1,461.10	MTP4 · DP4×TP2	1,636.08	+12.0%
Short Context	c = 64	MTP4 · DP4×TP1	6,048.40	MTP4 · DP8×TP1	7,153.35	+18.3%
Short Context	c = 256	MTP8 · DP4×TP1	8,238.99	MTP8 · DP8×TP1	9,207.40	+11.8%
Long Context	c = 1	MTP8 · DP2×TP2	452.68	MTP8 · DP2×TP4	436.26	−3.6% (Regression)
Long Context	c = 8	Suffix4 · DP4×TP1	2,193.59	MTP8 · DP4×TP2	2,427.24	+10.7%
Long Context	c = 64	Suffix4 · DP4×TP1	7,451.49	Suffix4 · DP4×TP2	9,153.70	+22.8%
Long Context	c = 256	Suffix4 · DP4×TP1	9,088.94	Suffix4 · DP4×TP2	9,825.28	+8.1%

The 8-GPU Cost Reality Check: Doubling your cluster from 4 to 8 B300 cards at c=256 yielded only an 8.1% to 11.8% throughput increase. 4× B300 already delivered 89% to 93% of the 8-GPU throughput. We did not test above 256 concurrent streams; up to that point, paying twice the hourly rate for an 8-GPU node bought at most 23% more throughput.

Best 8-GPU result vs. best 4-GPU result at the same workload and concurrency. Doubling the GPUs added at most 23%, and slightly slowed single streams.

Why did doubling GPUs not double throughput? We did not isolate the cause. Three candidates:

1. Gateway overhead: At 8,000+ tokens per second, the LiteLLM gateways and their Python event loops serialize JSON chunks and stream SSE events. Gateway CPU stayed near one core of four, which does not rule out a single-threaded bottleneck, and the 8-GPU deep-queue runs did hit gateway timeouts.
2. Client Serialization: Receiving and verifying 256 concurrent SSE streams under AIPerf could add client-side scheduling latency, although the 8-CPU client used only about 1.6 cores.
3. Queue Starvation at Fixed Concurrency: When 8 workers divide 256 concurrent requests, each worker averages only 32 active requests, which may not keep every B300 fully busy.

7. Head-to-Head: Fireworks AI vs. g factor Stack

We benchmarked our custom serving stack and a dedicated Fireworks AI deployment on the same GPU budgets, 2×, 4×, and 8× NVIDIA B300 SXM6, both serving Qwen 3.8 27B in FP8. Our stack quantized the BF16 checkpoint to FP8 at load time; we did not record which FP8 weights Fireworks used. Our stack was measured over in-cluster HTTP and Fireworks over HTTPS, so latency numbers include different network paths.

Neither system is uniformly faster across all workloads. Instead, the architectural trade-offs divide cleanly between interactive low-concurrency responsiveness and deep-queue batch saturation:

Hardware	Workload & System	c = 1 (Single Stream)	c = 8	c = 64	c = 256 (Deep Queue)
2× B300	g factor (MTP4 DP2 · Short)	244.91 tok/s	1,316.81 tok/s	4,464.12 tok/s	5,811.65 tok/s
2× B300	Fireworks AI (Dedicated · Short)	185.53 tok/s	1,138.03 tok/s	4,789.18 tok/s	6,953.51 tok/s
2× B300	g factor (Prefix+Suffix4 · Long)	325.53 tok/s	1,936.26 tok/s	5,704.54 tok/s	7,011.98 tok/s
2× B300	Fireworks AI (Dedicated · Long)	217.93 tok/s	1,345.03 tok/s	4,212.96 tok/s	5,637.99 tok/s
4× B300	g factor (MTP4 DP1×TP4 / DP4; MTP8 at c=256 · Short)	304.80 tok/s	1,461.10 tok/s	6,048.40 tok/s	8,238.99 tok/s
4× B300	Fireworks AI (Dedicated · Short)	186.74 tok/s	1,184.38 tok/s	5,745.23 tok/s	9,381.30 tok/s
4× B300	g factor (Prefix+Suffix4 DP4 · Long)	324.31 tok/s	2,193.59 tok/s	7,451.49 tok/s	9,088.94 tok/s
4× B300	Fireworks AI (Dedicated · Long)	218.27 tok/s	1,471.73 tok/s	6,549.06 tok/s	10,059.63 tok/s
8× B300	g factor (MTP4 DP2×TP4 / DP8; MTP8 at c=256 · Short)	288.12 tok/s	1,636.08 tok/s	7,153.35 tok/s	9,207.40 tok/s
8× B300	Fireworks AI (Dedicated · Short)	171.80 tok/s	1,300.37 tok/s	6,810.00 tok/s	16,930.96 tok/s
8× B300	g factor (DP4×TP2 Suffix4 · Long)	359.25 tok/s	2,361.05 tok/s	9,153.70 tok/s	9,825.28 tok/s
8× B300	Fireworks AI (Dedicated · Long)	218.47 tok/s	1,538.85 tok/s	8,499.90 tok/s	16,769.74 tok/s

Where g factor Outperforms

• Single-Stream Interactive Decode (c = 1): g factor delivers 245 to 305 tok/s on short prompts, beating Fireworks (172 to 187 tok/s) by +32% to +68% at the same GPU count.
• First-Token Response Latency (TTFT): At c=1, g factor’s p95 time to first token was 49–91 ms, compared to Fireworks at 237–940 ms. Part of that gap is the network: our stack was measured in-cluster, Fireworks over the internet.
• Structured Long-Context Workloads (c = 1 to 64): On 2.8k-token contexts with shared prefixes, g factor’s Prefix Caching and Suffix4 speculation beat Fireworks by +8% to +64%, with the largest gaps at c=1 and c=8 (e.g., 2,361 vs 1,539 tok/s at c=8 on 8 GPUs).
• Why: g factor leverages NVLink Tensor Parallelism (TP2/TP4) to accelerate individual token decodes, paired with MTP speculation and prefix caching. The Fireworks deployment ran one B300 per replica (no tensor parallelism) with 3 draft tokens.

Where Fireworks AI Outperforms

• Massive Batch Saturation (c = 256): Fireworks surges to an incredible 16,931 tok/s on 8× B300, outperforming g factor (9,207 to 9,825 tok/s) by +71% to +84%.
• 4 → 8 GPU Scaling Linearity: Fireworks doubles its throughput smoothly as GPUs double (from 9,381 to 16,931 tok/s, a 1.80× ratio), whereas g factor scaled only +8% to +12% under our test harness.
• Managed operations: Fireworks handled replicas and routing (with session affinity on) without any Kubernetes tuning on our side.
• Why: We cannot see Fireworks’ internals. On our side, the 8-GPU deep-queue runs hit gateway timeouts, and gateway or client overhead is the leading suspect, but we did not isolate it.

Our stack vs. a dedicated Fireworks AI deployment on 8× B300. Ours was faster up to 64 streams; Fireworks pulled far ahead at 256.

8. The Systems Architecture Playbook: What to Deploy When

Based on our verified findings, here are four concrete architectural rules for deploying Qwen 3.8 27B in production:

Scenario A: Interactive Chat & Coding Agents

Workload: Concurrency 1–4, low latency critical (Time-To-First-Token and Inter-Token Latency).

• Topology: TP2 (on 2 GPUs) or TP4 (on 4 GPUs).
• Speculation: MTP4 enabled.
• Prefix Cache: Off (unless long system prompts are shared).
• Outcome: 280–305 tok/s single-stream speed, about 2.4–2.8 ms per output token.

Scenario B: High-Throughput Batch APIs & Synthetic Data

Workload: Concurrency 64–256, maximizing aggregate completion tokens per dollar.

• Topology: Pure Data Parallelism (DP4×TP1 or DP8×TP1).
• Speculation: MTP4 or MTP8.
• Prefix Cache: Off for randomized data; on if prompts share templates.
• Outcome: 7,000–9,200 tok/s on 4–8 B300 GPUs.

Scenario C: Document Analysis & Catalog Extraction

Workload: 2k–8k token context, common system prompt, structured repetitive outputs.

• Topology: Balanced DP4×TP2 on 8 GPUs, or DP4×TP1 on 4 GPUs.
• Speculation: Suffix4 (Prefix + Suffix cache).
• Prefix Cache: Mandatory (85%+ hit rate saves massive prefill).
• Outcome: 9,000–9,800 tok/s with 99.7% draft acceptance.

Scenario D: Cluster Sizing & Hardware Economics

Decision: Choosing between renting a 4× B300 node vs. an 8× B300 node.

• Guideline: Start with 4× B300. It delivered 89–93% of 8-GPU throughput at c=256.
• When to buy 8 GPUs: Only after measuring your own load above 256 streams; we did not test beyond that.
• ROI: 4× B300 costs half as much per hour and gave up 7–11% of throughput at c=256 in our tests.

Configuring These Knobs in g factor: The main options from this benchmark (data and tensor parallelism, MTP4 or suffix speculation, prefix caching) are exposed in our CLI, SDK and Cloud Deploy interface, and we use the same stack for a private LLM deployment in your own VPC: gft deploy cloud Qwen/Qwen3.8-27B --dp 4 --tp 1 --speculation mtp4 --prefix-cache

All custom stack benchmarks conducted on dedicated NVIDIA B300 SXM6 AC nodes in Nebius uk-south1, September 20–21, 2026. AIPerf 0.12.0; our stack over in-cluster HTTP, Fireworks over HTTPS. Each cell is the mean of 3 repeats on a warm engine, not 3 cold starts. Our stack served about 300,000 measured requests; the 8-GPU run had 12 request errors in 3 cells, which we report as failed instead of averaging.

---

Applying this to your own models? We deploy and tune open models on teams' own GPUs, in their VPC or on-premises. Private LLM deployment.

Originally published at g-ftech.com.