Deploying Qwen3.8-2.4T-A95B with vLLM: Verified GPU Pods, Quants, and Serving Recipes

Nick K — Sat, 15 Aug 2026 03:19:40 +0000

Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B parameters active for each token. If you're planning to self-host it, the first thing to know is that this is a genuinely large distributed model: even the low-precision checkpoints are measured in terabytes.

The official open checkpoint is:

Qwen/Qwen3.8-2.4T-A95B

The model has 512 routed experts and selects 10 of them per token alongside one shared expert. Its 92-layer backbone mixes 69 Gated DeltaNet linear-attention layers with 23 full-attention layers, with full attention appearing every fourth layer.

Native context is 262,144 tokens, with an extended configuration available up to roughly 1.01 million tokens.

The open checkpoint is text-only and always uses reasoning. This is different from Qwen's hosted Qwen3.8-Max service, which adds features such as vision input and non-thinking mode.

For GPU deployment, the main decision is not whether 2.4T parameters will somehow fit. It is which precision format gives you a documented configuration on the hardware you actually have.

Start with the checkpoint that matches your GPUs

The practical options today are:

Your GPUs	Checkpoint	Documented setup
8× B300	`Inferact/Qwen3.8-2.4T-A95B-NVFP4`	TP8
8× GB300	`Inferact/Qwen3.8-2.4T-A95B-NVFP4`	TP8 across two NVL4 trays
16× B300	`Qwen/Qwen3.8-2.4T-A95B-FP8`	TP16
16× GB300	`Qwen/Qwen3.8-2.4T-A95B-FP8`	TP16
12× GB300	`Qwen/Qwen3.8-2.4T-A95B-FP8`	TP4 × PP3
8× MI355X	`Inferact/Qwen3.8-2.4T-A95B-MXFP4`	TP8

The full BF16 checkpoint is roughly 4.45 TiB. The official FP8 version is around 2.27 TiB, while the NVFP4 checkpoint used in the NVIDIA eight-GPU recipe is around 1.32 TiB.

That is why NVFP4 is the most approachable NVIDIA deployment if your goal is simply to get Qwen3.8 running without moving immediately to a 16-GPU cluster.

H100, H200, A100, B200 and smaller GPU configurations are not included here. Current vLLM material contains sizing information for some of those GPUs, but not equivalent end-to-end serving recipes. This guide does not turn memory estimates into deployment claims.

Before starting the model

The current vLLM recipe recommends a recent nightly build rather than an older stable release.

Create an environment with:

uv venv
source .venv/bin/activate

uv pip install -U vllm \
  --extra-index-url https://wheels.vllm.ai/nightly

uv pip install -U "transformers>=5.4.0"

Once you have a working build, pin it. Qwen3.8 relies on recently added model and kernel support, so continuously upgrading a production server to whatever nightly happens to be current is unnecessary risk.

Running Qwen3.8 on 8× B300 or GB300

For NVIDIA, the smallest documented configuration uses:

Inferact/Qwen3.8-2.4T-A95B-NVFP4

with eight GPUs.

Start with:

vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \
  --tensor-parallel-size 8 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

This configuration applies to an eight-GPU B300 system and is also documented across eight GB300 GPUs spanning two NVL4 trays.

vLLM's optimized NVFP4 path also uses:

--linear-backend flashinfer_cutedsl

but that flag depends on having the matching FlashInfer environment. Add it when reproducing the corresponding vLLM container/software stack rather than assuming every vLLM installation has the required backend.

Keep MTP enabled when you benchmark

Qwen3.8 contains a built-in Multi-Token Prediction head. vLLM's published results show that using three speculative tokens can make a substantial difference.

On its low-latency tests:

FP8 TP16
Without MTP: 130 output tok/s/user
MTP-3:       307 output tok/s/user

NVFP4 TP8
Without MTP: 133 output tok/s/user
MTP-3:       304 output tok/s/user

Those are measurements from vLLM's hardware and workload, not promised performance for every server. They do make MTP-3 worth testing from the beginning:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Using only one speculative token performed much less convincingly in the same testing. vLLM measured 64.8% acceptance for MTP-1, and at higher concurrency the extra speculative work could actually hurt throughput.

Running the official FP8 checkpoint

If you want Qwen's official FP8 checkpoint:

Qwen/Qwen3.8-2.4T-A95B-FP8

the documented low-latency configuration moves to 16 GPUs.

On B300, that normally means two eight-GPU nodes using TP16.

The head node runs:

vllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_ADDR \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The second node uses the same distributed configuration with:

--node-rank 1 \
--headless

Only rank 0 should expose the API server.

There is also a verified 12× GB300 FP8 configuration using:

TP4 × PP3

Why not TP12? Qwen3.8 has 64 full-attention heads, so its tensor-parallel size needs to divide 64. TP12 does not.

vLLM verified the TP4 × PP3 layout on 12 GB300 GPUs, including model loading, CUDA graph capture and generation. It is a useful option when you have three four-GPU GB300 trays, although the ordinary TP8 and TP16 setups remain simpler.

Running it on AMD

The documented AMD route uses:

Inferact/Qwen3.8-2.4T-A95B-MXFP4

on 8× MI355X.

vllm serve Inferact/Qwen3.8-2.4T-A95B-MXFP4 \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

One difference from the NVIDIA command is worth preserving: the current vLLM recipe does not recommend blindly adding FP8 KV cache to this ROCm setup.

Do not assume:

--kv-cache-dtype fp8

works with every ROCm/vLLM combination. Add it only after confirming it on the exact software build running on the node.

AMD also publishes:

amd/Qwen3.8-2.4T-A95B-Quark-MXFP4

for MI350 and MI355 hardware.

In that conversion, routed experts use OCP MXFP4 while components such as attention, routers, the shared expert, LM head and MTP layer remain at higher precision.

AMD's published GSM8K reproduction reported:

FP8 baseline: 97.49
MXFP4:        97.49

That result was reproduced with SGLang at TP8 on MI35x hardware. It is evidence for the quality of AMD's quantization, not a vLLM throughput benchmark.

How much context should you actually allocate?

The open checkpoint natively supports:

262,144 tokens

and can be extended to about:

1,010,000 tokens

with vLLM:

export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

and:

--max-model-len 1010000 \
--hf-overrides '{"max_position_embeddings":1010000}'

There is little reason to enable 1M context automatically.

Long context consumes memory that could otherwise be used for concurrent requests. vLLM's own NVFP4 testing illustrates the tradeoff: a configuration sized around the 262K window could hold roughly 25 concurrent requests in the available KV-cache budget, while a much shorter workload around 8K input plus 1K output allowed hundreds.

If you're serving repository-scale coding agents, very long context may be worth the cost. If most requests are 10K or 20K tokens, reserving for one million tokens mostly reduces how many users the GPUs can serve at once.

Set:

--max-model-len

for the workload you actually expect.

Reasoning and tool calling

Qwen3.8 is a reasoning model. The open checkpoint does not expose a normal non-thinking mode.

It supports three reasoning-effort settings:

xhigh
medium
low

xhigh is the default.

Qwen also preserves thinking across turns by default, which matters for agentic sessions where earlier reasoning and tool interactions are part of the ongoing context.

Start vLLM with:

--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder

Qwen's recommended generation settings include:

temperature = 1.0
top_p = 0.95
top_k = 20
min_p = 0.0
presence_penalty = 0.0
repetition_penalty = 1.0

A normal OpenAI-compatible client looks like:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
    timeout=3600,
)

response = client.chat.completions.create(
    model="Inferact/Qwen3.8-2.4T-A95B-NVFP4",
    messages=[
        {
            "role": "user",
            "content": "Review this distributed queue design and identify its failure modes."
        }
    ],
    reasoning_effort="medium",
    temperature=1.0,
    top_p=0.95,
    max_tokens=16384,
)

print(response.choices[0].message)

For hard coding and agent tasks, leave enough output budget for the reasoning trace. An aggressive max_tokens limit can terminate the generation before the model reaches its final answer.

Loading a model this large

At Qwen3.8's scale, startup time becomes part of operating the server.

vLLM's NVFP4 testing found that:

--load-format fastsafetensors \
--safetensors-load-strategy lazy

reduced model load time from 545 seconds to 306 seconds on the shared storage used in that test.

That result will vary with storage performance, but persistent model storage is clearly preferable to downloading or repeatedly copying a terabyte-scale checkpoint whenever a machine restarts.

vLLM also uses a larger engine startup timeout:

export VLLM_ENGINE_READY_TIMEOUT_S=3600

And don't make your own readiness decision solely from whether the process is alive. Send a small request to:

/v1/chat/completions

and verify that the model can actually generate.

Which GPU setup should you use?

If you're renting GPUs specifically for Qwen3.8-2.4T-A95B, the current documented choices are fairly straightforward.

8× B300 + NVFP4 is the simplest NVIDIA configuration in the current vLLM recipe.

8× GB300 + NVFP4 gives you the equivalent Grace Blackwell path across two NVL4 trays.

Use 16× B300 or GB300 + FP8 when you specifically want the official FP8 checkpoint and have enough hardware for TP16.

If you have three GB300 trays, 12× GB300 FP8 with TP4 × PP3 is an upstream-verified alternative.

On AMD, use 8× MI355X + MXFP4.

H100, H200, A100, B200 and smaller GPU counts are intentionally not recommended here. There may be enough aggregate memory in some of those configurations, but the current sources do not provide the same reproducible end-to-end deployment evidence.

For Qwen3.8, choosing the GPUs is only part of the deployment. The exact quant, tensor-parallel layout, KV-cache budget, vLLM build, network topology and MTP configuration all affect whether the resulting server is useful once real traffic starts hitting it.

Sources

Qwen

Qwen3.8-2.4T-A95B official model card

https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B
Official FP8 checkpoint

https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8

vLLM

Official Qwen3.8-2.4T-A95B deployment recipe

https://recipes.vllm.ai/Qwen/Qwen3.8-2.4T-A95B
vLLM Qwen3.8 launch notes and benchmarks

https://vllm.ai/blog/2026-08-12-qwen3.8

NVIDIA / Inferact

Qwen3.8 NVFP4 checkpoint

https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4
Qwen3.8 MXFP4 checkpoint

https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4

AMD

AMD Qwen3.8 Quark MXFP4 checkpoint https://huggingface.co/amd/Qwen3.8-2.4T-A95B-Quark-MXFP4

DEV Community: Nick K