DEV Community: Nick K

Kimi K3 Is 2.8T Parameters. That’s Not the Hardest Part of Serving It.

Nick K — Sun, 16 Aug 2026 18:12:21 +0000

When I first looked at Kimi K3, the obvious number was 2.8 trillion parameters.

That sounds like the whole deployment story.

It isn't.

After digging through the checkpoint, the architecture, and the current vLLM serving recipes, I think the more interesting story is this:

Kimi K3 is a good example of how modern LLM deployment is becoming a memory-layout and systems-engineering problem, not simply a parameter-count problem.

Kimi K3 is a Mixture-of-Experts model with:

2.8T total parameters
roughly 104B active parameters per token
896 routed experts
16 experts selected per token
native multimodal support
a 1,048,576-token context window

And yet the released model is nowhere near a hypothetical 5.6 TB BF16 checkpoint.

The reason is one of the first interesting deployment details.

The model already arrives quantized

Kimi K3 was trained to use MXFP4 weights with MXFP8 activations.

That matters.

The public checkpoint is roughly 1.56 TB, so you are not taking a giant BF16 model and deciding afterward whether to quantize it enough to make deployment practical.

The low-precision representation is part of the model's intended inference path.

That's a meaningful shift.

For infrastructure planning, the question becomes less:

"Which community quant should I download?"

and more:

"Which hardware can execute the model's native format efficiently?"

That immediately pushes you toward newer accelerators with the memory capacity and kernels required for MXFP4.

Sparse compute does not mean sparse storage

This is probably the easiest thing to misunderstand about large MoE models.

Kimi K3 only activates around 104B parameters for each token.

That's great for compute.

But it does not mean you only need enough GPU memory for 104B parameters.

The other experts still exist.

Their weights still have to be available.

So there are really two different numbers to think about:

Compute per token: ~104B active parameters.

Weights that need to live somewhere: the full ~1.56 TB checkpoint.

That distinction is why a 2.8T MoE can have reasonable per-token compute while still requiring datacenter-scale hardware to self-host.

The current practical floor is still an 8-GPU machine

This is where the model stops looking like something you casually spin up on a spare inference server.

Current vLLM guidance starts with 8× GB300 on NVIDIA.

For AMD, the documented ROCm path starts with 8× MI355X or MI350X.

vLLM's launch documentation also describes 16× B200 as a supported configuration.

So even though the model is heavily compressed, we're still talking about roughly two terabytes of aggregate accelerator memory before you start thinking seriously about serving headroom.

And that last part matters.

Weight capacity is only the beginning.

A 1M context window changes the memory conversation

Kimi K3 supports more than one million tokens of context.

It's tempting to see that number and immediately launch vLLM with:

--max-model-len 1048576

That's probably not how most production deployments should start.

Every context token has a serving cost.

Longer sequences mean more cache state, lower concurrency, and less room for simultaneous requests.

Kimi K3 does make long context more interesting than a conventional Transformer because its architecture isn't simply one giant stack of ordinary full-attention layers.

It uses a hybrid design:

69 Kimi Delta Attention layers + 24 Gated MLA layers.

Kimi Delta Attention maintains recurrent state rather than allowing a conventional KV cache to grow identically at every layer.

The periodic full-attention layers still need their own KV state.

That means vLLM has to manage two different kinds of cache state inside the same model.

This isn't just an architecture curiosity.

It affects how prefix caching, scheduling, memory allocation, and long-context serving actually work.

Prefix caching gets weird with recurrent attention

Normal prefix caching is conceptually simple.

If multiple requests share:

system prompt
+
large common document
+
user-specific question

you don't want to recompute the common prefix every time.

With a normal Transformer, you can reuse cached KV blocks.

Kimi K3 complicates this because the KDA layers maintain recurrent state while the full-attention layers maintain regular token-level KV state.

vLLM therefore had to make its cache manager understand both.

This is one of the things I find more interesting about Kimi K3 than the 2.8T headline.

New model architectures are increasingly forcing serving engines to become architecture-aware systems rather than generic "load weights and run attention" frameworks.

The basic vLLM command is surprisingly normal

Once the infrastructure is ready, the user-facing part still looks familiar:

vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

That gives you an OpenAI-compatible endpoint.

The complexity is mostly underneath:

MXFP4 kernels
expert routing
hybrid attention
prefix caching
tensor/expert parallelism
GPU-to-GPU communication
multimodal input handling
tool-call parsing

Which is exactly what an inference engine should hide from the application developer.

The most interesting optimization might be speculative decoding

For interactive applications, fitting the model isn't enough.

A model can technically run and still feel painfully slow to users.

vLLM's Kimi K3 work includes support for DSpark speculative decoding.

Instead of asking the full 2.8T model to generate every next token sequentially, a smaller draft model proposes several candidates and Kimi K3 verifies them.

The published GB300 benchmark is pretty dramatic:

Without DSpark: 118 output tokens/sec/user

With DSpark: 370 output tokens/sec/user

That's roughly a 3.14× improvement in that single-user benchmark.

The speculative configuration uses seven proposed tokens:

--speculative-config '{
  "model":"Inferact/Kimi-K3-DSpark",
  "method":"dspark",
  "num_speculative_tokens":7,
  "attention_backend":"FLASHINFER_MLA",
  "draft_sample_method":"probabilistic",
  "rejection_sample_method":"block"
}'

To me, this is an important part of the deployment story.

We spend a lot of time comparing quantization formats and GPU memory.

But once a huge model already fits, decode strategy can have a bigger effect on user experience than shaving another few percent from the checkpoint.

Production Kimi K3 is really a topology problem

An 8-GPU box answers:

Can I load and serve this model?

It doesn't automatically answer:

Can I serve hundreds of agent sessions economically?

At higher traffic levels, Kimi K3 starts becoming a cluster architecture problem.

You have several dimensions to work with:

Tensor parallelism

Split model operations across GPUs.

Expert parallelism

Distribute MoE experts across devices.

Data parallelism

Run additional replicas for concurrency.

Prefill/decode disaggregation

Let different GPU groups specialize in processing prompts and generating tokens.

Once you cross machine boundaries, networking becomes part of model performance too.

vLLM's K3 recipes explicitly distinguish between high-bandwidth NVLink-style deployments and RDMA-connected multi-node setups.

At that point, saying:

"This cluster has enough VRAM"

isn't enough.

You need to know where that VRAM is and how quickly the GPUs can communicate.

What I would decide before renting GPUs

If I were planning a Kimi K3 deployment today, I wouldn't start by asking "what is the cheapest cluster with 1.56 TB of VRAM?"

I'd start with four workload questions.

1. How much context do I actually need?

If requests normally stay below 32K or 64K, don't reserve resources as though every request will consume 1M tokens.

2. Is the workload latency-sensitive?

Interactive coding agents may benefit enormously from speculative decoding.

Batch workloads may care more about aggregate throughput.

3. How much concurrency do I expect?

One developer testing K3 and a production agent platform serving hundreds of sessions are completely different deployments.

4. Am I optimizing for minimum hardware or production efficiency?

The smallest cluster that loads a model isn't necessarily the cluster with the best cost per token.

That's especially true for MoE models where communication and expert routing start dominating scaling decisions.

My main takeaway

Kimi K3 is interesting partly because it's enormous.

But the more useful lesson is what it says about the next generation of open models.

We are moving from:

parameter count
→ precision
→ VRAM
→ launch server

toward something more like:

model architecture
→ native numerical format
→ memory topology
→ cache architecture
→ parallelism strategy
→ interconnect
→ speculative decoding
→ workload-specific context

That's a much more complicated deployment stack.

It's also a much more interesting one.

A 2.8T-parameter open-weight model would have sounded almost absurd to self-host not long ago.

Today, it can be served through an OpenAI-compatible vLLM endpoint.

You just need a couple of terabytes of very fast GPU memory first. :)

I put together a more infrastructure-focused breakdown with the checkpoint sizes, verified GPU layouts, vLLM configurations, and deployment recipes here:

Full Kimi K3 deployment guide:

https://blog.gpus.market/deploying-kimi-k3-with-vllm-verified-gpu-pods-quants-and-serving-recipes

Deploying Qwen3.8-2.4T-A95B with vLLM: Verified GPU Pods, Quants, and Serving Recipes

Nick K — Sat, 15 Aug 2026 03:19:40 +0000

Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B parameters active for each token. If you're planning to self-host it, the first thing to know is that this is a genuinely large distributed model: even the low-precision checkpoints are measured in terabytes.

The official open checkpoint is:

Qwen/Qwen3.8-2.4T-A95B

The model has 512 routed experts and selects 10 of them per token alongside one shared expert. Its 92-layer backbone mixes 69 Gated DeltaNet linear-attention layers with 23 full-attention layers, with full attention appearing every fourth layer.

Native context is 262,144 tokens, with an extended configuration available up to roughly 1.01 million tokens.

The open checkpoint is text-only and always uses reasoning. This is different from Qwen's hosted Qwen3.8-Max service, which adds features such as vision input and non-thinking mode.

For GPU deployment, the main decision is not whether 2.4T parameters will somehow fit. It is which precision format gives you a documented configuration on the hardware you actually have.

Start with the checkpoint that matches your GPUs

The practical options today are:

Your GPUs	Checkpoint	Documented setup
8× B300	`Inferact/Qwen3.8-2.4T-A95B-NVFP4`	TP8
8× GB300	`Inferact/Qwen3.8-2.4T-A95B-NVFP4`	TP8 across two NVL4 trays
16× B300	`Qwen/Qwen3.8-2.4T-A95B-FP8`	TP16
16× GB300	`Qwen/Qwen3.8-2.4T-A95B-FP8`	TP16
12× GB300	`Qwen/Qwen3.8-2.4T-A95B-FP8`	TP4 × PP3
8× MI355X	`Inferact/Qwen3.8-2.4T-A95B-MXFP4`	TP8

The full BF16 checkpoint is roughly 4.45 TiB. The official FP8 version is around 2.27 TiB, while the NVFP4 checkpoint used in the NVIDIA eight-GPU recipe is around 1.32 TiB.

That is why NVFP4 is the most approachable NVIDIA deployment if your goal is simply to get Qwen3.8 running without moving immediately to a 16-GPU cluster.

H100, H200, A100, B200 and smaller GPU configurations are not included here. Current vLLM material contains sizing information for some of those GPUs, but not equivalent end-to-end serving recipes. This guide does not turn memory estimates into deployment claims.

Before starting the model

The current vLLM recipe recommends a recent nightly build rather than an older stable release.

Create an environment with:

uv venv
source .venv/bin/activate

uv pip install -U vllm \
  --extra-index-url https://wheels.vllm.ai/nightly

uv pip install -U "transformers>=5.4.0"

Once you have a working build, pin it. Qwen3.8 relies on recently added model and kernel support, so continuously upgrading a production server to whatever nightly happens to be current is unnecessary risk.

Running Qwen3.8 on 8× B300 or GB300

For NVIDIA, the smallest documented configuration uses:

Inferact/Qwen3.8-2.4T-A95B-NVFP4

with eight GPUs.

Start with:

vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \
  --tensor-parallel-size 8 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

This configuration applies to an eight-GPU B300 system and is also documented across eight GB300 GPUs spanning two NVL4 trays.

vLLM's optimized NVFP4 path also uses:

--linear-backend flashinfer_cutedsl

but that flag depends on having the matching FlashInfer environment. Add it when reproducing the corresponding vLLM container/software stack rather than assuming every vLLM installation has the required backend.

Keep MTP enabled when you benchmark

Qwen3.8 contains a built-in Multi-Token Prediction head. vLLM's published results show that using three speculative tokens can make a substantial difference.

On its low-latency tests:

FP8 TP16
Without MTP: 130 output tok/s/user
MTP-3:       307 output tok/s/user

NVFP4 TP8
Without MTP: 133 output tok/s/user
MTP-3:       304 output tok/s/user

Those are measurements from vLLM's hardware and workload, not promised performance for every server. They do make MTP-3 worth testing from the beginning:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Using only one speculative token performed much less convincingly in the same testing. vLLM measured 64.8% acceptance for MTP-1, and at higher concurrency the extra speculative work could actually hurt throughput.

Running the official FP8 checkpoint

If you want Qwen's official FP8 checkpoint:

Qwen/Qwen3.8-2.4T-A95B-FP8

the documented low-latency configuration moves to 16 GPUs.

On B300, that normally means two eight-GPU nodes using TP16.

The head node runs:

vllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_ADDR \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The second node uses the same distributed configuration with:

--node-rank 1 \
--headless

Only rank 0 should expose the API server.

There is also a verified 12× GB300 FP8 configuration using:

TP4 × PP3

Why not TP12? Qwen3.8 has 64 full-attention heads, so its tensor-parallel size needs to divide 64. TP12 does not.

vLLM verified the TP4 × PP3 layout on 12 GB300 GPUs, including model loading, CUDA graph capture and generation. It is a useful option when you have three four-GPU GB300 trays, although the ordinary TP8 and TP16 setups remain simpler.

Running it on AMD

The documented AMD route uses:

Inferact/Qwen3.8-2.4T-A95B-MXFP4

on 8× MI355X.

vllm serve Inferact/Qwen3.8-2.4T-A95B-MXFP4 \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

One difference from the NVIDIA command is worth preserving: the current vLLM recipe does not recommend blindly adding FP8 KV cache to this ROCm setup.

Do not assume:

--kv-cache-dtype fp8

works with every ROCm/vLLM combination. Add it only after confirming it on the exact software build running on the node.

AMD also publishes:

amd/Qwen3.8-2.4T-A95B-Quark-MXFP4

for MI350 and MI355 hardware.

In that conversion, routed experts use OCP MXFP4 while components such as attention, routers, the shared expert, LM head and MTP layer remain at higher precision.

AMD's published GSM8K reproduction reported:

FP8 baseline: 97.49
MXFP4:        97.49

That result was reproduced with SGLang at TP8 on MI35x hardware. It is evidence for the quality of AMD's quantization, not a vLLM throughput benchmark.

How much context should you actually allocate?

The open checkpoint natively supports:

262,144 tokens

and can be extended to about:

1,010,000 tokens

with vLLM:

export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

and:

--max-model-len 1010000 \
--hf-overrides '{"max_position_embeddings":1010000}'

There is little reason to enable 1M context automatically.

Long context consumes memory that could otherwise be used for concurrent requests. vLLM's own NVFP4 testing illustrates the tradeoff: a configuration sized around the 262K window could hold roughly 25 concurrent requests in the available KV-cache budget, while a much shorter workload around 8K input plus 1K output allowed hundreds.

If you're serving repository-scale coding agents, very long context may be worth the cost. If most requests are 10K or 20K tokens, reserving for one million tokens mostly reduces how many users the GPUs can serve at once.

Set:

--max-model-len

for the workload you actually expect.

Reasoning and tool calling

Qwen3.8 is a reasoning model. The open checkpoint does not expose a normal non-thinking mode.

It supports three reasoning-effort settings:

xhigh
medium
low

xhigh is the default.

Qwen also preserves thinking across turns by default, which matters for agentic sessions where earlier reasoning and tool interactions are part of the ongoing context.

Start vLLM with:

--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder

Qwen's recommended generation settings include:

temperature = 1.0
top_p = 0.95
top_k = 20
min_p = 0.0
presence_penalty = 0.0
repetition_penalty = 1.0

A normal OpenAI-compatible client looks like:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
    timeout=3600,
)

response = client.chat.completions.create(
    model="Inferact/Qwen3.8-2.4T-A95B-NVFP4",
    messages=[
        {
            "role": "user",
            "content": "Review this distributed queue design and identify its failure modes."
        }
    ],
    reasoning_effort="medium",
    temperature=1.0,
    top_p=0.95,
    max_tokens=16384,
)

print(response.choices[0].message)

For hard coding and agent tasks, leave enough output budget for the reasoning trace. An aggressive max_tokens limit can terminate the generation before the model reaches its final answer.

Loading a model this large

At Qwen3.8's scale, startup time becomes part of operating the server.

vLLM's NVFP4 testing found that:

--load-format fastsafetensors \
--safetensors-load-strategy lazy

reduced model load time from 545 seconds to 306 seconds on the shared storage used in that test.

That result will vary with storage performance, but persistent model storage is clearly preferable to downloading or repeatedly copying a terabyte-scale checkpoint whenever a machine restarts.

vLLM also uses a larger engine startup timeout:

export VLLM_ENGINE_READY_TIMEOUT_S=3600

And don't make your own readiness decision solely from whether the process is alive. Send a small request to:

/v1/chat/completions

and verify that the model can actually generate.

Which GPU setup should you use?

If you're renting GPUs specifically for Qwen3.8-2.4T-A95B, the current documented choices are fairly straightforward.

8× B300 + NVFP4 is the simplest NVIDIA configuration in the current vLLM recipe.

8× GB300 + NVFP4 gives you the equivalent Grace Blackwell path across two NVL4 trays.

Use 16× B300 or GB300 + FP8 when you specifically want the official FP8 checkpoint and have enough hardware for TP16.

If you have three GB300 trays, 12× GB300 FP8 with TP4 × PP3 is an upstream-verified alternative.

On AMD, use 8× MI355X + MXFP4.

H100, H200, A100, B200 and smaller GPU counts are intentionally not recommended here. There may be enough aggregate memory in some of those configurations, but the current sources do not provide the same reproducible end-to-end deployment evidence.

For Qwen3.8, choosing the GPUs is only part of the deployment. The exact quant, tensor-parallel layout, KV-cache budget, vLLM build, network topology and MTP configuration all affect whether the resulting server is useful once real traffic starts hitting it.

Sources

Qwen

Qwen3.8-2.4T-A95B official model card

https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B
Official FP8 checkpoint

https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8

vLLM

Official Qwen3.8-2.4T-A95B deployment recipe

https://recipes.vllm.ai/Qwen/Qwen3.8-2.4T-A95B
vLLM Qwen3.8 launch notes and benchmarks

https://vllm.ai/blog/2026-08-12-qwen3.8

NVIDIA / Inferact

Qwen3.8 NVFP4 checkpoint

https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4
Qwen3.8 MXFP4 checkpoint

https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4

AMD

AMD Qwen3.8 Quark MXFP4 checkpoint https://huggingface.co/amd/Qwen3.8-2.4T-A95B-Quark-MXFP4