When I first looked at Kimi K3, the obvious number was 2.8 trillion parameters.
That sounds like the whole deployment story.
It isn't.
After digging through the checkpoint, the architecture, and the current vLLM serving recipes, I think the more interesting story is this:
Kimi K3 is a good example of how modern LLM deployment is becoming a memory-layout and systems-engineering problem, not simply a parameter-count problem.
Kimi K3 is a Mixture-of-Experts model with:
- 2.8T total parameters
- roughly 104B active parameters per token
- 896 routed experts
- 16 experts selected per token
- native multimodal support
- a 1,048,576-token context window
And yet the released model is nowhere near a hypothetical 5.6 TB BF16 checkpoint.
The reason is one of the first interesting deployment details.
The model already arrives quantized
Kimi K3 was trained to use MXFP4 weights with MXFP8 activations.
That matters.
The public checkpoint is roughly 1.56 TB, so you are not taking a giant BF16 model and deciding afterward whether to quantize it enough to make deployment practical.
The low-precision representation is part of the model's intended inference path.
That's a meaningful shift.
For infrastructure planning, the question becomes less:
"Which community quant should I download?"
and more:
"Which hardware can execute the model's native format efficiently?"
That immediately pushes you toward newer accelerators with the memory capacity and kernels required for MXFP4.
Sparse compute does not mean sparse storage
This is probably the easiest thing to misunderstand about large MoE models.
Kimi K3 only activates around 104B parameters for each token.
That's great for compute.
But it does not mean you only need enough GPU memory for 104B parameters.
The other experts still exist.
Their weights still have to be available.
So there are really two different numbers to think about:
Compute per token: ~104B active parameters.
Weights that need to live somewhere: the full ~1.56 TB checkpoint.
That distinction is why a 2.8T MoE can have reasonable per-token compute while still requiring datacenter-scale hardware to self-host.
The current practical floor is still an 8-GPU machine
This is where the model stops looking like something you casually spin up on a spare inference server.
Current vLLM guidance starts with 8× GB300 on NVIDIA.
For AMD, the documented ROCm path starts with 8× MI355X or MI350X.
vLLM's launch documentation also describes 16× B200 as a supported configuration.
So even though the model is heavily compressed, we're still talking about roughly two terabytes of aggregate accelerator memory before you start thinking seriously about serving headroom.
And that last part matters.
Weight capacity is only the beginning.
A 1M context window changes the memory conversation
Kimi K3 supports more than one million tokens of context.
It's tempting to see that number and immediately launch vLLM with:
--max-model-len 1048576
That's probably not how most production deployments should start.
Every context token has a serving cost.
Longer sequences mean more cache state, lower concurrency, and less room for simultaneous requests.
Kimi K3 does make long context more interesting than a conventional Transformer because its architecture isn't simply one giant stack of ordinary full-attention layers.
It uses a hybrid design:
69 Kimi Delta Attention layers + 24 Gated MLA layers.
Kimi Delta Attention maintains recurrent state rather than allowing a conventional KV cache to grow identically at every layer.
The periodic full-attention layers still need their own KV state.
That means vLLM has to manage two different kinds of cache state inside the same model.
This isn't just an architecture curiosity.
It affects how prefix caching, scheduling, memory allocation, and long-context serving actually work.
Prefix caching gets weird with recurrent attention
Normal prefix caching is conceptually simple.
If multiple requests share:
system prompt
+
large common document
+
user-specific question
you don't want to recompute the common prefix every time.
With a normal Transformer, you can reuse cached KV blocks.
Kimi K3 complicates this because the KDA layers maintain recurrent state while the full-attention layers maintain regular token-level KV state.
vLLM therefore had to make its cache manager understand both.
This is one of the things I find more interesting about Kimi K3 than the 2.8T headline.
New model architectures are increasingly forcing serving engines to become architecture-aware systems rather than generic "load weights and run attention" frameworks.
The basic vLLM command is surprisingly normal
Once the infrastructure is ready, the user-facing part still looks familiar:
vllm serve moonshotai/Kimi-K3 \
--tensor-parallel-size 8 \
--trust-remote-code \
--load-format fastsafetensors \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser kimi_k3 \
--reasoning-parser kimi_k3
That gives you an OpenAI-compatible endpoint.
The complexity is mostly underneath:
- MXFP4 kernels
- expert routing
- hybrid attention
- prefix caching
- tensor/expert parallelism
- GPU-to-GPU communication
- multimodal input handling
- tool-call parsing
Which is exactly what an inference engine should hide from the application developer.
The most interesting optimization might be speculative decoding
For interactive applications, fitting the model isn't enough.
A model can technically run and still feel painfully slow to users.
vLLM's Kimi K3 work includes support for DSpark speculative decoding.
Instead of asking the full 2.8T model to generate every next token sequentially, a smaller draft model proposes several candidates and Kimi K3 verifies them.
The published GB300 benchmark is pretty dramatic:
Without DSpark: 118 output tokens/sec/user
With DSpark: 370 output tokens/sec/user
That's roughly a 3.14× improvement in that single-user benchmark.
The speculative configuration uses seven proposed tokens:
--speculative-config '{
"model":"Inferact/Kimi-K3-DSpark",
"method":"dspark",
"num_speculative_tokens":7,
"attention_backend":"FLASHINFER_MLA",
"draft_sample_method":"probabilistic",
"rejection_sample_method":"block"
}'
To me, this is an important part of the deployment story.
We spend a lot of time comparing quantization formats and GPU memory.
But once a huge model already fits, decode strategy can have a bigger effect on user experience than shaving another few percent from the checkpoint.
Production Kimi K3 is really a topology problem
An 8-GPU box answers:
Can I load and serve this model?
It doesn't automatically answer:
Can I serve hundreds of agent sessions economically?
At higher traffic levels, Kimi K3 starts becoming a cluster architecture problem.
You have several dimensions to work with:
Tensor parallelism
Split model operations across GPUs.
Expert parallelism
Distribute MoE experts across devices.
Data parallelism
Run additional replicas for concurrency.
Prefill/decode disaggregation
Let different GPU groups specialize in processing prompts and generating tokens.
Once you cross machine boundaries, networking becomes part of model performance too.
vLLM's K3 recipes explicitly distinguish between high-bandwidth NVLink-style deployments and RDMA-connected multi-node setups.
At that point, saying:
"This cluster has enough VRAM"
isn't enough.
You need to know where that VRAM is and how quickly the GPUs can communicate.
What I would decide before renting GPUs
If I were planning a Kimi K3 deployment today, I wouldn't start by asking "what is the cheapest cluster with 1.56 TB of VRAM?"
I'd start with four workload questions.
1. How much context do I actually need?
If requests normally stay below 32K or 64K, don't reserve resources as though every request will consume 1M tokens.
2. Is the workload latency-sensitive?
Interactive coding agents may benefit enormously from speculative decoding.
Batch workloads may care more about aggregate throughput.
3. How much concurrency do I expect?
One developer testing K3 and a production agent platform serving hundreds of sessions are completely different deployments.
4. Am I optimizing for minimum hardware or production efficiency?
The smallest cluster that loads a model isn't necessarily the cluster with the best cost per token.
That's especially true for MoE models where communication and expert routing start dominating scaling decisions.
My main takeaway
Kimi K3 is interesting partly because it's enormous.
But the more useful lesson is what it says about the next generation of open models.
We are moving from:
parameter count
→ precision
→ VRAM
→ launch server
toward something more like:
model architecture
→ native numerical format
→ memory topology
→ cache architecture
→ parallelism strategy
→ interconnect
→ speculative decoding
→ workload-specific context
That's a much more complicated deployment stack.
It's also a much more interesting one.
A 2.8T-parameter open-weight model would have sounded almost absurd to self-host not long ago.
Today, it can be served through an OpenAI-compatible vLLM endpoint.
You just need a couple of terabytes of very fast GPU memory first. :)
I put together a more infrastructure-focused breakdown with the checkpoint sizes, verified GPU layouts, vLLM configurations, and deployment recipes here:
Full Kimi K3 deployment guide:
https://blog.gpus.market/deploying-kimi-k3-with-vllm-verified-gpu-pods-quants-and-serving-recipes
Top comments (1)
The topology point is the part I would weight most here. Once prefill and decode split across nodes, the constraint is less raw VRAM and more variance in the slowest path, especially for agent workloads with long context and uneven tool waits. Spec sheets don't price that very well.