Kimi K3 Open Weights Are Here: How to Self-Host the 2.8T-Parameter Model (Hardware, vLLM, and Data Sovereignty)

Lola Lin — Mon, 27 Jul 2026 05:47:59 +0000

Kimi K3 Open Weights Are Here: How to Self-Host the 2.8T-Parameter Model

On July 27, 2026, Moonshot AI released the open weights for Kimi K3 -- a 2.8-trillion-parameter Mixture-of-Experts model that has been turning heads across the AI industry since its initial launch. The open-weight release means organizations can now deploy K3 on their own infrastructure, unlocking the same frontier-level reasoning capabilities without routing prompts through external APIs. This guide covers everything you need to know to get K3 running on your hardware, from downloading the weights to serving an OpenAI-compatible API.

What the Open-Weight Release Includes

Moonshot published the K3 weights on their Hugging Face organization under the Apache 2.0 license. The release includes:

MXFP4 quantized weights (~594 GB total download) -- the same format Moonshot uses internally, achieving near-lossless quality at 4-bit precision
Tokenizer and model config -- ready to load with minimal configuration
Kimi Delta Attention (KDA) reference implementation -- K3's novel attention mechanism that made the 128K effective context window possible without the quadratic memory cost of standard attention
Inference code examples for vLLM and SGLang

The weights are available in the moonshotai/Kimi-K3-MXFP4 repository on Hugging Face. Moonshot co-founders confirmed during a July 27 AMA on r/LocalLLaMA that this is the exact same quantization used for the hosted API, so self-hosted inference quality should be identical.

Hardware Requirements

K3 is a big model. Even with 4-bit quantization and MoE sparsity (only ~10% of parameters active per token), you need serious hardware. Here is what you are looking at:

Component	Minimum	Recommended	Notes
GPU Memory (total)	~700 GB	~1.4 TB	MXFP4 is ~594 GB; you need headroom for KV cache and activations
Accelerator Count	8x H100 (80 GB)	16x H100 / 8x H200	H200's 141 GB HBM makes a big difference
Native MXFP4 Hardware	N/A	8x Blackwell B200 / 4x AMD MI400	Blackwell and MI400 have native MXFP4 support -- ~2x throughput vs emulation
System RAM	1 TB	2 TB	For weight loading and preprocessing
Disk (NVMe)	1 TB free	2 TB free	Weights alone are ~600 GB; add space for checkpoints
Interconnect	400 Gbps (NVLINK / InfiniBand)	800 Gbps	Critical for tensor parallelism across nodes

The short version: you need at least 64 accelerators (8 nodes of 8x H100) if you are running at full precision with tensor parallelism across nodes, or a single node of 8x H200 if you can tolerate the MXFP4 quantization. For teams with Blackwell or MI400 access, a single 8-GPU node with native MXFP4 support can serve K3 at interactive latencies.

If that hardware list made you wince, you are not alone. Self-hosting K3 is an infrastructure commitment. For teams that want K3's capabilities without the hardware burden, managed API access is the pragmatic path.

Step 1: Downloading the Weights

Start by installing the Hugging Face Hub CLI and authenticating:

pip install huggingface_hub[hf_transfer]
huggingface-cli login

K3's weights are gated -- you need to accept the license agreement on the Hugging Face model page before downloading. Once accepted, pull the weights:

# Download the full MXFP4 weights (~594 GB)
huggingface-cli download moonshotai/Kimi-K3-MXFP4 \
  --local-dir /data/models/kimi-k3-mxfp4 \
  --local-dir-use-symlinks False \
  --resume-download

The download will take a while depending on your connection. On a 10 Gbps line, expect roughly 8-10 minutes. On a standard 1 Gbps corporate connection, budget 80-90 minutes. The --resume-download flag is your friend -- connection drops happen.

After downloading, verify integrity:

# Verify all shard files are present and match expected sizes
ls -lh /data/models/kimi-k3-mxfp4/
# You should see model-00001-of-000XX.safetensors files totaling ~594 GB

Step 2: vLLM Setup with KDA Attention

K3 uses Kimi Delta Attention (KDA) -- a sparse attention mechanism that Moonshot designed to make extremely long contexts practical. Standard attention has O(n^2) memory cost with respect to sequence length; KDA reduces this substantially, which is how K3 achieves its 128K effective context window.

vLLM 0.7.0+ includes native KDA support. Install the latest version:

pip install vllm>=0.7.0

If you are building from source (recommended for maximum performance on your specific hardware):

git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.7.2  # latest stable with KDA optimizations
pip install -e . --no-build-isolation

Create a vLLM serving configuration. Here is a production-ready setup for an 8x H200 node:

# k3_serve_config.py
from vllm import LLM, SamplingParams
from vllm.config import ModelConfig, CacheConfig, ParallelConfig

# K3-specific configuration
model_config = ModelConfig(
    model="/data/models/kimi-k3-mxfp4",
    tokenizer="/data/models/kimi-k3-mxfp4",
    tokenizer_mode="auto",
    trust_remote_code=True,  # required for KDA custom ops
    dtype="float8_e4m3fn",  # MXFP4 compatible
    max_model_len=131072,    # 128K context
    quantization="mxfp4",   # native MXFP4 support in vLLM 0.7+
)

# Memory and parallelism
cache_config = CacheConfig(
    block_size=16,
    gpu_memory_utilization=0.92,  # aggressive but KDA is memory-efficient
    swap_space=64,                 # 64 GB CPU swap for overflow
)

parallel_config = ParallelConfig(
    pipeline_parallel_size=1,
    tensor_parallel_size=8,  # one per H200 GPU
)

Key settings to understand:

trust_remote_code=True -- KDA is implemented as a custom CUDA kernel. You must trust the Moonshot repository to load these ops. Audit the code if your security policy requires it.
quantization="mxfp4" -- vLLM 0.7+ supports MXFP4 natively. On Blackwell hardware, this maps directly to the hardware format; on H100/H200, vLLM emulates it efficiently.
max_model_len=131072 -- the full 128K context. Lower this to 32768 or 65536 if you want to trade context length for higher throughput or smaller KV cache.
gpu_memory_utilization=0.92 -- KDA's memory efficiency means you can push GPU memory utilization higher than with standard attention models.

Step 3: Starting the OpenAI-Compatible Server

Launch the vLLM API server:

python -m vllm.entrypoints.openai.api_server \
    --model /data/models/kimi-k3-mxfp4 \
    --trust-remote-code \
    --dtype float8_e4m3fn \
    --quantization mxfp4 \
    --max-model-len 131072 \
    --tensor-parallel-size 8 \
    --gpu-memory-utilization 0.92 \
    --port 8000 \
    --host 0.0.0.0

Once the server starts (expect 30-60 seconds for weight loading and CUDA graph compilation), test it:

# Health check
curl http://localhost:8000/health

# Test completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k3",
    "messages": [{"role": "user", "content": "Explain the KDA attention mechanism in three sentences."}],
    "max_tokens": 512,
    "temperature": 0.7
  }'

The server exposes the standard /v1/chat/completions endpoint, so any OpenAI-compatible client works out of the box -- the OpenAI Python SDK, LangChain, LlamaIndex, Claude Code with a custom provider, Cursor, Continue, and hundreds of other tools.

Data Sovereignty: Why Self-Hosting Matters

For many organizations, the decision to self-host K3 is not just about cost -- it is about data sovereignty. When you send prompts to a cloud-hosted K3 API operated by a Chinese company, your data transits through and is processed on servers subject to Chinese data regulations, including the Personal Information Protection Law (PIPL), the Data Security Law (DSL), and the Cross-Border Data Transfer Security Assessment requirements.

This is not a hypothetical concern. Organizations in finance, healthcare, defense, legal, and any sector handling personally identifiable information face compliance obligations that make routing prompts through PRC-based infrastructure complex at best and impermissible at worst.

Self-hosting K3 on your own infrastructure -- whether on-premises or in a cloud region of your choice -- eliminates this concern entirely. Your prompts never leave your network. Your fine-tuning data stays yours. Your user conversations are governed by your jurisdiction's data protection framework, not someone else's.

This is the core tension in the current AI landscape: frontier models increasingly come from labs in jurisdictions with data laws that may conflict with your own. Self-hosting resolves it, but at the cost of significant infrastructure complexity.

What Moonshot Revealed in the r/LocalLLaMA AMA

The July 27 AMA on r/LocalLLaMA with Moonshot's co-founders was unusually candid. Over 40 questions were answered, covering architecture details, training decisions, and future plans. Highlights relevant to self-hosters:

KDA is not just about memory. The co-founders explained that KDA was designed primarily to address the "attention sink" problem -- where long-context models allocate disproportionate attention to early tokens. KDA's delta mechanism naturally counteracts this, improving retrieval accuracy at long ranges without post-hoc fixes like attention scaling.
H800 training cluster. K3 was trained on a cluster of H800 GPUs -- the export-controlled variant of the H100 with reduced interconnect bandwidth. This constraint actually shaped KDA's design: Moonshot needed an attention mechanism that performs well under limited inter-GPU bandwidth, which serendipitously makes K3 more amenable to self-hosting on non-NVIDIA interconnects.
Fine-tuning support is coming. The team confirmed they are working on an official LoRA fine-tuning recipe, expected later in Q3 2026. In the meantime, the community has already produced working LoRA adapters using the unsloth library with KDA compatibility patches.
Apache 2.0 means exactly that. Commercial use, modification, distribution -- all permitted. No additional restrictions. The co-founders explicitly stated they want K3 to become "the Linux of foundation models."

When Self-Hosting Makes Sense (and When It Does Not)

Self-hosting K3 is the right call when:

You have regulatory requirements that prohibit or complicate cloud API usage
You operate at scale where per-token API pricing exceeds your infrastructure costs (roughly 10B+ tokens/month for K3-class models)
You need fine-tuning or model modification beyond what hosted APIs offer
You have existing GPU capacity that is underutilized
Latency to external APIs is unacceptable for your use case (e.g., real-time applications)

Self-hosting is probably not the right call when:

Your team is small (under ~20 engineers) and cannot spare the DevOps bandwidth
Your usage is moderate (under ~1B tokens/month)
You need to move fast and cannot afford the 1-2 week setup and burn-in period
You want managed failover across multiple models without building it yourself

The Middle Ground: Managed Access via API Gateway

For teams that want K3's power without the infrastructure burden, there is a pragmatic middle ground. Services like TeamoRouter provide managed API access to K3 alongside 500+ other providers through a single API key. You swap one line of configuration and get:

Agentic Routing -- automatically selects the best model for each request based on capability requirements and cost
Competitive pricing -- aggregated demand means better rates than direct API contracts for most teams
No infrastructure to manage -- no GPU clusters, no vLLM configs, no CUDA driver updates
Built-in failover -- if one provider has an outage, requests route to alternatives transparently

The mental model: self-host for sovereignty, use an API gateway for convenience. Many teams do both -- self-hosted K3 for sensitive workloads, routed API access for everything else.

Quick-Start Checklist

Accept the license on moonshotai/Kimi-K3-MXFP4
Provision hardware with at least ~700 GB total GPU memory
Install vLLM 0.7.0+ with pip install vllm>=0.7.0
Download weights with huggingface-cli download
Configure --trust-remote-code, --quantization mxfp4, and --tensor-parallel-size matching your GPU count
Launch the OpenAI-compatible server on port 8000
Point your OpenAI-compatible clients at http://your-server:8000/v1

Self-hosting a 2.8T-parameter model is not trivial, but Moonshot's open-weight release, vLLM's mature KDA support, and the permissive Apache 2.0 license make it more accessible than anyone expected six months ago. Whether you self-host, use an API gateway, or do both, K3's capabilities are now available on your terms.

DEV Community: Lola Lin