How to Run Kimi K3 Locally (and When You Shouldn't)

Moonshot AI released the open weights for Kimi K3 on July 27, and the download counter on Hugging Face is already close to 100,000. The pitch is obvious: a 2.8 trillion parameter model that beat Claude Opus 4.8 on every benchmark Moonshot published, and now you can host it yourself.

Try Apidog today

The catch is also obvious once you look at the numbers. Full-precision inference needs 1.57 TB of disk. Even the released MXFP4 weights are a 594 GB download. This is a model you can own, but “local” means something different at this scale than it did for an 8B Llama.

This guide shows what it takes to run K3 on your own hardware, what the community has managed on consumer machines, and how to connect a self-hosted K3 endpoint to your API workflow with Apidog.

What you’re downloading

Before choosing a deployment path, understand the model shape. For the full background, see What Is Kimi K3?. The short version:

• 2.8T total parameters, 104B activated per token. K3 is a Mixture-of-Experts model with 896 experts. Each token routes through 16 selected experts plus 2 shared experts, so compute per token is lower than the total parameter count suggests.
• 93 layers. These include 69 Kimi Delta Attention (KDA) layers and 24 Gated MLA layers. KDA is what makes the 1-million-token context window practical.
• Native vision. A 401M-parameter MoonViT-V2 encoder handles text, image, and video input.
• MXFP4 weights and MXFP8 activations. Moonshot used quantization-aware training, so the 4-bit release is the intended serving format. Further compression has limited headroom.
• Thinking-only. K3 reasons before answering, with low, high, and max effort levels. There is no instant mode.

The weights are gated behind the Kimi K3 License in the Hugging Face repository. Accept the license before pulling the model with huggingface-cli.

At 1 Gbps, plan for roughly 80–90 minutes to download the 594 GB MXFP4 release.

Option 1: Serve K3 on datacenter-class hardware with vLLM or SGLang

Moonshot recommends vLLM, SGLang, and TokenSpeed. Because KDA prefill-cache support shipped in vLLM alongside the weights, vLLM is the most straightforward starting point.

Start a server with tensor parallelism across eight GPUs:

vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --max-model-len 131072

Use this as the initial deployment baseline:

1. Start with eight GPUs. Moonshot evaluated K3 on H20 clusters. In practice, an 8-GPU node with tensor parallelism is the minimum realistic configuration.
2. Set a conservative context limit first. K3 supports up to 1,048,576 tokens, but the KV cache at full context is around 27 GB by itself. Start with --max-model-len 131072 and raise it only when your workload requires it.
3. Use Moonshot’s sampling defaults. Start with temperature 1.0 and top-p 0.95. For agentic workloads, keep temperature at 1.0 and consider increasing top-p to 1.0.
4. Measure throughput before expanding context. B200-class hardware can exceed 100 tokens/s, but actual throughput depends on context length, concurrency, and cache use.

This is “local” in the data-sovereignty sense: you control the infrastructure, logs, and compliance boundary. It is not laptop-local, and quantization does not change that for interactive use.

Option 2: Run GGUF quants on a large workstation

Unsloth published GGUF conversions for llama.cpp. These dynamic quants are the practical way to shrink K3 below the official release size.

Quant	Size	What it means
UD-IQ1_M	~345 GB	The floor: aggressive 1-bit dynamic quantization.
UD-IQ1_S	~650 GB	Unsloth’s recommended balance point.
UD-Q4_K_XL	~1.55 TB	Near full precision.
UD-Q8_K_XL	~1.6 TB	Effectively lossless.

A useful capacity rule is:

RAM + VRAM should roughly equal the quantized model size.

llama.cpp can offload when memory is insufficient, but every missing gigabyte costs performance. A Mac Studio paired with a 128 GB machine, or a DGX Station, sits at the practical low end.

Run a GGUF model with the vision projector enabled:

./llama.cpp/llama-cli \
  --model unsloth/Kimi-K3-GGUF/UD-IQ1_S/Kimi-K3-UD-IQ1_M-00001-of-00015.gguf \
  --mmproj unsloth/Kimi-K3-GGUF/mmproj-F16.gguf \
  --temp 1.0 \
  --top-p 0.95

Before committing to this route:

• Verify that your combined RAM and VRAM can hold the selected quant.
• Keep the model files on fast local storage.
• Benchmark tokens per second with your real prompt lengths.
• Include --mmproj when you need image or video input support.

If your hardware cannot meet the memory target, do not force it. The best local LLMs of 2026 list includes open models that fit in 24–128 GB and can respond in real time. K3 at 1-bit on insufficient RAM cannot.

The M1 Max experiment: yes, but 16 seconds per token

A Hacker News thread documented K3 running on a 64 GB M1 Max by streaming weights from a 2 TB SSD instead of keeping them in memory.

It works, but the performance tradeoff is severe:

• K3 has roughly 115 GB of dense parameters that every token touches, plus around 25 GB of routed expert weights per token.
• The dense parameters alone exceed the machine’s RAM, so the SSD becomes very slow memory.
• Reported performance was around 16 seconds per token. Some configurations exceeded one minute per token.
• Disk throughput determines usability. M1-era SSDs are slower than current Apple silicon, and streaming experts over a network is slower again.

This is an interesting demonstration of MoE sparsity plus memory mapping, not a practical K3 workstation setup. If you want K3 output on a MacBook, free tiers or the hosted API are the better option.

Wire your local K3 into an API workflow

Whether you run vLLM or llama.cpp server mode, the result is an OpenAI-compatible HTTP endpoint on localhost. Treat it like any other API.

The same process used for testing local LLMs as APIs applies here.

1. Create an environment for the local endpoint

For vLLM, create an environment variable such as:

base_url = http://localhost:8000/v1

Use that variable in your requests instead of hard-coding the host. You can then switch between a local deployment and Moonshot’s hosted endpoint without rewriting requests.

2. Send an OpenAI-compatible chat completion request

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/Kimi-K3",
    "messages": [
      {
        "role": "user",
        "content": "Explain tensor parallelism in three bullet points."
      }
    ],
    "temperature": 1.0,
    "top_p": 0.95,
    "stream": true
  }'

3. Inspect the thinking stream

K3 is thinking-only, so responses include reasoning content before the final answer. Apidog’s SSE debugging view can render the stream as it arrives, making it easier to compare reasoning-effort levels and identify streaming issues.

4. Test response structure, latency, and usage fields

Do not validate LLM output only by reading a few answers. Add automated assertions for:

• Expected response schema
• Required message fields
• Streaming event format
• Latency budgets
• Token usage fields
• Error behavior under invalid requests

This catches regressions caused by quant changes, engine upgrades, or serving configuration updates before users report them.

5. Mock K3 while the model is loading

A 594 GB model can take time to load. Record representative responses once, then use a mock server for frontend and integration work while GPUs are busy.

Download Apidog to configure mocks and tests against any OpenAI-compatible server.

The request format matches the Kimi K3 API guide, so tests written for the hosted API can transfer directly to a local deployment.

So, should you run K3 locally?

Your situation	Recommendation
8+ GPU node with data-sovereignty or compliance requirements	Yes. Use vLLM with tensor parallelism and MXFP4 weights.
Workstation with 350 GB+ RAM/VRAM	Workable. Use Unsloth 1-bit GGUFs with tempered expectations.
64–128 GB Mac or PC	No. Expect seconds per token, not tokens per second.
You only need K3 in your product	Use the hosted API; it is OpenAI- and Anthropic-compatible.

K3’s open weights matter because you can audit, fine-tune, and self-host a frontier-class model—not because most developers should run it on local hardware.

For teams with the required infrastructure, the vLLM path is practical today. For everyone else, hosted access is more realistic. In either case, the endpoint is where the model meets your code: test schemas, inspect streams, and use mocks to keep development moving while the model thinks.