Thurmon Demich

Posted on Jul 30 • Originally published at bestgpuforllm.com

Can You Run Kimi K3 Locally? No — Here's the Exact Math

#kimik3 #localllm #moe #cloudgpu

From the Best GPU for LLM archive. The canonical version has interactive calculators, an up-to-date GPU comparison table, and live pricing.

The weights dropped on July 27. Moonshot put Kimi K3 on Hugging Face under a Modified MIT license, the download button works, and somewhere right now a person with an RTX 4090 is watching a 594GB MXFP4 checkpoint crawl toward a 2TB SSD, convinced that llama.cpp will sort out the rest. It won't. I want to kill this dream cleanly, with numbers, before you waste a weekend on it.

Short answer: no. Not on a 4090, not on a 5090, not on a Mac Studio, not on the 8x H100 box your startup rents. As of late July 2026, Kimi K3 is a multi-node deployment or it is an API call. There is no third option yet.

The memory math, tier by tier

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 896 experts. The native MXFP4 release is already aggressively compressed — about 594GB just to download — and the total weight footprint lands around 1.4TB once scales and runtime overhead are counted. Here's what that means against real hardware:

Tier	Memory you have	Kimi K3 needs	Coverage
RTX 4090	24GB	~594GB (MXFP4 weights alone)	~4% of the download
RTX 5090	32GB	~594GB	~5%
4x RTX 5090 rig	128GB	~594GB	~22%
Mac Studio 512GB	512GB unified	~594GB	short before overhead
8x H100 80GB (one node)	640GB	~1.4TB total footprint	not enough
Multi-node H100/H200 cluster	1.5TB+	~1.4TB	the actual floor

Sit with those middle rows for a second. A full DGX-class node — eight H100s, the machine most people picture when they say "serious AI hardware" — cannot hold the full weights. The realistic floor is multiple interconnected nodes of H100s or H200s. That's not a homelab. That's a rack with a networking budget.

"But it's MoE — only 16 experts are active!"

This is the myth doing the heavy lifting in every optimistic Reddit thread. Yes, K3 routes each token through just 16 of its 896 experts. And yes, that makes inference compute dramatically cheaper than a dense 2.8T model would be. But the router picks different experts every single token, which means every expert has to be sitting in memory, ready to fire. MoE sparsity saves FLOPs, not bytes. All 896 experts are resident, all the time.

Smaller MoE models like Kimi K2 get away with local deployment because their total quantized footprint squeezes into prosumer territory. K3's footprint is roughly 20x too large for that trick, and its native format is already 4-bit — there's no magic quant waiting to shrink it another 95%. Days after release, no full-fidelity GGUF port has been confirmed either, so the "wait for community quants" plan currently has nothing on the calendar that changes the math.

What you can actually do (three lanes)

Lane 1 — hosted API. Moonshot's own API, OpenRouter, Together, and Fireworks all serve K3, and day-one vLLM support meant hosted capacity spun up fast. If you want K3's quality, this is the honest path for anyone without a cluster line item.

Lane 2 — rent the cluster. If you genuinely need K3 under your own control — fine-tuning experiments, private data, weight surgery — rent multi-GPU capacity by the hour instead of pretending a workstation will do it. Our cloud GPU for LLM guide walks the provider math; for scale context on why datacenter cards are a different species, see RTX 5090 vs H100.

Lane 3 — self-host the K2 class instead. This is what your money actually buys locally. A 24GB card runs Kimi K2 at Q4 with real agent-grade reliability, and Kimi K2.7 Code is the current lighter self-host path in the same family. The best GPU for Kimi K2 guide covers the exact builds.
See the recommended pick on the original guide
The same logic applies to any big-model itch: running 70B on a single GPU is a solved problem; running 2.8T is not.

Common mistakes I'm already seeing

Buying hardware for a model you'll run twice. Even if a heroic community effort eventually squeezes some K3 variant onto a 512GB unified-memory machine, ask how often you'd load it versus a fast K2.7 you use daily. Novelty inference is a terrible reason to spend five figures.
Confusing "open weights" with "runnable weights." Modified MIT means you may run it anywhere. The 1.4TB footprint decides whether you can.
Extrapolating from K2's local success. K2 fitting on a 4090 at Q4 was a happy accident of architecture and size. K3 is 2.8T with a 4-bit native format — the compression lever is already pulled.

Verdict

As of late July 2026: use K3 through Moonshot, OpenRouter, Together, or Fireworks; rent a multi-node cluster if you truly need custody of the weights; and put your local budget into the K2 class, which is the best Moonshot model that actually fits inside a house.

Kimi K3 being open-weight is genuinely great for the ecosystem. It is not an invitation to your GPU. Run K2.7 at home, rent H100s when you need the weights, and let the API handle the rest.

Kimi K3 hardware: quick answers

How much VRAM does Kimi K3 need?

Far more than any single machine offers. The native MXFP4 download alone is roughly 594GB, and the total weight footprint with scales and overhead lands around 1.4TB. Even an 8x H100 80GB node (640GB) can't hold the full weights, so realistic deployment means multiple interconnected H100 or H200 nodes — datacenter territory, not a workstation.

Can a quantized Kimi K3 run on consumer GPUs?

No. The release format is already 4-bit MXFP4, so the usual "wait for Q4 quants" play doesn't apply — that compression is baked in. As of late July 2026 there are no confirmed full-fidelity GGUF ports, and even hypothetical aggressive quants would still measure in hundreds of gigabytes, more than ten times what a 32GB flagship card offers.

What's the closest model you can actually run at home?

Kimi K2.7 Code is the lighter self-host path in the same Moonshot family, and the broader Kimi K2 class runs well at Q4 on 24GB cards like the RTX 4090 or a used RTX 3090. You keep the agentic strengths and the permissive license, at a hardware cost measured in hundreds or low thousands of dollars instead of a cluster.

Related guides on Best GPU for LLM

The full version lives on Best GPU for LLM — VRAM calculator, GPU comparison table, and live Amazon pricing.

DEV Community