The Problem
Moonshot AI's Kimi K3 has 2.78 trillion parameters. Stored naively at bfloat16, that's 5,560 GB — more than the combined memory of two fully-loaded DGX H100 nodes. Deploying it typically requires dozens of H100 GPUs.
Fareed Khan asked a different question: can you run the exact same model checkpoint, with no quantization, distillation, or weight dropping, on a single CPU with 8 GB of RAM?
The answer is kimi-k3-in-c: a 176 KB pure C99 binary, seven source files, zero GPU dependencies. It runs the unmodified 1.56 TB checkpoint and produces output that is byte-for-byte identical to the PyTorch reference. At roughly 33 seconds per token, it's impractical as a chatbot — but that's not why it matters.
The Four Reductions
The engine exploits a structural property of Mixture-of-Experts models: Kimi K3 has 93 layers, 92 of which route to the top 16 of 896 experts. Only ~3.7% of parameters (~104 billion) are active for any single token. The other 96.3% must exist somewhere reachable but don't need to be in RAM.
Reduction 1 — Experts ship small. Kimi K3's 82,432 routed experts occupy 1.447 TB at roughly 0.53 bytes per weight — packed 4-bit nibbles with a shared E8M0 scale. The engine multiplies directly out of this packed form without dequantizing to float first. Baseline: 5,560 GB → 1,560 GB.
Reduction 2 — Routing sparsity removes experts. Expert weights are never memory-resident — loaded on demand from NVMe with an LRU cache. What remains is the 113.49 GB dense trunk. 1,560 GB → 113.49 GB.
Reduction 3 — Trunk streaming. The 93 dense layers are repacked into a single 109 GB trunk.bin where each layer lives at a known offset. The engine pins as many layers as the memory budget allows and streams the rest via O_DIRECT, bypassing the OS page cache. 113.49 GB → configurable, as low as 8.24 GB peak RSS.
Reduction 4 — Expert LRU cache. Routed experts are loaded on demand with a configurable cache size. The author provides a trace-based capacity simulator for tuning.
Total: a 676× reduction from the bf16 baseline, with the output at the bottom of this ladder being byte-for-byte identical to the output at the top.
Validation
The make test target requires no model download. It builds a 13-layer model with the same tensor graph, validates against a committed PyTorch reference across three paths — teacher forcing (32/32 positions), greedy decode (20/20 tokens), incremental decode (20/20 tokens) — and ends with "ENGINE MATCHES THE REFERENCE EXACTLY."
The build disables FMA contraction (-ffp-contract=off) so that scalar, OpenMP, and AVX2 paths produce bit-identical results. Every memory budget from 8 GB to 224 GB emits the same token stream. Memory is a performance dial, not a correctness variable.
Performance and Bottlenecks
Measured on dual AMD EPYC 7763 (124 cores, 228 GB RAM, NVMe):
| Preset | Peak RSS | Speed |
|---|---|---|
| laptop | 8.24 GB | 32.69 s/token |
| desktop | 31.9 GB | 28–31 s/token |
| server | 127.92 GB | 10.69 s/token |
The bottleneck is unambiguous: sustained trunk reads at 5,373–6,064 MB/s, with I/O accounting for 41–61% of wall-clock time. On spinning rust, performance would degrade several-fold.
Limitations (Be Honest)
- v0.1.0, 28 commits, days old
- Linux x86-64 only (O_DIRECT, posix_memalign, getrusage)
- Requires ~1.7 TB free NVMe storage
- No chat template (raw continuations), no sampling, no batching, no GPU path
- ~33 s/token at the minimum preset — generating 200 tokens takes ~2 hours
- The electricity cost of a multi-hour run can exceed hosted API pricing
Why Study This
kimi-k3-in-c is not a practical inference server. It is, explicitly, a teaching artifact — the author built it to understand Kimi K3's architecture after deploying it on 32 H100 GPUs at work and being unable to debug on personal hardware.
For engineers working on model inference, compression, or edge deployment, it offers three transferable findings:
- Storage bandwidth, not RAM or FLOPs, is the real bottleneck for frontier MoE inference — a measured finding with direct implications for hardware selection.
- Memory is a dial, not a floor — the same model runs correctly at 8 GB and 224 GB, only wall-clock time changes. This reframing matters for edge deployment of sparse models.
- A complete, auditable reference implementation — the README is structured as a five-part technical paper, building every component (RMSNorm, KDA attention, MLA, MXFP4 matmul, expert cache) from first principles in runnable C. For understanding MoE internals at the byte level, this is more valuable than most papers.
The broader point: the wall for running frontier models locally isn't compute — it's capacity. And most of the model is asleep for any given token. That structural fact makes the impossible tractable.
Repo: https://github.com/FareedKhan-dev/kimi-k3-in-c (Apache-2.0, v0.1.0)
Not tested — this analysis is based on reading the public README, source tree, CHANGELOG, and independent technical reviews (andrew.ooo, essamamdani.com, securityonline.info). No local build or inference run was performed.
Top comments (0)