TL;DR
An open-source C engine called kimi-k3-in-c runs the 2.78-trillion-parameter Kimi K3 on one CPU in 8.24 GB of RAM. It is slow, 26.5 seconds per token on the project's fastest-drive machine, and its own numbers say something useful for anyone shipping on-device models: going from 8 GB to 224 GB of RAM (28 times the memory) made it only 1.70 times faster. We did not run the engine. We took the project's reported byte counts and one reported disk bandwidth and did the arithmetic. It predicts three of the four rows in the project's headline table to within 2%, and it explains the fourth. The practical rule: tokens per second on a streaming model is bytes read per token divided by random-read bandwidth, so spend RAM on the bytes you read with certainty before you spend it on caches.
What does kimi-k3-in-c actually do?
The repository (FareedKhan-dev/kimi-k3-in-c, Apache-2.0, release v1.1.0 from October 2, 2026, about 10.7k GitHub stars on November 5) is portable C99 with no BLAS, no framework and no GPU. Kimi K3 is published as moonshotai/Kimi-K3 on Hugging Face and ships as a 1.56 TB checkpoint in 96 shards. The README describes the model as 93 layers, of which 92 route tokens to 16 of 896 experts each, with about 104 billion parameters active per token (3.7% of the total).
The engine splits the checkpoint into two kinds of bytes and treats them differently:
- The trunk: the always-active dense path, attention and shared layers, stored in bfloat16. The README repacks it into one file of about 109 GB where each layer sits at a known offset and can be read in one call. The engine can pin as many layers as the RAM budget allows and stream the rest through a small ring of buffers.
-
The routed experts: 82,432 of them (896 times 92), each exactly 17,547,264 bytes in MXFP4, 1.45 TB in total. They are never resident. Each token fetches its 16 per layer from NVMe with
O_DIRECTreads.
Per token that is 16 experts times 92 layers, or 1,472 expert reads. Multiply by the per-expert size and you get 25.83 GB, which is the number the project prints in its logs. The trunk adds 108.81 GB if nothing is pinned. Everything else in this post follows from those two figures.
Why does 28 times more memory buy only 1.7 times more speed?
The project capped memory with a cgroup (MemoryMax plus MemorySwapMax=0, so an over-budget run dies instead of swapping and measuring swap) and ran twelve budgets from 8 GB to 224 GB. Every run produced byte-identical token ids. The timings:
Eight gigabytes took 32.69 s/token, 64 GB took 28.60 and 224 GB took 19.21. That is 1.70 times from 28 times the memory, and the jump from 8 GB to 64 GB is only 14%. The curve is not even monotonic: the 128 GB rung is slower than the 96 GB rung. The README itself puts the run-to-run spread of identical configurations at 33.1% (three repeats gave 14.78, 14.67 and 20.14 s/token), so several of those small steps are inside the noise.
One caution about absolute numbers. The README's headline table (26.5, 24.2, 19.8 and 5.6 s/token for 8, 32, 64 and 128 GB or more) comes from v1.1.0 on a fast NVMe drive, while the ladder above is an earlier dataset on a slower setup. The two do not line up in absolute seconds. Compare ratios inside one table, not seconds across tables.
Can byte arithmetic predict seconds per token?
If the engine is I/O bound, time per token should be bytes read divided by bandwidth. The README gives us a bandwidth from a different part of the document, an excerpt of a run log: 25.83 GB in 5.06 s (5104 MB/s while loading). That is 5.1 GB/s of random 17.5 MB reads at up to 16 in flight. We did not fit it to the table.
The pinned-layer counts for the 32 GB and 64 GB budgets (11 and 27 layers) come from the project's ladder. Each pinned layer removes about 1.17 GB per token (108.81 GB divided by 93 layers), which at 5.1 GB/s is 0.23 s.
def s_per_token(bw_gbs, trunk_gb=108.81, layers=93, pinned=0, expert_gb=25.83):
streamed = trunk_gb * (1 - pinned / layers)
return (streamed + expert_gb) / bw_gbs
for pinned in (0, 11, 27, 93):
print(pinned, round(s_per_token(5.104, pinned=pinned), 1))
# 0 26.4 | 11 23.9 | 27 20.2 | 93 5.1
| RAM budget | Trunk layers pinned | GB read per token | Predicted s/token | Reported s/token | Error |
|---|---|---|---|---|---|
| 8 GB | 0 | 134.64 | 26.38 | 26.5 | -0.5% |
| 32 GB | 11 | 121.77 | 23.86 | 24.2 | -1.4% |
| 64 GB | 27 | 103.05 | 20.19 | 19.8 | +2.0% |
| 128 GB+ | 93 (assumed) | 25.83 | 5.06 | 5.6 | -9.6% |
Three rows land within 2%, which is better than the run-to-run noise the same README admits to. The fourth row is informative because it misses. The README says that at 128 GB or more "the model fits entirely in memory". It cannot: the experts alone are 1.45 TB. What fits is the trunk. With the trunk fully pinned, every token still streams 25.83 GB of experts, and at 5.1 GB/s that alone is 5.06 s. The reported 5.6 s is half a second above that floor, which is the compute that did not overlap with I/O. So the fast row is not "no disk", it is "only the expert stream left".
Here is what each token reads from disk in the four configurations:
The caveats are the ones you would expect. The pinned counts for 32 and 64 GB come from the older ladder and we assume they carry over to v1.1.0. The 93 pinned layers in the last row is our assumption. One bandwidth figure from one log excerpt is not a bandwidth characterization. This is a consistency check on the project's reported numbers, not an independent measurement.
Why did the expert cache do nothing?
This is the part of the README worth reading in full. Across seven consecutive budgets from 8 GB to 64 GB the engine read exactly 25.83 GB per token, even as the expert cache grew from 28 slots to 1,344. A cache that grows 48 times and moves the byte count by nothing is not working inefficiently, it is not participating.
The README's explanation is in the model. Kimi K3 uses a load-balancing scheme the README calls Quantile Balancing that flattens expert usage across the pool by design, and flat usage defeats a least-recently-used cache: with no hot subset, a few gigabytes of arena retain nothing worth keeping. The cache starts to matter only around a 36 GB arena, and at the 96 GB rung the per-step reads fall from 99.70 to 23.58 to 19.90 to 14.07 GB over the first four steps as the arena fills.
The metric trap is worth copying into your own logs. The engine's first hit counter reported 100.00% hits while the true resident hit rate was 0.00%, because a batch prefetch had put each expert in the arena microseconds earlier by reading it from disk. Only a retention figure (1 - evictions/requests gives 29.89% retained at the 36.39 GB rung) agreed with the bytes actually read (70.1% still read). If your edge runtime reports a cache hit rate, check it against bytes read from storage before you believe it.
Where should the RAM go instead?
The README tested this directly by fixing the total at 128 GB and varying only the split between trunk and expert cache:
| Trunk GB | Cache GB | s/token | Cache hit % | GB read |
|---|---|---|---|---|
| 12.3 | 110.7 | 28.38 | 44.02 | 14.46 |
| 49.2 | 73.8 | 25.69 | 33.97 | 17.06 |
| 73.8 | 49.2 | 18.37 | 32.20 | 17.51 |
| 110.0 | 13.0 | 16.80 | 0.0 | 25.83 |
The fastest row has a 0% hit rate and reads the most expert bytes. The slowest has a 44% hit rate. The reason is in the arithmetic above: the trunk is 4.2 times the bytes of the experts and is re-read in full every token, so a gigabyte of pinned trunk removes about 1.17 GB of guaranteed traffic, while a gigabyte of cache below the knee removes nothing measurable. That is 28.38 down to 16.80 s/token (1.69 times) at identical total memory. The README is careful to say the magnitude was not replicated three times; the direction rests on rank correlations of -0.886 and -0.714 plus the mechanism.
What does this mean for on-device LLMs?
Most edge projects will never stream a 1.56 TB checkpoint, but the same structure shows up at every scale where weights exceed fast memory: a sparse MoE on a laptop, a model memory-mapped from flash on a phone, a 70B dense model on a unified-memory board. Five habits transfer.
Habit 1: write the roofline first. Seconds per token is streamed bytes divided by measured bandwidth. If your prediction is within 10% of the measurement you understand the system. If it is off by 2x, something else is the bottleneck.
Habit 2: measure bandwidth the way the model reads. The README warns that dd is one sequential stream at queue depth 1, while the expert path is random 17.55 MB reads with up to 16 outstanding, and on network storage the two diverge. It reports 31.71 s/token on local NVMe against 70.62 on a network volume, 2.2 times from the device alone. A minimal probe:
import os, sys, time, random
from concurrent.futures import ThreadPoolExecutor
SZ = 17_547_264 # one routed expert, in bytes
def read_at(path, off):
with open(path, "rb", buffering=0) as f:
f.seek(off)
return len(f.read(SZ))
def probe(path, qd, n=40):
size = os.path.getsize(path)
offs = [random.randrange(0, size - SZ) // 4096 * 4096 for _ in range(n)]
t = time.perf_counter()
with ThreadPoolExecutor(qd) as ex:
got = sum(ex.map(lambda o: read_at(path, o), offs))
return got / (time.perf_counter() - t) / 1e9
for qd in (1, 4, 16):
print(f"queue depth {qd:2d}: {probe(sys.argv[1], qd):.2f} GB/s")
Point it at a file larger than your RAM, or the page cache will report memory speed instead of drive speed. The engine uses O_DIRECT to avoid exactly that, and on Linux you should too.
Habit 3: pin the certain bytes before caching the probable ones. Dense-path weights are read every token. Experts, KV pages and embeddings are read with some probability. Certainty first.
Habit 4: count bytes, not hit rates. Log bytes read from storage per token. It is the only counter that cannot be gamed by a prefetcher.
Habit 5: repeat before you claim. The README declined to claim a 5.4% hugepage speedup because it sat inside a 33% noise floor. Three identical runs and a spread check cost minutes.
What would quantizing the trunk buy?
The same arithmetic prices the idea. This is a projection, not a measurement, and it assumes the engine stays bandwidth bound.
| Trunk precision | GB read per token | Projected s/token at 5.1 GB/s |
|---|---|---|
| bfloat16 (as shipped) | 134.6 | 26.4 |
| int8 | 80.2 | 15.7 |
| int4 | 53.0 | 10.4 |
The project deliberately offers no such knob. It sampled 31 attention tensors from the released checkpoint (384 rows each, symmetric per-row scaling) and found a mean weight reconstruction error of about 1% at int8 and about 17% at int4, with the worst int4 rows reaching 65%. It also points to the Kimi K3 technical report, which the README quotes as saying the experts are MXFP4 with quantization-aware training while non-expert components stay in higher precision. The README states its own limits: this is weight error rather than output quality, it covers attention projections only, and no downstream token comparison was run at int4. So the int8 row is plausible and the int4 row is a number nobody has shown to be usable. Time bought back with RAM is lossless. Accuracy lost to rounding is not.
FAQ
Can a 2.78-trillion-parameter model really run on an 8 GB machine?
According to the project, yes: peak resident memory was 8.24 GB while generating text from the 1.56 TB checkpoint, with a 1.7 TB disk requirement. It takes about 26 to 33 seconds per token depending on the drive and software version, so it is a proof of life rather than a chat experience.
Does more RAM change the answer?
The README reports byte-identical token ids at every budget from 8 GB to 224 GB. Memory changes only the clock.
What limits speed when a model streams from disk?
Bytes read per token divided by random-read bandwidth. For Kimi K3 that is about 134.6 GB per token with nothing pinned, of which 108.81 GB is the dense trunk and 25.83 GB is the sixteen-of-896 expert reads.
Is an LRU expert cache worth it for mixture-of-experts models?
It depends on the router. The README found no benefit below about a 36 GB arena for Kimi K3, attributing that to a balancing scheme that spreads expert usage evenly. Measure true resident hits against bytes read before sizing one.
Are the numbers in this article independent measurements?
No. Timings, hit rates and quantization errors are as reported in the project's README on November 5, 2026. The byte arithmetic, the predictions and the quantization projection are our own and are labeled as such.
Further reading
- Why a Sparse 35B MoE Runs Faster on a CPU Than a Dense 27B Model
- A 763B MoE on a 12 GB GPU: Our Speed Ceiling Versus What Strata Reports, and What Closes the Gap
Sources: kimi-k3-in-c repository and README, moonshotai/Kimi-K3 on Hugging Face. All timings, hit rates, memory figures and quantization errors are as reported by the project as of November 5, 2026. Predicted seconds per token, bytes per token and the trunk-quantization projection are our own illustrative arithmetic, not measurements.
Top comments (0)