DEV Community

Mohammad Nazibul Islam
Mohammad Nazibul Islam

Posted on

I made a language model that lives in your CPU's L3 cache

#opensource #performance #cpp #machinelearning

In January I wrote a spec for myself: ">50 tokens/second on one CPU core."

That's a deliberately boring number — llama.cpp does that on a laptop. So I kept pulling the thread, and nine months later the number is 3,405 tok/s on a single core of a $60 CPU, with a 1.9 MB model I trained from scratch.

The interesting part isn't the number. It's the one-line explanation:

I stopped treating the CPU cache as an implementation detail and made it the design target.

The arithmetic that decides everything

Every forward pass of a language model reads its weights once. That read is the cost.

  • My model: 1.91M parameters, int8 → 1.97 MB of weights
  • A typical desktop L3 cache: 4 MB (mine: 4 MB on the Ryzen 5 3400G)
  • 1.97 / 4 = 49.8% of L3

If the weights fit in cache, decode is a cache-streaming problem: ~6.7 GB/s of sequential reads served by L3. If they don't, every pass goes to DRAM, and you fight bandwidth, latency, and contention.

Most inference work optimizes compute. Cache-fit makes compute the non-bottleneck by removing the memory problem entirely.

The model isn't an accident of size — I sized it to the cache on purpose: 4 gated short-conv blocks + 2 grouped-query attention blocks (kv=1, so the KV cache is one vector per layer, not seq_len×d), vocab 4096 so the output head doesn't dominate, int8 export so bytes = params.

The measurement that proves it

A number without a control isn't a measurement. So the benchmark has an evict_read mode: a second process hammers the L3 with random reads while the model decodes.

tok/s
Warm (weights resident in L3) 3,405
Evicted (L3 polluted mid-run) 1,681
Penalty 2.03×

Same binary, same model, same positions. That 2.03× is what cache residency is worth — measured, not argued.

The stream command runs the opposite probe (128 MB sequential scan) to show your box's raw DRAM bandwidth, so you can compare the two numbers yourself.

Quality didn't get sacrificed for speed

Speed claims are cheap; parity claims aren't. The C runtime must match the PyTorch reference it was exported from, enforced as hard gates:

  • Perplexity: 26.8407 (C) vs 26.8188 (PyTorch) → 0.082% delta, gate ≤1%
  • Greedy token match: 99.41%, gate ≤1% mismatch
  • Determinism + AddressSanitizer suites: clean
  • All gates pass from JSON records checked into the repo — phase0/phase5_results.json

The int8 story: per-input activation quantization amortized across all rows, sign-extended dot products (vpmovsxbw + vpmaddwd, 4 accumulators), AVX2 attention, degree-7 polynomial exp. The float→int kernel push alone was 1,494 → 3,696 tok/s.

The whole pipeline, reproducible 

python data/clean.py && python data/generate.py && python data/pack.py
python train.py --train data/train.bin --val data/val.bin --out run --d 144 ...
python phase0_export.py --ckpt run/best.pt --out phase0/model.i8
phase0\build_p0.bat
phase0\phase0.exe phase0\model_p3.i8 bench data\val.bin 15.0 65
Enter fullscreen mode Exit fullscreen mode

Gutenberg → cleaning → synthetic task streams → BPE → train → QAT-lite int8 export → C runtime. Corpus build ~80 s. The benchmark prints one JSON line and you either reproduce my number or you don't.

What it can't do (read this part)

  • The outputs are weak. 1.9M parameters is a design study, not a chatbot. It's a benchmark vehicle for the cache-fit thesis, and I'm not going to pretend otherwise.
  • Windows x64 only today. MSVC build works; a Makefile for Linux/macOS is issue #2. (If this post gets any traction, that becomes priority #1.)
  • One model, one machine class. The numbers are from a Ryzen 5 3400G (4 MB L3). Bigger L3 = bigger headroom; the repo's info command tells you fit_half_l3 for your CPU.

Why publish this

The interesting claims in ML inference are usually about GPUs. But the memory-hierarchy argument generalizes: if your weights fit where the compute is, the whole problem changes shape. Small models sized to cache are the cheapest possible place to demonstrate that.

Everything is MIT: training code, kernels, measurements.
https://github.com/mdnazib963/CoreFlow — weights on Hugging Face at nazib61/coreflow-1.9m.

If you reproduce (or fail to reproduce) the numbers on your hardware, open an issue with your JSON output. Hardware diversity is the point.

Top comments (0)