I made my programming language run a 1B LLM at 20 tok/s on a laptop CPU (it beat the reference C by 4x)

I build machin — a small "machine-first" language that compiles through C, designed for AI agents to write. Its roadmap rule is simple: every feature has to be earned by dogfooding, by building something real that breaks.

So I gave it an unreasonable challenge: run TinyLlama-1.1B on a 4-core laptop CPU at 20 tokens/second, in pure MFL. No FFI. No BLAS. No llama.cpp. The matmul included.

The correctness bar

LLM demos are easy to fake, so speed only counted if greedy decoding stayed token-for-token identical to karpathy's llama2.c (run.c / runq.c). Every optimization below was re-verified with a per-position logits diff against the reference before it counted.

The engine is ~600 lines: RoPE, GQA attention + KV cache, SwiGLU, int8/Q8_0 group quantization, and a from-scratch sentencepiece BPE encoder. (Fun find along the way: llama2.c's HuggingFace export silently breaks on GQA models — n_kv_heads is hardcoded to n_heads.)

The arc: 2.1 → 21.8 tok/s

2.1   naive engine, single thread
2.5   -O3 -march=native on the generated C
6.1   goroutine worker pool (jobs as packed ints over channels)
8.4   peek_i8 (byte loads the autovectorizer understands)
9.9   512-bit vector width
15.7  int8 activations (i8 x i8 dot, half the loads)
21.8  dot_i8 builtin + fused qkv / w1-w3 dispatch

Two lessons worth stealing:

1. Deterministic parallelism is free performance you can trust. Matmul rows fan out to a worker pool as packed-int jobs over channels; each row is computed by exactly one worker, so output is bit-identical at any thread count. I never had to choose between "fast" and "still provably correct."

2. Your accumulator width is your vector width. The language's int is 64-bit — and a 64-bit reduction forces the autovectorizer into half-width lanes. The same loop with an int32 accumulator is 40% faster and inexpressible in the source language. So it became a builtin: dot_i8(a, b, n) — the signed-byte dot product that is to quantized inference what sha256 is to crypto. Plain C inside, zero intrinsics; gcc's autovectorizer does the rest.

Results (TinyLlama-1.1B, Q8_0, greedy)

runq.c -O2 reference (1 thread)      1.67 tok/s
pure-MFL engine (1 thread)           6.8  tok/s   <- 4x the C
6 threads, cold                     21.8  tok/s
6 threads, 200-tok sustained        19.0  tok/s   (91C thermal throttle)

Honest asterisk included: sustained generation on this thermally-limited laptop is 19.0; the same binary holds 21.8 whenever the package is under 60°C, which is what interactive bursts look like.

The part I keep staring at: a garbage-collected-feeling, type-inferred language that compiles through C beat the hand-written reference C by 4x single-threaded — because the compiler emits the loop shapes vectorizers want, and iterating was cheap enough to find them.

Full story with the compiler-internals details: blog.intrane.fr
Engine + reproduction protocol: github.com/javimosch/machin-colibri
The ecosystem: awesome-machin