I ran a 7B Mixture-of-Experts LLM in a language I built — token-identical to fp32

#ai #machinelearning #opensource #programming

I've been building machin (MFL), a machine-first language, and using it to write an LLM inference engine with zero dependencies — no PyTorch, no llama.cpp, no BLAS, no Python at runtime. Just a static binary.

The wall

A dense 1B model got me ~20 tok/s on a laptop CPU. That's the ceiling, and it's fundamental: decode speed is set by bytes moved per token, and on weak hardware you're pinned to the memory bus. I built and measured every trick — speculative decoding, int4, contextual sparsity, early-exit, continuous batching. Every one topped out at ~1.35×, because the box is balanced: save bandwidth and you go compute-bound, and vice-versa.

The disruption isn't in the engine. It's in the model.

Mixture-of-Experts

A dense model prices every token at its total params. An MoE decouples quality from speed: only a few experts fire per token. OLMoE-1B-7B is 6.9B total but 1.3B active (top-8 of 64 experts/layer). And since only a handful fire, you mmap the checkpoint and let the OS page cache stream cold experts from disk — total size bounded by disk, not RAM.

It works — and it's exact

I wrote an fp32 numpy reference reading the original weights, and checked my pure-MFL int8 engine against it token for token:

numpy fp32 : The capital of France is Paris. The capital of the United States is Washington
pure MFL   : The capital of France is Paris. The capital of the United States is Washington

12/12 tokens identical. Quantization didn't flip one.

experts	lm_head	size	tok/s
int8	int8	7.65 GB	14.5
int4	int8	4.43 GB	11.2

int8 lm_head buys speed (memory-bound); int4 experts buy footprint. Then I wrapped it in an OpenAI-compatible server — tokenizer and all in pure MFL — and pointed the official openai client at it. Works.

7B-class quality, ~1B speed, on hardware you own, no numeric libraries anywhere.

Full write-up: https://blog.intrane.fr/a-7b-moe-llm-at-1b-speed-in-pure-machin
Code (engine, converters, tokenizer, every dead end): https://github.com/javimosch/machin-colibri