Learning how LLMs actually work by building 18 of them in Zig

Most people who ship features on top of LLMs have never seen the inside of one. I
was in that group. I could call an API, tune a prompt, quantize a GGUF file, and
still not tell you what RMSNorm does or why RoPE exists. The model was a black box
with a temperature knob.

The tutorials did not help. They either hand you a 40-line PyTorch script that hides
everything behind nn.Linear, or they drop you into a production C++ engine where
the math is buried under kernel dispatch and memory pools. Neither one lets you see
the whole path from a tensor to a token.

So I built zigllm: an educational implementation of transformer architectures in
Zig, from raw tensor operations all the way up to text generation. It implements 18
model families (LLaMA, Mistral, GPT-2, Falcon, Mamba, BERT and more) and ships 285+
tests that double as executable documentation. Every component is written to teach
why it works, not just to run fast.

The core idea: six layers, each built on the last

The whole project is organized as a stack. You start at the bottom and work up, and
each layer only depends on the ones below it:

 6. Inference         Text generation, sampling, KV caching, streaming
 5. Models            LLaMA, GPT-2, Mistral, Falcon, GGUF loading, tokenization
 4. Transformers      Multi-head attention, feed-forward networks, full blocks
 3. Neural Primitives Activations (SwiGLU, GELU), normalization (RMSNorm), RoPE
 2. Linear Algebra    SIMD matrix ops, K-quantization, IQ-quantization (18+ formats)
 1. Foundation        Tensors, memory management, memory mapping

That ordering is the pedagogy. You cannot understand attention until you understand
matrix multiplication. You cannot understand a full LLaMA block until you understand
attention plus normalization plus RoPE. By the time you reach layer 6 and generate
your first token, nothing above you is magic, because you built every layer under it.

The 18 architectures on top of that stack cover roughly 80% of real-world LLM usage.
The point is not to run all of them in production. In practice LLaMA is the one path
wired end to end for real inference; the other families are there to read and
compare, which is exactly the point. You get to see that LLaMA, Mistral, Falcon and
GPT-2 are mostly the same machine with different knobs, and that Mamba and BERT are
genuinely different animals.

Why Zig, and how the engineering holds up

Zig is an odd choice until you sit with it. It gives you manual memory control,
comptime generics, and first-class SIMD, all with no runtime and no garbage
collector. That combination is exactly what inference wants. You get deterministic
memory (which matters a lot when you are learning, because you can see every
allocation) and you get SIMD intrinsics without dropping into inline assembly.

The optimizations in the repo are the real ones you find in production engines,
just written to be readable:

• KV caching, so you do not recompute attention over the whole sequence for every new token.
• SIMD acceleration (AVX, AVX2, NEON auto-detection), for a 3-5x speedup on matrix ops.
• 18+ quantization formats, up to 95% memory reduction.
• Memory-mapped model loading, so a large GGUF file does not have to be read into RAM up front.

Each of those is a lesson. KV caching teaches you why generation is quadratic
without it. Quantization teaches you the trade between memory and precision in a way
no blog post ever made click for me.

What using it looks like

Getting started is one clone and one command, because the tests are the tutorial:

git clone https://github.com/cognisoc/zigllm.git
cd zigllm
zig build test                    # All 285+ tests
zig build test-foundation         # Foundation layer only
zig build test-linear-algebra     # Linear algebra layer only

Running a single layer's tests is how you study that layer. Each test demonstrates a
concept and validates the math, so a failing assertion is a lesson about what the
math actually requires.

The examples are the guided tour, and the inference layer's own docs
(docs/06-inference/) walk the per-token cost down as each optimization comes on:

Without optimizations:  ~200ms/token   (naive)
With KV caching:         ~50ms/token    (4x)
With all optimizations:  ~5ms/token     (40x total)

That is the whole value of building it yourself. The number is not a claim in a
README, it is something you can reproduce and then explain: KV caching alone is the
4x, and the rest comes from SIMD and quantization stacked on top.

The same docs are honest about memory, and this is worth internalizing. For a
7B-class model in Q4 the parameters are ~3.5GB, but the KV cache for a 2k context
adds ~1.5GB per sequence, so the real floor is ~8GB, not 3.5GB. The parameter count
is the number people quote; the working set is the number that OOMs your box. Seeing
both fall out of code you wrote is the point.

There are more examples in examples/: simple_demo.zig for the overview,
gguf_demo.zig for loading a real pre-trained model, and
model_architectures_demo.zig for comparing all 18 families side by side.

Where this does NOT fit

This is the part that matters, so I will be blunt.

zigllm is not a production inference engine, and you should not deploy it as one.
Its own parity analysis rates it around 40% of llama.cpp on production concerns:
quantization coverage is a fraction of what llama.cpp offers, hardware acceleration
is CPU-focused, and broad multi-model serving is not the goal. If you need to serve
a 70B model to real users, use llama.cpp or vLLM. That is what they are for.

It is also not the fastest thing you can write in Zig. The README is explicit that
educational clarity takes priority over micro-optimization. When there was a choice
between a readable loop and a clever one, the readable one won. That is correct for
a teaching project and wrong for a serving one.

And if you do not want to learn the internals, this buys you nothing over just
running Ollama. There is no shortcut here. The value is entirely in reading the code
and running the tests. If you skip that, you skip the whole thing.

Takeaways

• The fastest way to stop treating LLMs as magic is to build one from tensors up, in a language that does not hide memory from you.
• A progressive layer stack (foundation, linear algebra, primitives, transformers, models, inference) is a better teaching order than any single top-down script.
• Tests as executable documentation is underrated. 285+ tests that each prove one concept beat any amount of prose.
• Zig turns out to be a genuinely good fit for this: comptime, SIMD, and manual memory, no runtime in the way.

The code, the docs for all six layers, and every example are here:
https://github.com/cognisoc/zigllm

If you have wanted to actually understand attention and KV caching instead of just
using them, clone it and run zig build test-transformers. If a layer's explanation
is unclear, that is a bug in the teaching, and I would genuinely like the issue.
Kick the tyres, PRs that improve educational clarity are the most welcome kind.