A 16 GB Mac mini gives its GPU about 12.7 GB. Google's Gemma 4 26B-A4B, in its official 4-bit file, is 14.4 GB. On paper it doesn't fit. In practice it writes TypeScript at 31 tokens a second, offline, from that one file, untouched.
The trick is that a mixture-of-experts model never uses all of itself at once. This post is about how I pick which part to keep.
What "26B-A4B" means
Gemma 4 26B-A4B has 26 billion parameters, but only about 4 billion are used for any one word. Each of its 30 layers holds 128 small "experts", and a router in each layer picks 8 of them per word. The other 120 sit idle for that word.
So the real question is never "does the model fit?" It is "do the experts this word needs fit?"
The heat map
I ran Gemma on TypeScript prompts and logged every pick the router made: which expert, in which layer, with what weight. Counted up, that log is a heat map. For each layer, a ranked list of the experts TypeScript work actually uses.
At load, Kraits reads the full model file but puts only each layer's 80 hottest experts on the GPU. The router still makes its own choice, just among those 80. Nothing is read from the SSD while it writes.
Two things follow:
- The file is never changed. It's Google's file, byte for byte. A different heat map points the same file at different work.
- The model's maths is unchanged. Every word still uses 8 experts per layer, picked by the model's own router.
What it scores
Measured by my release tester on a Mac mini M4 with 16 GB, temperature 0, thinking off:
| Reading the prompt | ~105 tokens/s |
| Writing | 31–33 tokens/s |
| Five TypeScript tasks, scored by strict compile plus executed tests | 33 / 40 |
| Same prompt twice | byte-identical output |
| A 19K-token document, five facts at the start | 5 / 5 recalled |
The weak spot is a task that asks the model to write a Jest test suite: 2 of 8. Its tests compile, but one wrong expectation fails the suite on correct code, and the scoring then gives nothing for catching bugs.
What a bigger model taught me
The obvious next move was a bigger model: Qwen3.5-122B-A10B, with 256 experts per layer. On a 16 GB Mac only about 30 per layer fit on the GPU. It ran at 14 tokens a second and wrote broken code.
The interesting part was finding out where it broke. I split the model's 48 layers in half:
| Early layers 0–20 | Late layers 21–47 | Result on one coding task |
|---|---|---|
| hot 30 only | hot 30 only | broken, loops |
| every expert | hot 30 only | broken, repeats the prompt |
| hot 30 only | every expert | clean, correct code |
The late layers carry the quality. And the late layers needed far more experts than fit, so the 122B could not be saved by moving slots around. On the same Jest task it scored 2 of 8, the same as Gemma, at about 2 tokens a second. I parked it.
The lesson: for a mixture-of-experts model on a small machine, size isn't the number that matters. Coverage is: how much of what the router wants is already on the GPU. Gemma keeps 80 of 128 per layer, and that's enough. The 122B could keep 30 of 256, and that wasn't.
Try it
Kraits is a plugin for stock llama.cpp (build b11715), Apache 2.0. It needs an Apple silicon Mac with 16 GB and Google's gemma-4-26B_q4_0-it.gguf from their Hugging Face page:
cd kraits/bin
LLAMA_SERVER=/path/to/llama-server ./kraits-server -m /path/to/gemma-4-26B_q4_0-it.gguf -c 8192 --reasoning off
It worked if it prints Kraits loaded. Repo, binaries and hashes: https://github.com/1picassoai/kraits
What would you want to run on a 16 GB Mac next?
Top comments (0)