The number that should stop you
An 8GB MacBook Air. A 26-billion-parameter model. 2GB of RAM used at runtime.
Read that again. Not "we quantized it down to something small." Not "we used a distilled 3B version." The actual, full-fat Gemma 4 26B-A4B, on hardware Apple stopped selling as "for real work" years ago, generating tokens right now.
This is TurboFieldfare, and it hit #1 on Hacker News with 784 points because the trick isn't a hack — it's just correct engineering that almost nobody bothers to do.
Why this shouldn't be possible (and why it is)
Gemma 4 26B-A4B is a Mixture-of-Experts model. 26 billion total parameters, but only ~3.88 billion activate per token — the "A4B" is "Active 4 Billion." That means at any given moment, roughly 15% of the model is doing real work. The other 85% is just sitting there, dead weight, waiting for a token that might route to it.
The naive approach — and the approach every llama.cpp tutorial teaches — is to load the whole checkpoint into memory because "you might need any expert at any time." That's true. It's also a terrible reason to keep 12.6GB of tensors resident when you're only going to touch ~2GB of them per forward pass.
TurboFieldfare's move: stop treating the SSD like a boot disk and start treating it like the next tier of a memory hierarchy that already has L1, L2, L3, and RAM in it. Experts aren't loaded — they're paged.
How it actually works
Metal computes attention + the router from resident weights
→ CPU takes the router's top-8 expert IDs
→ checks a 16-slot LFU cache
→ misses get bounded parallel pread() into Metal-visible buffers
→ GPU consumes them next layer
What stays resident:
- A 1.35GB shared core (attention, router, shared expert)
- The KV cache — FP16, split into 25 sliding-window layers (circular buffer) and 5 full-attention layers (linear buffer)
What gets streamed, per token, per layer: only the routed experts the router actually picked. Chunked prefill (up to 128 tokens per chunk) amortizes the fetch cost so you're not paying SSD latency per token during prompt processing.
Quantization is doing its normal job here too — 4-bit MLX affine quantization on embeddings, attention, and expert weights, 8-bit on the router, group size 64. But quantization alone gets you from 52GB down to ~14GB. It does not get you to 2GB of runtime memory. The paging architecture is what gets you there. People conflate these two techniques constantly and it's why so many "run big models on small hardware" posts underdeliver.
The numbers
| Hardware | Speed |
|---|---|
| M2 MacBook Air, 8GB | 5.1–6.3 tok/s |
| M5 Pro, 24GB | 31–35 tok/s |
Notice what's not driving that 6x gap: RAM. The Air has 8GB, the Pro has 24GB, but the model only ever needs ~2GB either way. The gap is SSD and Metal GPU throughput. That's the tell that the architecture is working as designed — once you decouple "model fits in RAM" from "model runs fast," SSD bandwidth becomes the bottleneck that actually matters, not capacity. That's a genuinely different constraint than the one every "how much RAM do I need" blog post is optimizing for.
Try it yourself
git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release
.build/release/TurboFieldfareMac
First run pulls ~14.3GB of weights via range requests from Hugging Face — it never materializes the full checkpoint on disk before repacking it, which is its own small feat of not being wasteful. There's also a CLI, and an OpenAI-compatible loopback server if you want to point existing tooling at it.
Requirements: Apple Silicon, macOS 26, Metal 4. This is Arm64-only, text-only — no images, no audio, no tool execution yet (the server hands tool calls back to the client for authorization instead of running them). If you're on an Intel Mac or anything pre-Metal-4, this isn't for you yet.
Why you should care even if you never run this repo
This isn't really a "cool Mac app" story. It's a preview of where every serious local-inference runtime is headed. MoE architectures made "load the whole model" the wrong default assumption, and most tooling hasn't caught up — llama.cpp's -ncmoe flag for expert-pinning is the same idea in earlier form, and there's active work like Dwarf Star doing similar SSD-targeted expert offloading. TurboFieldfare is just the version that took the idea all the way to its logical conclusion and shipped it as a polished product with a CLI, a server, and a native app in one release.
If you're building or evaluating local inference tooling in 2026, the question to ask isn't "how much RAM does this model need." It's "how much of this model does a single token actually touch, and is my runtime smart enough to only load that." Most runtimes still aren't. This one is, and it's open source — go read the Metal kernels.
Sources: TurboFieldfare on GitHub · Show HN discussion
Top comments (0)