Memory got expensive. So one developer gave a 26-billion-parameter model a ~2 GB budget.
TurboFieldfare (6,483 stars, Apache-2.0) is a custom Swift + Metal runtime that runs instruction-tuned Gemma 4 26B-A4B on any Apple Silicon Mac — including the 8 GB models — without loading the full 14.3 GB model into memory.
The core idea: stream experts, don't load everything
Gemma 4 26B-A4B is a Mixture-of-Experts (MoE) model: 26 billion total parameters, but only ~3.88 billion are activated per token.
TurboFieldfare exploits this sparsity with a three-layer strategy:
- Resident core — 1.35 GB of shared weights + FP16 KV cache stays in memory permanently.
- Streamed experts — for each token, only the few experts hit by the router are read from SSD, used, then released.
- 4-bit quantization — weights ship in MLX affine 4-bit (group size 64), with 8-bit routers.
This isn't compressing the model — it's moving the model on demand. The model itself is unchanged, so output quality is identical to a full load. You're just paying a latency cost for SSD reads.
Model-specific, not a wrapper
TurboFieldfare is deliberately model-specific rather than a wrapper around MLX or llama.cpp. That matters: a general-purpose framework has to compromise across hundreds of models and dozens of quantization formats, while TurboFieldfare tunes every decision — expert sharding, routing, which weights to keep resident — for a single model.
The author also published a curated experiment log with 103 measured results covering kernels, caching, I/O, prefill, and decode. Every optimization, every rejected approach, is documented.
Measured decode speeds
| Machine | RAM | Decode |
|---|---|---|
| M2 MacBook Air | 8 GB | 5.1 - 6.3 tok/s |
| M5 Pro | 24 GB | 31 - 35 tok/s |
Same model, same code — the 5-6× gap comes down to memory bandwidth and SSD speed, not the model.
Quick start
git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release
.build/release/TurboFieldfareMac
The first run downloads and repacks the ~15 GB pinned model. Then: Download → Load Model → type a prompt → Generate.
Constraints
- macOS 26+ only, Apple Silicon only
- Model-specific (Gemma 4 26B-A4B)
- 5 tok/s on an 8 GB M2 is "it runs", not "production"
The takeaway
Alongside projects like kimi-k3-in-c (a 2.78T model in 8 GB on CPU), TurboFieldfare points at the same trend: LLM inference is shifting from "throw more compute at it" to "squeeze every byte of resources". When MoE sparsity, 4-bit quantization, and KV-cache tricks mature, "can it run?" is no longer about how much RAM you have — it's about whether you've done the memory math.
The Chinese version of the documentation is available at https://github.com/yangshun2005/turbo-fieldfare-cn
Top comments (0)