This article was originally published on runaihome.com
TL;DR: Local models didn't just get bigger in 2026 — they got faster at the same quality. Three techniques did the heavy lifting: multi-token prediction (~1.8× throughput, lossless), speculative decoding (1.5–3× on consumer GPUs), and sparse MoE routing (35B of weights, only 3B active per token). Together they put GPT-4-class output on a single 24GB GPU at usable speeds.
| Multi-token prediction | Speculative decoding | Sparse MoE | |
|---|---|---|---|
| What it speeds up | Decode (built into the model) | Decode (runtime trick, any model) | Decode + memory pressure |
| Typical gain | ~1.8× (85–90% accept rate) | 1.5–3× depending on accept rate | 3B active vs 35B total → ~30B-dense feel at 3B-dense speed |
| The catch | Model must be trained with it | Needs a good small draft model + extra VRAM | Still needs VRAM for all weights |
Honest take: None of these are magic — they all trade extra compute or memory for fewer sequential steps. But stacked together on a model like Qwen3.6 35B-A3B, they're the reason a used RTX 3090 now does everyday coding and writing that needed a cloud API call 18 months ago.
The thing that actually changed
Ask anyone who ran local models in 2024 and they'll tell you the same story: a 7B model was fast but dumb, and a 70B model was smart but unusably slow on consumer hardware. You picked your poison. The honest answer to "should I run this locally or just call an API?" was usually "call the API."
That tradeoff broke in 2026, and it broke for a specific, technical reason. The models people run at home — Qwen3.6, Gemma 4, GPT-OSS, Nemotron-Cascade — aren't just bigger or better-trained than their predecessors. They're architected so the expensive part of generation (decoding one token at a time) costs far less per token than the raw parameter count suggests.
There are three distinct techniques doing this, and they're often confused because they all attack the same bottleneck. This article separates them, shows what each one actually buys you, and explains why the combination — not any single one — is what closed the gap to cloud.
If you just want to know which model to pull for your GPU, the open-source LLM shootout has the picks by VRAM tier. This article is the why behind those picks.
The bottleneck: why decoding is slow in the first place
Every autoregressive LLM generates text one token at a time. To produce token N, the model runs a full forward pass using tokens 1 through N−1 as context. Then it appends token N and runs another full forward pass for token N+1. The passes are strictly sequential — you can't start computing token N+1 until you know token N.
Here's the part that surprises people: that forward pass is memory-bandwidth-bound, not compute-bound. For a single user generating one token, the GPU spends most of its time reading the model's weights out of VRAM, not doing math. A dense 32B model at Q4 has roughly 18GB of weights, and the GPU has to stream a large fraction of those through its memory bus for every single token.
That's why a used RTX 3090, with 936 GB/s of memory bandwidth, does roughly 95 tok/s on a 7B model but only a fraction of that on a dense 32B — the model is the same architecture, there are just more weights to read each step. The short version, which the VRAM-tier guide leans on throughout: decode speed tracks bandwidth ÷ active-weight-size far more than it tracks raw TFLOPS.
All three 2026 techniques are different answers to one question: how do we produce more tokens without reading all the weights that many times in sequence?
Technique 1: Sparse MoE — read fewer weights per token
Mixture-of-Experts is the most consequential of the three, and it's the easiest to understand once you frame it around bandwidth.
A dense 32B model reads ~32B parameters' worth of weights for every token. A Mixture-of-Experts model splits most of its parameters into "expert" sub-networks and adds a small router that picks which experts to use for each token. The model still stores all the weights, but it only reads the active ones per token.
Two of the most-run local models in 2026 are built this way:
- Qwen3.6 35B-A3B: 35 billion total parameters, but only ~3 billion active per token.
- Gemma 4 26B-A4B: 26 billion total, ~4 billion active per token.
The "A3B" / "A4B" suffix literally means "active 3 billion / active 4 billion." That naming convention is itself a sign of how central this idea became.
The payoff shows up directly in benchmarks. On an RTX 4090, a dense 32B model at Q4 lands near 60 tok/s, while a ~30B MoE model with 3B active runs around 110 tok/s at 32K context — nearly double, from a model with more total parameters. Reported numbers vary by runtime and context length (one careful Q4_K_M measurement put Qwen3.5 35B-A3B at ~78 tok/s decode on a 4090), but the direction is consistent: you get the throughput of a ~3B model with the knowledge capacity of a much larger one.
The catch — and it's a real one — is VRAM. MoE saves bandwidth, not capacity. You still have to hold all 35B parameters in memory, so Qwen3.6 35B-A3B needs a 24GB card just like a dense model of that size would. MoE makes the smart model fast; it doesn't make it fit on less. That distinction trips up a lot of buyers, which is why the VRAM-tier guide leads with total size, not active size.
Technique 2: Speculative decoding — guess ahead, verify in parallel
Speculative decoding is a pure runtime trick. It doesn't change the model's weights or quality at all — it changes how you run inference.
The idea: pair your big "target" model with a small, fast "draft" model. The draft model cheaply generates a short run of candidate tokens — say the next 4. Then the target model does one forward pass that verifies all 4 candidates at once (verification is parallel; generation is not). Every candidate the target agrees with is accepted for free; the first disagreement is corrected, and you start the next round from there.
The crucial property is that the output is bit-for-bit identical to what the target model would have produced alone. Speculative decoding is lossless — tuning how many tokens the draft proposes changes speed only, never the text. That's what separates it from quantization or distillation, which trade quality for speed.
Real-world gains:
- General reports put speculative decoding at 2–3× speedup with no quality change.
- In llama.cpp specifically, users see 1.5×–3× tokens/sec depending on how often the draft model's guesses are accepted.
- NVIDIA has demonstrated up to 3.6× throughput on H200-class hardware with tuned draft models.
The acceptance rate is everything. If your draft model agrees with the target 80% of the time, most of your speculative tokens stick and you get a big speedup. If the draft is poorly matched and only agrees 30% of the time, you've paid for draft passes that get thrown away and you might even go slower. This is why picking a draft model from the same family (e.g. a 0.5B Qwen drafting for a 32B Qwen) matters — they make similar predictions.
By late 2025 this moved from research curiosity to production default: vLLM and TensorRT-LLM ship native support, and llama-server (the backbone of many local setups) supports several implementations. The cost is extra VRAM for the draft model and some tuning of the speculative length — a small price for a free 1.5–3×.
Technique 3: Multi-token prediction — bake the drafting into the model
Multi-token prediction (MTP) is the technique people most often confuse with speculative decoding, because the mechanism at inference time looks similar. The difference is where
Top comments (0)