If your MoE model is too big for your GPU, llama.cpp fills VRAM with as many whole layers of experts as fit and runs the rest on the CPU, from system RAM. Every token then waits on the slowest part of your machine. Since build b11480 there's a better use for that VRAM: a cache of the experts the model actually keeps picking, from every layer.
We ran it on a free 16 GB Tesla T4 with Qwen3-30B-A3B, an 18.6 GB model that doesn't fit on the card. Decode went from 22.6 tokens/s with llama.cpp's default placement to 43.2 tokens/s with every expert in RAM and a 10,000 MiB cache. The llama.cpp developer who wrote the feature measured 2.2x on an RTX 5090 with a 93.7 GiB model that doesn't fit an 80 GB H100 at all. Holding that in VRAM on one GPU takes a 141 GB H200, listed at $5.29 to $6.31 per GPU-hour on on-demand cloud VMs in InferPrice's October 17 data.
Hits come from VRAM, misses cross PCIe
The cache went into llama.cpp's master branch on October 7 as commit d6cf9ac, squashed from PR #29887 (whose page still said "Open" on October 17), and shipped in build b11480. Multi-GPU support followed a day later in b11507. We checked everything here against b11712. The flag is --moe-cache-mib N (env LLAMA_ARG_MOE_CACHE_MIB), default 0, and llama.h still marks it experimental.
The mechanics, from src/llama-moe-cache.cpp: llama.cpp takes the layers whose experts all live in host memory, groups those whose expert tensors share types and shapes, and gives each group a pool of slots in VRAM. When the router picks experts, hits are read straight from VRAM and only misses are uploaded over PCIe, evicting the least recently used. A pool is shared, so a hot expert in layer 3 can push out a cold one in layer 40. Only the ffn_*_exps tensors are cached, not the router, attention or shared experts.
The catch is batch size. The cache only serves batches of 32 tokens or fewer, and only if the batch's experts fit in the slots at once. Bigger batches, which means prompt processing, take the old offload path, though they copy any experts that happen to be cached instead of uploading them.
1. Measure your experts before you pick a number
You need three things from your model: how many bytes of experts it has, how big one cache slot is, and which layers share a pool. The GGUF header has all three. This stdlib script reads it and repeats llama.cpp's slot arithmetic, so you see what a budget buys before you load anything:
"""Size llama.cpp's --moe-cache-mib from a GGUF model (standard library only).
Usage: python3 moe_cache_plan.py model-00001-of-00002.gguf [model-00002-of-00002.gguf] [--mib N] [--n-cpu-moe N]
Models one CUDA or ROCm GPU with --cpu-moe (or --n-cpu-moe N), as in llama.cpp b11712.
"""
import argparse
import re
import struct
EXPS = re.compile(r"blk\.(\d+)\.ffn_(gate_up|gate|up|down)_exps\.weight$") # what the cache holds
ALIGN = 128 # ggml's CUDA buffer alignment (ROCm builds use the same code)
SCALAR = {0: "B", 1: "b", 2: "H", 3: "h", 4: "I", 5: "i", 6: "f", 7: "?", 10: "Q", 11: "q", 12: "d"}
def read_gguf(path):
"""Return (metadata, {tensor name: (dims, ggml type, bytes)}) from one GGUF file or split part."""
with open(path, "rb") as f:
def rd(fmt):
return struct.unpack("<" + fmt, f.read(struct.calcsize(fmt)))[0]
def string():
return f.read(rd("Q")).decode("utf-8", "replace")
def value(vtype):
if vtype == 8:
return string()
if vtype == 9:
etype, n = rd("I"), rd("Q")
return [value(etype) for _ in range(n)]
return rd(SCALAR[vtype])
if f.read(4) != b"GGUF":
raise ValueError(f"{path}: not a GGUF file")
_version, n_tensors, n_kv = rd("I"), rd("Q"), rd("Q")
meta = {}
for _ in range(n_kv):
key = string()
meta[key] = value(rd("I"))
infos = []
for _ in range(n_tensors):
name = string()
dims = [rd("Q") for _ in range(rd("I"))]
infos.append((name, dims, rd("I"), rd("Q"))) # sizes come from offsets: no table of quant types to go stale
align = meta.get("general.alignment", 32)
data_start = -(-f.tell() // align) * align
data_len = f.seek(0, 2) - data_start
ends = sorted(o for *_, o in infos) + [data_len]
nxt = dict(zip(ends, ends[1:]))
sizes = {}
for name, dims, gtype, off in infos:
size = nxt[off] - off
rows = dims[1] * dims[2] if len(dims) == 3 else 1
sizes[name] = (dims, gtype, size - size % rows) # experts are rows x row bytes; their padding is < rows bytes
return meta, sizes
def pad(n):
return -(-n // ALIGN) * ALIGN
def plan(paths, budget=None, n_cpu_moe=None):
"""budget is in bytes; None means 10% of the model's expert bytes, rounded down to whole MiB.
n_cpu_moe=None models --cpu-moe (every expert in host memory); N models --n-cpu-moe N (blk.0 to blk.N-1)."""
meta, tensors = {}, {}
for p in paths:
m, t = read_gguf(p)
meta.update(m)
tensors.update(t)
arch = meta["general.architecture"]
n_expert, k = meta[f"{arch}.expert_count"], meta[f"{arch}.expert_used_count"]
layers = {} # layer -> [(type, ne0, ne1, bytes per expert)] of its expert tensors
for name in sorted(tensors):
m = EXPS.match(name)
if m:
dims, gtype, size = tensors[name]
layers.setdefault(int(m.group(1)), []).append((gtype, dims[0], dims[1], size // dims[2]))
if not layers:
raise SystemExit("no MoE expert tensors found: the MoE cache only helps MoE models")
total = sum(n_expert * sum(t[3] for t in ts) for ts in layers.values())
groups = {} # same types and shapes = same cache banks, as in llama.cpp
for il, ts in sorted(layers.items()):
if n_cpu_moe is None or il < n_cpu_moe:
groups.setdefault(tuple(ts), []).append(il)
if not groups:
raise SystemExit("no MoE layer keeps its experts in host memory, so llama.cpp disables the cache")
host = sum(len(ls) * n_expert * sum(t[3] for t in ts) for ts, ls in groups.items())
budget = total // 10 // 2**20 * 2**20 if budget is None else budget
out = {"arch": arch, "n_expert": n_expert, "k": k, "layers": len(layers), "expert_gib": total / 2**30,
"host_gib": host / 2**30, "budget": budget, "groups": []}
for ts, ls in groups.items():
sz = [t[3] for t in ts]
g_host = len(ls) * n_expert * sum(sz)
share = int(budget * g_host / host) # split by expert bytes, as in llama.cpp
slots = 0
while slots < len(ls) * n_expert and sum(pad(x * (slots + 2)) for x in sz) <= share:
slots += 1 # each bank keeps one spare slot at the end
slots = slots if slots >= k else 0
max_batch = max([n for n in range(1, 33) if min(n * k, n_expert) <= slots], default=0)
need = -(-sum(pad(x * (k + 1)) for x in sz) * host // g_host) # smallest budget that gives this group k slots
out["groups"].append({"layers": len(ls), "slots": slots, "pct": 100 * slots / (len(ls) * n_expert),
"max_batch": max_batch, "min_bytes": need})
return out
if __name__ == "__main__":
ap = argparse.ArgumentParser(description="Size llama.cpp's --moe-cache-mib from a GGUF model.")
ap.add_argument("gguf", nargs="+", help="the model file, or every part of a split model")
ap.add_argument("--mib", type=int, help="the --moe-cache-mib to check (default: 10%% of the experts)")
ap.add_argument("--n-cpu-moe", type=int, help="model --n-cpu-moe N instead of --cpu-moe")
a = ap.parse_args()
r = plan(a.gguf, None if a.mib is None else a.mib * 2**20, a.n_cpu_moe)
print(f"{r['arch']}: {r['layers']} MoE layers, {r['n_expert']} experts, {r['k']} used per token, "
f"{r['expert_gib']:.2f} GiB of experts" + ("" if a.n_cpu_moe is None else f", {r['host_gib']:.2f} GiB on the CPU"))
print(f"--moe-cache-mib {r['budget'] // 2**20}" + ("" if a.mib is not None else " (10% of the experts)"))
for i, g in enumerate(r["groups"], 1):
print(f" group {i}: {g['layers']} layers, {g['slots']} slots ({g['pct']:.1f}%), "
f"batches up to {g['max_batch']} tokens use the cache, cached at all from {-(-g['min_bytes'] // 2**20)} MiB")
Run it on your model (pass every split part) to get the 10% starting point, or with --mib to test a budget. Add --n-cpu-moe N if that's how you run, because then only the first N layers are cached:
python3 moe_cache_plan.py Qwen3-30B-A3B-Q4_K_M.gguf --mib 10000
It prints one line per cache group: the slots you'll get, their share of the experts, the largest batch that can use them, and the budget below which the group isn't cached at all. For Qwen3-30B-A3B Q4_K_M that's 16.35 GiB of experts (48 layers, 128 experts, 8 per token) in two groups of 24 layers, because the Q4_K_M mix doesn't quantize every layer's experts the same way. 10,000 MiB buys 1,834 slots per group (59.7%). On the T4, llama-server's startup lines reported exactly those slots and 16,740 MiB of host experts, and they matched again with --n-cpu-moe 32.
The floor matters. Below it, llama.cpp leaves that group uncached with only a warning (MoE cache budget is too small for N layers, they are not cached), and if no group qualifies, llama-server won't start. Our model's floor is 50 MiB, and at 49 MiB the T4 run stopped with "MoE cache is too small to hold the experts of one token".
2. Give the VRAM to the cache, not to whole layers
There are two ways to run it. The easy one keeps llama.cpp's automatic placement (--fit, on by default), which decides how many expert layers stay in VRAM, and it counts the cache: it sizes things with a dry-run context built from your flags, and the cache reports its VRAM into that context's memory breakdown. On our T4, --fit kept 1,670 MiB less of the model in VRAM (11,952 down to 10,283 MiB) to make room for a 1,671 MiB cache.
llama-server -m model-00001-of-00003.gguf --moe-cache-mib 6544 --port 8080
The faster one, on both of the developer's test machines and on our T4, puts every expert in RAM with --cpu-moe (-cmoe for short) and hands the freed VRAM to the cache:
llama-server -m model-00001-of-00003.gguf --cpu-moe --moe-cache-mib 19000 -c 16384 --port 8080
I'd start with the second form: a whole layer in VRAM only helps when the router picks one of its experts, while the cache holds whichever experts are hot, from every layer. On the T4 it nearly doubled decode, where a 10% cache under --fit barely moved it (23.6 against 22.6 tokens/s). But you take over the sizing. --cpu-moe is a tensor override, and --fit won't move layers once one is set: it only checks that your settings fit (shrinking the context if you didn't pass -c), and otherwise gives up with a warning ending in tensor_buft_overrides already set by user, abort. Asking the T4 for 15,000 MiB got that warning, then "failed to allocate the MoE cache buffers". Leave room for the dense weights, the KV cache for your -c and the compute buffers.
Pass --port while you're upgrading. Build b11521 moved llama-server's default port from 8080 to 9931 (our T4 run without --port listened there), and the same commit pinned LLAMA_ARG_PORT=8080 in the official Dockerfiles, so containers keep 8080 while a bare binary or systemd unit moves.
3. You won't see the hit rate without -lv 4
The cache logs its size at startup and its hit rate at shutdown, and llama-server hides both. They're libllama INFO messages, which its logger files under verbosity 4, above the default threshold of 3 (common/log.cpp). Our first T4 run printed none of them. Add -lv 4:
llama-server -m model-00001-of-00003.gguf --cpu-moe --moe-cache-mib 19000 -c 16384 --port 8080 -lv 4
On the T4 that printed this at startup (timestamps cut), followed by one line per group with its layer count, quant type and slots:
I impl: CUDA0 MoE cache size = 9999.32 MiB for 16740.00 MiB of host experts
And this when the server stopped:
I llama_moe_cache: ubatch <= 8: hits = 234520, misses = 10452, hit rate = 95.73%, uploaded = 28587.06 MiB
I llama_moe_cache: ubatch > 8: hits = 2668, misses = 1785, hit rate = 59.91%, uploaded = 4862.46 MiB
I llama_moe_cache: large batches: 46881 experts copied from the cache, 42544.79 MiB
The hit rates appear only when the context is torn down, so you read them after stopping the server. ubatch <= 8 counts batches of up to 8 tokens (generation) and ubatch > 8 those of 9 to 32. Bigger batches don't use the slots, but when llama.cpp uploads host experts for them it copies already-cached ones on the GPU instead, and the last line counts those. -lv 4 prints plenty of other trace output, so filter for moe.
What a free T4 showed
| Qwen3-30B-A3B Q4_K_M on one T4 | Decode tokens/s | Prompt tokens/s |
ubatch <= 8 hit rate |
|---|---|---|---|
--fit (default) |
22.6 | 349 | |
--fit --moe-cache-mib 1674 (10%) |
23.6 | 357 | 75.6% |
--cpu-moe |
9.6 | 150 | |
--cpu-moe --moe-cache-mib 50 (the floor) |
3.8 | 152 | 0.0% |
--cpu-moe --moe-cache-mib 10000 |
43.2 | 382 | 95.7% |
| the same, four requests at once | 37.0 in total | 92.3% |
Decode is the mean of four 128-token replies at temperature 0 after a warm-up request, from llama-server's own timings, and prompt speed is one 1,436-token prompt. It's one run on one shared Kaggle VM (4 vCPUs, 31 GB of RAM), so don't read much into the decimals. --cpu-moe alone is slow because every expert runs on four CPU cores; with 10,000 MiB of cache it's the fastest, because 95.7% of expert lookups were served from VRAM. The 10% cache under --fit hit 75.6% of the time and still barely moved decode: it only covers the 20 layers --fit left in RAM, and --fit gave up resident layers to pay for it. Prompt processing got faster too, 382 against 150 tokens/s without the cache, and the cache's counter shows why: for those big batches llama.cpp copied 46,881 cached experts on the GPU instead of uploading them.
The developer's rule of thumb: start at 10%
The developer who wrote the feature found "10% of the total expert size is good" as a starting point, and more cache kept paying off on their RTX 4090 and 5090 (the 5090's 67.8 tokens/s came with 19,000 MiB and an 89% hit rate). That's one model, one prompt set and one host. Your gain depends on how often your prompts reuse experts and on PCIe bandwidth, since every miss is an upload, so treat 10% as where to start measuring.
Where it stops helping
Concurrency eats the cache. llama-server decodes all active requests together, one token each, with 4 slots by default. Four users generating at once means four tokens per step, each picking its own experts, so a step needs up to 4 times as many slots and churns the LRU harder. Past 32 tokens in a step the cache is skipped, and a step that also carries a new request's prompt is a large batch too. On the T4, four requests at once made 37.0 tokens/s between them, less than one request managed alone, and the hit rate slipped from 95.7% to 92.3%. This is a feature for one person or a small team on one box, not for a busy endpoint.
Speculative decoding grows the batch the same way, since verifying a draft feeds several tokens per request at once, and the draft model never gets a cache of its own.
On several GPUs, the budget is split like the layers (by --tensor-split, or by free memory), and llama.cpp turns off pipeline parallelism while the cache is on. Tensor-parallel split mode (-sm tensor) refuses to start with it; on our two T4s the error read MoE cache requires a GPU backend.
A budget near the floor is worse than none. At the script's "cached at all from" number, a group holds about one token's experts for one layer, and all its layers share that pool, so the next layer evicts them before they're used again. On the T4, 50 MiB gave 3.8 tokens/s against 9.6 with no cache, and 2 hits in 243,904 lookups.
Don't expect identical replies at temperature 0 either. With the cache, those experts run in CUDA kernels instead of on the CPU, and the results aren't bit-for-bit the same, so a greedy pick can flip and the text diverges from there. Under --cpu-moe, 1 of our 4 replies matched the no-cache run word for word and the others diverged after 15 to 367 characters; under --fit, 2 of 4 matched. Judge quality on your own evals.
Checks before you trust it
- Run the script on your GGUF for your
--moe-cache-mib. It models--cpu-moeand--n-cpu-moe; under--fit, only the layers left in RAM are cached. - Start llama-server with
-lv 4and confirm the startup lines show the same slots and no "budget is too small" warning. - Replay real prompts with
--parallelset to what you'll run, stop the server, and read theubatch <= 8hit rate. - Compare decode tokens/s against the same replay with
--moe-cache-mib 0.
Do this today
If you run an MoE model on one GPU that can't hold it, update to b11712 or later, run the script, and try --cpu-moe with most of your free VRAM as cache, plus -lv 4 so you can see what it hits. On a T4 that was one flag and a restart for nearly twice the decode speed.
What hit rate do you get on your own prompts, and does it hold up once a second user starts generating?
InferPrice ran the script and these commands on a free Kaggle T4 (16 GB, CUDA 12.8) on October 17, 2026, with llama.cpp b11712 built from source and Qwen3-30B-A3B Q4_K_M from Qwen's Hugging Face repo, and checked the rest against the b11712 source. The RTX 4090 and 5090 figures were measured by the llama.cpp developer who wrote the feature, in pull request #29887. The script is tested against GGUF files written by llama.cpp's own gguf-py and cross-checked against a line-by-line port of llama-moe-cache.cpp.
Sources
- llama.cpp PR #29887: GPU cache for MoE experts kept in host memory (description and benchmarks)
- llama.cpp commit d6cf9ac: add a GPU cache for MoE experts kept in host memory
- llama.cpp commit c811cb8: support MoE cache over multiple GPUs (#30112)
- llama.cpp commit b42b7e6: server uses port 9931 by default (#30159)
- llama.cpp source at b11712: src/llama-moe-cache.cpp
- llama.cpp source at b11712: common/fit.cpp
- llama.cpp source at b11712: common/log.cpp (log levels)
- llama.cpp server README (flags and environment variables)
- Qwen3-30B-A3B GGUF (the model we ran)
- InferPrice GPU prices (H100 and H200, checked October 17, 2026)
Originally published at inferprice.pages.dev.
Top comments (0)