Gemma 4 26B CPU Inference Benchmark: 5 tok/s Production Math [2026]

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

Gemma 4 26B CPU inference is the practice of running Google's 26-billion-parameter Mixture-of-Experts (MoE) model entirely on a server CPU, with no GPU at all. A viral benchmark from Ryan Findley of Neomind Labs proved it works — 5.2 tok/s decode on a 13-year-old Xeon costing under $300. But the production questions the benchmark left unanswered are more important than the headline number.

Key takeaways:

• Gemma 4 26B-A4B only activates ~4 billion parameters per token thanks to its MoE architecture, making CPU inference viable where a dense 26B model would be unusable.
• The 5.2 tok/s decode rate is not arbitrary — it's directly predicted by DDR3 memory bandwidth math (~40–50 GB/s reading ~8 GB of Q8_0 weights per token).
• CPU inference works for single-user production workloads like async RAG pipelines and air-gapped deployments, but collapses past 2-3 concurrent users.
• Dropping from Q8_0 to Q4_K_M nearly doubles throughput while halving RAM, but quality loss is model-family-specific and must be tested.
• At low concurrency, a $300 used Xeon pays for itself in under 3 months versus GPU cloud rental at $0.50–$1.50/hour.

CPU inference isn't a compromise — it's the right architecture for any LLM workload where latency tolerance exceeds 200ms per token.

The Hardware: What a $300 Xeon Server Actually Gives You

The Neomind Labs benchmark ran on a repurposed HP StoreVirtual — a storage appliance, not a compute server. As Ryan Findley, Engineer and Founder at Neomind Labs, describes it: dual Xeon E5-2690 v2 processors (Ivy Bridge, 2013 vintage), DDR3 memory, and no GPU whatsoever. Total cost: under $300 on the used server market.

What matters for inference is what that $300 buys you in compute terms. Each E5-2690 v2 has 10 cores at 3.0 GHz base, 25 MB of L3 cache, and support for DDR3-1866 across four memory channels. With dual sockets, you get 8 memory channels total, yielding a theoretical memory bandwidth ceiling of roughly 119 GB/s (8 channels × 14.9 GB/s per DDR3-1866 channel). In practice, sustained bandwidth on aged DDR3 DIMMs lands closer to 40–50 GB/s due to memory controller overhead, NUMA effects across two sockets, and the reality of 13-year-old hardware.

This is important context. When people on Hacker News called 5 tok/s "slow," they were comparing it to GPU inference on hardware that costs 10–30× more. For a $300 server running in a closet, the real question isn't "is it fast?" It's "is it fast enough for my use case?" That's a different question entirely.

You can find comparable used Xeon servers (Haswell v3 or Broadwell v4 with DDR4) for $400–$600 on eBay today, and those will push the bandwidth ceiling higher. The point isn't that you must use a 2013 box — it's that the floor for viable CPU inference is remarkably low.

Why Gemma 4 26B-A4B Is Uniquely Suited to CPU Inference

The "A4B" in Gemma 4 26B-A4B stands for "4 billion active parameters." This is the key architectural detail that makes the entire benchmark possible.

Gemma 4 uses a Mixture-of-Experts architecture. The model has 26 billion total parameters, but on any given token, only about 4 billion are activated — the router selects a subset of expert layers while the rest sit idle. As the Hugging Face Gemma 4 announcement from April 2, 2026 explains, this comes with per-layer embeddings (PLE), shared KV cache, and long context window support.

For GPU inference, the MoE architecture is mostly about efficiency — you get 26B-quality outputs at 4B-level compute cost. But for CPU inference, it's transformative. A dense 26B model would need to read all 26 billion parameters from memory for every single token. With MoE, the model only reads the ~4B active weights per forward pass, dramatically reducing the memory bandwidth demand per token.

This is why Gemma 4 26B runs at 5 tok/s on a CPU where a dense 26B model would crawl at under 1 tok/s. It's also why Qwen3.6-35B-A3B, another MoE model with only 3B active parameters, manages 7–9 tok/s on a 16 GB MacBook Air with no discrete GPU, as HN user dwa3592 demonstrated. MoE is the enabling architecture for practical CPU inference in 2026.

Gemma 4 is also released under a clean Apache 2.0 license with no additional restrictive terms — unlike prior Gemma versions which carried a separate "Gemma Terms of Use." For production deployment, this matters. You can redistribute, modify, and commercially deploy without legal friction. As Simon Willison noted, this makes Gemma 4 genuinely open in a way its predecessors were not.

The Memory Bandwidth Math: Why 5 tok/s Is Exactly What Physics Predicts

Here's the thing nobody in the original post or the HN thread explained: 5.2 tok/s isn't a random number. It's a direct consequence of memory bandwidth.

During autoregressive decode (generating one token at a time), the bottleneck is reading model weights from RAM. For each token, the inference engine must load the active parameters from memory, multiply them against the input, and produce the next token. On CPU, this is entirely memory-bound — the compute cores sit idle waiting for data most of the time.

Let's do the math. Gemma 4 26B-A4B at Q8_0 quantization stores each parameter in 8 bits (1 byte). With ~4B active parameters per token, that's roughly 4 GB of weight data read per forward pass. On the Neomind Labs server with ~40–50 GB/s of usable memory bandwidth, you get:

• 45 GB/s ÷ 4 GB per token ≈ 11 tok/s theoretical maximum
• Real-world overhead (KV cache reads, attention computation, NUMA latency, OS scheduling) cuts this roughly in half
• Result: ~5–6 tok/s — exactly what was measured

This is why prompt evaluation (processing the input context in parallel) runs at ~16 tok/s. During prompt eval, the engine batches multiple tokens together, amortizing the weight-read cost across all of them. The same memory bandwidth serves more useful work per read cycle.

Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, this bandwidth-bound pattern holds across hardware tiers. On Apple Silicon with unified memory delivering 200+ GB/s, the same MoE models hit 20–40 tok/s. On NVIDIA GPUs with HBM delivering 1+ TB/s, you see 100+ tok/s. The ratio tracks memory bandwidth almost linearly — confirming that CPU decode is a bandwidth problem, not a compute problem.

Instruction Set Requirements: The AVX2 Gotcha That Breaks Old Xeons

The Neomind Labs benchmark surfaced one of the most practical gotchas in CPU inference: instruction set compatibility.

Ryan Findley discovered that ik_llama.cpp's optimized CPU kernels require AVX2 and FMA3 instruction sets, which Intel introduced with the Haswell architecture ("v3" Xeons) in 2014. His Ivy Bridge chips ("v2", 2013) only support AVX1 and have no FMA3 at all. The optimized inference paths simply wouldn't execute.

He used Claude as a coding agent to diagnose the build failure and rewrite the hot paths to fall back cleanly on pre-AVX2 hardware. It worked, but there's a performance penalty. AVX2 doubles the SIMD width (256-bit operations on integers, not just floats), and FMA3 fuses multiply-accumulate into single instructions — both directly impact how fast you can process matrix operations on CPU.

Here's the practical instruction set matrix for anyone shopping for used servers:

CPU Generation	Year	AVX Level	FMA3	AVX-512	ik_llama.cpp Fast Path	Estimated Speed Penalty
Sandy Bridge (v1)	2012	AVX1	No	No	No	~40-50% slower
Ivy Bridge (v2)	2013	AVX1	No	No	No (needs fallback)	~30-40% slower
Haswell (v3)	2014	AVX2	Yes	No	Yes	Baseline
Broadwell (v4)	2016	AVX2	Yes	No	Yes	~5-10% faster
Skylake-SP	2017	AVX2	Yes	AVX-512	Yes + 512-bit paths	~15-25% faster

The takeaway: if you're buying a used server specifically for local LLM inference, Haswell (v3) is the minimum generation worth targeting. You'll find dual-socket Haswell servers with 128 GB DDR4 for $350–$500 on the used market. The extra $100–$200 over an Ivy Bridge box buys you both the AVX2 fast path and DDR4 bandwidth — easily a 2× throughput improvement.

Quantization Sweet Spot: Q4_K_M vs Q5_K_M vs Q8_0 on CPU

Ryan Findley ran the benchmark at Q8_0 quantization, but never explained why — and never compared alternatives. For anyone actually deploying this, quantization choice is the single biggest lever you have.

Quantization reduces model precision from the original 16-bit floats to fewer bits per weight. Less precision means smaller files, less RAM, and faster inference (less data to read from memory). The tradeoff is output quality degradation.

For Gemma 4 26B-A4B, here's the estimated breakdown across the three most common quantization levels:

Quant Level	Bits/Weight	Est. GGUF Size	RAM Needed	Est. CPU tok/s (DDR3)	Quality vs FP16
Q8_0	8	~26 GB	32+ GB	~5 tok/s	~99%
Q5_K_M	5.5 (avg)	~18 GB	24+ GB	~7–8 tok/s	~97%
Q4_K_M	4.5 (avg)	~15 GB	20+ GB	~9–10 tok/s	~94-95%

The Q4_K_M sweet spot is compelling: nearly double the throughput of Q8_0, fits in 20 GB of RAM, and retains ~95% of full-precision quality for most tasks. But there's a critical caveat. When building the Walmart conversational commerce chatbot at Firework, I learned that retrieval quality dominated answer quality at scale — model quantization mattered far less than whether the right context was retrieved. For a RAG pipeline, Q4_K_M is almost certainly good enough because the retrieval stage, not the generation stage, is where quality lives or dies.

However, for tasks demanding precise reasoning or numerical accuracy, the quality gap between Q4_K_M and Q8_0 is noticeable. The right quantization depends on your specific use case, not a blanket recommendation. This is something I've confirmed across multiple model families through the benchmark work on this site — quantization quality cliffs are model-family-specific, and a blanket Q4 recommendation is wrong.

For a deeper dive on quantization formats beyond GGUF, see the GGUF vs GPTQ vs EXL2 comparison.

How Fast Is Gemma 4 on CPU vs GPU?

Let's put the 5 tok/s number in context against the full speed spectrum.

Simon Willison documented that Ornith-1.0, a 35B MoE model built on Gemma 4 architecture, runs its Q4_K_M GGUF (20 GB) at 103 tok/s on a consumer GPU via LM Studio. DiffusionGemma, the diffusion-based variant of the same 26B-A4B architecture, hits 500+ tok/s on NVIDIA's NIM cloud infrastructure.

That's the speed landscape:

Platform	Speed	Cost	Multiplier vs CPU
13-year-old Xeon (DDR3, no GPU)	~5 tok/s	$300 one-time	1×
Modern Xeon / Ryzen (DDR5, no GPU)	~15–20 tok/s	$800–$1,200 one-time	3–4×
Apple Silicon M4 Max (unified memory)	~25–40 tok/s	$2,500–$3,500 one-time	5–8×
Consumer GPU (RTX 4090, 24 GB VRAM)	~80–103 tok/s	$1,600–$2,000 GPU alone	16–20×
Cloud GPU (A100/H100)	~200–500 tok/s	$0.50–$3.00/hour	40–100×

The 20× gap between CPU and consumer GPU is real. But the cost gap is also real. A used Xeon server costs $300. A GPU cloud instance at $1.00/hour costs $720/month. For a local LLM deployment serving a single user on internal tools, the CPU path pays for itself before you finish your first quarterly OKR review.

For more on GPU hardware tradeoffs, I've covered the RTX 4060 Ti vs RTX 4070 for local inference and the Apple Silicon vs NVIDIA decision in detail.

Is 5 Tokens Per Second Fast Enough for Production?

This is the question everyone in the HN thread was really asking. And the answer depends entirely on concurrency.

At 5 tok/s, a 200-token response takes 40 seconds. That's slow for a chatbot. But for an async document processing pipeline — summarizing PDFs, extracting entities from contracts, classifying support tickets — 40 seconds per document is fine. Most batch workloads don't care about latency measured in seconds.

Here's the concurrency math that nobody in the original coverage provided:

• 1 concurrent user: 5.2 tok/s — usable for interactive RAG queries if you stream the response
• 2 concurrent users: ~2.6 tok/s each — tolerable with streaming, but pauses are noticeable
• 3 concurrent users: ~1.7 tok/s each — painful for interactive use, fine for async
• 5 concurrent users: ~1 tok/s each — unusable for anything interactive
• 10+ concurrent users: queue depth explodes, effective throughput collapses

CPU inference on this class of hardware is fundamentally a single-user architecture. If you need to serve even 3 concurrent users interactively, you need a GPU or multiple CPU servers behind a load balancer.

As one HN commenter (nkozyra) put it: "We're smashing ants with hammers most of the time. We're asking frontier models to classify text and build frontend code." That's exactly right. For internal tooling where one engineer at a time queries a knowledge base, 5 tok/s with streaming is a perfectly adequate experience — and the cost savings versus GPU cloud are dramatic.

I've written more about the cost math of local vs cloud inference and the latency budgets that matter for production AI agents.

CPU vs GPU: The Production Cost-Benefit Calculation

Let's do the actual break-even math for a 12-month internal RAG deployment at low concurrency.

Option A: $300 Used Xeon (CPU-only)

• Hardware: $300 one-time
• Electricity: ~200W × 24h × 365 days × $0.12/kWh = ~$210/year
• Total 12-month cost: ~$510
• Performance: 5 tok/s, single user

Option B: GPU Cloud Instance (e.g., Lambda Labs A10G)

• Hourly rate: ~$0.75/hour
• 24/7 operation: $0.75 × 24 × 365 = ~$6,570/year
• Even at 8 hours/day, 5 days/week: $0.75 × 8 × 260 = ~$1,560/year
• Performance: 80–150 tok/s, handles 5–10 concurrent users

Option C: Consumer GPU (RTX 4070 + system)

• Hardware: ~$1,200 total system cost
• Electricity: ~300W typical = ~$315/year
• Total 12-month cost: ~$1,515
• Performance: 40–60 tok/s, handles 3–5 concurrent users

The break-even point between the Xeon CPU path and a GPU cloud instance is roughly 25–30 days of continuous operation at typical cloud GPU rates. If you need the server running more than a month, the CPU box is cheaper. If you need multi-user concurrency or sub-second responses, the GPU path is the only viable option.

For a deeper look at LLM cost optimization, including when API calls beat self-hosting entirely, see my cost-reduction techniques guide.

When CPU Inference Wins: The Right Use Cases

CPU inference isn't a poor man's GPU. It's the right architecture for specific workloads. Here's the decision framework:

CPU inference is the right choice when:

1. Single-user internal tools — one engineer querying an internal knowledge base during working hours. The RAG pipeline processes documents asynchronously, and queries come one at a time.
2. Async batch processing — nightly document summarization, weekly report generation, ETL pipelines that extract structured data from unstructured text. Latency doesn't matter; cost per token does.
3. Air-gapped or regulated environments — government, healthcare, and financial services deployments where data cannot leave the premises and cloud GPU instances are not an option. A physical server in a locked room is sometimes the only compliant path.
4. Development and testing — running evals, testing prompt templates, iterating on RAG configurations before deploying to a GPU-backed production stack.
5. Budget-constrained startups — early-stage teams validating an AI feature before committing to GPU infrastructure. A $300 server lets you ship a prototype without burning runway.

CPU inference is the wrong choice when:

• You need to serve more than 2-3 concurrent users interactively
• Response latency under 2 seconds matters (chatbots, real-time assistants)
• You're running large context windows (32K+ tokens) — KV cache pressure on CPU is brutal
• You need multimodal inference (image + text) at interactive speeds

For those workloads, look at the local LLM hardware guide or the vLLM vs Ollama production comparison.

The ik_llama.cpp Fork: What It Adds and Why Mainline llama.cpp Is Slower on CPU

The Neomind Labs benchmark used ik_llama.cpp, a fork of Georgi Gerganov's llama.cpp (121K+ GitHub stars). The fork matters because mainline llama.cpp, while excellent for GPU inference, leaves significant CPU performance on the table.

ik_llama.cpp adds four CPU-specific optimizations:

1. CPU-aware MoE routing — the expert selection logic is optimized for cache locality on x86 CPUs, ensuring the activated expert weights are read sequentially rather than scattered across memory pages.
2. Flash attention ported to CPU — the standard flash attention implementation targets GPU SRAM. ik_llama.cpp adapts it for CPU L2/L3 cache hierarchies, reducing memory round-trips during attention computation.
3. Run-time weight repacking — on startup, the model weights are rearranged in memory to align with the CPU's cache line size and NUMA topology. This costs a few seconds at load time but pays dividends on every subsequent token.
4. Speculative decoding for CPU — a smaller draft model proposes several tokens at once, and the main model verifies them in a single forward pass. This amortizes the per-token weight-read cost, effectively batching single-user decode.

The catch: these optimizations require AVX2 and FMA3 (Haswell 2014+). On older CPUs, you need the fallback paths that Ryan Findley engineered — or you're stuck with mainline llama.cpp's more conservative CPU code paths.

For most people starting with CPU inference today, Ollama wrapping mainline llama.cpp is the easiest entry point. If you're optimizing for maximum CPU throughput on known hardware, ik_llama.cpp with the right flags is worth the setup complexity. The original benchmark post used roughly 25 carefully tuned flags — this isn't a one-click experience.

Benchmark Results: Prompt Eval vs Decode Speed Explained

The Neomind Labs results showed two very different numbers: ~16 tok/s for prompt evaluation and ~5.2 tok/s for decode. Many readers conflated these, but they measure fundamentally different operations.

Prompt eval (~16 tok/s) processes the entire input context — your system prompt, retrieved documents, and user query — in parallel. All input tokens are known upfront, so the engine can batch the matrix multiplications, reading weights once and applying them across multiple tokens simultaneously. This is compute-bound, not memory-bound, and benefits directly from having 20 CPU cores (dual 10-core Xeons).

Decode (~5.2 tok/s) generates output tokens one at a time, autoregressively. Each new token depends on the previous one, so there's no parallelism to exploit. Every token requires a full read of the active model weights from RAM. This is purely memory-bandwidth-bound.

The 3× gap between prompt eval and decode is consistent with what I see across hardware tiers in the benchmarks on this site. On Apple Silicon with higher memory bandwidth, the absolute numbers go up but the ratio stays similar. On GPUs with HBM, the decode speed increases dramatically because HBM delivers 10–20× the bandwidth of DDR3.

Practically, the 16 tok/s prompt eval means a 2,000-token RAG context (system prompt + retrieved chunks + query) processes in about 125 seconds. That's the real startup cost per query. Once the context is loaded, streaming the response at 5 tok/s is the interactive part. For document-heavy RAG workloads, prompt eval speed matters as much as decode speed — and it's the number that improves most with better hardware.

What Comes Next: The Trajectory That Matters

The Gemma 4 CPU benchmark is a snapshot of mid-2026. The trajectory is what should get your attention.

As HN user dwa3592 predicted: "By mid-2027, we will have >200B MoE models running on basic consumer hardware." That prediction isn't wild speculation. Prism's Bonsai 27B already runs as a ternary model (~7 GB, 2-bit quantization) at 44+ tok/s on an M4 Max. Recursive architectures like HRM are reducing parameter counts needed for equivalent capability. And DDR5 servers hitting the used market in 2026–2027 will push the bandwidth ceiling from 40–50 GB/s to 80–100 GB/s, roughly doubling CPU inference speeds for the same dollar.

The combination of sparsely activated MoE architectures, aggressive quantization techniques, and dropping hardware costs is making the GPU-required assumption for production AI increasingly wrong for low-concurrency workloads. Not for ChatGPT-scale serving. Not for real-time multimodal agents. But for the internal tools, batch pipelines, and air-gapped deployments that represent most enterprise AI workloads? The $300 CPU path is already viable and getting better fast.

If you're building an internal RAG tool for a small team, stop defaulting to GPU cloud. Do the concurrency math. Check your latency tolerance. And consider that a server in a closet, running a genuinely open Apache 2.0 model, might be exactly the production AI architecture your use case demands.

Frequently Asked Questions

Can you run a 26B parameter model on a CPU without a GPU?

Yes, if the model uses a Mixture-of-Experts (MoE) architecture like Gemma 4 26B-A4B. The MoE design activates only ~4 billion parameters per token, dramatically reducing the memory bandwidth needed per forward pass. A server with 32+ GB of RAM and a modern x86 CPU can run the Q8_0 quantized version at approximately 5 tokens per second.

What is the minimum RAM needed to run Gemma 4 26B?

At Q8_0 quantization, the model file is approximately 26 GB, so you need at least 32 GB of system RAM. At Q4_K_M quantization, the file shrinks to roughly 15 GB, making it runnable on systems with 20+ GB of RAM. You should always leave headroom above the model file size for the KV cache and operating system overhead.

What is Q4_K_M quantization and how does it affect model quality?

Q4_K_M is a mixed-precision quantization format in GGUF that averages about 4.5 bits per weight. It retains approximately 94–95% of the original model's quality while cutting file size by nearly half compared to Q8_0. The "K" indicates k-quant grouping (which preserves important weights at higher precision), and the "M" indicates medium granularity. For most RAG and summarization tasks, the quality loss is imperceptible.

What is the difference between ik_llama.cpp and regular llama.cpp?

ik_llama.cpp is a fork of llama.cpp optimized specifically for CPU inference. It adds CPU-aware MoE routing, flash attention adapted for CPU cache hierarchies, run-time weight repacking for cache-line alignment, and CPU-optimized speculative decoding. These features require AVX2 and FMA3 instruction sets (Intel Haswell 2014 or newer). Mainline llama.cpp works on CPUs but doesn't include these aggressive CPU-specific optimizations.

When should I use CPU inference instead of GPU for LLM production?

CPU inference is the right choice for single-user internal tools, async batch processing pipelines, air-gapped environments where cloud GPUs are not an option, and budget-constrained prototyping. If you need to serve more than 2-3 concurrent users interactively or require sub-2-second response times, GPU inference is the better path.

How much does it cost to self-host a 26B LLM vs using a cloud API?

A used Xeon server capable of running Gemma 4 26B costs approximately $300 upfront plus ~$210/year in electricity. GPU cloud instances run $0.50–$3.00 per hour, costing $4,380–$26,280 per year for 24/7 operation. The CPU self-hosting path breaks even against cloud GPU rental in roughly 25–30 days of continuous operation, making it dramatically cheaper for always-on low-concurrency workloads.

---

Originally published on kunalganglani.com