Picking a Gemma 4 Quantization: VRAM Math That Actually Matters

ethanlin — Fri, 24 Jul 2026 19:00:12 +0000

Every "run this model locally" guide tells you to grab a Q4 GGUF and move on. That advice is fine right up until you try a long-context run and your machine starts swapping.

The weights are the part everyone budgets for

Quantization maths is straightforward. A model's weight footprint is roughly params x bits / 8:

Quant	Bits/param	12B model	Quality note
Q8_0	~8.5	~12.8 GB	Near-lossless, rarely worth it
Q6_K	~6.6	~9.9 GB	Very close to Q8
Q4_K_M	~4.8	~7.2 GB	The usual sweet spot
Q3_K_M	~3.9	~5.9 GB	Noticeable degradation

Below Q4 the loss stops being subtle. Instruction-following degrades before raw perplexity does, which is why benchmark numbers can look fine while the model quietly stops respecting your system prompt.

The KV cache is the part that bites

Here is what the guides skip. The KV cache scales with context length, and it is not quantized by default:

kv_bytes ~= 2 (K and V) x layers x kv_heads x head_dim x seq_len x dtype_bytes

The practical consequence: a model that loads in 7 GB can need well over twice that at long context. Grouped-query attention helps a lot — kv_heads is much smaller than attention heads — but the term still grows linearly with sequence length while your weights stay fixed.

Two knobs matter more than picking a fancier quant:

--ctx-size: do not allocate 128K if your prompts are 8K. You are reserving memory you will never touch.
KV cache quantization (q8_0 for K/V): roughly halves cache memory for a quality hit most workloads never notice. Underused.

A decision order that works

Start at Q4_K_M
Set context to what you actually use, not the model maximum
If you are still tight, quantize the KV cache before dropping to Q3
Only move up to Q6/Q8 if you have headroom left over

That ordering matters: dropping to Q3 to buy context is the most common mistake, and it trades a permanent quality loss for memory you could have gotten from the cache instead.

Per-quantization benchmarks and deployment notes for the Gemma 4 family are collected at gemma-4.net — every number self-run, raw data published.

Caveat

These are rules of thumb, not guarantees. Backends differ in how they allocate, and Apple Silicon unified memory behaves differently from discrete VRAM. Measure on your own hardware before trusting any table, including mine.

DEV Community: ethanlin

Picking a Gemma 4 Quantization: VRAM Math That Actually Matters

The weights are the part everyone budgets for

The KV cache is the part that bites

A decision order that works

Caveat