Pick Q4_K_M by default; go Q6_K or Q8_0 when you have VRAM to spare and need the last few percent of quality. GGUF quantization shrinks a model's weights from 16 bits to fewer — Q4_K_M stores roughly 4.85 bits per weight, so a 7B model drops from ~14 GB to ~4.1 GB with perplexity typically less than 1% worse than the original. The question "which gguf quantization to use" has a boringly stable answer that most of the drama online obscures. Below: what quantization actually does to weights, how much quality each level costs, how to do the size math yourself, and one command to measure the damage on your own hardware instead of trusting a stranger's benchmark.
What does GGUF quantization actually do?
A model is trained in 16-bit floating point (FP16 or BF16): every one of its billions of weights is a 2-byte number. Quantization compresses each weight into fewer bits. The naive way — round every weight to a 4-bit integer — destroys small-but-important values, so GGUF uses two tricks:
- Block scaling. Weights are grouped into blocks (usually 32), and each block gets its own scale factor. The 4-bit values are offsets within that block, so a wide range of magnitudes survives.
- Importance-aware k-quants. The "K" in Q4_K_M means super-blocks of scales, plus treating attention and feed-forward layers differently from each other, because they tolerate compression unequally.
The "I" family (IQ4_XS and friends) goes further with information-theoretic codebooks borrowed from image compression. Same idea, fancier encoding: fewer bits per weight at similar quality, at the cost of slightly slower inference on some backends.
One clarification that prevents most confusion: quantization changes only the stored weights. Architecture, tokenizer, and context handling are untouched. A Q4 file and a Q8 file of the same model are the same model, wearing different coats.
Q4 vs Q8: is higher quantization better?
Yes, technically; no, perceptually. Using llama.cpp's own perplexity runs on Llama models as the reference: Q8_0 lands within ~0.02% of FP16 — for any practical purpose, lossless. Q6_K is near-indistinguishable. Q4_K_M gains roughly 1–2% perplexity, Q4_0 a bit more, and Q2_K is where coherent answers start falling apart on small models.
Two rules the numbers imply:
- Model size buys quantization headroom. A 70B model survives Q2/Q3 far better than a 7B model does, because larger models are more redundant. Quantizing a 7B to Q2 is amputation; quantizing a 70B to Q3 is tailoring.
- The quality floor moves with the task. Chat tolerates Q4. Exact code generation, math, and RAG over precise documents expose quantization noise sooner. If a Q4 model keeps writing subtly wrong code, test the same model at Q6_K before you blame the model.
| Level | Bits/weight | Size vs FP16 | Quality loss | Use it when |
|---|---|---|---|---|
| Q2_K | ~3.4 | ~21% | Severe on sub-13B | Nothing else fits, large models only |
| Q3_K_M | ~3.9 | ~25% | Noticeable | Tight VRAM, ≥14B models |
| Q4_K_S | ~4.6 | ~29% | Small | Q4_K_M won't fit and it's close |
| Q4_K_M | ~4.85 | ~30% | ~1% perplexity | The default. Best quality/size trade |
| Q5_K_M | ~5.7 | ~35% | ~0.5% | VRAM available, quality-critical tasks |
| Q6_K | ~6.6 | ~41% | Near-nil | Code/math, still fits comfortably |
| Q8_0 | ~8.5 | ~53% | Effectively none | Reference runs, fine-tune bases |
| IQ4_XS | ~4.3 | ~27% | ≈Q4_K_M | Q4_K_M slightly too big, backend supports i-quants |
How much VRAM does each level need?
Do the size math yourself instead of memorizing tables — it is one line:
size_GB ≈ (bits_per_weight × params) / 8
# 8B model @ Q4_K_M: 4.85 × 8 / 8 ≈ 4.9 GB
# 8B model @ Q8_0: 8.50 × 8 / 8 ≈ 8.5 GB
Then add the parts the formula leaves out: the KV cache (grows with context length — a few hundred MB to multiple GB), activations, and compute buffers. Practical margin: a "4.9 GB" model wants a 6 GB card at 4k context, and flash-attention plus KV-cache quantization to stay there at 16k. Weights are the headline, not the whole bill.
Which GGUF quantization should you use?
Decision order, no exceptions worth memorizing:
- Compute your context + KV budget first, then weights. Context you can't fit is worse than quality you can't measure.
- Default to Q4_K_M. It is the community's default for a reason — roughly 1% perplexity for 70% of the size. Every registry, including Ollama's, ships it as the baseline.
- Step up to Q6_K when the task punishes noise: code, math, extraction, anything you feed to a pipeline unattended.
- Use Q8_0 only for reference — A/B tests, quantization-damage measurement, or a fine-tune base. As a daily driver it mostly buys warmth in your VRAM sensors.
- Go below Q4 only under duress, and only on large models. Test with a known-hard prompt before trusting it.
If you are choosing which file to download on Hugging Face, prefer a single Q4_K_M.gguf over sharded splits unless the uploader only ships the latter — fewer moving parts. And if you are choosing where to run it, the engine choice is separate: see llama.cpp vs Ollama for that axis.
How do you measure quantization damage yourself?
Benchmarks differ; your prompt is constant. Build llama.cpp once, download two levels of the same model, and measure both perplexity (lower is better) and tokens/second:
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release -j
huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf Meta-Llama-3.1-8B-Instruct-Q8_0.gguf \
--local-dir models
# perplexity on a wiki-text chunk (lower = closer to the original model)
./build/bin/llama-perplexity -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99
./build/bin/llama-perplexity -m models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf -ngl 99
# and speed on the same hardware
./build/bin/llama-bench -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99
The Q4_K_M number will be a few hundredths of a point worse than Q8_0 and the file will be ~40% smaller. If your downstream task can't tell the difference — and for most, it can't — you have your answer without reading anyone's leaderboard.
Does quantization hurt privacy or local-only claims?
No — it is arithmetic on weights, entirely offline, and the quantized file is just a smaller container of the same parameters. Running a Q4_K_M locally leaks exactly as much (or little) as running the full-precision model locally: nothing leaves the machine. The privacy-relevant variable is where inference runs, not the bit width. The usual caveats about model provenance apply equally to every quant level: a stolen-base "uncensored" fine-tune in Q8 is not safer than the same weights in Q4. For the file-format side of this, see how to run GGUF models locally.
Which level should you pick?
Q4_K_M, and stop reading forums about it. Upgrade to Q6_K for precision-hungry work if VRAM allows, keep one Q8_0 around for A/B comparisons, and treat anything under Q4 as an emergency ration for large models only. The one mistake worth avoiding is symmetric: worrying about Q4-vs-Q5 while ignoring context length, which wrecks more local setups than any quant ever did.
Top comments (0)