q4_k_m vs q8_0: which GGUF quant should you use?
Q4_K_M is the best choice for speed and memory efficiency, while Q8_0 is better if you need maximum fidelity and have enough VRAM to spare. On Llama-3.1-8B, Q4_K_M uses 4.58 GiB of space, whereas Q8_0 requires 7.95 GiB.
How do Q4_K_M and Q8_0 differ in performance?
Q4_K_M offers much faster generation speeds and a smaller footprint, but Q8_0 maintains higher accuracy by staying closer to the original FP16 model's perplexity and KL divergence. You trade a small amount of logic precision for significantly more headroom on your GPU.
The difference comes down to bits per weight. According to the llama.cpp quantize documentation, Q4_K_M is approximately 4.89 bits per weight (https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize). Q8_0 uses roughly 8.5 bits per weight (https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize).
I look at perplexity to see how much the model "stumbles" compared to the original. In a llama.cpp perplexity test on Llama-3 8B, Q8_0 achieved a PPL of 6.234284, while Q4_K_M was higher at 6.407115 (https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/perplexity/README.md). Lower is better.
An independent study in arXiv:2601.14277v1 shows that Q8_0 is a high-fidelity baseline, but its compression is weaker than 4 or 5 bit options (https://arxiv.org/html/2601.14277v1). In task benchmarks, the average score for Q4_K_M was 69.15 and Q8_0 was 69.41 (https://arxiv.org/html/2601.14277v1).
Which quant should I choose for my hardware?
Choose Q4_K_M if your GPU VRAM is limited or you need high tokens per second. Choose Q8_0 if you have plenty of VRAM and the task requires the highest possible accuracy from the model.
The choice depends on your specific bottleneck. If you are running local LLM inference on consumer GPUs, VRAM is your most precious resource.
Use this guide to decide:
| Feature | Q4_K_M | Q8_0 |
|---|---|---|
| Llama-3.1-8B Size | ~4.58 GiB | ~7.95 GiB |
| Generation Speed | Faster (71.93 t/s) | Slower (50.93 t/s) |
| Accuracy (PPL) | 6.407 | 6.234 |
| Best Use Case | High speed / Low VRAM | High fidelity / High VRAM |
Note: size and speed rows are the Llama-3.1-8B run in the llama.cpp quantize README (https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/quantize/README.md). The perplexity row is the Llama-3 8B run in the perplexity README (https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/perplexity/README.md), the rows measured without an imatrix. They are two different measurements.
If you are unsure about the math behind these sizes, read my GGUF quantization explained post or use the quant compare tool to see how different weights affect your model.
How do I create these quants myself?
You can use the llama.cpp quantize tool to generate these files from an existing F16 or F32 GGUF. You just need to specify the target type and the number of threads for the process.
Run this command in your terminal:
./llama-quantize input-model-f32.gguf q4_k_m 8
The q4_k_m part is the quantization type. The 8 at the end is the number of CPU threads you want to use. To make a Q8_0 version, simply replace q4_k_m with q8_0. Be careful when using the --allow-requantize flag, as it can reduce quality compared to quantizing directly from 16-bit (https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize).
What should you do with this?
- Check your available VRAM using
nvidia-smibefore downloading a new quant. - Download the Q4_K_M version first to ensure the model fits and runs at your target speed.
- Only move to Q8_0 if you notice logic errors in the smaller quant and have the extra GiB of VRAM available for context.
Accompanying prompt
What the prompt does: Helps you decide between specific GGUF quantization levels based on your hardware constraints.
Copy/paste this prompt:
Role:
Hardware-aware LLM Engineer
Context:
I am running local LLM inference and need to choose between different GGUF quantization levels (e.g., Q4_K_M, Q6_K, Q8_0) for a specific model size.
Task:
1. Analyze my available VRAM and the model parameter count I am using.
2. Compare the estimated memory footprint of Q4_K_M vs Q8_0 for this model.
3. Recommend a quantization level that balances generation speed (tokens per second) and model perplexity.
Output:
- A recommendation for which quant to use.
- An estimate of how much VRAM will be left for context window/KV cache.
- A brief justification based on the trade-off between precision and speed.
Constraints:
- Prioritize preventing Out-of-Memory (OOM) errors.
- Assume I want to maximize tokens per second unless I state otherwise.
Copy the block above.
Weekly measured local runs: https://bmdpat.com/5090-reports
Related reading
Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=q4-k-m-vs-q8-0-which-gguf-quant-should-you-use-2026
Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.
Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.
Top comments (0)