Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

LLM Quantization Levels Compared: Q4_K_M vs Q8_0 vs FP16 [2026]

Local LLM quantization is the process of reducing a model's weight precision — from 32-bit or 16-bit floating point down to 8-bit, 4-bit, or even lower — so it fits in consumer-grade VRAM without requiring a datacenter GPU. Every developer pulling a model through Ollama or LM Studio today faces the same decision: which quantization level actually gives the best trade-off between quality, speed, and memory? This local LLM quantization levels comparison of Q4, Q8, and FP16 is the guide I wish existed when I started running models locally.

Key takeaways:

• Q4_K_M is the Pareto-optimal default for most consumer setups — roughly 95-97% of FP16 quality at ~35% of the VRAM cost and 2-3× the throughput.
• Q8_0 is effectively lossless (within 0.02-0.05 perplexity points of FP16) and worth the 2× VRAM cost for coding and structured reasoning tasks.
• Running a larger model at Q4_K_M usually beats a smaller model at Q8_0 — model size matters more than quantization precision for overall capability.
• K-quant mixed precision (_S, _M, _L) is fundamentally different from naive Q4_0 — not all 4-bit quantization is equal.
• IQ-quants (importance-matrix quantization) now outperform K-quants at the same bit width and are the upgrade path for 2026.

What Is LLM Quantization and Why It Matters for Local AI

Quantization maps a model's weights from high-precision floating-point representations (typically FP32 or FP16) to lower-precision integers or narrower floats. The motivation is simple: a 7B parameter model at FP16 consumes roughly 14 GB of memory. At Q4_K_M, that same model fits in about 4 GB. At Q8_0, about 7 GB.

As Tim Dettmers, PhD researcher and creator of bitsandbytes, documented: BLOOM-176B requires 8× 80GB A100 GPUs — approximately $15,000 each — to run inference at full FP32 precision. Quantization isn't a nice-to-have. It's the only reason local LLM inference exists on consumer hardware at all.

Quantization quality is a triangle: you trade VRAM, speed, and accuracy — and you can only optimize two.

The explosion of quantization research is real. As of 2026, the Hugging Face Transformers library supports over 20 distinct quantization schemes natively — including GGUF, GPTQ, AWQ, bitsandbytes, HQQ, AQLM, FP8, MXFP4, and torchao. But for the practitioner running models through Ollama or llama.cpp on a consumer GPU or Apple Silicon Mac, the decision comes down to a handful of GGUF quantization levels: Q4_K_M, Q5_K_S, Q8_0, and FP16.

The Quality-Speed-VRAM Triangle

Every quantization choice lives on a three-way trade-off that I call the quality-speed-VRAM triangle. You cannot maximize all three simultaneously:

• Quality (measured by perplexity — lower is better) decreases as you quantize more aggressively.
• Speed (tokens per second) increases as models get smaller, because inference on consumer hardware is memory-bandwidth-bound.
• VRAM usage drops proportionally with bit width.

The reason Q4_K_M dominates as the community default is that it sits at the knee of this curve. Going from FP16 to Q4_K_M cuts VRAM by ~65% and roughly doubles throughput, while only sacrificing 3-5% of measurable quality. Going from Q4_K_M to Q2_K saves another ~50% VRAM but quality falls off a cliff.

Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, unified memory on Apple Silicon changes the VRAM-is-the-limit intuition — big models load but throughput becomes the real constraint. A 70B model at Q4_K_M will load into a 64 GB M3 Max's unified memory, but it generates at 8-12 tokens/sec versus 40-60 tok/s for a 7B model at the same quantization.

This is the insight that changes hardware buying decisions: on limited VRAM, running a larger model at Q4_K_M often beats running a smaller model at Q8_0. A Llama 3 70B at Q4_K_M will outperform a Llama 3 8B at Q8_0 on virtually every benchmark, even though the 8B at Q8_0 has better per-token precision. Model capability scales with parameter count far more than it scales with quantization precision.

How Floating-Point Precision Works: FP32, FP16, BF16, INT8, INT4

To understand why quantization works — and where it breaks — you need to understand what you're throwing away.

Maarten Grootendorst, machine learning engineer and author of Hands-On Large Language Models, explains it clearly: "The more bits we use to represent a value, the more precise it generally is." The dynamic range of a representation determines what weight magnitudes can be stored without clipping.

FP32 uses 32 bits per weight: 1 sign bit, 8 exponent bits, 23 mantissa bits. This is full precision — the training default — but at 4 bytes per parameter, a 7B model consumes 28 GB.

FP16 / BF16 use 16 bits (2 bytes per parameter). FP16 has higher mantissa precision; BF16 has the same exponent range as FP32 but lower mantissa precision, making it better for training stability. For inference, both produce essentially identical results. A 7B model at FP16 is ~14 GB.

INT8 (Q8_0) uses 8 bits (~1 byte per parameter). A 7B model fits in ~7 GB. The key innovation here, as Tim Dettmers showed with LLM.int8(), is that naive INT8 quantization causes catastrophic quality loss because ~0.1% of weights — outlier feature channels in attention layers — have magnitudes that blow up the quantization error. LLM.int8() solves this with mixed-precision decomposition: those outlier channels stay in FP16 while the rest goes to INT8. This is why Q8_0 in GGUF is effectively lossless while naive INT8 is not.

INT4 (Q4_K_M) uses ~4.5 bits on average (~0.5 bytes per parameter). A 7B model fits in ~4 GB. This is where the K-quant mixed-precision strategy becomes critical.

GGUF K-Quant Naming Decoded: What _S, _M, _L Actually Mean

The naming convention in GGUF quantization files confuses almost everyone. Here's the actual system:

The K-quant family was introduced by contributor ikawrakow in llama.cpp and uses importance-matrix-weighted quantization. Instead of quantizing every layer to the same bit width, K-quants assign higher precision to layers that matter more for output quality — particularly attention matrices and the first/last transformer blocks.

The suffixes indicate the aggressiveness of this mixed-precision strategy:

• _S (Small): Most layers at the target bit width, minimal higher-precision layers. Smallest file size, lowest quality within the family.
• _M (Medium): A balanced mix — some layers (especially attention) get bumped to 6-bit while the rest stay at 4-bit. This is the community sweet spot.
• _L (Large): More layers kept at higher precision. Largest file, best quality, but diminishing returns versus _M.

This means Q4_K_M is fundamentally different from Q4_0. Q4_0 is naive 4-bit quantization with uniform precision. Q4_K_M uses a mix of 4-bit and 6-bit layers, resulting in perplexity within ~0.1-0.15 of FP16 — far better than Q4_0, which can degrade 0.5+ perplexity points according to benchmarks in llama.cpp Discussion #406.

Tom Jobbins ("TheBloke"), the most prolific GGUF quantization uploader on Hugging Face with over 3,800 quantized model repositories, established the community convention of labeling these variants. His model cards — which include file size, use-case recommendations, and RAM/VRAM estimates — became the de-facto reference millions of users rely on.

Quick Reference: Quantization Levels Comparison Table

This table summarizes the key numbers across quantization levels for the most common model sizes. These figures are derived from community benchmarks, llama.cpp perplexity measurements, and TheBloke's model cards.

Quant Level	Bits/Weight	VRAM for 7B	VRAM for 13B	VRAM for 70B	Perplexity Δ vs FP16	Tokens/sec (RTX 4090, 7B)	Best Use Case
FP16	16	~14 GB	~26 GB	~140 GB	Baseline (0.00)	12-18 tok/s	Reference / fine-tuning
Q8_0	8	~7 GB	~14 GB	~70 GB	+0.02-0.05	20-30 tok/s	Coding, structured output
Q5_K_M	5.5	~5.5 GB	~10 GB	~50 GB	+0.05-0.08	30-40 tok/s	RAG, long-context tasks
Q5_K_S	5	~5 GB	~9.5 GB	~48 GB	+0.06-0.10	32-42 tok/s	RAG, balanced quality
Q4_K_M	4.5	~4 GB	~8 GB	~40 GB	+0.10-0.15	40-60 tok/s	General chat, creative writing
Q4_K_S	4	~3.8 GB	~7.5 GB	~38 GB	+0.15-0.20	42-62 tok/s	Chat when VRAM is tight
Q3_K_M	3.5	~3.2 GB	~6 GB	~32 GB	+0.25-0.40	48-68 tok/s	Experimental, VRAM-starved
Q2_K	2.5	~2 GB	~4 GB	~20 GB	+0.50-1.00+	55-75 tok/s	Not recommended

Key observation: Can you run a 70B model in Q4_K_M on a 24 GB GPU like the RTX 4090? No — 40 GB exceeds 24 GB VRAM. You need either an Apple Silicon machine with 64+ GB unified memory, a multi-GPU setup, or the RTX 5090 with 32 GB VRAM. A 70B at Q4_K_M fits comfortably on a Mac Studio M4 Max with 128 GB unified memory, though throughput will be limited by memory bandwidth.

Per-Use-Case Recommendations: Coding vs RAG vs Chat

This is the section no other guide provides. Different tasks have different sensitivity to quantization error, and the right level depends on what you're actually doing with the model.

Coding and Structured Output: Use Q8_0

Code generation is the most precision-sensitive local LLM task. A single incorrect token — a misplaced bracket, a wrong variable name, an off-by-one in an index — makes the entire output useless. Quantization error at the 4-bit level occasionally causes exactly these kinds of subtle mistakes.

When building the Walmart conversational commerce chatbot at Firework, I learned that retrieval quality dominated answer quality at scale. But for code generation specifically, model precision matters more because there's no retrieval layer to compensate for token-level errors. If your VRAM can fit Q8_0, use it for coding. The 2× VRAM cost pays for itself in fewer broken outputs.

For local AI coding workflows, this means: run a 7B coding model at Q8_0 (~7 GB) rather than a 13B at Q4_K_M (~8 GB) when the task is pure code completion. But for broader coding assistance that includes reasoning and explanation, the 13B at Q4_K_M will be stronger overall.

RAG and Long-Context Tasks: Use Q5_K_S or Q5_K_M

Quantization error compounds over long context windows. At 2K tokens, the difference between Q4_K_M and Q8_0 is negligible. At 8K+ tokens, Q4_K_M's per-token error accumulates enough to subtly shift attention patterns and degrade retrieval-augmented outputs.

If you're building RAG pipelines where the model processes large retrieved chunks — say, 4-6 documents of 1,000 tokens each — Q5_K_S hits the sweet spot. It's roughly 25% smaller than Q8_0, fast enough for interactive use, and maintains quality across long contexts.

For vector database-backed retrieval systems, the retrieval quality matters more than generation precision. But Q5 gives you insurance against context-length degradation without the VRAM cost of Q8.

General Chat and Creative Writing: Use Q4_K_M

Conversational tasks are the most tolerant of quantization. Creative writing, brainstorming, summarization, and general Q&A all work excellently at Q4_K_M. The 0.10-0.15 perplexity delta is imperceptible in conversational use.

This is why Ollama defaults to Q4_K_M for most model pulls — it's the right default for the majority use case. If you're using Ollama for a local AI voice assistant or general-purpose chat, Q4_K_M is the correct choice.

Perplexity Benchmarks: Real Numbers From the llama.cpp Community

Perplexity measures how "surprised" a model is by a sequence of text. Lower perplexity means the model better predicts the next token, which correlates with output quality. Perplexity is measured on standardized datasets like Wikitext-2.

The canonical community benchmark source is llama.cpp Discussion #406, started by contributor Green-Sky and maintained by Georgi Gerganov's team (Gerganov is the creator of llama.cpp and GGML). Here are representative perplexity scores for a Llama 2 7B model on Wikitext-2:

• FP16 baseline: ~5.80
• Q8_0: ~5.82 (Δ +0.02)
• Q5_K_M: ~5.86 (Δ +0.06)
• Q5_K_S: ~5.87 (Δ +0.07)
• Q4_K_M: ~5.93 (Δ +0.13)
• Q4_K_S: ~5.96 (Δ +0.16)
• Q4_0 (naive): ~6.30 (Δ +0.50)
• Q3_K_M: ~6.10 (Δ +0.30)
• Q2_K: ~6.80 (Δ +1.00)

The jump from Q4_K_M to Q4_0 is dramatic — 0.13 vs 0.50 delta. This is the K-quant advantage. The mixed-precision strategy keeps quality remarkably close to FP16 while achieving nearly the same compression ratio as naive 4-bit.

One important caveat: quantization quality cliffs are model-family-specific. Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, a blanket Q4 recommendation is wrong — some model families (particularly those with fewer parameters but more aggressive training) degrade faster at Q4 than others. Always check perplexity numbers for your specific model.

Throughput Benchmarks: Tokens/sec on Consumer Hardware

Throughput on consumer hardware scales near-linearly with quantization level for memory-bound inference. Lower quantization means smaller model, which means better memory bandwidth utilization.

Here are representative token generation speeds across hardware tiers:

Hardware	Q4_K_M (7B)	Q8_0 (7B)	FP16 (7B)	Q4_K_M (13B)	Q8_0 (13B)
RTX 4090 (24 GB)	50-60 tok/s	25-30 tok/s	14-18 tok/s	30-38 tok/s	16-20 tok/s
RTX 4060 Ti (16 GB)	35-45 tok/s	18-22 tok/s	10-14 tok/s	22-28 tok/s	N/A (OOM)
Apple M3 Max (64 GB)	30-40 tok/s	18-24 tok/s	10-15 tok/s	20-28 tok/s	12-16 tok/s
Apple M3 Max (128 GB)	30-40 tok/s	18-24 tok/s	10-15 tok/s	20-28 tok/s	12-16 tok/s

Notice that more unified memory on Apple Silicon doesn't increase throughput — it only lets you load larger models. The M3 Max with 128 GB unified memory generates at the same speed as the 64 GB variant for models that fit in both. The bottleneck is memory bandwidth (400 GB/s on M3 Max), not capacity.

For the RTX 5090 vs RTX 4090, the 5090's 32 GB VRAM opens up the Q8_0 tier for 13B models that previously required offloading, and its higher memory bandwidth pushes Q4_K_M throughput above 70 tok/s for 7B models.

VRAM Requirements by Model Size and Quantization Level

The formula is straightforward: VRAM ≈ (parameters × bytes_per_weight) + context overhead. Context overhead varies by sequence length but typically adds 500 MB to 2 GB.

A 7B parameter model needs approximately:

• FP16: ~14 GB + context = ~15-16 GB total
• Q8_0: ~7 GB + context = ~8-9 GB total
• Q4_K_M: ~4 GB + context = ~5-6 GB total

For local LLM hardware requirements, this means:

• RTX 4060 Ti (16 GB): Fits 7B at Q8_0, 13B at Q4_K_M. Cannot run 13B at Q8_0.
• RTX 4090 (24 GB): Fits 13B at Q8_0, 30B at Q4_K_M. Cannot run 70B at any quantization.
• Apple M3 Max (64 GB): Fits 70B at Q4_K_M (barely), 30B at Q8_0.
• Apple M3 Max (128 GB): Fits 70B at Q8_0 comfortably.

Q4_K_M on an RTX 4090 is particularly sweet: you can run a 13B model with room for an 8K context window, getting strong general-purpose performance at 30-38 tok/s. This is why Q4_K_M is the default recommendation for the most popular local LLM hardware setup.

Quantization and Long Context: Why It Matters for RAG

Here's something most quantization guides skip: quantization error isn't static across context length. It compounds.

Each quantized weight introduces a small error. During attention computation, these errors propagate through the softmax and get amplified as the context window grows. At 2K tokens, the accumulated error is negligible. At 8K tokens, it starts to shift attention patterns. At 32K+ tokens, Q4_K_M can produce noticeably different outputs from FP16 on the same prompt.

This has practical implications for retrieval-augmented generation. If your RAG pipeline retrieves 5 documents of 1,000 tokens each and prepends them to a 500-token query, you're operating at 5,500 tokens of context. At this length, Q5_K_S provides a measurable quality improvement over Q4_K_M for retrieval-dependent answers.

When I built the RAG pipeline for Walmart's conversational commerce chatbot at Firework, retrieval quality dominated answer quality — not model precision. But that was with FP16 inference on Azure. For local AI RAG at quantized precision, the retrieval-generation quality balance shifts, and Q5 becomes the minimum viable quantization for production-grade results.

IQ-Quants: The 2026 Upgrade Path From K-Quants

The IQ-quant family — IQ4_XS, IQ3_M, IQ2_M — was stabilized in llama.cpp through 2024-2025 and represents the next evolution in GGUF quantization. IQ stands for "importance-matrix quantization" (also called imatrix calibration).

The key difference: K-quants assign precision based on layer type (attention gets more bits, FFN gets fewer). IQ-quants go further by using a calibration dataset to measure which individual weight groups matter most for output quality, then allocating bits accordingly.

The result is that IQ4_XS achieves quality comparable to Q4_K_M at a smaller file size, and IQ3_M matches Q4_K_S quality at 3-bit compression. For VRAM-constrained setups, this means you can fit a larger model in the same memory budget without the quality cliff that traditional Q3 quantization produces.

The trade-off: imatrix-calibrated quants require a calibration step during quantization (adding ~10 minutes to the process). Pre-quantized IQ models are increasingly available on Hugging Face, so most users won't need to do this themselves. If you see an IQ4_XS file alongside a Q4_K_M for the same model, the IQ variant is almost always the better choice.

How to Switch Quantization in Ollama

Ollama makes quantization selection easy once you know the syntax. Most models on the Ollama registry ship with multiple quantization tags.

To pull a specific quantization, append the tag:

• ollama pull llama3:8b — pulls the default, typically Q4_K_M
• ollama pull llama3:8b-q8_0 — pulls 8-bit quantization
• ollama pull llama3:8b-fp16 — pulls full FP16 precision
• ollama pull llama3:8b-q5_K_M — pulls 5-bit K-quant medium

To see available tags for any model, check the model page on ollama.com or run ollama show llama3 --modelfile to inspect what you currently have.

For custom GGUF files (from Hugging Face or self-quantized), create a Modelfile that points to the file:

FROM ./your-model.Q8_0.gguf

Then run ollama create mymodel -f Modelfile. This works with any GGUF quantization variant, including IQ-quants. For more on Ollama vs llama.cpp workflows, the key difference is that Ollama handles model management while llama.cpp gives you direct control over inference parameters.

Common Mistakes When Choosing a Quantization Level

Mistake 1: Treating all Q4 variants as equivalent. Q4_0, Q4_1, Q4_K_S, and Q4_K_M are dramatically different in quality. Q4_K_M is ~0.37 perplexity points better than Q4_0 on typical 7B models. Always prefer K-quant variants.

Mistake 2: Using Q8_0 when Q4_K_M would let you run a bigger model. If you have 16 GB VRAM, running a 13B model at Q4_K_M will almost always produce better outputs than a 7B at Q8_0. Model scale trumps quantization precision.

Mistake 3: Ignoring context length when choosing quantization. If you're using 2K context for chat, Q4_K_M is perfect. If you're stuffing 16K tokens into a RAG prompt, Q5_K_S or Q8_0 will produce noticeably better results.

Mistake 4: Assuming Apple Silicon VRAM works like GPU VRAM. Unified memory lets you load bigger models, but Apple Silicon memory bandwidth is 2-3× lower than discrete GPU memory bandwidth. A model that generates at 50 tok/s on an RTX 4090 might only hit 25 tok/s on an M3 Max at the same quantization.

Mistake 5: Not checking model-specific quantization behavior. As Maxime Labonne, ML engineer and LLM researcher, explains: Post-Training Quantization (PTQ) degrades differently across architectures. Some model families (like Mixtral MoE) are more robust to aggressive quantization than dense models. Check perplexity numbers for your specific model before committing to a quantization level.

Which Quantization Level Should You Pick?

Here's the decision framework:

Start with your VRAM budget. List the model sizes your hardware can fit at each quantization level using the table above.

Then match to your use case:

1. General chat, brainstorming, creative writing: Q4_K_M. This is the right default. Don't overthink it.
2. Code generation, structured output, function calling: Q8_0 if VRAM allows. The precision matters for token-exact tasks. If Q8_0 doesn't fit, use Q5_K_M.
3. RAG with long context (>4K tokens): Q5_K_S or Q5_K_M. The extra bits pay for themselves in context-dependent quality.
4. Fine-tuning base: FP16 or BF16. You need full precision for LoRA and QLoRA training.
5. VRAM-starved experimentation: IQ4_XS. Better quality than Q4_K_S at smaller size.

The cascade rule: Always prefer a larger model at lower quantization over a smaller model at higher quantization — unless your task is coding, where token precision matters more than general capability.

The quantization landscape will keep shifting. The RTX 50-series brings native FP4 hardware support. IQ-quants are making 3-bit practical where it was previously unusable. And Quantization-Aware Training (QAT) — where the model is trained to be robust to specific quantization levels — is increasingly common in new model releases.

But the fundamental triangle hasn't changed: quality, speed, VRAM — pick two. The numbers in this guide will shift as hardware and methods improve, but the decision framework won't. Know your VRAM budget, know your use case, and let the numbers — not vibes — drive your quantization choice.

---

Originally published on kunalganglani.com