You download llama-3.1-8b-instruct-q4_k_m.gguf. It's 4.7 GB. The original BF16 weights were 16 GB. You run it, it works, and you move on.
But here is what actually happened to those weights.
The Core Trick: Linear Quantization
Quantization is just a reconstruction problem. You have a continuous value w (a weight in FP16). You want to store it as an integer q using fewer bits. You pick a scale s and a zero point z.
q = clip(round(w / s) + z, q_min, q_max)
At inference, you reconstruct:
ŵ = s * (q - z)
The error is bounded by s/2. Smaller scale = finer grid = less error. But the scale must be large enough to cover the largest weight in the group. One outlier forces a coarse grid for everyone else.
That is the entire game: how you group weights and choose scales.
The Outlier Problem
Imagine a row of 128 weights. 127 of them live in [-1, 1]. One sits at +12.
Naive per-row quantization: scale = 12 / 7 ≈ 1.71 (for INT4, range -8..7).
Now your 127 well-behaved weights can only take values {-1.71, 0, +1.71}. You have crushed the signal.
This is why per-tensor and per-row quantization fail. The solutions are all about isolating outliers.
Three Approaches to the Same Problem
1. GPTQ: Minimize Output Error, Not Weight Error
GPTQ (Frantar et al., 2022) does not ask "how close is ŵ to w?" It asks "how close is ŴX to WX on real activations?"
It quantizes weights one column at a time, using the layer's Hessian (second-order curvature) to measure how much each weight's error propagates to the output. After quantizing a column, it updates the remaining unquantized columns to absorb the introduced error.
Minimize ||WX - ŴX||² (not ||W - Ŵ||²)
The calibration dataset (usually ~128 samples from C4) provides the activations X. The Hessian approximation H ≈ X Xᵀ tells GPTQ which input directions matter.
Result: Better quality than round-to-nearest at the same bit width. But it is slow (hours for 175B), GPU-only, and needs the Marlin kernel for fast inference.
2. AWQ: Protect the Channels That Activations Actually Use
AWQ (Lin et al., 2023) observed: activations have outlier channels too. A tiny fraction of channels (often <1%) carry huge activation values. The corresponding weight columns get amplified in the matmul.
Y = WX
If channel i has activations around 100, a 0.1 weight error becomes 10 output error. Same error on a quiet channel (0.1 activation) → 0.01 output error.
AWQ's trick: rescale before quantizing.
W X = (W · diag(s)) · (diag(s)⁻¹ X)
Scale down the important weight columns by s, quantize them (now they fit in a tighter grid), then scale up the corresponding activations by 1/s at runtime. Mathematically identical, but the quantizer sees a tamer distribution.
Result: Better quality than GPTQ, fewer calibration samples (128-512), faster quantization. Pure INT4 weights — no mixed precision awkwardness.
3. GGUF / K-Quants: Two-Level Hierarchical Scaling for CPU
GGUF is a file format, not a quantization algorithm. The quantization inside is the K-quant family (Q2_K through Q6_K, plus I-quants).
The K-quant innovation: super-blocks.
Legacy (Q4_0): 32 weights → 1 FP16 scale (0.5 bits/weight overhead)
K-quant (Q4_K): 256 weights (super-block)
→ 1 FP16 super-scale
→ 8 sub-blocks of 32 weights
→ each gets a 6-bit sub-scale (quantized against super-scale)
→ each gets a 6-bit sub-min (for asymmetric quantization)
Overhead drops from 0.5 to ~0.4 bits/weight, and local fitting is better, not worse.
The _S / _M / _L suffix is a per-layer mixed-precision policy:
| Variant | Typical Policy (Q4_K_M) |
|---|---|
attn_q, attn_k
|
Q4_K |
attn_v, attn_output
|
Q6_K ← escalated |
ffn_gate, ffn_up
|
Q4_K |
ffn_down |
Q6_K ← escalated |
norms, embeddings
|
F16 |
Why those layers? attn_v and ffn_down sit at the residual stream — errors accumulate across depth. The output projection and embeddings are also sensitive. The policy is hand-tuned from perplexity experiments, not learned.
Result: Q4_K_M at 4.5 bits/weight beats legacy Q4_0 (also 4.5 bits/weight) by 1-3 perplexity points on WikiText. Runs on CPU/Metal via llama.cpp with hand-tuned SIMD kernels.
The I-Quant Frontier: Below 4 Bits
Below ~3 bits/weight, scalar quantization (round each weight independently) hits a wall. The error becomes uniform and irreducible.
I-quants (IQ1 through IQ4) replace evenly spaced integer levels with learned codebooks:
- Groups of 8 weights → index into a table of allowed sign/magnitude patterns
- An importance matrix (diagonal Hessian approximation from calibration) weights the rounding decisions
- Precision goes where the model is sensitive
IQ4_XS: 4.25 bits/weight, beats Q4_K_S (4.5 bits/weight)
IQ3_XXS: 3.06 bits/weight, coherent output where Q3_K_M degrades
IQ1_M: 1.75 bits/weight, barely usable but exists
The catch: i-quants require an importance matrix (.imatrix file from llama-imatrix). Without it, the quantizer refuses to run below ~3 bits.
Reading a GGUF Filename
Q4_K_M
│ │ │
│ │ └─ M = medium mix policy (S=small, L=large, XL=extra large)
│ └─── K = k-quant (two-level hierarchical scaling)
└───── 4 = ~4 bits per weight (nominal)
Effective bits/weight is always higher — scale overhead + promoted sensitive tensors.
| Format | Nominal | Effective | 7B Size | Quality vs FP16 |
|---|---|---|---|---|
| Q8_0 | 8 | 8.5 | ~8.1 GB | <0.1% (indistinguishable) |
| Q6_K | 6 | 6.6 | ~6.3 GB | ~0.1% (transparent) |
| Q5_K_M | 5 | 5.7 | ~5.4 GB | ~0.3% (excellent) |
| Q4_K_M | 4 | 4.8 | ~4.7 GB | ~0.8% (very good) |
| Q4_K_S | 4 | 4.5 | ~4.5 GB | ~1.2% (good) |
| Q3_K_M | 3 | 3.9 | ~3.8 GB | ~3-4% (noticeable) |
| IQ3_XXS | 3 | 3.1 | ~3.0 GB | ~6% (degraded but coherent) |
| Q2_K | 2 | 2.6 | ~2.5 GB | ~10%+ (clearly degraded) |
The knee is at 4 bits. Q4_K_M is the last "boring" quant — quality loss under 1%, 58% faster than Q8_0 on the same hardware (memory-bandwidth bound).
What Actually Happens at Inference
GGUF / llama.cpp (CPU / Metal / partial GPU)
1. mmap(model.gguf) → kernel maps tensor data into virtual address space
2. Parse header + metadata + tensor info → small in-RAM index
3. For each generation step:
a. Tokenize input via embedded BPE/SentencePiece
b. For each layer:
- SIMD dequantize Q4_K block of 256 weights into FP32 scratch
- Multiply with FP16/FP32 activation, accumulate in FP32
- Apply norm, residual, attention, FFN
c. Sample next token, append to context
4. KV cache stays in RAM in FP16 (or quantized via --kv-cache-dtype)
The dequantize-then-matmul path is intentional: on CPU, the bottleneck is weight memory bandwidth. Dequantizing into a register tile is fast; the matmul runs in FP32.
AWQ / GPTQ (GPU with Tensor Cores)
1. Load safetensors (quantized INT4 weights + FP16 scales)
2. For each layer:
a. Fused INT4 GEMM kernel (Marlin / exllama / TRT-LLM)
- Read packed INT4 weights + scales
- Compute in INT4 Tensor Cores (Hopper+) or dequantize to FP16 on older GPUs
- Apply scale per output tile
b. Activation remains FP16/BF16 (W4A16)
Marlin kernel makes GPTQ-INT4 712 tok/s on H200 (54% faster than FP16). AWQ with Marlin: 741 tok/s — fastest production format.
Practical Decision Guide
| Your Hardware | Your Task | Recommendation |
|---|---|---|
| Consumer GPU (8-16 GB) | Chat / general | Q4_K_M (GGUF via Ollama/LM Studio) |
| Consumer GPU | Coding / Math | Q5_K_M or Q6_K if VRAM allows |
| Apple Silicon (unified) | Any | Q4_K_M for 70B, Q8_0 for 34B |
| 24+ GB VRAM / A100 | Production serving | AWQ 4-bit (vLLM + Marlin) |
| CPU only / limited RAM | Must fit larger model | Q4_K_M → Q3_K_M → IQ3_XXS (last resort) |
| Archival / paranoia | Quality above all | Q8_0 or F16 |
Two rules that save you from guessing:
Fitting in VRAM > quant level. A Q4_K_M that fits in VRAM crushes a Q6_K that spills to system RAM.
Bigger model at lower quant > smaller model at higher quant. A 13B Q4_K_M almost always beats a 7B Q8_0.
What Developers Get Wrong
| Misconception | Reality |
|---|---|
| GGUF is a quantization method | GGUF is a container. The quantization is K-quants, I-quants, or legacy types inside the file. |
| Q4_K_M means every tensor is 4-bit | It means most tensors are Q4_K. Sensitive ones (attn_v, ffn_down, output, embeddings) are Q6_K or F16. |
| Lower bits = linearly worse quality | Quality is flat from 16→8→5→4 bits, then falls off a cliff below 4. The knee is sharp. |
| Quantized models can be fine-tuned | The round() function has zero gradient everywhere. Fine-tuning starts from BF16/FP16, then you re-quantize. |
| AWQ/GPTQ models run on llama.cpp | Different ecosystems. GGUF for CPU/Metal/partial GPU. AWQ/GPTQ safetensors for GPU-only (vLLM, TGI, TensorRT-LLM). |
Quantization is not "make weights smaller." It is "allocate your bit budget where the model is sensitive."
- GPTQ: uses Hessian to allocate precision where output error hurts most
- AWQ: uses activation statistics to protect high-traffic channels
- K-quants: use two-level hierarchical scales + hand-tuned per-layer policy
- I-quants: use codebooks + importance matrix to squeeze below 4 bits
The file name (Q4_K_M, awq, gptq) tells you which allocation strategy was used. The bit count tells you the budget. The combination tells you whether it will work for your use case.
Next time you see Q4_K_M, you will know: 256-weight super-blocks, 6-bit sub-scales, attn_v and ffn_down promoted to Q6_K, dequantized on-the-fly by AVX2/NEON/AMX SIMD into FP32 for the matmul. That is what is actually happening.
Top comments (2)
MLX's default quantizer is flatter than this, per-group affine, scale and bias per group, group_size 64 by default, no super-block. But it also ships mixed-precision recipes now (mixed_2_6, mixed_3_4, etc.) that explicitly mirror llama.cpp's Q4_K_M policy: v_proj, down_proj and lm_head get bumped to more bits, chosen by a layer-position heuristic (early layers, late layers, every third layer in the middle) instead of hand-tuned perplexity sweeps. Default mlx_lm.convert -q still applies one bit width everywhere though, you have to opt into a recipe. Fine-tuned and quantized a 27B Qwen model with mlx_lm recently and used the flat default, hadn't noticed the mixed recipes existed until just now. Curious whether that position-based heuristic tracks the same residual-stream intuition your K-quant policy is doing by hand, or if it's cruder.
Good piece. Actually, there is an alternative — not the only one, but for the sake of the argument: DFloat11. It keeps the full 16 bits, and instead of quantizing it just Huffman-compresses the weights. The model ends up about 30% smaller, with no loss at all. A bit slower, but on the other hand… maybe now the whole thing fits in VRAM, which by your own rule beats the quant level.
I know it's not quantization, but… same family, so to speak.