Choosing Between Q4_K_M and Q4_K_S

#ai #llm #machinelearning #programming

Both files say Q4. Neither is four bits per weight, and the difference between them is not a quality dial — it is a short list of tensors that one variant stores at higher precision and the other does not.

Where 4.5 bits comes from

A Q4_K tensor is stored in superblocks of 256 weights, subdivided into eight blocks of 32. Each superblock carries one 16-bit scale, one 16-bit minimum, twelve bytes of six-bit per-block scales and minimums, and then the 4-bit quantized weights themselves. llama.cpp asserts the layout in ggml/src/ggml-common.h, and it adds up:

block_q4_K = 2 (d) + 2 (dmin) + 12 (scales) + 128 (256 weights x 4 bits)
           = 144 bytes per 256 weights
           = 4.50 bits per weight

block_q5_K = 2 + 2 + 12 + 128 + 32  = 176 bytes / 256 =  5.50 bpw
block_q6_K = 2 + 16 + 192           = 210 bytes / 256 =  6.5625 bpw
block_q8_0 = 2 + 32                 =  34 bytes /  32 =  8.50 bpw

So the floor for anything called Q4_K is 4.5 bits, not 4. That half-bit is the block metadata, and it is not optional — it is what lets each group of 32 weights have its own scale instead of sharing one across the whole tensor.

The M and the S are a tensor list

Neither variant stores every tensor at Q4_K. The rules live in llama.cpp’s llama_tensor_get_type in src/llama-quant.cpp, and reading them is the fastest way to understand what you are choosing between. Both start from Q4_K as the default type and then promote specific tensors:

Attention value projections (attn_v). Q4_K_M promotes them to Q6_K on layers selected by a helper called use_more_bits, which is true for the first eighth of layers, the last eighth, and every third layer in between. Q4_K_S promotes only the first four to Q5_K and leaves the rest at Q4_K.
Feed-forward down projections (ffn_down). Q4_K_M promotes the same use_more_bits selection to Q6_K. Q4_K_S promotes the first eighth of layers to Q5_K.
Fused QKV projections, where an architecture has them. Q4_K_M moves these to Q5_K; Q4_K_S does not touch them.
Architecture special cases. On a 70B model, whose eight query heads share one value head, any tensor that landed on Q3_K or Q4_K is bumped to Q5_K — the value tensor is eight times smaller than the query tensor there, so the accuracy is nearly free. On an eight-expert model, attn_v goes to Q8_0 outright.

That is the entire difference. M is not “a better quantizer”; it is the same quantizer spending its extra bits on the value and down-projection tensors, which is where llama.cpp’s authors concluded the error hurts most.

You do not have to take the source’s word for what a particular file contains, and on a downloaded file you should not, because the encoding in a GGUF filename is written by a person while the tensor table is written by the tool. Dumping the header lists every tensor with its actual type:

python gguf-py/gguf/scripts/gguf_dump.py Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  | grep -E "attn_v|ffn_down" | head -20

On a genuine Q4_K_M you will see Q6_K appearing on a scattered subset of the attn_v and ffn_down tensors and Q4_K on the rest, in exactly the first-eighth, last-eighth, every-third pattern that use_more_bits describes. On Q4_K_S you will see Q5_K on only the first few and nothing above it after that. If a file labelled Q4_K_M shows a uniform type across every tensor, it was produced with --pure or by a different tool, and it is not the thing its name claims to be.

The size gap, in bytes

llama.cpp’s quantize README publishes bits-per-weight for each type on Llama-3.1-8B: 4.67 bpw for Q4_K_S and 4.89 bpw for Q4_K_M. The difference is 0.22 bits per weight, and over roughly 8.03 billion parameters that is:

0.22 bits x 8.03e9 params / 8 bits per byte  =  ~221 MB

Which is what the published files show. In the Hugging Face repository bartowski/Meta-Llama-3.1-8B-Instruct-GGUF, checked on 2026-08-11, Q4_K_S is 4,692,673,952 bytes (4.37 GiB) and Q4_K_M is 4,920,739,232 bytes (4.58 GiB). The gap is 228,065,280 bytes — 218 MiB, or 4.9% larger — and the derivation above lands within three percent of it, which is the right kind of agreement to expect when the parameter count is rounded.

Both bits-per-weight figures and both file sizes are properties of one llama.cpp version quantizing one model. The tensor selection rules are edited from time to time, so re-derive from the current README rather than carrying these numbers forward.

Which one to take

Framed correctly the question is nearly always answered by memory, not by quality. Two hundred megabytes decides nothing on a machine with headroom, and decides everything on a machine without it. So:

If both fit with room to spare, take M. You are paying 4.9% of file size for higher precision on the tensors that were selected precisely because they carry the error. There is no argument for S here.
If only S fits, take S rather than dropping to Q3. The step from Q4_K_S to Q3_K_M is a much larger quality change than the step from M to S, because it moves the default type rather than a handful of promoted tensors.
If neither fits, the variant is not your problem. You need a smaller model or more context budget, and which quantization level suits the task is the question to answer first.

One practical note on speed: the two are close enough that the difference is not the reason to choose. M has more Q6_K tensors, which are slightly more work to dequantize, but it is also 4.9% more bytes to stream per token — and since decode is bandwidth-bound, that is the term that dominates.

What --pure would cost you

llama-quantize takes a --pure flag that disables the k-quant mixtures entirely and quantizes every tensor to the same type. It exists for experiments, and it is a good way to see what the mixtures buy: a pure Q4_K file is smaller than either S or M and is missing every promotion described above, including the ones the 70B and mixture-of-experts special cases add.

The general lesson is worth carrying to the rest of the encoding ladder. What distinguishes S, M and L within a level is never a different arithmetic — it is which tensors were judged too important to round hard. If you find yourself comparing two files whose names differ by one letter, the answer is in that list, and the list is in the source.

DEV Community