Genrius AI

Posted on Oct 25

How to Compare Quantized LLMs Fairly in llama.cpp: Same Bytes, Same Windows, Same GPU

#machinelearning #llm #ai #tutorial

Here are two GGUF files of Qwen3-4B, both sold as "2-bit". One scores a WikiText-2 perplexity of 28.61, the other 16.64. Is the second one better? Maybe. Before you believe a gap like that, you need three facts: how many bytes each file really takes, which text windows were scored and how, and whether the speed numbers came from one GPU in one session.

This is a how-to for comparing quantized models in llama.cpp so that the numbers mean something. Every command below is stock llama.cpp and works on any GGUF file. Our worked example is Pingala-8, a 2-bit format we built (the two numbers above are llama.cpp's IQ2_XXS and ours). Its files need our patched llama.cpp build: stock llama.cpp, Ollama and LM Studio cannot load them. How the format works is in our first article; this one is about measuring.

Step 1: Match file sizes in bytes, not labels

A quant name describes a block format, not a file. llama-quantize lists IQ2_XXS as "2.06 bpw", but an IQ2_XXS file stores the output layer as Q5_K, the token embeddings as Q2_K and, on models such as Qwen3 and Llama 3, the attention value tensors as Q4_K. The label tells you little about the file.

The arithmetic for Qwen3-4B: the 16-bit file is 8.05 GB at 2 bytes per weight, so it holds about 4.0 billion weights. Our IQ2_XXS file is 1,246,621,312 bytes, about 2.5 bits per weight over the whole model, not 2.06. Pingala-8 is 1,243,153,792 bytes, also about 2.5.

Why bytes? Bytes are what you download and what has to fit in VRAM, and they are the quality budget: a file 10% bigger can win simply by being bigger. GSQ's Q2_K of Qwen3-4B has a better perplexity than ours (14.37 against 16.64), but at 1.67 GB it is 34% larger, so that pair tells you little about either method.

Measure on disk, and add up every part of a split file:

stat -c '%s %n' model.gguf                  # one file, exact bytes
du -cb qwen3-4b-pingala8/*.gguf | tail -1   # split GGUF: total of all parts

Two traps. Run any llama.cpp tool with -lv 4 (recent builds hide loader messages by default) and the loader prints file size = ... GiB (... BPW). That line counts tensor bytes only (no metadata) and uses GiB: 1,243,153,792 bytes is 1.24 GB on Hugging Face but 1.16 GiB in the log. Use stat for matching sizes, and the loader's BPW, which includes the embeddings and output tensors, for bits per weight.

Our rule: the candidate may not be larger than the file it is compared with. Pingala-8 is 3,467,520 bytes under IQ2_XXS and 2,760,960 bytes under the IQ2_KT file below.

Step 2: Same text, same context, same windows

Perplexity depends on the text, the tokenizer, the window length and which tokens in each window are scored. Fix all four.

Same text, byte for byte. Check the sha256 of the file you pass with -f.
Same base model. Windows are counted in tokens, so the same text gives 73 windows of 4,096 tokens on Qwen3-4B and 70 on Llama-3.1-8B. Perplexity is comparable only between files of one model.
Same mode. llama-perplexity has two, and they give different numbers:

# default: every window starts with a BOS token; the second half of each window is scored
./llama-perplexity -m model.gguf -f wiki.test.raw -c 4096 -ngl 99

# strided: every token is scored; windows after the first start mid-text
./llama-perplexity -m model.gguf -f wiki.test.raw -c 2049 --ppl-stride 4095 -ngl 99

The -c 2049 is not a typo. In strided mode the tool adds half the stride to -c, and the log says so: "adjusting context size from 2049 to 4096".

One pair of Llama-3.1-8B files under both modes (the IQ2_KT file was quantized from Q8_0):

Llama-3.1-8B, WikiText-2	Pingala-8	IQ2_KT (ik_llama.cpp)
Default, BOS at every window	8.44	9.36
Strided	9.14	10.04

Same ranking, different numbers, because two things change at once. The default mode scores only the second half of each window, where every token has at least 2,048 tokens of context; strided mode also scores the poorly predicted early tokens. And in strided mode, windows after the first begin with an ordinary token: llama.cpp writes the BOS into the token list only after the window has been decoded. Models that use the BOS as an attention sink behave differently without it.

That detail cost us a build. An earlier Llama-3.1-8B file of ours did well with a BOS and fell apart in strided mode, because every calibration sequence it had seen started with a BOS. The fix was to start every second calibration sequence mid-text. So compare with someone else's number only if the mode matches, and if your method calibrates on data, test both modes.

One more check when formats need different runtimes. IQ2_KT lives in the ik_llama.cpp fork, so we ran a shared file through both binaries: our IQ2_XXS file scores 28.6083 in llama.cpp and 28.6037 in ik_llama.cpp. The runtimes are not the difference.

Compare per window, not two averages

Both files score the same windows, so subtract window by window: shared difficulty cancels and the error bar shrinks. llama-perplexity prints a running value after each window ([1]...,[2]...), which is enough to recover each window's loss:

# paired.py A.log B.log: per-window NLL differences between two llama-perplexity runs
import re, math, statistics as st, sys

def windows(path):
    ppl = [float(v) for v in re.findall(r'\[\d+\]([0-9.]+)', open(path, errors='ignore').read())]
    ln = [math.log(p) for p in ppl]
    # running perplexity after k windows -> NLL of window k (every window has the same token count)
    return [ln[0]] + [(k + 1) * ln[k] - k * ln[k - 1] for k in range(1, len(ln))]

a, b = windows(sys.argv[1]), windows(sys.argv[2])
d = [x - y for x, y in zip(a, b)]
print(f"A - B: {st.mean(d):+.4f} +/- {1.96 * st.stdev(d) / math.sqrt(len(d)):.4f} nats/token, "
      f"A better on {sum(x < 0 for x in d)} of {len(d)} windows")

On our Qwen3-4B logs it gives -0.542 +/- 0.020 nats per token against IQ2_XXS, better on 73 of 73 windows. Adjacent windows are correlated, so treat the interval as optimistic when a margin is small.

Step 3: Use more than one quality metric

Perplexity says how well a file predicts real text. It is cheap and sensitive, but it is an average over tokens.

KL divergence says how far the quantized model's next-token distribution is from the original's, token by token (0 means identical). It needs a reference run that saves logits:

# 1. reference: save the logits of the fp16 (or Q8_0) model
./llama-perplexity -m qwen3-4b-q8_0.gguf -f wiki.test.raw -c 512 --chunks 72 -ngl 99 --kl-divergence-base q8.kld
# 2. each quant: the tokens and context are read back from q8.kld
./llama-perplexity -m quant.gguf -c 512 -ngl 99 --kl-divergence-base q8.kld --kl-divergence

The logits file is huge (the llama.cpp docs quote 37 GiB for Llama 3 on the full WikiText-2 test set), so cap --chunks. If fp16 does not fit on your GPU, Q8_0 is a fair reference: on Qwen3-4B it scores 12.50 against 12.46 for fp16.

The two can disagree. Of the IQ2_KT files of this size we built for Qwen3-4B, the one with the lowest perplexity (18.15) had the highest KL divergence: right on average, yet furthest from the original model. Report both.

Task scores check whether answers survive. llama-perplexity has zero-shot modes built in:

./llama-perplexity -m model.gguf -c 2048 -ngl 99 --hellaswag --hellaswag-tasks 1000 -f hellaswag_val_full.txt
./llama-perplexity -m model.gguf -c 2048 -ngl 99 --winogrande -f winogrande-debiased-eval.csv
./llama-perplexity -m model.gguf -c 2048 -ngl 99 -np 5 --multiple-choice -f arc-easy-validation.bin

ARC questions can have five options, and our first ARC run aborted partway without -np 5. Our five-task average on Qwen3-4B (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, MMLU) is 50.8 for Pingala-8 against 41.9 for IQ2_XXS: 90% of the 8-bit Q8_0's 56.3, not 100%. A smaller file gets more answers wrong.

Step 4: Measure speed at batch 1, prompt and generation apart

llama-bench measures two different jobs. pp (prompt processing, -p 512 -n 0) reads many tokens at once and is limited by compute. tg (token generation, -p 0 -n 128) makes one token at a time and is limited by how fast the weights stream from memory. A smaller file usually helps tg; extra decode work in a complex format shows up in pp.

Change nothing but the file: one GPU, one session, flash attention and KV cache type set explicitly, and a rotated order so that warm-up and clock drift do not always favour one file:

models=(iq2_xxs.gguf candidate.gguf q4_k_m.gguf)
for r in 0 1 2; do
  for k in 0 1 2; do
    m=${models[$(( (k + r) % 3 ))]}
    for t in "pp:-p 512 -n 0" "tg:-p 0 -n 128"; do
      ./llama-bench -m "$m" -ngl 99 -fa 1 -ctk f16 -ctv f16 ${t#*:} -r 5 -o json > "${t%%:*}_r${r}_${m%.gguf}.json"
    done
  done
done

Each JSON file holds avg_ts, tokens per second. Divide it by the baseline's from the same round, then average the ratios over rounds.

Our Qwen3-4B result on one RTX 5070 Ti Laptop GPU: about 200 tokens per second in generation, 39% faster than Q4_K_M (a file about twice the size), but prompt processing at 0.74x IQ2_XXS. Both belong in the same sentence. Batch size matters too: with 2 to 8 parallel sequences our file is much slower than IQ2_XXS, so benchmark the batch you will actually serve.

Step 5: Write the prediction down before the run

If you tune on the test text, the final number is optimistic. Our Qwen3-4B file is the last of many builds we checked on WikiText-2, so we label it tuned and treat C4 as the only held-out text. For Llama-3.1-8B and Qwen3-14B we wrote a prediction and a decision rule into a timestamped file before the run. A template:

prereg.txt       written before the run, never edited after
compare:    candidate vs IQ2_KT, no more bytes than IQ2_KT
metric:     WikiText-2, llama-perplexity strided, all windows
rule:       win only if the paired dNLL is below 0 and its 95% interval excludes 0
prediction: PPL <value> [<low>, <high>]

Hash it (sha256sum prereg.txt) and publish the hash with the result. Both of our predictions missed, in opposite directions: on Llama-3.1-8B the result beat our interval, and on Qwen3-14B we predicted a win and got a loss.

Step 6: Put the losses in the same table

Qwen3-4B at equal size (WikiText-2, strided, 73 windows):

File	Bytes	Perplexity	5-task quiz
fp16	8.05 GB	12.46	-
Q8_0	-	12.50	56.3
IQ2_XXS (llama.cpp)	1,246,621,312	28.61	41.9
IQ2_KT (ik_llama.cpp)	1,245,914,752	18.15	46.5
Pingala-8	1,243,153,792	16.64	50.8

And next to it, the losses:

Qwen3-14B: IQ2_KT wins, 9.40 against our 9.57, paired +0.018 +/- 0.008 nats per token, with our file better on only 12 of 73 windows. On held-out C4 the two tie (15.25 against 15.23).
Speed: prompts at 0.74x and batched decoding at 0.18x to 0.34x IQ2_XXS.
Runtime: a patched llama.cpp, CUDA only, measured on one GPU.
Retrained models: Ternary Bonsai 4B scores 11.59 at 1.07 GB. It took a training run, so it belongs in a different comparison than post-training quantization.

The checklist

Sizes from stat, in bytes, split parts added; the candidate no larger than the baseline.
Bits per weight over the whole file, embeddings and output tensors included.
Same text (sha256), same -c, same perplexity mode, same window count; the mode stated next to every number.
Perplexity compared only within one base model.
A shared file through every runtime before comparing formats across runtimes.
Paired per-window differences with an interval, not two averages.
Perplexity, KL divergence against fp16 or Q8_0, and at least one task score.
pp and tg reported apart, at the batch size you will use, on one GPU in one session, in rotated order, with -fa and the KV cache type fixed.
Prediction and decision rule written and hashed before the run; anything tuned on the test text labelled as tuned, with a held-out text.
Losses in the same table as wins.

DEV Community