DEV Community

Private Local AI
Private Local AI

Posted on Originally published at blog.localai.hex7.com

Model quantization, read carefully: what the bit-widths really cost you

#ai #performance #storage #quantization

Draft for blog.localai.hex7.com and Ghost · Private Local AI · 2026-11-10

Every guide to running a model locally arrives at the same table: FP32 takes this much memory, FP16 half as
much, 4-bit a quarter, and then everyone moves on to which file to download. The table is not wrong, but it
is a summary of a process that has several decisions inside it, and each of those decisions costs you
something different — memory, accuracy, or time. Worth reading properly, because the choice you make changes
depending on whether your constraint is a card with 6 GB or a workstation you are not allowed to open.

What quantization does, precisely

A model's weights are numbers. Quantization replaces those numbers with lower-precision representations:
fewer bits per weight, and correspondingly less memory and less memory traffic, which is what actually limits
token generation on consumer hardware. A neural network in FP32 uses 32 bits per parameter; an INT8 version
uses a byte; a 4-bit version uses under half of that. The model does not change architecture — only the
storage format of its weights and, in some schemes, the numbers needed to decode them.

Two caveats the summary tables skip:

  • Not everything gets quantized. Layer norms, embeddings, rotary tables and small 1-D tensors stay at fp32 or fp16. That is why a "4-bit" file does not weigh four bits per parameter times N — it weighs about 4.5–4.85.
  • The quoted sizes are weight sizes. KV cache, activations and context buffers are not in them. At long context, the KV cache can be the difference between fitting and not fitting.

The ladder, with corrected numbers

A table circulating around recently gives this shape for a 7B model: FP32 ~28 GB, FP16/BF16 ~14 GB, INT8
~7 GB, Q4_0 ~3.8 GB. It is right about the orders of magnitude and slightly wrong in ways that matter when
you are choosing a file.

format bits stored per weight effective bits/weight 7B file, weights only what it is for
FP32 32 32.0 ~28 GB training state; nobody runs this to generate text
BF16 / FP16 16 16.0 ~14 GB the honest baseline for local inference
INT8 (per-tensor) 8 ~8.5 ~7.2 GB rarely used directly; llama.cpp's q8_0 is 8.5 bpw
Q4_0 4 4.5 ~4.3 GB the oldest 4-bit ggml format, block-32 with one fp16 scale per block
Q4_K_M 4 ~4.85 ~4.1–4.5 GB default "good enough" choice for most cards
Q5_K_M 5 ~5.66 ~4.8 GB noticeably better than Q4 at modest extra cost
Q6_K 6 ~6.56 ~5.5 GB near-BF16 quality, large file
Q8_0 8 8.5 ~7.2 GB quality ceiling of the ordinary formats

The correction that matters is effective bits per weight: side information — scales, and in K-quants extra
quantized min/max levels — is part of the format, so a "4-bit" quant needs 4.5 to 4.85 bits per weight of
storage. If you budget memory from the name rather than from the size, you will be wrong by about ten
percent, and ten percent is how a model fails to fit on a card with no headroom.

KV cache arithmetic is where people get surprised twice. A rough estimate: two bytes per token per layer per
head times the number of layers and heads, doubled because there are keys and values. On a 32-layer, 32-head,
128-dimension model that is 0.5 MB per token in BF16 — meaning a 32k-token context costs about 16 GB before
you notice it. This is why quantizing the cache (--cache-type-k/--cache-type-v, q8_0 or q4_0) buys back
a lot on long-context runs, and why "the file fits" and "the run fits" are separate questions.

Anatomy of a block: why group size costs memory

A ggml quant does not quantize the whole tensor at once; it chops it into blocks — 32 weights for Q4_0,
larger super-blocks for K-quants — and stores, for each block, a scale (and in richer formats a second-level
scale plus quantized minimum/offset values). Decoding multiplies the stored integer by the scale.

Three consequences worth internalising:

  1. Small groups cost memory. A 16-bit scale per 32 weights is 0.5 bits per weight of overhead — half a whole bit of payload. Wider blocks amortise better, which is part of why K-quants can beat Q4_0 at similar size: they spend their bits differently rather than having more of them.
  2. Two-level scales help quality. Q4_K and friends add sub-block scales; Q4_1-style formats add a stored offset per block. Same data, more side information, better reconstruction of outliers — the reason the "same 4 bits" variants have different sizes.
  3. Some formats are not fixed-width at all. The IQ types store their values entropy-coded (arithmetic coding) with an importance matrix (--imatrix, from a calibration corpus), so their bit-width depends on the content, and they need lookup-table decoders. They can hold more accuracy per bit; they cost time to run.

That third point is not theoretical. The model card behind the recent GSQ-RCO builds measured it directly:
two builds of nearly identical size, one using ordinary Q2_0 blocks and one using IQ2_XS, at 367.49 vs
108.19 prompt tokens/s and 93.79 vs 70.30 decode tokens/s on the same prompts — with the lookup-table
build varying by 45 tokens/s across task categories versus 2 for the ordinary one. The smaller file was the
fast one. If your interest is speed rather than benchmark scores, format choice matters more than bit width.

What a filename does and does not tell you

The name of a GGUF file states the main type — Q4_K_M, IQ3_XS — not the composition of the file. Real
files hold many tensors at higher types and some small tensors at fp32; and labels like UD, _XL or
GSQ-RCO are recipes, produced by someone's process, not formats with a fixed bit-width. Two files with
the same type label can be different sizes with different quality.

Which is exactly why the newest approach is to stop naming formats at all. In the GSQ-RCO builds, one
quantization type is chosen per tensor by a constrained optimiser under an explicit size budget, and the
resulting file carries an allocation record (rco-allocation.txt) listing which type every tensor got. The
name in the filename becomes the target average bit-width — 2.40, 3.00, 3.50 — because that is what was
actually controlled.

That distinction is the practical difference between picking a quant today and in a year: you stop guessing
which hand-tuned recipe is best for your card and start telling something your memory budget.

Choosing on real hardware

The honest answer depends on which part of the trade you are paying for.

  • You have 6–8 GB. 4-bit K-quants (Q4_K_M) are the starting point; expect a 7–8B model to fit with a short context. Use IQ types only if you have the patience for decode cost and want the extra accuracy.
  • You have 12 GB. Q5_K_M or a Q4 variant with more headroom for context. More context than precision is usually the better trade at this size.
  • You have 16–24 GB. Q6_K, or a quantized large model instead — often the big-model, lower-bit choice beats a small high-quality model on reasoning tasks. If you are near 24 GB and want the large models, quantizing KV cache is how you keep enough context.
  • You are deciding between "quantized" and "original." Original weights are not automatically better; careful low-bit quantization often matches it on standard benchmarks. The recent GSQ-RCO cards report a 3.00 bpw build at 92.57 vs the BF16 base's 93.12 on the task average, at 4.7× smaller — and are candid that recoverings above 100% on zero-shot averages are noise, not improvement.

Then measure on your own workload rather than trusting anybody's table: prompt length, batch size and which
layers a request touches change throughput more than the bit-width does. The two formats at the same size in
the example above differed by more than 3× in prompt speed. That is the useful number to know about
quantization: the parts you can see in the benchmark table are the least interesting part of what you will
actually run.

Read also

  • GSQ and RCO: the non-uniform quantization now sitting in GGUF files — posts/gsq-rco-blog.md, published at https://blog.localai.hex7.com/
  • Llama.cpp - What is it? (the engine that runs these formats) — same blog.

Sources

Top comments (0)