Genrius AI

Posted on Oct 24

How We Shrank an AI Model by 85% and Kept 90% of Its Smarts

#ai #machinelearning #opensource #llm

Qwen3-4B ships as an 8.05 GB file of 16-bit weights. In this post we walk through how a 2-bit quantizer gets the same model down to 1.24 GB (85% smaller) without retraining, and how to check whether the result is any good. Our worked example is Pingala-8, an open-source format we built on llama.cpp. Most of the ideas are not ours: the rotation and rounding come from QuIP and QuIP#, the trellis from QTIP and Ungerboeck's modem codes, and ikawrakow's IQ2_KT is its closest relative. We will also show which piece mattered less than we expected.

Why 2 bits is hard

At 2 bits per weight, each weight can take one of 4 values. Every weight is rounded to its nearest allowed value, and the difference is error. Two things make that error hurt.

It adds up. Each output of a layer is a sum of thousands of weight-times-input products. The rounding errors in those products do not cancel on their own, and they travel through dozens of layers.

Outliers. Weights are stored in small groups, each with a scale so its 4 levels cover the group's range. Take the group [3.0, 0.2, -0.1, 0.1]. To reach 3.0, the levels must be spread wide, say {-3, -1, 1, 3}. Now 0.2, -0.1 and 0.1 all land on -1 or +1, an error of about 1 each, larger than the weights themselves.

A good 2-bit quantizer must remove outliers, make errors cancel, and get more precision per bit. Those are the three tricks.

Trick 1: rotate the weights first

If R is an orthogonal matrix (R^T R = I), then W x = (W R^T)(R x). We can store the rotated weights W R^T, apply R on the input side, and the layer computes exactly the same result. A rotation keeps lengths but changes coordinates, so it can spread one large value across many coordinates.

The usual choice is a Hadamard matrix with random signs, because it is cheap to apply. The 4x4 version, scaled by 1/2:

      [ 1  1  1  1 ]
1/2 * [ 1 -1  1 -1 ]
      [ 1  1 -1 -1 ]
      [ 1 -1 -1  1 ]

Apply it to [3.0, 0.2, -0.1, 0.1] and you get [1.6, 1.3, 1.6, 1.5]. The sum of squares is 9.06 both times, but the largest value is now close to the others instead of 15 to 30 times bigger, and four evenly spaced levels fit it well. Real layers rotate many coordinates at once, and the rotated weights look roughly like a bell curve without extreme outliers.

Trick 2: round with error feedback

Plain rounding treats each weight on its own. The layer cares about the output, the sum of w_i * x_i.

Toy example: three weights [0.4, 0.4, 0.4] multiply inputs that are almost always equal, so the layer effectively uses their sum, 1.2. Rounding each to the nearest integer gives [0, 0, 0], off by 1.2. With feedback, the first 0.4 rounds to 0 and its error (0.4) is carried to the next weight, which becomes 0.8 and rounds to 1; that error (-0.2) is carried to the third, which becomes 0.2 and rounds to 0. The result [0, 1, 0] sums to 1, off by only 0.2.

LDLQ, from the QuIP paper, is the principled version. It quantizes the columns of a weight matrix one at a time and pushes each column's error onto the columns not yet quantized. How much goes where comes from H, the average of x x^T over inputs collected by running sample text through the model. If the inputs were uncorrelated, H would be diagonal and LDLQ would reduce to plain rounding.

Trick 3: trellis coding

IQ2_XXS, llama.cpp's smallest common 2-bit format, uses a codebook: each group of weights becomes the nearest entry of a fixed table, one group at a time. Trellis-coded quantization, used by QTIP and IQ2_KT, adds memory instead.

The encoder carries a small state from one group to the next. The state decides which subset (a "coset") of values the next group may use. The bits written for that group pick a value inside the subset and also set the next state. The decoder tracks the state for free, so the format can use twice as many values as the bits alone could address. The price: today's choice decides tomorrow's menu, so you cannot round each group on its own.

A toy with 1 bit per weight shows both sides. Four levels are split into two cosets: state 0 may use {-1.5, 0.5} and state 1 may use {-0.5, 1.5}. Bit 0 picks the lower value, bit 1 the higher, and the bit you wrote becomes the next state. Start in state 0 and encode [-0.4, 0.1, 1.5]:

Step	Target	Greedy: state, pick	Best path: state, pick
1	-0.4	0, 0.5	0, -1.5
2	0.1	1, -0.5	0, 0.5
3	1.5	0, 0.5	1, 1.5
Squared error		2.17	1.37

Greedy rounding takes the closest value at step 1 and is stuck with the wrong menus at steps 2 and 3. The best path accepts a worse first step to get the right menus later. Plain 1-bit rounding to {-1, 1}, with the same 3 bits, scores 1.42: greedy trellis rounding loses to it, the searched path beats it. On real data the advantage is an average over long sequences, not a guarantee at every step.

The search is the Viterbi algorithm: dynamic programming that keeps, for every state, the cheapest path ending there. Its cost grows linearly with length, and it runs once, offline, when the file is made.

Pingala-8 uses Ungerboeck's 8-state set-partitioned trellis over symbols of 4 weights, at about 2 bits per weight. Decoding is cheap (read bits, update state, look up values), and a fused CUDA kernel does it inside the matrix-vector product. Expensive search once, cheap decode on every token: that asymmetry is why trellis coding suits inference.

How we measured fairly

Quantization results are easy to flatter by accident. Our rules, which work for any comparison:

Equal file size, not equal "bits per weight". Formats count overhead differently (scales, embeddings, layers kept at higher precision). File bytes decide what fits in GPU memory, so we compared against files of the same size within a few megabytes: llama.cpp's IQ2_XXS and ik_llama.cpp's IQ2_KT.
Same harness for every file. WikiText-2 perplexity at context 4096 with the same tool and text windows. Five zero-shot quizzes (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, MMLU) through one harness. Speed on one RTX 5070 Ti Laptop GPU, in one session.
Two kinds of quality. Perplexity measures how well the model predicts real text (lower is better). Quizzes check whether answers survive.
Paired comparisons. Every file scores the same windows and questions, so we compare item by item; shared difficulty cancels and error bars tighten.
Say what was tuned. The Qwen3-4B file is the last of many builds we checked on WikiText-2, so treat it as tuned. For Llama-3.1-8B and Qwen3-14B we wrote predictions down before running.

Results

Perplexity against file size. B4 is the released Pingala-8 file. Left: Qwen3-4B, where "post hoc" marks the tuned result. Right: Llama-3.1-8B scored without a start token in each window; the figures below add one.

Qwen3-4B, WikiText-2 perplexity at context 4096 (lower is better):

File	Size	Perplexity
Original, fp16	8.05 GB	12.46
llama.cpp IQ2_XXS	same as ours	28.61
ik_llama.cpp IQ2_KT	same as ours	18.15
Pingala-8	1.24 GB	16.64

On the five quizzes, Pingala-8 averages 50.8 against 41.9 for IQ2_XXS (+21%) and keeps 90% of the 8-bit Q8_0 average (56.3).

Speed for one user (batch 1), Qwen3-4B on the RTX 5070 Ti Laptop:

Measure	vs IQ2_XXS (same size)	vs Q4_K_M (twice the size)
Generating text	1.10x	1.39x
Reading the prompt	0.74x	-

Generating one token at a time mostly means reading every weight from memory, so fewer bytes give more tokens per second while decoding stays cheap. Reading a prompt processes many tokens at once and is limited by compute, which is where the extra decode work shows.

On Llama-3.1-8B (a 2.39 GB file), Pingala-8 scores 8.44 against 9.36 for IQ2_KT at the same size.

Where it loses

Qwen3-14B. IQ2_KT wins, 9.40 against our 9.57. We had predicted a win in writing before the run.
Prompt speed. 0.74x IQ2_XXS, so long prompts take longer before the first word. Serving several chats at once is also much slower; this is a one-user, one-GPU format.
CUDA only. It needs our patched llama.cpp and an NVIDIA GPU. Stock llama.cpp, Ollama and LM Studio cannot load the files.
Retrained models. Ternary Bonsai 4B scores 11.59 at 1.07 GB, smaller and better. It is a retrained model, not a compression of the original Qwen3-4B, so it took a training run. Post-training quantization is for models nobody has retrained for you.

What the ablation says

An ablation removes one part and measures the damage. We removed the trellis memory at equal bytes. The cost was 0.076 nats per token of extra loss, about 8% higher perplexity (e^0.076 is about 1.08).

Set it against our two gaps:

Against the tuned IQ2_KT file, itself a trellis format, the trellis memory accounts for most of our margin.
Against IQ2_XXS, it accounts for only 14% of our gain. The other 86% comes from the rotation, the LDLQ rounding and the rest of the pipeline.

The lesson for anyone building a 2-bit quantizer: rotation and error-feedback rounding carry most of the quality. Trellis memory explains most of the small margin between formats that are already close, and little of the large gap to a plain codebook format. We expected the trellis to be the main story; the ablation says otherwise, which is why it is worth running.

How to reproduce

Try it: download the one-click build (links below), unzip, and run run_qwen3_4b.bat on Windows or ./run.sh qwen on Linux. It downloads the model once and opens a local chat page. You need an NVIDIA RTX 30, 40 or 50 series GPU.
Rerun our numbers: the repository has the patch, the commands and the logs behind every number here.
Measure your own quant the same way with stock llama.cpp tools, on one machine, for files of the same size:

# perplexity: same text file and context give the same windows
./llama-perplexity -m model.gguf -f wiki.test.raw -c 4096 -ngl 99

# batch-1 speed: prompt processing (pp) and text generation (tg)
./llama-bench -m model.gguf -p 512 -n 128

Why the name

Pingala's Chandahsutra, an ancient Sanskrit work on poetic metre, lists the 8 patterns of three long or short syllables, the ganas. Our trellis has 8 states, so we named the format after him. The name is a credit; the trellis labels are Ungerboeck's.

DEV Community