INT4 Accuracy Loss by Model Size, Derived

#ai #machinelearning #llm #programming

“Quantization hurts small models more” is repeated everywhere and sourced almost nowhere. There is a published table it comes from, the effect is real, and it is smaller and less monotone than the folklore version. Here is the table, the arithmetic on it, and the mechanism.

The published numbers, dated

The source is the GPTQ paper by Frantar, Ashkboos, Hoefler and Alistarh, published at ICLR 2023. It reports WikiText2 perplexity for the OPT family at FP16, and after round-to-nearest and GPTQ quantization at 4 and 3 bits. Quoted, not interpolated:

OPT      FP16     RTN 4-bit  GPTQ 4-bit  GPTQ 3-bit
125M     27.65    37.28      31.12       53.85
1.3B     14.63    48.17      15.47       20.97
2.7B     12.47    16.92      12.87       16.88
6.7B     10.86    12.10      11.39       14.86
13B      10.13    11.32      10.31       11.61
30B       9.56    10.98       9.63       10.27
66B       9.34   110         9.55        14.16
175B      8.34    10.54       8.37         8.68

Lower perplexity is better. The setting is the paper’s main one: uniform per-row asymmetric quantization on the min-max grid, without grouping. That last detail matters and is picked up below.

These figures are three years old at the time of writing and describe the OPT family, which has been superseded several times over. No equivalent table with the same methodological care exists for current model families, which is why this page uses these and says so rather than quoting a fresher number from a source that does not document its method. Read the paper for the full set, including the C4 results.

The deltas, computed

Subtracting the FP16 column from the GPTQ 4-bit column, and expressing each as a percentage of the baseline:

OPT      FP16     GPTQ-4    absolute Δ   relative Δ
125M     27.65    31.12     +3.47        +12.5%
1.3B     14.63    15.47     +0.84         +5.7%
2.7B     12.47    12.87     +0.40         +3.2%
6.7B     10.86    11.39     +0.53         +4.9%
13B      10.13    10.31     +0.18         +1.8%
30B       9.56     9.63     +0.07         +0.7%
66B       9.34     9.55     +0.21         +2.2%
175B      8.34     8.37     +0.03         +0.4%

The trend is downward and it is not monotone. 6.7B is worse than 2.7B; 66B is worse than 30B. The paper itself notes the exception — larger models are generally easier to quantize, with OPT-66B called out — and the round-to-nearest column shows why nobody should treat model size as the only variable: RTN at 66B produces a perplexity of 110, catastrophically worse than the 30B or 175B rows around it. That is a property of that checkpoint, not of its size.

The 3-bit column is the other half of the story and belongs to a different page: the 4-bit deltas above are between 0.03 and 3.47, while the 3-bit ones run from 0.34 at 175B to 26.2 at 125M. That bend is the argument in why 4-bit became the local-inference default.

Why the delta shrinks: the derivation

Three mechanisms contribute, and they are worth separating because only two of them are about the model getting better at absorbing error.

The denominator shrinks. Part of the effect is arithmetic rather than mechanism. Larger models have lower baseline perplexity, so the same absolute degradation is a larger relative one at the bottom of the table. Compare the absolute and relative columns above: the relative column falls less steeply than the absolute one, and it is the relative one that is closer to a statement about the model.
Parameter redundancy. A model with more parameters trained on a comparable amount of data represents each learned regularity across more weights. Quantization perturbs each weight by an independent, roughly zero-mean amount bounded by half a grid step. Perturbations spread across k redundant weights contributing to the same computation partially cancel — their sum grows as sqrt(k) while the signal they carry grows as k, so the signal-to-error ratio improves as sqrt(k). Redundancy rises with parameter count at fixed training data, which is the core of the argument and also its weakness: models trained far past compute-optimal on enormous token budgets have less spare capacity per parameter, and the argument predicts they should quantize worse.
Wider layers, better-conditioned reconstruction. GPTQ’s error compensation works by pushing each rounding error into the not-yet-quantized columns. A layer with more input channels has more columns to absorb each error, so the compensation has more room. This mechanism is specific to error-compensating methods, and it predicts — correctly, per the RTN column — that the size trend should be much weaker for round-to-nearest, which has no compensation at all.

What the table does not say

Four limits, each of which would make a naive reading of the numbers wrong in a different direction.

It is ungrouped. The main setting is one scale per output row. Essentially every 4-bit checkpoint distributed today uses group size 128, which is a much finer granularity and produces smaller deltas than the table shows. The numbers are a lower bound on modern 4-bit quality, not an estimate of it. See group size.
Perplexity is not your task. A 0.18 perplexity increase is a statement about next-token likelihood on WikiText2. It does not translate into a percentage on a coding benchmark, an exact match rate, or whether structured output still parses. Tasks with a sharp correctness boundary — valid JSON, a compiling function, an exact numeric answer — can degrade far more than a perplexity delta suggests, because they fail on a single wrong token.
It is one model family. Architecture matters independently of size: how pronounced a model’s outlier features are, whether it uses grouped-query attention, how wide its MLPs are relative to its hidden size. OPT-66B’s RTN result is a standing warning about generalising from one checkpoint.
Mixture-of-experts models are not covered. Total and active parameter counts diverge, so “model size” is two numbers, and the redundancy argument above applies to neither cleanly. Nothing in this table speaks to them.

Getting a number for your own model

Because no published table will answer the question for your model on your task, the useful output of this page is a method rather than a figure.

Build an evaluation set of a few hundred real requests from your own traffic or a realistic proxy, with the outputs you would accept. Perplexity is the wrong instrument here; you want your own correctness criterion.
Score the FP16 model on it. This is the baseline and it will not be 100%; the question is the gap, not the absolute.
Score the quantized model on the identical set with identical sampling parameters and a fixed seed. A difference produced by temperature is not a difference produced by quantization.
Look at the disagreements individually rather than at the aggregate. Quantization damage is usually concentrated — a format that fails slightly on everything and one that fails badly on a specific input class can produce the same average, and only one of them is acceptable.
Repeat for one bit width above and below. The absolute numbers are not comparable to anyone else’s; the ordering between your own candidates is what you needed.

A common outcome of that exercise is that the quantized model is fine for most of the traffic and wrong on an identifiable slice — long documents, one language, one output format. That is a routing result rather than a quantization result: keep the cheap local path for the bulk and send the identified slice to a larger hosted model. Multigrid is one API and one key across both, which is what makes the split a rule you can change rather than two integrations you have to maintain.