Integer quantisation replaces a real number x with a small integer q using q = round(x/s) + z, where the step size s and the zero-point z are computed from the tensor’s own minimum and maximum. Everything else follows from those two constants by arithmetic: the error is bounded by s/2, each extra bit buys about 6.02 dB of signal-to-noise, and “4-bit” weighs about 0.578 bytes per weight rather than 0.5. This page does all of that arithmetic in front of you.
The affine map, and where s and z come from
A float carries its own exponent, so it can represent both 1e-8 and 1e8. An integer cannot. Every floating-point format spends bits on that exponent; integer quantisation spends none, and buys the range back by attaching one shared scale to a whole block of numbers. That is the entire idea. The rest is bookkeeping.
Quantise: q = clamp(round(x / s) + z, q_min, q_max)
Dequantise: x' = s * (q - z)
s the step size, a positive real (stored as fp16 or fp32)
z the zero-point, an INTEGER in [q_min, q_max]
q_min -128 for int8, 0 for uint8, -8 for int4
q_max 127 for int8, 255 for uint8, 7 for int4
Given an observed range [x_min, x_max]:
s = (x_max - x_min) / (q_max - q_min)
z = round(q_min - x_min / s)
The two formulas for s and z are not conventions, they are the unique solution to two requirements: that x_min maps to q_min and x_max maps to q_max. Solving those two equations simultaneously gives exactly the lines above.
Worked on a real-shaped activation tensor. Take the output of a GELU somewhere in the middle of a network, with an observed minimum of -0.17 and a maximum of 3.41, quantised to uint8:
x_min = -0.17, x_max = 3.41, q_min = 0, q_max = 255
s = (3.41 - (-0.17)) / 255 = 3.58 / 255 = 0.01403922
z = round(0 - (-0.17 / 0.01403922))
= round(12.1090) = 12
Check the endpoints:
x = -0.17 -> round(-12.1090) + 12 = -12 + 12 = 0 ok
x = 3.41 -> round(242.8910) + 12 = 243 + 12 = 255 ok
x = 0.00 -> round(0) + 12 = 0 + 12 = 12
Dequantise that last one:
x' = 0.01403922 * (12 - 12) = 0.0 exactly zero
One more, in the middle:
x = 1.00 -> round(71.2291) + 12 = 71 + 12 = 83
x' = 0.01403922 * (83 - 12) = 0.99678
error = -0.00322
Zero surviving exactly is the reason z is an integer rather than a real offset. Padding tokens, masked positions and every output of a ReLU are exactly zero, and a scheme that turned them into 0.003 would leak a small signal into every place the network relies on nothing being there.
The price of that integer is small and worth naming, because almost nobody does. Rounding z from 12.1090 to 12 shifts the whole representable window:
Representable window after rounding z:
lowest = s * (0 - 12) = -0.16847
highest = s * (255 - 12) = 3.41153
Asked for [-0.17000, 3.41000]
Got [-0.16847, 3.41153]
Width is identical (3.58). The window slid up by 0.00153,
which is 0.109 of a step. Any value in [-0.17, -0.16847)
now clamps to q = 0 and dequantises to -0.16847.
Symmetric or asymmetric, and why it differs by tensor
Symmetric quantisation fixes z = 0 and derives the scale from the absolute maximum alone, so the map collapses to q = round(x/s) and a dequantisation of s*q. Asymmetric keeps the zero-point. The choice is not a matter of taste; it falls out of what the two tensors look like and out of what the matmul has to compute.
Symmetric, int8:
s = max(|x|) / 127
z = 0
Worked on one weight row with absmax 0.212:
s = 0.212 / 127 = 0.00166929
s/2 = 0.00083465 <- the error bound
w w/s q x' = s*q error
------------------------------------------------
0.031 18.5710 19 0.031717 +0.000717
-0.052 -31.1508 -31 -0.051748 +0.000252
0.018 10.7830 11 0.018362 +0.000362
0.212 127.0000 127 0.212000 0.000000
-0.007 -4.1934 -4 -0.006677 +0.000323
Every error is below 0.00083465. It has to be — see below.
Weights take the symmetric form because of what happens when the matmul is expanded. Substitute both dequantisation formulas into a single dot product over K terms and multiply out:
sum_k x_k * w_k
= sum_k s_x(q_x - z_x) * s_w(q_w - z_w)
= s_x*s_w * [ sum_k q_x*q_w <- the real work
- z_w * sum_k q_x <- depends on the input
- z_x * sum_k q_w <- constant per output channel
+ K * z_x * z_w ] <- a scalar
With z_w = 0 (symmetric weights), terms 2 and 4 vanish:
= s_x*s_w * [ sum_k q_x*q_w - z_x * sum_k q_w ]
sum_k q_w is a per-output-channel integer known at load
time. So the inner loop is a pure int8 dot product, and the
correction is one subtraction per output element.
Term 2 is the expensive one. It depends on the activations, so it cannot be precomputed, and it costs a full extra reduction over K per output. Setting z_w = 0 deletes it. That is the whole argument, and it is an argument about the shape of the matmul, not about the distribution of weights — although the distribution cooperates, since a trained weight row is roughly zero-centred and loses little to a symmetric range.
Activations go the other way because they are one-sided. A ReLU or a GELU output is mostly non-negative, and forcing symmetry throws away the levels below zero:
Post-ReLU activations, observed range [0, 6.0]
Symmetric int8: s = 6.0 / 127 = 0.047244
values live in q = 0..127
levels used: 128 of 256
Asymmetric uint8: s = 6.0 / 255 = 0.023529
values live in q = 0..255
levels used: 256 of 256
The symmetric version has exactly half the resolution:
one whole bit, thrown away on a range the tensor never
visits.
A second, quieter reason activations differ: their range is not known at load time. It is either measured on a calibration set beforehand (static quantisation, where an unrepresentative calibration set is the usual cause of a mysterious accuracy drop) or computed per batch at run time (dynamic quantisation, which is more robust and costs a min/max reduction over every tensor). Weights have no such problem: their range is a fact about a file.
Step size, the half-step bound, and 6.02 dB per bit
The step size is the whole error story. Rounding to the nearest multiple of s cannot be wrong by more than half a step, so for any x inside the representable window:
|x' - x| <= s / 2
For int8 over [-0.212, 0.212]:
s = 0.212 / 127 = 0.00166929
s/2 = 0.00083465
For int4 over the same range:
s = 0.212 / 7 = 0.03028571
s/2 = 0.01514286
Same tensor, 18.1x the worst-case error, because
127 / 7 = 18.1.
That bound holds only inside the window. Outside it there is no bound at all: a value of 0.9 in a tensor scaled for 0.212 clamps to 0.212 and the error is 0.688, which is 824 half-steps. Clipping error and rounding error are different failures with different fixes, and conflating them is the most common mistake in tuning a quantisation scheme.
Now the signal-to-noise result, derived rather than quoted. Model the rounding error as uniformly distributed across one step, which is accurate whenever the signal moves over many steps:
Noise power. e is uniform on [-s/2, s/2]:
var(e) = (1/s) * integral of e^2 from -s/2 to s/2
= (1/s) * (s^3 / 12)
= s^2 / 12
Signal power. Take x uniform over the full range R,
where R = 2^b * s for b bits:
var(x) = R^2 / 12 = (2^b * s)^2 / 12
Ratio:
SQNR = var(x) / var(e)
= [(2^b * s)^2 / 12] / [s^2 / 12]
= 2^(2b)
In decibels:
10 * log10(2^(2b)) = 20b * log10(2)
= 20b * 0.301030
= 6.0206 * b dB
So the constant is 20*log10(2) = 6.0206, and the mechanism behind it is one line: an extra bit halves s, halving s quarters s^2/12, and a factor of four in power is 10*log10(4) = 6.02 dB. Nothing about neural networks enters the derivation; it is the same result that governs audio sampling.
The uniform-signal assumption flatters the answer, and weights are not uniform — they are roughly Gaussian. Redo it for a Gaussian of standard deviation sigma, clipped at k standard deviations so the range is R = 2*k*sigma:
s = 2*k*sigma / 2^b
SQNR = sigma^2 / (s^2 / 12)
= 12 * sigma^2 * 2^(2b) / (4 * k^2 * sigma^2)
= 3 * 2^(2b) / k^2
In dB:
= 6.0206*b + 10*log10(3) - 20*log10(k)
= 6.0206*b + 4.771 - 20*log10(k)
At k = 4 (clip at four sigma):
= 6.0206*b + 4.771 - 12.041
= 6.0206*b - 7.270
b = 8 -> 40.9 dB
b = 4 -> 16.8 dB
b = 2 -> 4.8 dB
The 6.02 slope survives; only the offset changes. That is the useful part of the result: whatever your distribution and wherever you clip, going from 8 bits to 4 costs 24 dB, which is a factor of 16 in error amplitude. Any claim that 4-bit is “nearly free” is a claim about the network’s tolerance for a 16-fold larger perturbation, not about the arithmetic.
Per-tensor, per-channel, per-group: what grouping buys
One scale for a 4096×4096 weight matrix means one number controls the resolution of 16.8 million weights, and that number is set by the single largest of them. Every quieter row pays for the loudest one.
A 4096x4096 matrix. Suppose:
absmax over the whole tensor = 2.10
absmax of one ordinary row = 0.21 (10x smaller)
Per-tensor, symmetric int8:
s = 2.10 / 127 = 0.0165354
the ordinary row occupies q = -13..+13
(since round(0.21 / 0.0165354) = 13)
levels used: 27 of 255
effective bits: log2(27) = 4.75
log2(255 / 27) = 3.24 bits are unreachable.
Per-channel, one scale per row:
s = 0.21 / 127 = 0.00165354
the row occupies the full q = -127..+127
error bound falls from 0.008268 to 0.000827 — 10x
The bits recovered are exactly log2(tensor_absmax / row_absmax). A 10:1 spread across channels costs log2(10) = 3.32 bits per-tensor and nothing per-channel, which is why per-channel weight quantisation is essentially always on. The reason it is free is arithmetic too:
Scale storage, 4096x4096 = 16,777,216 weights,
fp16 scales at 16 bits each:
per-tensor 1 scale 16 bits 0.0000010 bits/weight
per-channel 4,096 scales 65,536 0.0039 bits/weight
per-group 128 131,072 scales 2,097,152 0.1250 bits/weight
per-group 32 524,288 scales 8,388,608 0.5000 bits/weight
Per-channel is free to four decimal places. Per-group is not, and this is where “4-bit” stops meaning four bits. Grouping exists because dynamic range varies along a row as well as between rows, and at 4 bits there are only 15 levels to spend, so a single local outlier inside a row is enough to flatten its neighbours.
Real bytes per weight at 4 bits:
4-bit, per-tensor scale only
(0 + 4)/1 = 4.0000 bits = 0.500000 bytes
4-bit, per-channel fp16 scale
(4*4096 + 16)/4096 = 4.0039 bits = 0.500488 bytes
4-bit, group 128, fp16 scale
(4*128 + 16)/128 = 4.1250 bits = 0.515625 bytes
4-bit, group 128, fp16 scale + 4-bit zero-point
(4*128 + 16 + 4)/128 = 4.1875 bits = 0.523438 bytes
4-bit, group 32, fp16 scale
(4*32 + 16)/32 = 4.5000 bits = 0.562500 bytes
4-bit, group 32, fp16 scale + 4-bit zero-point
(4*32 + 16 + 4)/32 = 4.6250 bits = 0.578125 bytes
On a 7B model (7.0e9 parameters, 1 GB = 1e9 bytes):
0.500000 -> 3.50 GB
0.515625 -> 3.61 GB
0.523438 -> 3.66 GB
0.562500 -> 3.94 GB
0.578125 -> 4.05 GB
The last is 15.6% larger than the nominal 3.50 GB.
0.578 bytes, not 0.5. That gap is why a downloaded 4-bit file is always larger than parameters divided by two, and it is a real 0.55 GB on a 7B model — enough to decide whether it fits. It is also the honest way to compare two 4-bit formats: one at group 128 and one at group 32 are not the same format competing on cleverness, they are 4.19 bits against 4.63 bits, and the larger one should win.
| Granularity | Description |
|---|---|
| per-tensor | One scale for everything. Cheapest to store and to execute, and the only option on some fixed-function accelerators. Loses log2(tensor_absmax / channel_absmax) bits on every quiet channel. |
| per-channel | One scale per output channel, for weights. Costs 0.0039 bits per weight on a 4096-wide matrix and folds into the per-output requantisation the matmul already performs, so it is free in both bytes and instructions. Effectively mandatory below 8 bits. |
| per-group | One scale per 32, 64 or 128 consecutive weights within a row. Costs 0.125 to 0.5 bits per weight and requires the kernel to reload a scale mid-row. This is the knob that makes 4-bit usable, and the reason 4-bit files vary in size between formats. |
| per-token (activations) | One scale per token rather than per tensor, computed at run time. The activation analogue of per-channel: it stops one unusual token in a batch from setting the scale for all of them, and costs a min/max reduction that the memory traffic hides. |
Where the error actually hurts
The half-step bound says the error is uniform and small. Both halves of that are misleading in a transformer, because the input is not uniform: a small number of activation channels carry values one or two orders of magnitude larger than everything else, and they set the scale for everything else.
A hidden state where 99.9% of values lie within +/-2.0,
and one channel holds 60.0.
Per-tensor symmetric int8:
s = 60.0 / 127 = 0.472441
s/2 = 0.236220
An ordinary value of 1.35:
round(1.35 / 0.472441) = round(2.8575) = 3
x' = 3 * 0.472441 = 1.417
error = +0.067
But look at the levels the ordinary values can reach:
the range [-2, 2] maps to q = -4 .. +4
9 levels of 255
effective bits: log2(9) = 3.17
One number destroyed 4.8 of the 8 bits, for 99.9%
of the tensor.
Clipping is the obvious response and it is usually wrong. Clip at ±4.0 and the ordinary values regain all eight bits, but the outlier becomes 4.0 with an error of 56.0 — and those large channels are not noise, they carry information the following layers depend on. The mitigations that work all avoid the choice rather than making it: per-token and per-channel scales so the outlier only poisons its own row, keeping the handful of outlier channels in 16-bit and quantising the rest, or migrating the magnitude from activations into weights beforehand so both end up in a friendly range. Which of those a given method uses is most of what distinguishes one int8 or int4 deployment recipe from another.
- Weights are the easy tensor. Their distribution is close to Gaussian, their range is known offline, and per-channel scales cost nothing. Almost every reported accuracy loss at 8 bits comes from the activations.
- The error is not random. Rounding is deterministic, so the same weight is wrong by the same amount on every token, and those errors compose through layers rather than averaging out. This is the argument for stochastic rounding and for error-compensating methods that adjust the remaining weights after each one is rounded.
- The last layer is not like the others. Quantising the embedding or the output projection tends to cost more than quantising a middle block, which is why many 4-bit formats keep those two in higher precision and why the resulting file is larger than the bits-per-weight table alone predicts.
Why the accumulator is int32
Two int8 values multiply to at most 127 * 127 = 16,129, which needs 15 bits. A dot product then adds K of those together, and the width required grows with K:
Worst-case magnitude of a K-term int8 dot product:
K * 127 * 127 = K * 16,129
int8 max 127 overflows at K = 1
int16 max 32,767 overflows at K = 3
int32 max 2,147,483,647 overflows at K = 133,145
For a 4096-wide hidden dimension:
4096 * 16,129 = 66,064,384
which is 3.1% of the int32 range.
Bits needed: log2(66,064,384) = 26.0
int16 fails after two terms, so the choice is really int32 or nothing. The 133,145 figure is the absolute worst case, where every product is maximal and every sign agrees; a real dot product has mixed signs and grows like sqrt(K) rather than K, so the headroom in practice is far larger than 3.1% suggests.
The property that matters more than the headroom is that the accumulation is exact. Integer addition below the overflow point loses nothing, so a 4096-term int8 dot product has precisely one source of error — the rounding applied to its inputs — and none at all from the summation. Compare that with an fp16 accumulator, which stops moving at 2,048 and rounds at every one of the 4,096 additions. This is the reason int8 inference is accurate at all, and it is the same reason every tensor core accumulates 16-bit floats into fp32.
The int32 result then has to become an int8 input for the next layer. That step is a requantisation, and it is where the scales finally arrive:
acc int32, = sum_k q_x * q_w (plus the z_x correction)
real = s_x * s_w * acc
q_out = clamp(round(real / s_y) + z_y, q_min, q_max)
Collect the constants into one multiplier M:
M = (s_x * s_w) / s_y
q_out = clamp(round(M * acc) + z_y, q_min, q_max)
M is known before the model runs and is normally in (0, 1),
so it is stored as a fixed-point integer and a right shift:
M ~= M0 * 2^-n, with M0 a 32-bit integer
which turns the whole rescale into one integer multiply and
one shift. No floating-point unit is touched anywhere in the
layer, which is the point of the exercise on hardware that
has no fast fp16.
That last block is what “integer-only inference” means literally: the scales exist, but they were folded into a 32-bit multiplier and a shift count before the first token arrived, and nothing in the inner loop knows they were ever real numbers.
One thing the arithmetic on this page cannot tell you: how much accuracy a given model loses at a given bit width. That depends on the model, the calibration data and the task, it is measured rather than derived, and any single number quoted for it — “4-bit costs 1% accuracy” — is a result about one model on one benchmark. What the arithmetic does give you is the shape of the trade: 24 dB per four bits, an error bound of s/2 inside the window and unbounded outside it, and the exact bytes each scheme costs. Those three are enough to rule most options out before measuring anything.
Top comments (0)