Every one of these formats splits its bits into a sign, an exponent and a mantissa. The exponent buys range — how large and how small a number can be — and the mantissa buys precision. bf16 exists because gradients were underflowing in fp16, and it fixed that by taking three bits from the mantissa and giving them to the exponent.
How a float is built out of bits
value = (-1)^S x 2^(E - bias) x 1.M
S one sign bit
E exponent field, an unsigned integer
bias a constant subtracted from E so it can go negative
1.M the mantissa, with an implied leading 1
bias = 2^(exponent_bits - 1) - 1
The implied leading 1 is the trick that gets one bit for free: any normalised binary number starts with a 1, so there is no point storing it. A 10-bit mantissa therefore gives 11 bits of significand.
Worked once, on the fp16 encoding of 1.5:
1.5 = 1.1 in binary = 1.1000000000 x 2^0
S = 0
E = 0 + bias = 0 + 15 = 15 = 01111
M = 1000000000
0 01111 1000000000
Read back: (-1)^0 * 2^(15-15) * 1.5 = 1.5
The layouts, drawn
fp32 S EEEEEEEE MMMMMMMMMMMMMMMMMMMMMMM
1 8 23 = 32 bits, bias 127
fp16 S EEEEE MMMMMMMMMM
1 5 10 = 16 bits, bias 15
bf16 S EEEEEEEE MMMMMMM
1 8 7 = 16 bits, bias 127
fp8 E4M3 S EEEE MMM
1 4 3 = 8 bits, bias 7
fp8 E5M2 S EEEEE MM
1 5 2 = 8 bits, bias 15
int4 S MMM
1 3 = 4 bits, no exponent
Line the three 16-bit rows up against fp32 and the design intent is visible at a glance. fp16 keeps a large mantissa and cuts the exponent from 8 bits to 5. bf16 keeps the fp32 exponent exactly — same 8 bits, same bias of 127 — and cuts the mantissa from 23 to 7. bf16 is literally the top 16 bits of an fp32 number, which is why converting between them is a truncation and costs nothing.
What each width buys, computed
Both properties follow from the field widths by arithmetic. Range comes from the exponent: the largest finite value is 2^(E_max - bias) * (2 - 2^-mantissa_bits). Precision comes from the mantissa: the smallest relative gap between neighbouring values is 2^-(mantissa_bits + 1).
| Format | Description |
|---|---|
| fp32 (1/8/23) | Max 3.403e38. Min normal 1.175e-38. Relative precision 2^-24 = 6.0e-8, about 7.2 decimal digits. Integers exact up to 2^24 = 16,777,216. |
| fp16 (1/5/10) | Max 65,504. Min normal 6.104e-5, min subnormal 5.96e-8. Relative precision 2^-11 = 4.9e-4, about 3.3 digits. Integers exact up to 2^11 = 2,048. |
| bf16 (1/8/7) | Max 3.390e38. Min normal 1.175e-38. Relative precision 2^-8 = 3.9e-3, about 2.4 digits. Integers exact up to 2^8 = 256. |
| fp8 E4M3 (1/4/3) | Max 448. Min normal 0.015625. Relative precision 2^-4 = 0.0625. Integers exact up to 16. The forward-pass format. |
| fp8 E5M2 (1/5/2) | Max 57,344. Min normal 6.104e-5. Relative precision 2^-3 = 0.125. More range, less precision — the gradient format. |
The “integers exact up to” column is the one that surprises people. bf16 cannot represent 257. It has 8 significand bits total, 257 needs 9, so it rounds to 256. Neither can it represent 1,000 exactly — the nearest bf16 values around it are 1,008 and 992. This does not matter for weights, which are small fractions, and matters enormously for anything counting.
Same number in each format, so the precision loss is visible:
1/3 = 0.3333333333333...
fp32 0.33333334 (24 significand bits)
fp16 0.33325195 (11 significand bits)
bf16 0.33398438 ( 8 significand bits)
Absolute errors:
fp32 0.00000001
fp16 0.00008138
bf16 0.00065105
bf16 is 8x coarser than fp16, which is exactly 2^3
for the 3 mantissa bits it gave away.
The trade bf16 made, and the bug it fixed
Training in fp16 broke, and it broke at the bottom of the range rather than the top. Gradients late in training are small — magnitudes around 1e-7 and below are ordinary — and fp16’s smallest normal value is 6.1e-5.
A gradient of 1e-8 in each format:
fp16 1e-8 is below the min subnormal 5.96e-8
-> flushes to 0.0
The weight receives no update. Ever.
bf16 min normal is 1.175e-38
-> represented fine, with 2 to 3 significant digits
fp32 -> represented fine
A gradient that becomes exactly zero is not a small error, it is a weight that stops learning, silently, with no warning in any log. The workaround for fp16 is loss scaling: multiply the loss by a large constant before the backward pass so every gradient is scaled up into representable territory, then divide the gradients by the same constant before the optimiser step.
scale = 1024 = 2^10
gradient 1e-8 x 1024 = 1.024e-5 representable in fp16
optimiser step: divide by 1024 again
Dynamic loss scaling: raise the scale until an inf or nan
appears, then halve it and skip that step. This is a control
loop, running inside the training loop, that exists purely
because fp16's exponent is 5 bits wide.
bf16 deletes that entire mechanism. It has fp32’s exponent, so nothing that fits in fp32 underflows in bf16, so no scaling is needed and no control loop can misbehave. The cost is 3 mantissa bits, and the reason that cost is acceptable is that gradient descent is averaging noisy estimates anyway — two or three significant digits per gradient, averaged over a large batch, is enough signal.
- Range mattered more than precision. The failure mode of too little range is silent zeros; the failure mode of too little precision is a slightly noisier gradient. One of those is recoverable.
- Conversion is free. fp32 to bf16 is dropping the low 16 bits; bf16 to fp32 is appending 16 zeros. No exponent rebiasing, no special cases.
- Inference is a different question. At inference there are no gradients, activations sit in a comfortable range, and fp16’s extra mantissa bits are genuinely useful. Plenty of inference stacks prefer fp16 for exactly that reason, and the outlier activations that do exceed 65,504 are handled by scaling specific layers rather than by changing format.
Zero, infinity, NaN and subnormals
Two exponent patterns are reserved, which is where every special value lives. The formula 2^(E - bias) * 1.M applies only when the exponent field is neither all-zeros nor all-ones.
| Encoding | Description |
|---|---|
| E all zeros, M zero | Zero. There are two of them, +0 and -0. They compare equal, but 1/+0 is +inf and 1/-0 is -inf, so a sign that no comparison can see still changes the result of a division. |
| E all zeros, M nonzero | Subnormal. The implied leading 1 becomes a leading 0, which fills the gap between zero and the smallest normal value at gradually reducing precision. |
| E all ones, M zero | Infinity, with a sign. Produced by overflow and by division by zero. |
| E all ones, M nonzero | NaN. Produced by 0/0, inf - inf, inf/inf, and sqrt of a negative. It propagates through every arithmetic operation it touches. |
Subnormals are the part with a real performance consequence. Without them, the smallest fp16 value would be 6.104e-5 and everything below would be zero. With them, the range extends down to 2^-24 = 5.96e-8:
fp16 subnormals: E = 00000, M = 1..1023
value = M * 2^-24
smallest: 1 * 2^-24 = 5.9605e-8
largest: 1023 * 2^-24 = 6.0976e-5
smallest normal = 6.1035e-5
1023 extra representable values in the gap, at
progressively fewer significant bits — the largest
subnormal has 10 bits of precision, the smallest has 1.
Many accelerators run in flush-to-zero mode, where subnormal results are set to zero rather than handled, because subnormal arithmetic is slow on some hardware and was historically trapped to software. The consequence for training is direct: a gradient of 1e-7 is representable as an fp16 subnormal in principle and becomes zero in practice, so the effective underflow floor is the smallest normal value rather than the smallest subnormal. It is one more reason bf16 won.
The one property to remember about NaN: it is the only value not equal to itself. x != x is true exactly when x is NaN, and that is how every NaN check in every language is implemented. It also means a sort over an array containing NaN has undefined behaviour, and a maximum computed with a naive comparison loop can silently return the wrong element.
int4 is not a float at all
There is no exponent field. int4 stores a signed integer in the range -8 to +7 — sixteen levels — and recovers a real number using a scale shared across a group of weights:
w ~= scale * (q - zero_point)
q 4-bit integer, -8..7 (or 0..15 unsigned)
scale one fp16 value per group
zero_point optional offset, for asymmetric ranges
group typically 32, 64 or 128 consecutive weights
Example, group of 4 real weights:
[0.031, -0.052, 0.018, 0.044]
absmax = 0.052, symmetric int4 range is 7 levels each way
scale = 0.052 / 7 = 0.0074286
q = round(w / scale) = [4, -7, 2, 6]
dequantised = [0.029714, -0.052, 0.014857, 0.044571]
errors = [0.0013, 0.0000, 0.0031, 0.0006]
The error is bounded by half a scale step, which is why group size is the knob that matters: a smaller group means the scale tracks the local magnitude more closely, at the cost of storing more scales. A group of 128 with one fp16 scale adds 16 / 128 = 0.125 bits per weight, so “4-bit” is really 4.125 bits, and adding a zero-point makes it 4.25.
This is also why quantisation quality depends so much on outliers. A single large weight in a group forces a large scale, and every other weight in that group loses resolution. The techniques that distinguish one 4-bit method from another are largely different answers to that one problem.
Bytes per weight, and what a model file weighs
file size ~= params * bytes_per_weight
For a 7B model (6.97e9 parameters, using 1 GB = 1e9 bytes):
fp32 4 bytes -> 27.9 GB
fp16/bf16 2 bytes -> 13.9 GB
fp8 1 byte -> 7.0 GB
int4 0.53 bytes -> 3.7 GB (4.25 bits with scales)
For a 70B model:
fp16 140 GB
fp8 70 GB
int4 37 GB
This single line is the most-used piece of arithmetic in local inference: it tells you whether a model fits, and the rest of the VRAM budget — the KV cache, the activations, the framework overhead — is added on top of it.
It also predicts decoding speed, because generating one token requires reading every weight once. At 3.3 TB/s of memory bandwidth, 13.9 GB of weights takes 4.2 ms per token and 3.7 GB takes 1.1 ms. Quantisation does not reduce the FLOP count at all; it reduces the bytes moved, and bytes moved is what actually limits single-stream generation.
Providers serve the same model name at different precisions, and the quantisation is not always stated on the model card. Multigrid lists what each provider exposes per model, which is the first thing to check when the same model name gives you visibly different output quality on two routes.
Top comments (0)