Our fp16 quantization post-mortem was wrong in the expensive direction: it told anyone with a mask-gated model to give up on half-precision, and that verdict would have kept a 52%-smaller download off the internet. The real cause was sixteen BatchNorm variance tensors larger than fp16's maximum representable value, silently clamped by the converter, leaving sixteen layers normalizing their output by the wrong divisor. The fix is two lines of algebra. The reward is a model that matches fp32 and downloads 52% smaller.
This is a correction post, so it comes with receipts: the lab notes for this round are checked in next to the code, every figure below is reproducible from them on a CPU-only ONNX Runtime install, and the whole investigation fits in a day or two of work.
The claim we published
Part 3 of this series ends with a case study called "When fp16 quantization quality loss is real: the LaMa failure". The measurements in it are real, and the failure is reproducible: rebuilding that conversion from the same weights today, the unfixed fp16 model averages 15.8 dB against fp32's 29.7 dB across our 14 acceptance cases, and its worst case scores 11.1 dB — the watermark surviving, exactly as described. Part 5 repeats it as "its catastrophic mirror image when we tried it on LaMa". Part 1's series index summarises the lesson as "mask-gated architectures and precision sensitivity".
We offered a mechanism too, and it was a good story. LaMa is mask-gated: the mask doesn't just mark where to work, it gates the network's behavior. A mask-gated inpainter therefore has a decision boundary inside it — which regions are trustworthy context, which must be regenerated — and half-precision noise pushes the model across that boundary. The gating fails softly, the model starts copying the watermarked pixels through instead of repairing them, and the output looks plausible except the watermark is still there.
That story explained the symptom exactly. It was consistent with everything else we'd learned (browser inference is bandwidth-bound, so moving half the bytes is free money). And it produced a tidy rule that we then repeated to readers:
whether quantization is safe is a property of the model's architecture, not of your tooling
The rule is wrong. Not "wrong with exceptions" — wrong as a predictor, and expensive: it would have kept a quality-identical model that is 48% smaller out of the product forever, and it tells every reader with an attention head, a gating mask, or a long residual chain to stop investigating. Part 8 of this series was about a quality bug that turned out to be ours, not the model's. This is the same shape of mistake, one layer lower in the stack: our conversion tooling, not the model's architecture.
Step one: prove that precision is not even the variable
"FP16" bundles two independent changes: mantissa precision (how finely you can split a value) and numeric range (the largest magnitude you can hold at all). If the failure were precision sensitivity, then rounding the mantissa while keeping fp32's range would degrade quality in proportion. So we built models that keep the graph and all arithmetic in fp32, and only round the weights' mantissas:
| Weights rounded to | Region PSNR (5 cases) | Output fidelity vs fp32 |
|---|---|---|
| fp32 (baseline) | 28.88 | — |
| 7-bit mantissa (bf16) | 28.83 | 66.9–76.9 dB |
| 10-bit mantissa (= fp16) | 28.88 | 74.5–96.6 dB |
| 13-bit mantissa | 28.88 | 95.9–109.0 dB |
fp16 has a 10-bit mantissa. The 10-bit row is 0.004 dB off the fp32 baseline, and even throwing away 16 of the 23 mantissa bits (7-bit, coarser than bf16) costs 0.054 dB. LaMa is not precision-sensitive — it is precision-indifferent. Whatever fp16 was doing to it, it wasn't the mantissa.
That left range, and range has a hard, checkable boundary: fp16's largest finite value is 65,504. So we dumped every fp32 initializer in the graph and sorted by magnitude.
Step two: find the numbers that don't fit
The graph is 17,480 nodes with 606 float32 initializer tensors, 51M parameters, 204 MB of weights — 197 MB of it in the FFT path (17,022 of those nodes are Fourier convolutions, 97%). Run the check below over the 606 tensors and sixteen come back above 65,504, and every single one is a BatchNorm running_var:
484322.56 model.generator.model.5.conv1.bn_l.running_var
440853.66 model.generator.model.25.running_var
240497.69 model.generator.model.22.conv1.bn_l.running_var
218544.31 model.generator.model.11.conv1.bn_l.running_var
174809.92 model.generator.model.6.conv1.bn_l.running_var
65564.05 model.generator.model.10.conv1.bn_l.running_var
... 16 of 606 tensors, every one of them a running_var
Note the last row: the marginal tensor clears the ceiling by 0.09%, and it gets clamped to exactly the same 65,504 as the one that clears it by 7×. That's the whole hazard — a clamped value sits exactly at fp16's ceiling, which is a value fp16 can barely hold, so the next fp16 pass over that tensor takes it to infinity.
This is a five-line check, and it is the first thing to run before believing any "this model can't be quantized" conclusion:
import numpy as np, onnx
from onnx import numpy_helper
m = onnx.load("lama_fp32.onnx", load_external_data=False)
worst = sorted(
((float(np.abs(numpy_helper.to_array(t)).max()), t.name)
for t in m.graph.initializer if t.data_type == 1), # 1 = float32
reverse=True,
)[:16]
for value, name in worst:
print(f"{value:>14.2f} {name}")
Sixteen tensors. Not sixteen layers of masking logic — sixteen numbers.
Why a BatchNorm variance is the thing that overflows
A BatchNorm layer computes y = (x − μ) / sqrt(var + ε) · γ + β. running_var is a scale quantity measured over the layer's input during training. If a layer's inputs live at large magnitudes, its variance lives at the square of those magnitudes, and 484,322 is entirely legitimate in fp32: sqrt(484322) ≈ 696.
Clamp that variance to fp16's ceiling of 65,504 and the divisor becomes sqrt(65504) ≈ 256. The layer now normalizes by 256 instead of 696 — its output is 2.7× too large — and the convolution that consumes it was trained against the correct scale. Sixteen of those, in series, and the network's internal magnitudes are destroyed while the graph still runs and still produces plausible-looking images.
That last part is the trap. There is no NaN, no warning, no crash: just a model that quietly keeps the watermark.
The clamp that never said a word
The conversion never told us anything, and this is the part worth internalising. onnxruntime.transformers.float16.convert_float_to_float16 clamps to max_finite_val=65504; onnxconverter_common.float16 defaults to 10000 (48× more aggressive). Both clamp silently — that's what "clamp" means: the fp16 model has a finite, wrong number in it, and nothing anywhere prints a line.
You can watch the corruption happen tensor by tensor, which is how we finally believed it:
| Stage | model.5.conv1.bn_l.running_var |
|---|---|
| fp32 source | 484322.5625 |
after keep_io_types fp16 conversion |
65504.0 (clamped to the fp16 ceiling) |
| after a further fp16 storage pass | inf |
| after our fix (see below) | 1892.0 (fits comfortably in fp16) |
Note the third row: the clamped value is already at the ceiling, so a second conversion overflows it to infinity. Whether your broken model lands on 65504 or inf depends on which converter pass touched it last — which is also why the failure looked so erratic in our old notes.
The fix is algebra
We tried the blunt version first: keep the sixteen offending BatchNorm nodes in fp32 and convert everything else. That works, costs 34 KB of fp32 parameters, and produced identical quality — but it leaves a graph with fp32 islands and sixteen permanent assertions about which layers are dangerous.
The better fix is to notice that a BatchNorm's variance and its scale are only meaningful relative to each other:
y = (x − μ) / sqrt(var + ε) · γ + β
≡ (x − μ) / sqrt(var/K² + ε/K²) · (γ/K) + β
So pick a power-of-two K per layer, divide the variance by K², divide the scale by K, and divide epsilon by K² — the layer is mathematically identical, and now var/K² fits in fp16. With K = 2^ceil(log2(max|var| / 40000)), that's a twenty-line script touching 19 of the 75 BatchNorm layers:
k = float(2 ** np.ceil(np.log2(max_abs_var / 40000.0))) # K = 16 for the worst layer
var_init.CopyFrom(numpy_helper.from_array((var / (k * k)).astype(np.float32), var_init.name))
scale_init.CopyFrom(numpy_helper.from_array((scale / k).astype(np.float32), scale_init.name))
# and the epsilon attribute: a.f = a.f / (k * k)
The result is worth stating precisely, because it kills the "mixed precision is inherently fuzzy" intuition: the converted model has 606 fp16 initializers and zero fp32 residue. It is 100% fp16, every tensor, with no casts added around the BatchNorms. And the arithmetic is exact — powers of two, so nothing is lost to rounding.
What this is worth — and what it is not
Three independent measurements on the same converted weights, plus a control and a harness sanity check:
| Check | fp32 | fp16 (fixed) |
|---|---|---|
| 14-case masked-region PSNR | 29.74 dB | 29.73 dB |
| Per-case ghost-score drift (mean / max) | — | 0.0014 / 0.005 |
| Browser runtime (ORT Web WASM), same tile, vs fp32 reference | — | 66.5 dB |
| Unfixed fp16, same test (control) | — | 22.6 dB |
| 16 real photo regions, production-detected boxes | — | 59.9 dB mean, 40.3 dB min |
The browser row is the one that matters for shipping: that's the exact runtime our production HQ tier uses, and 66.5 dB means the two models disagree at the level of sub-grey-step rounding. The control row is what the old mechanism should have looked like if it were real precision damage — 22.6 dB, i.e. visibly different. (Sanity check on the harness: browser fp32 vs a CPU fp32 reference on the same tile is 156 dB, so the comparison chain isn't the limiting factor.)
The size win is the point of the exercise:
| Raw | brotli q11 (what users download) | |
|---|---|---|
| fp32 | 198.4 MiB | 179.2 MiB |
| fp16 (fixed) | 102.8 MiB | 86.3 MiB (−52%) |
The brotli pipeline that makes a 198 MiB weight file a 179 MiB download in the first place is part 9's story — worth saying here because the win compounds: same compressor, half the bytes.
And now the honest part, which is the real lesson for anyone shipping fp16 in a browser: fp16 made our inpainting slower, not faster. 11.4 s → 17.4 s per tile on the WASM execution provider, 2.7 s → 9.4 s on CPU. The WASM EP has no native fp16 convolution kernels, so it inserts casts and computes in fp32 anyway, paying the copy tax for nothing. Part 3's 3.5× speedup came from WebGPU, which has real fp16 ALUs; on the WebGPU-disabled HQ path, half-precision buys bytes and nothing else. If a future ORT Web release ships fp16 kernels on WASM — or the JSEP gap that keeps HQ off WebGPU gets closed upstream — this model collects the speed win too. The fix is the precondition for that, not the payoff.
The int8 half, where the conclusion survives and the reason does not
The old post also reported int8 failing (24.5 → 15.6 dB on our dark-text case) and concluded that quantization is architecture-hostile in general. We re-ran both int8 recipes on the current five-case subset (our case definitions have drifted since 2026-10, so compare within a run, not across posts):
| Variant | Region PSNR | vs fp32 |
|---|---|---|
| fp32 | 28.88 | — |
| int8 dynamic (weights only, per-channel) | 17.82 | −11.1 dB |
| int8 QDQ (activations too), all 75 BatchNorms excluded | 19.11 | −9.8 dB |
So int8 is still dead for this model — the conclusion stands. But the mechanism is the opposite of the one we published: this isn't a weight-precision problem, and it isn't the BatchNorm overflow either. Explicitly excluding every BatchNorm from quantization doesn't rescue it, because the inputs to those 75 layers still arrive through DequantizeLinear. Counting the graph: 432 Einsum/MatMul nodes with 648 inputs wrapped in quantize/dequantize pairs. The FFT path is expanded into spectral matrix products during export, and that's the part that cannot survive 8-bit activations. Different failure, different size of blast radius, and it stays on the do-not-retry list.
The corrected decision order
Our old order was architecture → bit width → runtime. The new one starts one step earlier, because a conversion artifact can impersonate an architectural verdict:
-
Audit the values before you audit the architecture. Dump
max |value|for every initializer, compare against the target dtype's ceiling, and check the subnormal floor while you're there. In our model 16 of 606 tensors overflowed fp16 (0.087% of parameters sat below the subnormal floor — harmless, but the same scan finds both). This costs minutes and can invalidate a conclusion you'd otherwise keep for months. - Separate precision from range. Round the mantissa, keep the range. If quality doesn't move, precision is not your variable and no amount of "the model is sensitive" theorising will help.
- Then test the structure. Gating, long accumulation chains, adversarial spectral paths — all real failure modes, but they're hypotheses to test, not a diagnosis.
- Verify in the runtime you ship. A Python CPU pass proves the weights; it does not prove that ORT Web's WASM EP, with its cast insertion and kernel gaps, produces the same thing.
- Benchmarks decide admission. Unchanged, and it's the rule that caught this: the broken model looked plausible at 15.8 dB, the fixed one measures 66.5 dB against the fp32 reference, and only the benchmark could tell you which was which.
Two corrections to our own record, then: the LaMa fp16 entry comes off the do-not-retry list and onto the opt-in path (fp16 verified against fp32 on every check above, fp32 spec retained as the fallback), the int8 entry stays, and the sentence in parts 1 and 5 that makes quantization safety a property of architecture gets retired in favour of something less quotable and more true: a quantization failure is a measurement, and most measurements have a checkable cause.
Try it
Everything here runs client-side with no uploads. The fp32 model that this post improves powers the free watermark remover, and the inpainting engine is the same one behind the video watermark remover. The fp16 path that started all of this is live in our image upscaler, and the quality dashboard publishes the bench numbers these gates are built on.
Part 13 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.
Top comments (0)