Quantizing a Model for ONNX Edge Deployment

#ai #machinelearning #llm #tutorial

ONNX Runtime’s quantizer will produce a smaller model in one function call. Whether that model is fast, accurate, and acceptable to the accelerator you are targeting depends on three choices the one-call version makes for you.

Dynamic or static: pick before you start

The two APIs are not interchangeable and the difference is where the activation scale comes from. Dynamic quantization computes the scale and zero point for activations at inference time, which costs a little work per run and generally preserves accuracy well because the scale always fits the actual data. Static quantization computes those parameters ahead of time from calibration data and bakes them into the model as constants, removing the runtime cost and adding a dependency on the calibration set being representative.

The choice is often made for you by the target. NPU backends want a fully static QDQ graph — the Qualcomm HTP backend, for example, requires quantized models with fixed shapes, as set out in the Hexagon page. For a CPU-only edge target where you mostly want the file to be smaller, dynamic is the lower-effort start. The wider trade-off is in edge quantization.

There is a third choice hiding inside both, and it is the one that determines how much accuracy you lose: what gets quantized. Weights are the easy part — they are fixed, you can inspect their distribution, and per-channel scales handle most of the variation. Activations are the hard part, because their range depends on the input and a single outlier in calibration widens the scale for everything else. This is why weight-only schemes survive aggressive bit widths and full integer schemes do not, and it is the same asymmetry that makes Qualcomm’s HTP offer 16-bit activations alongside 8-bit weights.

Preprocess first, or the results will disappoint

ONNX Runtime’s quantization documentation is explicit that pre-processing should run before quantization, and skipping it is the most common reason a quantized model is barely smaller or noticeably worse than expected. The step performs symbolic shape inference, graph optimization and ONNX shape inference, which is what lets the quantizer recognise the patterns it can fuse.

python -m onnxruntime.quantization.preprocess \
  --input model.onnx \
  --output model-infer.onnx

For transformer models there is a second, earlier step: ONNX Runtime’s guidance is to run its transformer model optimization tool before quantizing, so that attention blocks are fused into the operators that have quantized implementations. Quantizing an unfused attention graph gives you a smaller model that is not meaningfully faster.

The dynamic path

Preprocess, as above, producing model-infer.onnx.
Call quantize_dynamic with an explicit weight type. Do not leave it to the default if you care which one you get.
Compare the file sizes.
Run both models on a held-out set and compare outputs before you believe the size number means anything.

import os
import numpy as np
import onnxruntime as ort
from onnxruntime.quantization import quantize_dynamic, QuantType

quantize_dynamic(
    model_input="model-infer.onnx",
    model_output="model-int8.onnx",
    weight_type=QuantType.QUInt8,
)

before = os.path.getsize("model-infer.onnx") / 1e6
after = os.path.getsize("model-int8.onnx") / 1e6
print(f"{before:.1f} MB -> {after:.1f} MB  ({after / before:.2f}x)")

ref = ort.InferenceSession("model-infer.onnx", providers=["CPUExecutionProvider"])
qnt = ort.InferenceSession("model-int8.onnx", providers=["CPUExecutionProvider"])

name = ref.get_inputs()[0].name
worst = 0.0
for x in held_out_batches:                       # real inputs, not noise
    a = ref.run(None, {name: x})[0]
    b = qnt.run(None, {name: x})[0]
    worst = max(worst, float(np.max(np.abs(a - b))))
print(f"worst absolute difference: {worst:.5f}")

The expected size ratio is roughly a quarter for the quantized tensors, since fp32 weights become 8-bit. The realised ratio is always worse, because dynamic quantization only converts the operator types it has quantized kernels for — ONNX Runtime keeps that list in its operator registry — and everything else stays fp32. A model that only shrinks by 20% is telling you that most of its weight mass is in operators the quantizer left alone.

The static path, with a calibration reader

Static quantization needs a CalibrationDataReader: an object that yields feed dictionaries of representative inputs. A few hundred real samples is the usual guidance, and “real” is load-bearing — calibrating on synthetic data produces ranges that do not match production and an accuracy cliff on the inputs that matter.

from onnxruntime.quantization import (
    quantize_static, CalibrationDataReader, QuantType, QuantFormat,
)

class Reader(CalibrationDataReader):
    def __init__(self, samples, input_name):
        self.it = iter([{input_name: s} for s in samples])

    def get_next(self):
        return next(self.it, None)

quantize_static(
    model_input="model-infer.onnx",
    model_output="model-int8-qdq.onnx",
    calibration_data_reader=Reader(calibration_samples, "input"),
    quant_format=QuantFormat.QDQ,
    activation_type=QuantType.QUInt8,
    weight_type=QuantType.QInt8,
)

QuantFormat.QDQ is the format to prefer. It inserts explicit QuantizeLinear and DeQuantizeLinear nodes around the quantized regions, which every accelerator backend knows how to read and fuse. The alternative, QuantFormat.QOperator, replaces the nodes with dedicated quantized operators such as QLinearConv — more compact, and less portable across execution providers.

Two things about the calibration set are worth stating plainly, because they are where static quantization goes wrong quietly. It should cover the range of inputs you expect in production, including the awkward ones: the very long document, the very quiet audio clip, the dark image. And it should not be your test set, because the quantized model has now been fitted to it and evaluating on the same data will tell you the compression was free. Hold out a separate evaluation split before you calibrate.

The other option worth knowing is the calibration method. The default picks activation ranges from observed minima and maxima, which is maximally sensitive to a single outlier; percentile and entropy-based methods clip the tail instead and usually give a better scale for the bulk of the distribution at the cost of saturating rare extremes. Which is right depends on whether those extremes carry signal in your task, and that is a question about your data rather than about ONNX Runtime.

Format and saturation traps

S8S8 with QOperator is slow on x86-64. ONNX Runtime’s documentation says so directly and recommends avoiding it in general; S8S8 with QDQ is the default and the balance it recommends (ONNX Runtime quantization docs).
U8S8 can saturate on AVX2 and AVX512 without VNNI. ONNX Runtime uses the VPMADDUBSW instruction for U8S8 on those machines, and that instruction can saturate, which shows up as accuracy loss rather than an error. The documented remedies are the reduce_range option or the U8U8 format, which does not have the problem. The documentation states this does not affect x64 with VNNI or Arm.
Your development machine is not your target. The saturation issue above is x86-specific; if you are quantizing on a laptop and deploying to an Arm board, you can be tuning around a problem that does not exist on the device and missing one that does. Validate on the target.
Per-channel weights are usually worth it. A single scale for an entire convolution weight tensor is set by its largest outlier. Per-channel scales cost a little metadata and frequently recover most of the accuracy gap, which is the same reason grouped granularity matters in Core ML palettization.