What I Learned Quantizing DistilBERT to ONNX for Browser Inference

What I Learned Quantizing DistilBERT to ONNX for Browser Inference

I built a support-ticket classifier — 77 banking intents, fine-tuned DistilBERT, Banking77 — and got it to 92.2% accuracy. Then I tried to ship it, and ran into the actual problem: the checkpoint was 256 MB and took ~9 ms per query on CPU. That's fine on a GPU server. It is not fine as a static site with no backend, which is what I wanted this to be — no server to pay for, no cold start, no infra to babysit.

The fix was exporting to ONNX and quantizing to int8, then running the whole thing client-side with Transformers.js. The end state — a live demo that downloads a 64 MB model once and classifies text entirely in the browser — sounds like a clean pipeline in one sentence. Getting there wasn't. This is the writeup of what quantization actually bought, what it didn't, and the four or five ways I got tripped up along the way. The full project (thresholding, error analysis, the case for shipping the transformer at all) is in the repo — this post is scoped to the conversion itself.

Why bother — the size problem, not the accuracy problem

Quick context on why this mattered. DistilBERT beat a TF-IDF + logistic regression baseline by 3.5 accuracy points at full coverage (92.2% vs 88.7%), which on its own is a mediocre argument for a 61×-larger, 46×-slower model. The real case for the transformer showed up once I added a confidence threshold and let the system escalate uncertain predictions instead of guessing: at a fixed 99% accuracy bar, DistilBERT auto-routed 72% of traffic against 53–55% for the cheaper models. That's the difference between "looks similar in a benchmark table" and "handles a third more volume without a human in the loop."

So the model was worth having. But "worth having" and "shippable as a static page" are different questions, and the second one is what this post is about.

Step one: ONNX export, and the first surprise

The conversion itself is a few lines with 🤗 Optimum:

python
from optimum.onnxruntime import ORTModelForSequenceClassification

ort_model = ORTModelForSequenceClassification.from_pretrained(SOURCE, export=True)
ort_model.save_pretrained(OUT)

I expected this step to be a no-op on performance — same weights, same math, just a different graph format. It wasn't. On identical inputs, identical predictions, bit-identical accuracy, the ONNX Runtime graph ran 1.58× faster than eager PyTorch (5.66 ms vs 8.95 ms p50, single-query, CPU, tokenization included). Nothing about the model changed. The runtime just executes the same computation graph better — operator fusion, more efficient memory layout, no Python-level overhead per op.

That's worth separating out explicitly, because it's easy to bundle "ONNX" and "quantization" into one mental step called "made it fast," and they're not the same lever. Export bought speed for free. Quantization is a different trade, and it did not behave the way I expected.

Step two: quantization bought size, not speed
python
from onnxruntime.quantization import QuantType, quantize_dynamic

quantize_dynamic(
model_input=str(fp32_path),
model_output=str(int8_path),
weight_type=QuantType.QUInt8,
)

One line, and the model dropped from 256 MB to 64 MB — a clean 4×. That was the part I was counting on. What I wasn't expecting: on the same machine, the int8 model was 8% slower than ONNX fp32 (6.14 ms vs 5.66 ms p50).

variant accuracy macro-F1 p50 latency size
PyTorch fp32 91.88% 91.94% 8.95 ms 256.3 MB
ONNX fp32 91.88% 91.94% 5.66 ms 255.7 MB
ONNX int8 91.75% 91.70% 6.14 ms 64.3 MB

(all numbers from a seeded 800-row sample of the held-out test set, latency measured single-query on CPU including tokenization — batching would have flattered the larger model and defeated the point of the comparison)

The reason, as far as I can tell, is architecture-specific: this ran on Apple Silicon, which already handles fp32 matmuls efficiently, so the quantize/dequantize overhead around each int8 operation ate most of the arithmetic savings. On x86 with AVX-512 VNNI — instructions built specifically for int8 dot products — I'd expect int8 to win on latency too, not just size. I haven't measured that; it's a gap in this benchmark, not a claim. The takeaway I'd actually stand behind: don't assume quantization is a speed optimization on your target hardware. Measure it there. It's reliably a size optimization. Speed is conditional.

In this case the size win was the only one that mattered anyway — a 64 MB download is something a user will wait for once; 256 MB is not. That's what made "ship it as a static page with no server" a real option instead of a nice idea.

The accuracy cost was real but tiny — and easy to measure wrong

Quantization cost 0.12 accuracy points (91.88% → 91.75%). On an 800-row sample, that's one flipped prediction. I'm treating it as noise, not a regression — but getting even that number required catching a mistake I almost shipped.

Banking77's test split is grouped by class — the raw ordering isn't shuffled. Slicing test[:800] for a faster eval, which I did on the first pass, silently covers only a fraction of the 77 intents. Accuracy on that slice looks fine; macro-F1 becomes meaningless, because it's averaging over classes that were never in the sample. The fix is one line — random.Random(0).shuffle(sample) before slicing, seeded so it stays reproducible — but it's the kind of bug that doesn't error, doesn't look wrong, and just quietly reports a number that isn't measuring what you think it's measuring.

The gotcha that had nothing to do with quantization

Partway through wiring up the ONNX inference session, predictions started failing with an error about an unexpected input. Not a quantization problem — a forward() signature mismatch. DistilBERT has no segment embeddings, so its forward() rejects token_type_ids. Some saved tokenizer configs emit that key anyway when you call the tokenizer normally. PyTorch's from_pretrained wrapper silently tolerates the extra key; a raw InferenceSession.run() does not.

python
encoded = tokenizer(text, truncation=True, padding="max_length",
max_length=MAX_SEQ_LENGTH, return_tensors="np")
feed = {k: v.astype(np.int64) for k, v in encoded.items() if k in input_names}

Filtering the feed dict down to input_names the session actually declares fixed it. The broader lesson: when you move from a high-level pipeline() call to a raw ONNX Runtime session, you lose every convenience wrapper that was quietly papering over small mismatches. Budget time for this category of bug — it shows up right when you're mid-conversion, so it's tempting to blame the quantization step, when the culprit is usually the interface between two libraries that used to agree by accident.

QUInt8, not QInt8

onnxruntime.quantization supports both signed (QInt8) and unsigned (QUInt8) int8 weights. This project uses QUInt8, and the reason is a one-line comment I left myself in the export script: it's what ONNX Runtime Web — the WASM runtime that actually executes the model client-side — expects. Get this wrong and the failure doesn't show up in your Python eval loop, where the model quantized with either type will happily load and score correctly. It shows up later, in the browser, which is a much more annoying place to debug a quantization flag.

Packaging for Transformers.js is opinionated

The output layout that made the browser demo work isn't the default save_pretrained() layout:

artifacts/onnx/
config.json tokenizer.json tokenizer_config.json ...
onnx/model.onnx
onnx/model_quantized.onnx

Transformers.js expects the graph files nested under an onnx/ subfolder alongside the tokenizer config, not sitting at the top level next to it. Miss this and pipeline() fails to find the model with an error that reads like a missing-file problem, not a wrong-directory-structure problem. This is the kind of detail that's obvious once you've hit it once and invisible until then — which describes most of this list, honestly.

Python latency and browser latency are different numbers

Native ONNX Runtime, int8, Apple Silicon CPU: ~6 ms p50. The same model, same weights, running via WebAssembly in Chrome: ~58 ms — roughly 10× slower. That gap is expected, not a regression: WASM runs sandboxed, without the native SIMD and threading parity a host runtime gets, and there's tokenization and a JS↔WASM boundary crossing on top. 58 ms is still completely invisible to someone who just clicked a button — but if I'd quoted the 6 ms Python number as "the latency" without separately measuring in-browser, I'd have been off by an order of magnitude on the number that actually matters for this deployment.

One real limitation worth stating plainly: this only works in browsers with a working WASM-based ONNX runtime. Chrome, yes. Safari currently fails to load it — that's a WebAssembly/runtime limitation on Safari's end, not something wrong with the model.

What actually mattered, in order

If I had to rank what this exercise bought, size dominates:

The size cut is what changed the deployment model. 256 MB isn't something you casually ship to a browser tab. 64 MB is. That's the difference between "needs a server" and "static page, zero hosting cost" — a bigger consequence than any latency number in this whole writeup.
The accuracy cost was real but negligible (−0.12 points) — worth measuring properly rather than assuming, but not worth losing sleep over once measured correctly.
The speed win came from the ONNX export, not the quantization — and conflating those two steps would have led to the wrong conclusion about what quantization is actually for on this hardware.
Every other failure was plumbing, not modeling — signature mismatches, a quantization dtype the browser runtime cares about, a directory layout convention, an unshuffled eval sample. None of it touched the model's actual behavior; all of it was capable of quietly breaking the pipeline or the numbers if I hadn't been looking for it.

None of these are exotic. They're the ordinary friction of taking a model from "trained and evaluated in a notebook" to "running somewhere a browser can load it," and I hadn't hit most of them before this project. Writing them down mainly because the failure mode in each case was silent — a number that's quietly wrong, or a model that quietly won't load — rather than a loud error pointing at the actual cause.