DEV Community

Cover image for Deploying a vision model to the edge: quantization and ONNX in practice
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

Deploying a vision model to the edge: quantization and ONNX in practice

#vision #onnx #edgedeployment

Running vision inference on-device — a phone, a browser, a small board — avoids the latency and privacy cost of a cloud round trip, but the model has to actually fit and run fast enough on far weaker hardware.

The two levers

  • Quantization: converting model weights from 32-bit to 8-bit (or lower) shrinks size and speeds up inference, usually with a small, measurable accuracy cost.
  • ONNX: an intermediate model format that runs across many devices and runtimes without depending on the original training framework, which is what actually makes 'runs on this specific phone' practical.

A realistic workflow

Train and validate normally, export to ONNX, quantize the ONNX model, then re-run your evaluation set on the quantized version before shipping — the accuracy drop from quantization is usually acceptable but should be measured, not assumed.

See ONNX Runtime Web: running ML in the browser for the specific case of shipping this to a browser rather than a native app.

About Pranjul Rathour

Pranjul Rathour pitching with a microphone in front of a projector slide at CSJMU, Kanpur
Pitching at CSJMU, Kanpur

Pranjul Rathour speaking from the podium at VSICS, Kanpur
At the VSICS podium, Kanpur

Pranjul Rathour holding a trophy and a certificate of merit after a win
Trophy and certificate after a win

Pranjul Rathour presenting evaluation criteria — feasibility, innovation, practicality, problem solving — on a projector screen
Walking a room through evaluation criteria

Pranjul Rathour presenting BrandHive on a projector screen
Presenting BrandHive

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-07.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)