TL;DR— Small language models get treated as portable artifacts— a single GGUF or ONNX file that behaves identically everywhere. It doesn't. Quantization kernels, compiler graph fusion, and accelerator-specific math mean the same weights produce different outputs on different devices. On-device AI's real bottleneck isn't model size— it's that evaluation has to happen per hardware target, and almost nobody is doing that.
Everyone treats a small language model as a file. You quantize it, you export it to GGUF or ONNX or a vendor-specific format, and you ship it. The assumption baked into that workflow is that the file is the model— that the same weights produce the same behavior wherever they land. On a server cluster running one accelerator type, that assumption mostly holds. On a device fleet spanning phones, laptops, wearables, and embedded boards, it's false, and the gap between those two worlds is the real unsolved problem in on-device AI.
The model is not a file. It's a stack: weights, a quantization scheme, a compiled execution graph, and an accelerator's specific arithmetic. Change any layer of that stack and you change the outputs— sometimes subtly, sometimes in ways that break a downstream agent loop or silently degrade a classifier's confidence. Nobody evaluates at that granularity. Almost everyone evaluates the checkpoint once, in one environment, and assumes portability that doesn't exist.
Quantization is not one thing
"We shipped an INT4 model" sounds like a single decision. It isn't. INT4 on one NPU uses per-channel scaling; on another it uses per-tensor scaling with a different calibration set. Some backends use symmetric quantization, others asymmetric with a zero-point offset. Rounding modes differ. Outlier handling differs— some runtimes clip, others use mixed precision for a handful of sensitive channels. Two "INT4" deployments of the identical checkpoint can diverge in perplexity by a meaningful margin, and that divergence is invisible unless you actually run both and compare.
This matters more for language models than for, say, image classifiers, because autoregressive generation compounds small numeric differences. A slightly different logit distribution at token five changes the sampled token, which changes the context for token six, which changes everything downstream. A model that's "basically the same" numerically can produce a noticeably different completion, tool call, or refusal decision depending on which chip generated it.
The compiler is a silent co-author
Beneath quantization sits the compiler that turns your graph into device-specific kernels. Operator fusion, memory layout decisions, and scheduling heuristics are not neutral— they change numerical precision at intermediate steps, especially in attention and normalization layers where operation order affects floating-point accumulation. Two compilers targeting the same hardware can produce measurably different outputs from the same ONNX graph, because they fuse operations differently and accumulate in different intermediate precisions.
This is the part teams underestimate. You didn't just pick a model and a quantization level. You picked a model, a quantization level, a compiler, and a compiler version, and each of those is a variable that can move independently of the others when a vendor ships an update. Your model didn't change. Its behavior did anyway.
Fragmentation is an evaluation problem, not a model problem
The instinct when facing this is to look for a magic format— one intermediate representation that guarantees identical behavior everywhere. That's not coming, for the same reason browser rendering never fully converged despite decades of standards work: the underlying hardware is genuinely different, vendors compete partly on their own kernel implementations, and "identical" isn't actually what accelerator vendors are optimizing for. They're optimizing for throughput and power on their own silicon, and numerical bit-for-bit parity with a competitor's chip is not on their roadmap.
So the fragmentation is permanent. The only thing that changes is whether you treat it as an evaluation problem. Web engineering solved its version of this with browser compatibility matrices and cross-browser CI— not by demanding browsers converge, but by testing against the real, divergent set of targets you actually ship to. On-device AI needs the same discipline and mostly doesn't have it. Most eval pipelines run once, on one reference build, and call it done.
What a real device-tier eval matrix looks like
Practically, this means building an evaluation matrix keyed by hardware tier, not just by model version. For each accelerator class you actually ship to, you need a held-out eval set run through the exact production stack— same quantization, same compiler, same runtime— not a proxy run on a dev machine with a generic backend. You're not evaluating the checkpoint. You're evaluating the deployed artifact on that specific tier.
You also need drift detection across compiler and runtime updates, the same way you'd gate a dependency bump in any other production system. A vendor SDK update that changes kernel fusion should trigger a re-run of your eval suite on affected devices before rollout, not after a support ticket tells you accuracy dropped on one phone model.
Telemetry from the fleet matters more here than in server deployments, precisely because you can't reproduce every device locally. Sampling real outputs from a representative slice of the device population— with consent and privacy constraints respected— is the only way to catch divergence you didn't anticipate in the lab. Server-side, one bad pod gets drained. Edge-side, a bad output pattern on one chip family might sit undetected for a release cycle.
Tiered fallback beats uniform expectations
The other implication is architectural: stop expecting uniform quality across your device fleet and design for tiers instead. A flagship phone's NPU and a five-year-old mid-range chip are not going to deliver the same effective model, even with identical weights, because the achievable quantization level and kernel efficiency differ. Build explicit tiers— full precision where the hardware supports it, more aggressive quantization with a validated accuracy floor where it doesn't, and a defined fallback (a smaller model, a cloud call, or a narrower feature set) below that floor. Pretending every device gets the same model is how you end up debugging a "random" quality complaint that's actually a specific chip generation running a specific compiler build that nobody put in the eval matrix.
Small models made on-device AI possible. They didn't make it uniform. The size of the weights was never the hard part— the hard part is that the same weights are not the same model once they leave your test environment, and evaluation has to catch up to that reality before on-device deployment can be trusted the way server deployment is.
Top comments (0)