On the N100, OpenVINO was not faster. And I almost measured it wrong

If you work with models on Intel hardware, you know the usual advice: for CPU inference, use OpenVINO. It is Intel's runtime, built for Intel's hardware, and on a mini PC with an N100 it would seem the obvious pick over ONNX Runtime.

I measured it. On the N100, across the three models I compared, OpenVINO was between 11% and 22% slower than ONNX Runtime.

But the result is not the interesting part. The interesting part is that before I got there, I almost measured something else, and the wrong measurement would have passed every check.

What happened in the rehearsal

Before touching the N100, I rehearsed the protocol on my Mac, which has an ARM CPU. MobileNetV2 in FP32, OpenVINO with its default settings.

It came out fast, and it came out right: the outputs matched the reference. The same top class on all twelve test images, and a cosine similarity of 0.999 or more against the expected result.

Then I looked at something almost nobody looks at: the precision OpenVINO had decided to run the model in.

An FP32 model, and OpenVINO running it in FP16. Without being asked, and without saying so.

INFERENCE_PRECISION_HINT = float16

It is a design decision in OpenVINO, not a bug: on that processor, FP16 runs faster. But it is not the one I asked for, and it shows up nowhere unless you go looking for it. In FP16, the model's p50 was 1.46 ms. Forcing f32, the same model on the same machine took 2.39 ms.

Had I kept the first number, I would have measured a precision change and credited it to the runtime.

Why validation does not catch it

This is what I think matters most here. The usual check—does it give the same answers?—passed. And it had to: for image classification, FP16 is enough. The output does not move far enough for an accuracy test to notice.

So same results, and faster does not prove the comparison is fair. It proves the precision change was small enough not to show on that model. On another one—deeper, a regression model, one that is already quantized—it may well show. And it may show in production.

How I measured it on the N100

After the rehearsal, the protocol changed. In every FP32 case, precision is pinned to f32 when the model is compiled, and after compiling, the effective precision is read back. If it is not f32, the measurement does not count: it is marked invalid, even if everything else passed.

compiled = core.compile_model(model, "CPU", {"INFERENCE_PRECISION_HINT": "f32"})
compiled.get_property("INFERENCE_PRECISION_HINT")   # checked, not assumed

With that, on an 8 GB Intel N100 running Ubuntu 24.04, freshly rebooted, one hundred measurements after twenty warm-up runs, batch 1 and default threading:

Model	ONNX Runtime 1.30.0	OpenVINO 2026.4.0	OpenVINO vs ORT
MobileNetV2 FP32	5.7 ms	7.0 ms	22% slower
ResNet18 FP32	22.8 ms	26.9 ms	18% slower
YOLO26n FP32	57.6 ms	64.0 ms	11% slower

These are the p50 of model inference. OpenVINO's effective precision was f32 in all three cases, and in all three the outputs matched the reference. In detection, YOLO26n found the same objects as the reference, not one more, not one fewer.

I had not predicted this. Before measuring, I wrote down what I expected from each case, and for this one I left the answer as unknown. The measurement settled it in a direction I did not expect.

It is not noise

The p50 is a single number. This is every one of the hundred inferences of each case:

The clouds do not touch. Comparing every ONNX Runtime inference against every OpenVINO one—ten thousand pairs per model—ONNX Runtime was faster in 100 % of the pairs for MobileNetV2 and ResNet18, and in 99 % for YOLO26n. In the first two, the slowest ONNX Runtime inference was faster than the fastest OpenVINO one. The difference is not measurement noise: it is systematic.

The raw data—every inference, the temperature during the run and the exact versions of each environment—is at crtdrops.mx/datos/round1, so you can check it or compare it with yours.

What this result does not say

This is a measurement, not a verdict on OpenVINO. It holds for what was measured, and nothing more:

• One device. One N100. On a processor with native FP16, or with the matrix instructions of Xeon chips, the story may be different.
• CPU only. The N100 has an integrated GPU, and OpenVINO knows how to use it. I did not measure it.
• Batch 1, latency. OpenVINO has a mode built for processing many inputs at once, with several parallel streams. I did not measure that either.
• Three models, in f32. With reduced precision, or with OpenVINO's own quantization, the numbers change.

What it does say is more modest and, I think, more useful: on this device and with this setup, the advantage you would expect did not show up.

The questions I left open, measured

On 7 October I went back to the N100, freshly rebooted, with three of those questions: whether it also lowered the precision on its own there, whether its many-inputs-at-once mode closed the gap, and what the integrated GPU did.

On the N100's CPU, the precision does not drop. Compiled without asking for anything, all three models stayed in f32. That is not OpenVINO's doing: that processor has no FP16, so there is nowhere to drop to. On the Mac it did drop because its processor does have it. The guard is still needed as soon as you change hardware.

Many inputs at once does not close the gap. In its mode for processing several requests in parallel, OpenVINO opens two streams on the N100 and gains between 4 and 9 %. Comparing the best of each—that mode against ONNX Runtime with four processes—ONNX Runtime stays ahead by 14 to 20 %.

On the integrated GPU, OpenVINO does win. In FP16. The N100's GPU does run FP16, and OpenVINO uses it in that precision without anyone asking: the same trap as the rehearsal, now on the N100. That way, YOLO26n goes from 17.8 inferences per second with ONNX Runtime to 24.7, 39 % more; ResNet18, from 43.4 to 55.9. With precision pinned to f32, the GPU gains nothing: 16.3 and 37.2. And MobileNetV2 is slower on the GPU than on the CPU, in any precision.

And is it the same? I checked it the way Round 1 does: the twelve real images, against the reference output, with the same tolerances. Both classifiers give the same class on all twelve. YOLO26n passes validation—it asks for 95 % agreement and got 96.6 %—but it is not the same: it found the same objects with lower confidence. In one image, a person went from 0.829 to 0.696; in another, a pair of skis from 0.468 to 0.376. If your system only accepts detections above 0.7, that person disappears on the GPU and not on the CPU.

It is this note's lesson again, on different hardware: passing validation does not prove nothing changed. The GPU's advantage is real, and it comes with a precision change you have to decide on knowingly, not discover later.

And the numbers repeat. The CPU measurements, taken on three different days, changed by less than 3 % from one time to the next. The data for this part is at crtdrops.mx/datos.

What I would do differently when comparing runtimes

If you are choosing a runtime for an edge device, four things I would have done from the start:

Pin the precision, and read back what you got. Not what you asked for: what the runtime chose after compiling. It is two lines of code, and it is the difference between comparing runtimes and comparing precisions.

Do not use accuracy as proof that the comparison is fair. Matching outputs say the change did not show on that model. They do not say there was no change.

Write down what you expect, first. Even if the answer is I don't know. If you do not write it down, any result seems to confirm what you already thought.

Measure on the device, not on your machine. Freshly rebooted, warmed up, and with p50 and p95, not an average. The Mac helped me find the problem; only the N100 gave the answer.

The guard, in code

This is what I use now so it does not slip past me again. It compiles the model asking for a precision, reads back the one it actually got and, if they differ, refuses:

import openvino as ov


class PrecisionMismatch(RuntimeError):
    """The compiled model runs in a precision other than the one requested."""


def effective_precision(compiled: ov.CompiledModel) -> str:
    return compiled.get_property("INFERENCE_PRECISION_HINT").get_type_name()


def compile_pinned(model, device="CPU", precision="f32", core=None, **config):
    core = core or ov.Core()
    compiled = core.compile_model(model, device, {"INFERENCE_PRECISION_HINT": precision, **config})
    got = effective_precision(compiled)
    if got != precision:
        raise PrecisionMismatch(f"asked for {precision} on {device}, the compiled model uses {got}")
    return compiled

With OpenVINO 2026.4.0 on a Mac with an Apple M5 Pro, compiling MobileNetV2 without asking for anything gives f16; with compile_pinned it gives f32. And if one day it does not, the measurement stops instead of carrying on in another precision. It works the same for the integrated GPU (device="GPU"), where OpenVINO uses FP16 by default.

Your turn

Have you caught a runtime changing precision behind your back? Which runtime, which hardware? And if you run these models on your own N100 or Pi 5, the raw data is at crtdrops.mx/datos: I would like to compare.

---

The full table—five models on a Raspberry Pi 5 and on the N100, with ONNX Runtime and OpenVINO—is on crtdrops.mx, next to an ONNX model check that runs in your browser without uploading the file anywhere. If you open one of the models I measured, the tool recognizes it and shows you its numbers.

Having your own model measured this way, on both devices and with the same protocol, is now possible: it is in the same section, for 149 USD, with a sample report so you can see what you get.

Try it: the full results table and the in-browser ONNX model check on crtdrops.mx.

Updated on 5 October 2026 with the full distribution, the raw data and the code for the guard, and on 7 October with the open questions, measured on the N100.

Israel Negrete Lepe is an electronics engineer. Twenty-five years making different systems work together under real conditions: from the sensor, the vehicle or the transaction to the backend, the data, security and operations. Today, increasingly, edge AI. I take on projects by the hour or by deliverable: [e-mail].

I send a short email when a new note or lab result comes out: subscribe here.