vmodal_ai

Posted on Aug 11

ONNX Runtime with Flutter: Run AI Models On-Device

Running machine-learning models directly on a mobile device can reduce latency, improve privacy, and allow an application to work without a permanent internet connection.

ONNX Runtime is a cross-platform inference engine that can execute ONNX models.

Flutter
  |
  v
ONNX Runtime
  |
  v
ONNX Model
  |
  v
Prediction

Why on-device inference?

Cloud inference:

Phone -> Internet -> Server -> Model -> Internet -> Phone

On-device inference:

Phone -> Model -> Prediction

Advantages include:

lower latency
offline operation
improved data privacy
reduced server costs
predictable response times

Step 1: Prepare an ONNX model

Export your trained model to ONNX using the exporter appropriate for your framework.

After export, inspect:

input names
input shapes
data types
output names
output shapes

These details are critical.

Step 2: Add the ONNX model to Flutter assets

flutter:
  assets:
    - assets/models/classifier.onnx

Then:

flutter pub get

Step 3: Load the model

The exact Dart API depends on the ONNX Runtime Flutter package you choose. The conceptual flow is:

final modelBytes = await rootBundle.load(
  'assets/models/classifier.onnx',
);

// Create the inference session using the package API.
final session = await createInferenceSession(
  modelBytes.buffer.asUint8List(),
);

Keep model initialization outside the widget build() method.

Step 4: Prepare input tensors

Suppose the model expects:

float32[1, 224, 224, 3]

Your Flutter preprocessing must produce exactly that shape.

For image models:

Image
  ↓
Resize
  ↓
RGB conversion
  ↓
Normalization
  ↓
Float32 tensor
  ↓
ONNX Runtime

For example:

normalized = (pixel / 255.0 - mean) / std

The exact preprocessing must match the training pipeline.

Step 5: Run inference

Conceptually:

final outputs = await session.run({
  'input': inputTensor,
});

final prediction = outputs['output'];

The exact method names depend on the Flutter ONNX Runtime binding you use.

Important: input shape matters

A common error is a dimension mismatch.

For example, the model expects:

[1, 224, 224, 3]

but Flutter provides:

[224, 224, 3]

These are different tensors.

Always inspect the ONNX model before implementing preprocessing.

Inspect the model in Python

Install ONNX:

pip install onnx

Then:

import onnx

model = onnx.load("classifier.onnx")

for input_tensor in model.graph.input:
    print("Input:", input_tensor.name)
    print(input_tensor.type)

for output_tensor in model.graph.output:
    print("Output:", output_tensor.name)
    print(output_tensor.type)

Netron is also useful for visually inspecting ONNX graphs.

Performance optimization

Avoid recreating the inference session for every prediction.

Bad:

Prediction
  -> Load model
  -> Create session
  -> Run

Better:

Application start
  -> Load model once
  -> Create session once
  -> Reuse session

Also consider:

reducing model size
quantization
smaller input resolution
platform-specific acceleration
minimizing unnecessary tensor copies

Threading and UI responsiveness

Inference should not make the Flutter UI unresponsive.

A practical architecture is:

UI
 |
BLoC
 |
Inference Repository
 |
ONNX Runtime
 |
Native acceleration

Measure inference time instead of assuming the model is fast.

Model versioning

Treat models like application dependencies:

assets/models/
  classifier_v1.onnx
  classifier_v2.onnx

Record:

model version
training dataset version
preprocessing version
expected input shape
output labels

Conclusion

ONNX Runtime makes it possible to move AI inference from the server to the device. The important part is ensuring that preprocessing, tensor shapes, model outputs, and runtime configuration exactly match the training pipeline.