DEV Community

vmodal_ai
vmodal_ai

Posted on

ONNX Runtime Android SDK Tutorial: Run AI Models Offline in Kotlin

Artificial Intelligence is no longer limited to cloud servers. With ONNX Runtime Android SDK, you can run machine learning models directly on Android devices, enabling offline AI, faster inference, improved privacy, and lower latency.

Whether you're building an OCR app, object detection system, image classifier, or AI-powered chatbot, ONNX Runtime provides a high-performance inference engine for Android.

In this tutorial, you'll learn how to integrate the ONNX Runtime Android SDK into your Kotlin application and perform on-device AI inference.

Why Use ONNX Runtime on Android?

ONNX Runtime is an open-source inference engine optimized for multiple hardware platforms. It allows developers to deploy trained AI models without depending on cloud APIs.

Key Benefits

  • 🚀 Fast on-device inference
  • 📱 Works completely offline
  • 🔒 Better user privacy
  • 🔋 Reduced network usage and latency
  • 🤖 Supports models from PyTorch, TensorFlow, and Scikit-learn (after conversion to ONNX)
  • ⚡ Hardware acceleration support on compatible devices

Common use cases include:

  • OCR (Optical Character Recognition)
  • Object Detection
  • Face Recognition
  • Image Classification
  • Speech Recognition
  • AI Chat Applications
  • Medical Image Analysis

Prerequisites

Before starting, make sure you have:

  • Android Studio
  • Kotlin
  • Android SDK
  • An ONNX model (.onnx)
  • Basic understanding of machine learning inference

Step 1: Add ONNX Runtime Dependency

Add the ONNX Runtime Android library to your build.gradle file.

dependencies {
    implementation("com.microsoft.onnxruntime:onnxruntime-android:1.22.0")
}
Enter fullscreen mode Exit fullscreen mode

Sync your Gradle project.

Step 2: Add Your ONNX Model

Create an assets folder if it doesn't already exist.

app/
 └── src/
      └── main/
           └── assets/
                └── model.onnx
Enter fullscreen mode Exit fullscreen mode

Place your trained ONNX model inside the assets directory.

Step 3: Load the Model

Read the model from the assets folder.

val environment = OrtEnvironment.getEnvironment()

val modelBytes = assets.open("model.onnx").readBytes()

val session = environment.createSession(modelBytes)
Enter fullscreen mode Exit fullscreen mode

Creating the session loads the model into memory and prepares it for inference.

Step 4: Prepare Input Data

Most models expect tensors as input.

Example:

val inputArray = FloatArray(224 * 224 * 3)

val inputTensor = OnnxTensor.createTensor(
    environment,
    FloatBuffer.wrap(inputArray),
    longArrayOf(1, 3, 224, 224)
)
Enter fullscreen mode Exit fullscreen mode

Ensure the tensor shape matches the model's expected input dimensions.

Step 5: Run Inference

Execute the model.

val outputs = session.run(
    mapOf("input" to inputTensor)
)
Enter fullscreen mode Exit fullscreen mode


The inference returns predictions generated by the model.

Step 6: Read the Output

Extract the output tensor.

val result = outputs[0].value

println(result)
Enter fullscreen mode Exit fullscreen mode

Depending on the model, the output may represent:

  • Classification scores
  • Bounding boxes
  • OCR text
  • Facial embeddings
  • Segmentation masks

Image Preprocessing

Most computer vision models require preprocessing before inference.

Typical steps include:

  • Resize the image
  • Normalize pixel values
  • Convert RGB channels
  • Convert the bitmap into a FloatArray
  • Match the model's expected tensor shape

Incorrect preprocessing is one of the most common causes of inaccurate predictions.

Performance Tips

For faster inference:

  • Reuse the OrtSession instead of creating it repeatedly.
  • Load the model once during application startup.
  • Perform inference on a background thread using Kotlin Coroutines.
  • Release tensors and sessions when they are no longer needed.
  • Choose a lightweight model for mobile devices.

These optimizations improve responsiveness and reduce memory usage.

Common Errors

Shape Mismatch 

Invalid input shape
Enter fullscreen mode Exit fullscreen mode

Verify the tensor dimensions expected by your model.

Wrong Input Name 

Input not found
Enter fullscreen mode Exit fullscreen mode

Ensure the input name in your code matches the model's input node.

Model Loading Failure 

Failed to load model
Enter fullscreen mode Exit fullscreen mode

Confirm the .onnx file exists in the assets folder and is included in the APK.

Slow Inference

Large models can increase inference time. Consider model optimization or quantization if performance is insufficient.

Real-World Applications

ONNX Runtime powers many intelligent Android applications:

  • 📷 OCR document scanners
  • 🚗 License plate recognition
  • 🏥 Medical imaging solutions
  • 🛒 Visual product search
  • 🎯 Object detection
  • 😊 Face recognition
  • 🌱 Plant disease detection
  • 📦 Barcode and QR code analysis
  • 🗣️ Speech and audio processing

Best Practices

  • Keep AI inference off the main thread.
  • Validate input tensor shapes before running the model.
  • Optimize models for mobile deployment.
  • Cache the inference session for reuse.
  • Protect sensitive models if distributing them with your app.
  • Test on multiple Android devices to evaluate performance.

Conclusion

The ONNX Runtime Android SDK makes it easy to bring powerful AI capabilities directly to Android devices. By running models on-device, you gain faster inference, offline functionality, and improved user privacy without relying on cloud services.

Whether you're developing OCR applications, image classifiers, object detectors, or AI-powered healthcare solutions, ONNX Runtime provides a flexible and efficient foundation for deploying machine learning models in production Android apps.

As mobile AI continues to grow, learning ONNX Runtime is a valuable skill for Android developers building intelligent, privacy-focused applications.

Frequently Asked Questions

What is ONNX Runtime?

ONNX Runtime is an open-source inference engine that runs machine learning models in the ONNX format across multiple platforms, including Android.

Can I use TensorFlow or PyTorch models?

Yes. Models created with TensorFlow or PyTorch can be converted to the ONNX format before deployment.

Does ONNX Runtime require an internet connection?

No. Models run locally on the device, making it ideal for offline applications.

Is ONNX Runtime suitable for production apps?

Yes. It is widely used in enterprise and commercial applications for mobile, desktop, cloud, and edge AI deployments.


SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Tags

android kotlin ai machinelearning onnx mobiledevelopment

Top comments (0)