DEV Community

Tushar ahmed
Tushar ahmed

Posted on

Building an Offline Food Scanner on Android with Google AI Edge LiteRT and CameraX

Running machine learning models completely offline on Android gives you zero latency, zero API costs, and full user privacy. Google's LiteRT (com.google.ai.edge.litert, the evolution of TensorFlow Lite) makes it straightforward to run .tflite models directly on-device.

In this tutorial, we'll walk through how NutriLens — a Jetpack Compose + CameraX food & nutrition tracker — uses LiteRT to identify dishes from a live camera feed and estimate calories, macros, and micronutrients 100% offline.


What We Are Building

The NutriLens scanning pipeline works like this:

CameraX ImageAnalysis (RGBA_8888)
        │
        ▼
Rotate upright & throttle (~850ms)
        │
        ▼
Center-crop & scale Bitmap (192×192)
        │
        ▼
LiteRT Inference (CompiledModel / Interpreter)
        │
        ▼
Dequantize UINT8 Softmax → Top-3 Food Labels
        │
        ▼
Temporal Smoothing + Nutrition Estimator → Compose UI
Enter fullscreen mode Exit fullscreen mode

Step 1: Add LiteRT and CameraX Dependencies

In gradle/libs.versions.toml:

[versions]
camerax = "1.4.2"
litert = "2.1.0"

[libraries]
androidx-camera-core = { group = "androidx.camera", name = "camera-core", version.ref = "camerax" }
androidx-camera-camera2 = { group = "androidx.camera", name = "camera-camera2", version.ref = "camerax" }
androidx-camera-lifecycle = { group = "androidx.camera", name = "camera-lifecycle", version.ref = "camerax" }
androidx-camera-view = { group = "androidx.camera", name = "camera-view", version.ref = "camerax" }
google-ai-edge-litert = { group = "com.google.ai.edge.litert", name = "litert", version.ref = "litert" }
Enter fullscreen mode Exit fullscreen mode

In app/build.gradle.kts, add the dependencies and disable compression for .tflite assets so Android can memory-map (mmap) the model file directly without decompressing it into RAM:

android {
    // ...
    androidResources {
        noCompress += "tflite"
    }
}

dependencies {
    implementation(libs.androidx.camera.core)
    implementation(libs.androidx.camera.camera2)
    implementation(libs.androidx.camera.lifecycle)
    implementation(libs.androidx.camera.view)
    implementation(libs.google.ai.edge.litert)
}
Enter fullscreen mode Exit fullscreen mode

Tip: Never skip noCompress += "tflite". If Gradle compresses your .tflite asset inside the APK, assets.openFd(...) will fail and prevent zero-copy memory mapping.


Step 2: Bundle Your Offline .tflite Model and Labels

Place your model and label files inside app/src/main/assets/:

app/src/main/assets/
├── vision_classifier_food.tflite   # aiy/vision/classifier/food_V1 (2,024 classes)
└── food_labels.txt                 # 2,024 class labels matching output indices
Enter fullscreen mode Exit fullscreen mode

Before writing Kotlin code, inspect your model's input and output tensor specs:

  • Input: [1, 192, 192, 3] (UINT8 RGB pixels 0..255)
  • Output: [1, 2024] (UINT8 quantized softmax probabilities)

Knowing whether your model uses UINT8 or FLOAT32 is critical — feeding FLOAT32 normalized data to a UINT8 model (or forgetting to dequantize UINT8 outputs) is one of the most common reasons an offline model outputs garbage.


Step 3: Initialize LiteRT (CompiledModel + Interpreter Fallback)

LiteRT 2.1.0 provides the modern CompiledModel + TensorBuffer API (com.google.ai.edge.litert.*) as well as a compatible Interpreter (org.tensorflow.lite.Interpreter bundled inside the litert artifact).

Older .tflite models (like aiy/vision/classifier/food_V1) sometimes lack SignatureDefs or use UINT8 quantization parameters (scale and zeroPoint). Initializing CompiledModel first and keeping Interpreter ready for tensor metadata & fallback gives you the best of both worlds:

class LiteRtFoodClassifier(
    private val context: Context
) {
    private var compiledModel: CompiledModel? = null
    private var inputBuffers: List<TensorBuffer>? = null
    private var outputBuffers: List<TensorBuffer>? = null

    private var interpreter: Interpreter? = null
    private var inputWidth = 192
    private var inputHeight = 192
    private var inputDataType = DataType.UINT8
    private var outputClassCount = 2024
    private var outputDataType = DataType.UINT8
    private var outputScale = 1f / 255f
    private var outputZeroPoint = 0
    private var labels: List<String> = emptyList()

    @Synchronized
    fun ensureInitialized(): Boolean {
        if ((compiledModel != null || interpreter != null) && labels.isNotEmpty()) {
            return true
        }

        labels = loadLabelsFromAssets()

        // 1. Modern LiteRT CompiledModel API
        val compiledReady = runCatching {
            val model = CompiledModel.create(
                context.assets,
                "vision_classifier_food.tflite",
                CompiledModel.Options(Accelerator.CPU)
            )
            val inputs = model.createInputBuffers()
            val outputs = model.createOutputBuffers()
            if (inputs.isNotEmpty() && outputs.isNotEmpty()) {
                compiledModel = model
                inputBuffers = inputs
                outputBuffers = outputs
                true
            } else {
                model.close()
                false
            }
        }.getOrDefault(false)

        // 2. Inspect tensor shapes, DataTypes, and quantization params via Interpreter
        val interpreterReady = runCatching {
            val modelBuffer = loadMappedModelBuffer("vision_classifier_food.tflite")
            val tflite = Interpreter(modelBuffer, Interpreter.Options().apply { setNumThreads(4) })

            val inTensor = tflite.getInputTensor(0)
            val inShape = inTensor.shape() // [1, 192, 192, 3]
            if (inShape.size >= 3) {
                inputHeight = inShape[1]
                inputWidth = inShape[2]
            }
            inputDataType = inTensor.dataType()

            val outTensor = tflite.getOutputTensor(0)
            outputClassCount = outTensor.shape().lastOrNull() ?: 2024
            outputDataType = outTensor.dataType()

            val qParams = outTensor.quantizationParams()
            if (qParams.scale > 0f) {
                outputScale = qParams.scale
                outputZeroPoint = qParams.zeroPoint
            }

            interpreter = tflite
            true
        }.getOrDefault(false)

        return compiledReady || interpreterReady
    }
}
Enter fullscreen mode Exit fullscreen mode

Memory-mapping the model from assets:

private fun loadMappedModelBuffer(assetName: String): ByteBuffer {
    return runCatching {
        context.assets.openFd(assetName).use { fd ->
            FileInputStream(fd.fileDescriptor).channel.use { channel ->
                channel.map(
                    FileChannel.MapMode.READ_ONLY,
                    fd.startOffset,
                    fd.declaredLength
                )
            }
        }
    }.getOrElse {
        val bytes = context.assets.open(assetName).use { it.readBytes() }
        ByteBuffer.allocateDirect(bytes.size).apply {
            order(ByteOrder.nativeOrder())
            put(bytes)
            rewind()
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Step 4: Preprocess Bitmaps & Run Inference

Before passing a camera Bitmap into LiteRT:

  1. Convert Bitmap.Config.HARDWARE to ARGB_8888 if needed (since getPixels cannot read hardware bitmaps).
  2. Center-crop to a square and scale to 192×192.
  3. Extract packed RGB bytes (R, G, B) in row-major order.
private fun centerCropAndScale(source: Bitmap, targetWidth: Int, targetHeight: Int): Bitmap {
    val safeBitmap = if (source.config == Bitmap.Config.HARDWARE) {
        source.copy(Bitmap.Config.ARGB_8888, false)
    } else {
        source
    }
    val minDim = min(safeBitmap.width, safeBitmap.height)
    val cropX = (safeBitmap.width - minDim) / 2
    val cropY = (safeBitmap.height - minDim) / 2
    val cropped = Bitmap.createBitmap(safeBitmap, cropX, cropY, minDim, minDim)
    return if (cropped.width == targetWidth && cropped.height == targetHeight) {
        cropped
    } else {
        Bitmap.createScaledBitmap(cropped, targetWidth, targetHeight, true)
    }
}
Enter fullscreen mode Exit fullscreen mode

When reading quantized UINT8 output bytes in Kotlin/Java, remember that Byte is signed (-128..127), so always mask with and 0xFF and apply the quantization formula:

val probability = ((rawByte.toInt() and 0xFF) - outputZeroPoint) * outputScale
Enter fullscreen mode Exit fullscreen mode

Running inference with CompiledModel + TensorBuffer:

val model = compiledModel
val inputs = inputBuffers
val outputs = outputBuffers
if (model != null && inputs != null && outputs != null) {
    val probs = runCatching {
        val rgbBytes = bitmapToRgbByteArray(croppedBitmap, inputWidth, inputHeight)
        if (inputDataType == DataType.FLOAT32) {
            val floatIn = FloatArray(rgbBytes.size) { i ->
                ((rgbBytes[i].toInt() and 0xFF) / 127.5f) - 1.0f
            }
            inputs[0].writeFloat(floatIn)
            model.run(inputs, outputs)
            outputs[0].readFloat()
        } else {
            inputs[0].writeInt8(rgbBytes)
            model.run(inputs, outputs)
            if (outputDataType == DataType.FLOAT32) {
                outputs[0].readFloat()
            } else {
                val rawOut = outputs[0].readInt8()
                FloatArray(rawOut.size) { i ->
                    (((rawOut[i].toInt() and 0xFF) - outputZeroPoint) * outputScale).coerceAtLeast(0f)
                }
            }
        }
    }.getOrNull()
}
Enter fullscreen mode Exit fullscreen mode

Always release TensorBuffer, CompiledModel, and Interpreter resources when your ViewModel clears (onCleared()).


Step 5: Stream Live Frames from CameraX

In CameraPreviewView.kt, configure ImageAnalysis with OUTPUT_IMAGE_FORMAT_RGBA_8888 and STRATEGY_KEEP_ONLY_LATEST so slow frames never queue up. Don't forget to rotate the Bitmap using imageProxy.imageInfo.rotationDegrees — otherwise your model receives sideways images and accuracy drops sharply:

val imageAnalysis = ImageAnalysis.Builder()
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
    .build()

imageAnalysis.setAnalyzer(analysisExecutor) { imageProxy ->
    try {
        val rawBitmap = imageProxy.toBitmap()
        val rotation = imageProxy.imageInfo.rotationDegrees
        val uprightBitmap = if (rotation != 0) {
            val matrix = Matrix().apply { postRotate(rotation.toFloat()) }
            Bitmap.createBitmap(
                rawBitmap, 0, 0, rawBitmap.width, rawBitmap.height, matrix, true
            )
        } else {
            rawBitmap
        }
        onFrameBitmap(uprightBitmap)
    } finally {
        imageProxy.close()
    }
}
Enter fullscreen mode Exit fullscreen mode

Step 6: Stabilize Live Detections in the ViewModel

Running inference on every camera frame can cause flickering or stale UI states if you aren't careful. Here are three practical rules we applied in ScanFoodViewModel:

  1. Throttle & guard concurrency: Skip incoming frames if less than 850ms has passed or if an AtomicBoolean (isClassifyingFrame) is already active. Run inference on Dispatchers.Default.
  2. Use honest confidence thresholds: A 2,024-class softmax spreads probability across many labels, so real top-1 confidence on a good match is often 28%–65%, not 90%+. Never artificially inflate confidence scores in code; instead calibrate your lock threshold (MIN_LIVE_CONFIDENCE_PERCENT = 28) to the model's real distribution.
  3. Smooth transitions instead of hard-locking:
    • First match (>= 28%) updates the UI right away.
    • Switching to a different food requires 2 consecutive frames (pendingCandidateStreak >= 2) so brief camera shake doesn't overwrite a good detection.
    • Pointing away from food (< 18% for 3 consecutive frames) clears the previous item so the user can seamlessly scan a new plate.

Step 7: Map Detected Labels to Offline Nutrition Data

Once LiteRT returns a food label (e.g., "Mango Sticky Rice", "Breakfast Burrito"), FoodNutritionEstimator maps the label offline to calories, protein/carbs/fat, serving weight, and micronutrients (fiber, sugar, sodium, potassium, calcium, iron, vitamin C, cholesterol). Users can also tap any macro card, adjust serving multipliers (0.5x, 1.0x, 1.5x, or custom grams), or switch between the top-3 predicted matches before logging to Room/SQLite.


Key Takeaways & Common Pitfalls

  • Always inspect your .tflite tensors: Check input size (192×192), channel order (RGB), and quantization (UINT8 vs FLOAT32) before writing preprocessing code.
  • Avoid overriding fine-grained classifiers with coarse segmenters: If you combine a classifier with a semantic segmenter, let the whole-image classifier name cohesive dishes instead of forcing segment categories into hardcoded fallback names.
  • Reuse direct ByteBuffers for heavy outputs: Large segmentation tensors (like [1, 513, 513, 26] FLOAT32 ≈ 27 MB) should reuse a cached direct ByteBuffer rather than allocating per frame.
  • Keep live scanning responsive: Use STRATEGY_KEEP_ONLY_LATEST, throttle frames, and use short multi-frame streaks instead of permanently pausing the camera after the first detection.

With LiteRT + CameraX, you get a fast, privacy-first, zero-cost AI scanner that works anywhere — even in airplane mode.
Github: https://github.com/tusharahmed11/NutriLens

Top comments (0)