Prem Singh

Posted on Oct 30

From Pixels to Text: Google ML Kit OCR on Android, with a KMP Core

#android

A receipt scanner looks simple in a demo: point the camera, recognize text, fill a form.

The engineering work starts when the receipt is rotated, the model is not ready, or an invoice number looks convincing but is wrong. A useful OCR feature handles those cases before it asks the user to trust the result.

Here is a practical architecture for Android capture and recognition, with a Kotlin Multiplatform core for the decisions that follow.

Choose the right job for each component

Google ML Kit Text Recognition v2 reads supported text from images. It is not an invoice parser or a guarantee that a recognized amount is correct.

Keep the workflow explicit:

Capture or import
      ↓
Prepare image + orientation
      ↓
Native OCR
      ↓
Shared normalization + candidate extraction
      ↓
Editable review screen
      ↓
User-confirmed save

For document capture on Android, ML Kit also has a separate Document Scanner API. Capture/cropping and text recognition are different steps; obtaining a scanned image or PDF does not mean your application already has structured OCR fields. Check that API's device and model requirements before selecting it.

Decide how the recognition model arrives

Android Text Recognition offers bundled and Google Play services-based options. Bundling increases app size but makes the model available with the app. An unbundled model must be downloaded before use; design a real preparation state rather than showing an unexplained empty result.

For Latin-script recognition, the bundled artifact is com.google.mlkit:text-recognition. The unbundled artifact is com.google.android.gms:play-services-mlkit-text-recognition. Choose one route and pin a compatible version from the official setup guide/release notes. Do not assume all target devices have Google Play services.

Text Recognition v2 supports Latin, Chinese, Devanagari, Japanese, and Korean scripts through the relevant SDKs. A Latin recognizer is not an Arabic recognizer. For Arabic or Urdu documents, evaluate an engine that explicitly supports the required script and test representative documents; do not silently promise coverage from a generic “OCR” label.

Keep the Android recognition adapter small

This integration sketch receives a prepared InputImage, reuses one Latin recognizer, and returns plain application data:

import com.google.mlkit.vision.common.InputImage
import com.google.mlkit.vision.text.TextRecognition
import com.google.mlkit.vision.text.latin.TextRecognizerOptions

data class OcrDocument(
    val fullText: String,
    val lines: List<String>,
)

class AndroidTextReader {
    private val recognizer = TextRecognition.getClient(
        TextRecognizerOptions.DEFAULT_OPTIONS
    )

    fun recognize(
        image: InputImage,
        onSuccess: (OcrDocument) -> Unit,
        onFailure: (Exception) -> Unit,
    ) {
        recognizer.process(image)
            .addOnSuccessListener { result ->
                onSuccess(
                    OcrDocument(
                        fullText = result.text,
                        lines = result.textBlocks.flatMap { block ->
                            block.lines.map { it.text }
                        },
                    )
                )
            }
            .addOnFailureListener { error -> onFailure(error) }
    }

    fun close() = recognizer.close()
}

This is an adapter sketch, not a complete scanner app. Its owner must manage request lifetime, errors, and teardown. Close the recognizer when finished with it, after in-flight work has settled; keep late callbacks from updating a screen that no longer owns the request.

For an imported image, InputImage.fromFilePath(context, uri) is an option; handle preparation failures and URI access lifetime. For CameraX frames, use the frame's actual rotation metadata when constructing the input image.

Flattened lines are enough for a simple candidate extractor. If layout matters, retain block/line bounds too and define their coordinate system. A visual document can have columns; do not assume flattening reconstructs the intended reading order.

Live camera OCR needs backpressure

Recognition is asynchronous. Starting work for every frame can create a queue of stale results.

With CameraX ImageAnalysis, use an appropriate backpressure strategy such as STRATEGY_KEEP_ONLY_LATEST. Keep the underlying image valid until recognition completes, then close the ImageProxy on success and failure. Close immediately if there is no usable image. Do not close the wrapped Media.Image directly.

If your app replaces an active scan with a new one, track request identity so an old callback cannot overwrite the new result. Dropping queued frames and ignoring obsolete UI results solve different problems.

For a first portfolio implementation, a captured still image is often easier to reason about than continuous OCR. Add live overlays only after the basic capture, review, and lifecycle paths work.

Share interpretation through KMP, not native image objects

Put a plain OcrDocument model and pure parsing functions in commonMain. The Android adapter maps ML Kit results into that model. An iOS adapter can use the native ML Kit iOS SDK or another suitable native recognizer and return the same application contract.

This does not mean Google's Android SDK runs in common Kotlin. Keep InputImage, Android Bitmap, and iOS image types at their platform boundaries. Agree on supported scripts, line ordering, coordinate conventions, and error meanings rather than assuming both engines behave identically.

For example, a deliberately narrow shared extractor could be:

fun invoiceIdCandidate(lines: List<String>): String? {
    val prefix = "Invoice ID:"
    return lines.asSequence()
        .map { it.trim() }
        .filter { it.startsWith(prefix, ignoreCase = true) }
        .map { it.substring(prefix.length).trim() }
        .firstOrNull { it.isNotEmpty() }
}

This only handles one label format. It returns a candidate, not a verified identifier. It will miss split labels, other languages, and some recognition errors. Keep fixtures for those cases and expand the rules intentionally instead of presenting a regex as document understanding.

Make the review screen part of the feature

Show the source image beside editable extracted fields. Distinguish “no text found” from processing failure and model preparation. Let the user recapture a blurred image without losing their other work.

Test rotations, small print, glare, multiple columns, mixed scripts, and ambiguous characters such as O/0 and I/1. Verify numeric formatting and currency conventions separately; do not automatically accept a financial value because OCR produced digits.

When recognition is poor, check focus, crop, lighting, and text size in the input before adding elaborate post-processing. A sharper capture often beats another parsing rule.

On-device OCR is not the whole privacy story

Google states that ML Kit processes input and recognition output on device rather than sending them to Google servers. Its SDK may still contact servers for models/updates and send performance/utilization metrics.

Your own app can still leak a receipt through logs, analytics, image backups, or an AI summarization request. Retain only what the feature needs, protect stored documents, and make any upload a separate deliberate action. On-device recognition does not authorize forwarding the extracted text elsewhere.

A portfolio demo worth showing

Record three short paths: a clean receipt, an unreadable capture that recovers gracefully, and a corrected field saved only after review. Add shared parser fixtures and platform integration checks. Explain model availability and supported scripts in the README.

That demonstrates a useful mobile product flow, not just a call to a recognition API.

DEV Community