On-Device AI on Android: Delegates, NPUs and Fragmentation

#devops #llm #tutorial #ai

The hard part of on-device AI on Android is not writing the inference call. It is that the same call runs on thousands of system-on-chip and driver combinations, some of which will accelerate your model beautifully, some will fall back to CPU without telling you, and a few will produce wrong numbers. The strategy that works is to find out at runtime and remember the answer.

The problem is not the API, it is the variance

On iOS you target a handful of silicon generations from one vendor. On Android you target several chip vendors, many generations each, with driver stacks shipped by device manufacturers on their own schedules. Two phones with the same headline chip can behave differently because one has an older GPU driver.

The consequence is a rule: never decide acceleration from a device model string. Allowlists by device name go stale within a release cycle, they cannot cover devices that did not exist when you shipped, and they encode a guess where a measurement is available. Decide by trying, on the device, once, and caching the result.

The layers you are choosing between

Broadly there are three levels of abstraction available, and picking the right one is mostly a question of how much control you need:

A managed on-device model provided by the platform. Google ships system-level generative AI capabilities that apps can call, with the model managed and updated outside your app. This gives you zero bundle cost and no control: availability is gated by device and by system component version, so your feature must degrade when it is absent. Check current availability programmatically rather than assuming a minimum API level.
A runtime you bundle, with delegates. LiteRT — the runtime formerly published as TensorFlow Lite — or ONNX Runtime, with an acceleration delegate or _execution provider_selected at load time. This is the mainstream choice. You control the model, the version and the fallback.
A vendor SDK targeting one silicon family directly. The chip vendors publish their own neural SDKs, which typically extract more performance from their own NPUs than a generic delegate does, at the cost of a separate integration and a separate model artefact per vendor. Worth it when inference is the product; rarely worth it when inference is a feature.

Which acceleration paths are current, and which are deprecated in favour of vendor SDKs, has changed more than once and continues to. Treat the delegate you choose as a parameter of your build, not as a fact about Android, and re-check it at each major platform release. The probe below is written so that changing the delegate is a one-line change.

The capability probe

The pattern is the same regardless of which runtime you picked: attempt to construct the accelerated interpreter, run a fixed input, compare against a CPU reference, and time both. If construction throws, if the numbers differ beyond tolerance, or if the accelerated path is not actually faster, use CPU.

enum Accel { NNAPI_OR_VENDOR, GPU, CPU }

data class ProbeResult(val accel: Accel, val medianMs: Double, val valid: Boolean)

fun probe(context: Context, modelBytes: ByteBuffer): Accel {
    val golden = loadGoldenInputOutput(context)   // fixed input + expected output
    val results = mutableListOf<ProbeResult>()

    // Always establish the CPU reference first: it is the tie-breaker
    // for both correctness and speed.
    val cpu = timeRun(modelBytes, Accel.CPU, golden)
    results += cpu

    for (accel in listOf(Accel.NNAPI_OR_VENDOR, Accel.GPU)) {
        val r = try {
            timeRun(modelBytes, accel, golden)
        } catch (t: Throwable) {
            // Delegate construction failing is normal, not exceptional.
            ProbeResult(accel, Double.MAX_VALUE, valid = false)
        }
        results += r
    }

    val best = results
        .filter { it.valid }
        .minByOrNull { it.medianMs } ?: cpu

    // Require a real margin. A 5% win is not worth a second code path.
    return if (best.medianMs < cpu.medianMs * 0.8) best.accel else Accel.CPU
}

Run this once, on a background thread, the first time the feature is used — not at app launch, where it competes with everything else for startup budget. Persist the verdict keyed by your app version, the model version and the OS build number, and re-probe when any of those three change. A system update can add or remove a working driver, and your cached answer must not outlive it.

Validating against a golden reference

The correctness half of the probe is the part most integrations omit, and it is the part that catches the worst bug class: a delegate that runs, is fast, and is wrong.

Pick a fixed input that exercises the model meaningfully — not zeros, which many broken kernels handle correctly by accident.
Compute the expected output once, offline, with the reference implementation you trust, and ship it as an asset.
On device, compare element-wise with a tolerance appropriate to the precision. A delegate that computes in fp16 will not match an fp32 reference exactly, so an absolute tolerance around 1e-2 on normalised outputs is usually right; an exact-match test will fail on every healthy device.
For a classifier, also assert that the top-1 label matches. Numeric tolerance can hide a class flip, and a class flip is what your user sees.

Log probe outcomes with the SoC identifier and OS build. Over a few weeks that log becomes the only accurate map of your own install base, and it is worth far more than any published compatibility matrix.

Designing for tiers instead of devices

Rather than one experience that must work everywhere, define two or three tiers and let the probe assign the device:

Tier	Description
accelerated	Probe found a valid, materially faster path. Full feature: larger model, longer context, real-time processing.
cpu-capable	No acceleration, but CPU inference completes within the interaction budget. Smaller model, batched or on-demand rather than live.
unsupported	CPU inference is too slow, or memory does not allow the model. Feature is hidden or served remotely. Hiding it is the honest option; a feature that takes nine seconds is worse than no feature.

Deciding by measurement means the tier assignment stays right as your install base changes, and it means a new phone released after your launch gets the good experience automatically. It also gives you a clean place to put the kill switch: if a delegate turns out to be broken on some population, you demote that population to CPU with a remote flag rather than shipping an emergency build. The pattern is the same one described in feature flags for models and prompts.

The failures you will actually hit

Silent CPU fallback. The delegate is constructed, reports no error, and executes most of the graph on CPU anyway because some operator was unsupported. Only timing reveals it. This is why the probe compares against CPU rather than merely checking that construction succeeded.
Delegate initialisation cost. Building an accelerated interpreter can take hundreds of milliseconds or longer, because drivers compile the graph. Amortise it: keep the interpreter alive for the session rather than constructing one per inference, and never build one on the main thread.
Quantisation mismatch. Many NPUs execute integer models only. A float model will either be rejected by the delegate or silently run elsewhere. If the NPU path matters to you, ship a quantised artefact and verify it against the golden reference — quantisation is a numerics change, and which layers tolerate it is not uniform.
Background execution limits. Android will not let you grind the NPU indefinitely behind a backgrounded app. Long jobs belong in a foreground service with a visible notification, or in a deferrable background work queue that the system schedules when the device is idle and charging.
Memory pressure on entry-tier devices. Handle onTrimMemory by releasing the interpreter and reloading lazily. A model held resident through a backgrounded session is a common cause of being killed and then blamed for a slow cold start.

If the unsupported tier falls back to a hosted model, the two paths need to be interchangeable at the call site — Multigrid exposes one API across providers with per-request cost and latency, which keeps the remote branch a single implementation rather than one per vendor you might route to.