The on-device AI industry feels like it is in its "single app demo" phase right now. Most of what I see - hackathon projects, conference talks, developer blogs, including my own - showcases one app running one model on one phone. The demo works. The audience claps. Everyone moves on.
But it left me with an uncomfortable question I have not seen discussed much: what happens when 15 apps on the same phone each ship their own multi-GB model?
I started thinking about this while building Redacto, a zero-trust document redaction app I built for the Qualcomm x Google LiteRT hackathon. Redacto runs Gemma 4 E2B entirely on-device: no internet permission, no cloud calls, no BAA required. It works beautifully in isolation. But the moment I started measuring what it actually consumes - storage, memory, NPU time, I/O bandwidth - I started to wonder whether we are building toward a systemic crunch that the tooling has not caught up with yet.
This piece lays out the math and the questions it raised for me. On the numbers: the measurements here come from a single directional benchmarking session in May 2026.
A note on where I am standing. This is my naive, practitioner-level read from limited exposure - one app, one device, one hackathon. I have not surveyed every tool, runtime, or architecture out there, so treat this as "here is what I observed and what I suspect," not a verdict on the state of the industry. Some of the gaps I describe may already be solved by tools I simply have not come across yet; others may be in active development as I write this; and some may be genuinely missing, the kind of thing the ecosystem will grow into. I plan to dig further, test some of this in practice, and I would genuinely like to hear from readers who know of work that already addresses these problems. Consider this the start of a conversation, not the last word.
The Assumptions Behind This
The whole argument rests on three things I believe will be true as on-device AI scales, based on what I saw building one app. If these do not hold, the crunch looks different, so it is worth stating them plainly.
1. Apps will not converge on one shared model. In my experience, an out-of-the-box model is not always capable of what a specific app needs. A redaction app, a translation app, and a coding assistant are asking genuinely different things of the model. So I expect developers to reach for fine-tuning: some will ship a fully fine-tuned model, many will ship a base model plus a task-specific adapter (LoRA-style), and different teams will start from different bases (one team likes Gemma, another Llama, another Qwen). The realistic picture is not "everyone bundles the same stock Gemma" but a mix: several distinct base models, plus a spread of adapters layered on the common ones. That mix is exactly what makes naive deduplication hard - which I come back to later.
2. It is not only LLMs competing for the hardware. The "15 apps" framing is really shorthand for on-device AI in general. Long before the LLM wave, apps already shipped classic ML models: camera scene detection, computational photography, face and object recognition, speech-to-text, OCR, recommendation and ranking models. Those are smaller than a 2B LLM, but there are a lot of them, many run continuously, and they draw on the same finite NPU, memory bandwidth, storage, and I/O. So the real contention picture is a mix of a few heavy LLMs and many lighter classic-ML models all summing against one device's budget - which makes the crunch arrive sooner than an LLM-only count suggests.
3. Apps will not all be in the foreground. Some AI apps are foreground-only: you open them, run inference, close them (Redacto is one). Others will run as persistent background services: always-on assistants, continuous audio processing, real-time translation. The contention problems below are most acute precisely because these two kinds of apps coexist and compete for the same NPU, memory, and I/O without any arbiter deciding who yields.
The Storage Math: ~84 GB of Weights on a 256 GB Phone
Let me start with what Redacto actually ships.
The app needs two model files to support its backend cascade (try NPU, fall back to GPU, fall back to CPU):
| File | Size | Purpose |
|---|---|---|
gemma4.litertlm |
2.59 GB | GPU and CPU inference (standard tflite ops) |
gemma4_npu.litertlm |
3.02 GB | NPU inference (QNN-compiled DISPATCH_OP custom ops for Hexagon V79) |
| Total | 5.61 GB | One app, one model architecture, two hardware targets |
These two files exist because the NPU variant contains chip-specific compiled operations that only run on Hexagon V79. You cannot use the GPU model on the NPU or vice versa. The dispatch delegate architecture requires separate compilation targets, so any app that wants to support both fast-path (NPU) and fallback (GPU/CPU) needs to ship both files. This is not a packaging inefficiency: it is a fundamental consequence of how heterogeneous compute works on mobile SoCs.
Now scale that to a phone with multiple AI-powered apps. Assume a 256 GB phone, of which roughly 250 GB is usable after the OS, system apps, and firmware:
| Scenario | Model Storage | Share of Usable Storage (~250 GB) |
|---|---|---|
| 1 app (Redacto) | 5.61 GB | 2.2% |
| 5 apps | ~28 GB | 11% |
| 10 apps | ~56 GB | 22% |
| 15 apps | ~84 GB | 34% |
This conservatively assumes every app ships the same 5.61 GB footprint (one 2B INT4 model, two hardware targets). Per assumption 1 above, real apps will vary - some larger, some sharing a base model - but the aggregate pressure is the point. And 250 GB usable is the ceiling before real content: photos, videos, and existing apps typically consume another 40-60% on top of the OS.
At 15 apps, you have burned through a third of your phone's storage on AI model weights alone. And this assumes every app uses a 2B parameter model quantized to INT4. Larger models - 4B, 7B, or multimodal variants with vision encoders - would be significantly worse. A single 7B INT4 model can exceed 5 GB. Two variants of that for NPU and GPU, and one app consumes 10+ GB.
This is not a problem you can solve by buying a bigger phone. The 512 GB tier adds cost, and the storage pressure grows linearly with the number of AI apps users install.
NPU Contention: No Scheduler, No Priority, No Recovery
Here is something I learned the hard way during the hackathon: the Hexagon DSP's QNN Protection Domain cannot be re-acquired within the same process after it has been released.
The constraint, as I hit it during the hackathon:
Once a non-NPU Engine has been instantiated and closed in the process, the Hexagon DSP's PD (Protection Domain) reservation can't be re-acquired by a subsequent NPU Engine in the same process. This is a QNN/LiteRT-LM constraint, not something fixable from app code without restarting the process.
The error looks like this:
E QnnDsp: Failed to find available PD for contextId 5 ... err: 1002
E tflite : Encountered unresolved custom op: DISPATCH_OP.
This is a single-app, single-process limitation. Now consider the multi-app scenario.
During the hackathon, I observed teams building apps that run models as persistent background services: always-on assistants, continuous audio processing, real-time translation. Other apps, like Redacto, are foreground-only: the user opens the app, runs inference, closes it.
What happens when three apps need the Hexagon DSP simultaneously?
As far as I could find, there is no clean answer to that question today. From what I saw, the QNN stack has:
- No multi-tenant scheduler. There is no system-level arbiter deciding which app gets NPU time and which app waits.
- No priority API. A foreground app actively responding to user input has no way to preempt a background service that is running a low-priority classification loop.
- No graceful eviction. If one app holds the Protection Domain, another app's NPU init will fail with error 1002. There is no queue, no callback, no "try again when available" mechanism.
- No recovery without process restart. Even within a single app, losing the PD means you must kill the process and relaunch. Across apps, this means the system would need to terminate another app's process to free the NPU.
Compare this to how Android handles GPU contention today. The GPU has a kernel-level scheduler (on Qualcomm: the KGSL driver) that multiplexes compute contexts across processes. An app does not need to know or care whether another app is using the GPU. The driver handles context switching, priority management, and memory isolation transparently. This infrastructure took years to mature: Adreno's GPU scheduler evolved through multiple Android releases in the 2013-2017 era, driven by the demands of mobile gaming.
The NPU feels like it is roughly where the GPU was around 2010. We have the hardware. What I could not find is the software infrastructure for multi-tenant access - though I would be glad to be pointed to it if it exists.
Memory Contention: 1.9 GB Per App, and the LMK Knows It
Redacto's peak RSS (Resident Set Size) on the NPU backend was 1,934 MB: nearly 2 GB of physical memory consumed by a single app running a single 2B parameter model.
On the GPU backend, it was 1,375 MB, still over 1.3 GB.
These come from my own benchmark runs (peak RSS via /proc/self/status VmRSS), and as above are directional figures from a single May-2026 session, not a multi-run average.
The NPU variant uses approximately 560 MB more RSS than the GPU variant. This delta comes from the larger NPU model file (3.02 GB vs 2.59 GB on disk, with a proportional difference in runtime memory mapping) and the QNN runtime's internal buffer allocations for the Hexagon DSP communication pathway.
Now do the multi-app math:
| Scenario | Estimated Combined RSS | Typical Flagship RAM (12 GB) |
|---|---|---|
| 1 AI app (NPU) | ~1.9 GB | 16% of total RAM |
| 2 AI apps (NPU) | ~3.8 GB | 32% of total RAM |
| 3 AI apps (NPU) | ~5.7 GB | 48% of total RAM |
| 2 AI apps + Android OS + system services | ~6-7 GB | 50-58% of total RAM |
Android's Low Memory Killer (LMK), specifically lmkd, the userspace daemon that replaced the older kernel-level OOM killer, assigns oom_score_adj values to processes based on their importance. Foreground apps get the lowest scores (highest priority), cached background apps get the highest (first to die). When memory pressure crosses configured thresholds, lmkd starts killing processes in ascending priority order.
Here is the problem: AI apps all request android:largeHeap="true" in their manifests. They have to - not for the model weights, which are memory-mapped, but for the Java-side allocations (tokenizer state, tensor buffers, intermediate activations) that a 2B model pushes past the default heap ceiling. But largeHeap does not give an app more physical memory: it raises the per-process Java heap ceiling. The model weights themselves are memory-mapped from the filesystem, consuming physical RAM pages regardless of the Java heap setting. And lmkd does not distinguish between "this app is using 1.9 GB because it loaded an AI model" and "this app is using 1.9 GB because it has a memory leak." Both get the same oom_score_adj treatment.
The practical consequence: with two AI apps running, Android will aggressively kill background processes to maintain free memory headroom. Users will see their non-AI apps being evicted and cold-restarting constantly. And if both AI apps are in the foreground (split-screen, picture-in-picture, or rapid task-switching), the system has to choose which one to kill. There is no mechanism for the OS to say "unload this app's model but keep the process alive": the granularity is process-level, not model-level.
Every AI app developer will independently arrive at the same conclusion: request largeHeap, memory-map the model, hold it in RAM for fast inference. And every one of them will be correct for their app in isolation. The tragedy-of-the-commons failure only emerges at the system level.
I/O Pressure: Multi-GB Cold Reads on a Shared Bus
Model loading is not instant. On first launch, Redacto reads the full 2.59 GB (GPU) or 3.02 GB (NPU) model file from flash storage into memory. Even on UFS 4.0, which delivers theoretical sequential read throughput of around 4.2 GB/s, a single model load takes measurable time.
Redacto's ahead-of-time (AoT) compilation cache (roughly 327 MB of compiled kernels) speeds up subsequent launches; on the NPU path the engine initialized in about 2.5 seconds once the cache was warm. The cache itself consumes additional storage. And the first launch, the one that determines whether a user keeps or uninstalls the app, always pays the full cold-read cost.
Now picture three AI apps launching in sequence after a phone reboot. Each reads 2.59-3.02 GB from flash. That is roughly 7.8 to 9 GB of sequential reads hitting the UFS controller in rapid succession. UFS storage is a shared bus: the same controller handles app installs, photo writes, database commits, and system updates. Sustained large sequential reads from multiple processes will compete with the random I/O patterns that the rest of the system depends on.
The AoT cache approach, which Redacto uses to accelerate re-launches, is itself a storage trade-off. Each cache consumes additional flash space. Multiply that across 15 apps and you are adding another layer of storage pressure on top of the model files themselves.
What the Ecosystem Needs
These do not feel like hypothetical problems to me. They look like a likely consequence of the current architecture as on-device AI adoption scales. Every number I have cited is from one app with one 2B model on current hardware, so read the trajectory as a hypothesis I want to test further, not a foregone conclusion.
Here is what I think the ecosystem, specifically the Android/LiteRT and QNN/Hexagon stacks, needs to build:
1. Shared model runtime. If five apps all use Gemma 4 E2B, the phone should load one copy of the model into memory and serve inference requests from all five apps through a shared service. This is architecturally similar to how Android's SurfaceFlinger composites GPU output from multiple apps through a single display pipeline, or how the mediaserver process manages hardware codec access. A shared inference service would reduce the per-app memory overhead from ~1.9 GB to a fraction of that (the model weights would be shared; only per-app context buffers would be private).
2. OS-level model deduplication - keyed on more than the whole file. If three apps bundle the same gemma4.litertlm file, the OS should recognize the duplication at install time and store one physical copy. Android already does this for shared native libraries via the uses-native-library manifest tag and the linker namespace system. But per assumption 1, whole-file dedup only catches the easy case. The harder, more realistic case is apps that share a base model but layer different fine-tunes or adapters on top: naive hashing sees these as entirely different files. Real deduplication would need to work at the level of shared base weights plus per-app adapter deltas - a system-managed model registry that understands "same base, different adapter," not just "identical bytes." That is a bigger ask, and it is the one that actually matches how I expect apps to ship models.
3. NPU scheduling and priority APIs. The Hexagon DSP needs a multi-tenant scheduler analogous to what KGSL provides for Adreno GPUs. At minimum: a queue-based access model, foreground/background priority levels, and a preemption mechanism so that a foreground app's inference request does not block behind a background service's batch job. The Protection Domain limitation (error 1002 on re-acquisition) needs to become a recoverable condition, not a process-fatal one.
4. Model-aware memory management. lmkd needs to understand that memory-mapped model weights are a different category than application heap. Ideally, the OS could evict model pages under memory pressure and transparently reload them from flash when needed - similar to how Linux handles page cache eviction for memory-mapped files, but with awareness of the latency cost (reloading a 2.59 GB model is not the same as re-reading a 4 KB config file). A model-level eviction API would let the system reclaim memory from background AI apps without killing their processes entirely.
5. Storage-aware model delivery. Instead of each app bundling its own model files, the OS could manage a shared model cache - similar to how Android's ART manages ahead-of-time compiled dex files in a system-managed directory. Apps would declare their model dependencies (model ID, version, quantization level, target hardware), and the OS would download, store, and deduplicate them centrally. Google Play's on-demand asset delivery already has the infrastructure for large file management; extending it to model files with deduplication semantics is a natural evolution.
The GPU Analogy: We Have Been Here Before
In the early days of Android gaming (2010-2013), GPU compute on mobile was in a remarkably similar state to where NPU compute is today.
Multiple apps could submit GPU workloads, but there was no real scheduling: whichever app got the GPU first held it. Context switching was expensive and poorly defined. Power management was coarse-grained. Memory isolation between GPU contexts was incomplete. Developers had to manage GPU memory manually because the system would not do it for them.
Over the next several years, Qualcomm's KGSL (Kernel Graphics Support Layer) driver evolved to handle multi-context scheduling, priority-based preemption, per-context memory accounting, and cooperative power management. ARM's Mali and Imagination's PowerVR went through similar evolutions. By the time Android gaming matured, the GPU software stack had caught up to the hardware's capabilities.
The NPU is at the beginning of that same arc. The hardware - Qualcomm's Hexagon V79, MediaTek's APU, Google's Tensor TPU - is already capable. In my own single-session measurements, the NPU path decoded Gemma 4 E2B at about 42 tokens per second with a time-to-first-token in the low-100ms range. NPU decode here is memory-bandwidth-bound, so a roughly flat token rate is expected rather than surprising. The point is that the hardware is not the bottleneck.
The software infrastructure - scheduling, memory management, multi-tenancy, developer APIs - is where the gap lives. And unlike the GPU transition, which played out over 5+ years of gradual adoption, the NPU transition is going to be compressed. The AI app ecosystem is growing faster than the mobile gaming ecosystem did, which means the contention problems will arrive sooner and hit harder.
Why This Matters Now
I am writing this piece not because these problems are blocking me today. Redacto works. On the NPU it decoded at about 42 tok/s with a time-to-first-token around 100ms in my one benchmarking session, and it handles document redaction on-device with zero network access.
I am writing it because I can see the wall from here. Every developer building on-device AI apps is making the same rational, individually correct decisions: bundle the model, claim the NPU, request largeHeap, load the weights into memory. And each of those decisions is fine in isolation. The crisis is emergent. It only becomes visible when you multiply by the number of apps on a phone and ask: who arbitrates?
As far as I can tell from where I sit, the answer today is nobody. I did not find an arbiter, a scheduler, a shared runtime, or model deduplication. From the outside, every app looks like an island. If any of these already exist and I missed them, that is genuinely the thing I most want a reader to tell me.
The teams at Google and Qualcomm are building extraordinary hardware and runtime capabilities. LiteRT-LM is a genuine leap forward for on-device LLM inference. The Hexagon V79 is remarkable silicon. But the system-level infrastructure for multi-tenant AI - the layer that makes 15 apps coexist gracefully on one device - is not something I have come across yet. That gap, if it is one, is not any one vendor's fault; it is what happens when several independent, individually sound design choices collide at the system level.
That is the 15-app problem. And the time to start solving it is before we get to 15, not after.
Related in this series of "Edge AI from the Trenches"
- Why My LLM Runs 4x Faster on Hardware I Had Never Heard Of - foundational context on the hardware at the center of the contention problem
- One Model, Three Chips, Two Files: How LiteRT Delegates Really Work - why separate model files and vendor libraries create the storage multiplication
- What I Learned Benchmarking the Same LLM on My Phone's NPU, GPU, and CPU - the memory and performance measurements behind the per-app resource estimates
Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware. Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.
Last updated: July 2026
21st of 23 posts in the "Edge AI from the Trenches" series
Top comments (0)