DEV Community

Jaydeep Shah (JD)
Jaydeep Shah (JD)

Posted on

The Deployment Boundary: Why On-Device AI Breaks After Colab

#edgeai #android #litertlm #privacy

Your model runs perfectly in Colab. Loss converged. Eval looks good. You export to a mobile-friendly format, deploy to a phone, and nothing works.

Not a crash - something worse. A silent failure, a cryptic error from a layer you have never heard of, or behavior subtly different from what you validated in the notebook. You spend three days debugging something that has nothing to do with your model and everything to do with the layers between your Python code and the silicon that runs inference.

I built Redacto, an on-device PII redaction app running Gemma 4 E2B on Snapdragon 8 Elite. Every serious problem we hit came from the gap between "it works in Colab" and "it works on a phone" - and almost none of those problems are documented.

This is the edge AI developer's blind spot. The model is not the hard part. The deployment boundary is.

If you have been following this series, the five failure modes in the next section will be familiar territory - I covered the NPU init failures and the chat template trap in detail in earlier posts. Feel free to skip ahead to "The Deployment Boundary: Where Developers Get Stuck", which is where this post goes somewhere new: naming the structural line between what a developer can do alone and what is locked behind vendor tooling. If you are landing here first, read on - the five failures are the concrete evidence that boundary exists.

The "Last Mile" Breaks in Five Specific Ways

Developers talk about "optimization" and "model size" as the deployment challenges. Those are real but easy. The hard problems are structural - things that break because the on-device runtime is fundamentally different from the Python environment where the model was developed.

Here are five ways our model broke going from Colab to the Samsung Galaxy S25 Ultra. Each is a class of problem, not a one-off bug.

1. Quantization Changes Model Behavior, Not Just Speed

INT4 quantization (4-bit integer weights with FP32 activations) is not a lossless compression. It is a lossy transformation that changes how the model responds to inputs.

In our on-device benchmarks, the standard Gemma 4 E2B model (INT4, 2.59 GB) scored 83.7% on text preservation. Our fine-tuned model (INT4, 4.7 GB, GPU-only) scored 65.9%: it was over-redacting, destroying non-PII context that should have been preserved. (All the benchmark numbers in this post come from a single directional session in May 2026 on one Galaxy S25 Ultra, with no raw logs retained and the device no longer available. Treat them as directional, not as a rigorous multi-run study.)

It is tempting to blame quantization, and quantization does change behavior. But that was not the main story here. The dominant cause was the fine-tune itself: it was an under-resourced run. We trained on only 3,000 of the 400,000 ai4privacy/pii-masking-400k samples, for a single epoch, and the training data taught it to emit generic [REDACTED] and [REDACTED NAME] labels rather than Redacto's structured [CATEGORY_N] format that the benchmark scores against. So a good prompt beat an under-resourced fine-tune. With the full dataset and format-aligned labels, that result could easily flip.

Quantization still matters, and it is a real constraint. When you compress 32 bits of precision into 4 bits, you lose information, and the loss is not uniform across the weight space - it hits edge cases hardest, which are exactly the cases that matter for domain-specific tasks. You must quantize for mobile (Gemma 4 E2B is 5.1B total / 2.3B effective parameters - the E is for effective, via Per-Layer Embeddings - and at FP32 across its 5.1B total parameters it would be well over 8 GB, far too large for phone RAM), but the lesson is that quantization changes your model's behavior in ways your Colab evaluation will not predict.

2. Dispatch Library Versioning: dlopen Fails Silently

On-device inference on Qualcomm hardware requires a chain of shared libraries. libLiteRtDispatch_Qualcomm.so registers a custom op called DISPATCH_OP via static-init constructors when dlopened at runtime. If the dispatch library version does not match the LiteRT runtime version, dlopen fails silently - no crash, no exception, just a missing op registration.

The symptom: Encountered unresolved custom op: DISPATCH_OP. We spent hours looking at model files and export settings. The actual problem was that the .so files in our jniLibs/ directory came from the device's vendor partition - slightly different ABI than the official sample's libraries. Different bytes, same filename. The fix was replacing all six QNN (Qualcomm AI Engine Direct) libraries with the exact QAIRT 2.42 versions from the official sample.

In Colab, you pip install a package and it works. On-device, you are linking native shared objects at runtime, and version mismatches manifest as silent no-ops.

3. DSP Protection Domains: A One-Shot Deal Per Process

The Hexagon NPU is a separate processor with its own firmware, running in a Protection Domain (PD) allocated by the QNN runtime. Once your process acquires a PD and then releases it (by switching to a different backend), you cannot re-acquire it without restarting the entire process.

Switch from NPU to GPU and back to NPU: Failed to find available PD for contextId 5 ... err: 1002. The NPU is gone for that process lifetime.

This is not a software bug. The DSP firmware does not support re-entry after teardown. Our workaround is a cascade: try NPU first on app launch, fall back to GPU or CPU if it fails, never return to NPU within the same session. If the user wants NPU back, they kill the app and relaunch.

No amount of clever engineering changes this. It is how the silicon works.

4. Chat Template Parser Subset: Jinja Features That Break On-Device

Gemma 4's chat template on HuggingFace uses map.get() - a standard Jinja2 method that works in every Python environment. LiteRT-LM's on-device template parser is not a full Jinja2 implementation. It supports a subset. map.get() is not in that subset.

The error: Failed to apply template: unknown method: map has no method named get (in template:238).

This is a deployment trap because the failure is completely invisible during training and export. You fine-tune the model, merge the LoRA adapter, call tokenizer.save_pretrained(), and the HuggingFace-native template gets bundled into the exported .litertlm file automatically. Everything works in Python. The template only fails when the on-device parser tries to execute it at runtime on the phone.

Our fix was to create a re-export notebook that swaps the Gemma 4 chat template with the older Gemma 3 template before calling litert_torch.generative.export_hf. This is undocumented. We found it by reading the error message, guessing that the parser was limited, and testing older templates until one worked.

5. Vision Encoder Assumptions: Text-Only Model Meets Multimodal Runtime

Gemma 4 E2B is a multimodal model - it supports text and images. When you fine-tune it for a text-only task (like PII redaction) and export to .litertlm, the exported file does not include the vision encoder section (TF_LITE_VISION_ENCODER). But the default LiteRT-LM engine configuration expects a vision backend to be present.

The error: NOT_FOUND: TF_LITE_VISION_ENCODER not found in the model.

The fix is to pass visionBackend = null and maxNumImages = null in the engine config when loading a text-only fine-tuned model. But there is a compounding issue: even for the standard multimodal model, setting visionBackend = Backend.GPU() (as the developer guide recommends) crashes on Android 16 with UNIMPLEMENTED: CreateSharedMemoryManager is not implemented. The GPU's OpenGL backend has an unimplemented codepath on Android 16, and it kills the entire engine initialization - even if you are running inference on the NPU and never actually processing an image.

We set all vision and audio sub-backends to CPU across every backend configuration. The cost is zero for text-only workloads, but finding this required reading source-location traces from litert::ml_drift - internal runtime code paths that no application developer should have to debug.

Silicon-Level Reality: What an NPU Actually Is

Most documentation describes the NPU as an "accelerator." This is technically correct and practically useless. Understanding what the NPU physically is explains why deployment problems exist.

An NPU is dedicated matrix multiply silicon. On the Snapdragon 8 Elite, this is the Hexagon V79: a separate processor on the SoC die with its own instruction set, firmware, and memory management. Not a GPU with different drivers - a fundamentally different processor designed for one job: executing neural network operations (matrix multiplication, convolution, activation functions, normalization) at maximum throughput per watt.

The Hexagon V79 supports INT4, INT8, and FP16 precision natively. It is widely reported at around 45 TOPS. In our single-session measurements, it delivered roughly 42 tok/s on Gemma 4 E2B versus about 25 tok/s on the Adreno 830 GPU - close to 1.7x throughput. Time-to-first-token was around 92ms on NPU versus 366ms on GPU - roughly a 4x improvement.

The consequence of dedicated silicon: the NPU has its own instruction set. You cannot run arbitrary TFLite ops on it. The model must be compiled through the QNN SDK into a binary using Hexagon V79-specific operations. This is what DISPATCH_OP is - a single TFLite node wrapping an entire QNN-compiled subgraph. When LiteRT encounters DISPATCH_OP, it hands computation to the Hexagon via the dispatch library. If that library is missing or wrong-versioned, the NPU does not exist to your application.

A model compiled for an earlier Hexagon generation (say V73) will not run on V79; a model built against an earlier QAIRT release may not match the 2.42 runtime libraries. The ops are not portable across chip generations because the silicon is different.

The Deployment Boundary: Where Developers Get Stuck

There is a line in the edge AI stack separating what developers can do independently from what requires vendor access. This is the most important structural reality in the ecosystem, and it is almost never discussed explicitly.

What is public and works:

  • LiteRT-LM SDK (com.google.ai.edge.litertlm): the Android library for loading and running .litertlm models. Available on Maven Central.
  • litert_torch export tool: converts HuggingFace models to .litertlm format with quantization. Available via pip.
  • Pre-compiled models: Google and Qualcomm publish .litertlm files for popular models (Gemma, Llama) compiled for specific chipsets. Available from litert-community repos on HuggingFace.
  • Sample apps: working reference implementations for GPU and NPU inference. Available on GitHub (google-ai-edge/litert-samples).

What is required for NPU compilation of custom models but not part of the public LiteRT-LM SDK:

  • AIMET (AI Model Efficiency Toolkit): Qualcomm's quantization tool for preparing models for QNN compilation.
  • QNN-AOT compiler: converts a quantized model into Hexagon-specific binary ops, producing the DISPATCH_OP subgraphs that run on the NPU.
  • Chip-specific calibration data and compilation profiles: parameters that map operations to the specific Hexagon version on the target SoC.

Everything before this boundary works. We fine-tuned a model in Colab, exported with litert_torch, and ran it on the phone's GPU. Everything after requires vendor toolchain access. A developer cannot independently compile a fine-tuned model for the NPU.

This is the deployment boundary.

The Fine-Tuned Model NPU Blocker: A Real Case Study

This is not theoretical. It is what happened with Redacto.

We fine-tuned Gemma 4 E2B using QLoRA in Colab (2.85M trainable parameters, 217 seconds on an NVIDIA RTX PRO 6000), exported with litert_torch using dynamic_wi4_afp32 quantization. The fine-tuned model runs on the phone's GPU at about 9 tok/s. The standard model runs on the NPU at roughly 42 tok/s. That looks like a large gap, but be careful reading it: it combines two variables at once - a different model and a different processor (GPU versus NPU) - so it is not a clean apples-to-apples comparison.

But our fine-tuned model cannot run on the NPU. The exported .litertlm contains standard TFLite ops, not QNN-compiled DISPATCH_OP subgraphs. To get it onto the NPU, we would need the QNN-AOT (ahead-of-time) compiler - which is not part of the public SDK.

This is not an engineering shortcut we missed. It is a hardware-team integration boundary. The Qualcomm team that compiled the standard model for Hexagon V79 used internal tools we do not have access to. Every developer who fine-tunes will hit this wall if they want NPU performance.

The consequence: fine-tuning buys better accuracy on GPU at the cost of being locked out of the fastest hardware. Even in our under-resourced run, the fine-tuned model improved entity recall by about 13% on both FIELD_SERVICE and TACTICAL - meaningful quality gains, stuck at roughly 9 tok/s because it cannot reach the NPU.

What This Means for the Ecosystem

The edge AI ecosystem has a bottleneck, and it is not model quality. Gemma 4 E2B scores 80.5% overall on our 85-entry PII redaction benchmark with nothing but prompt engineering. The 2B-class (2.3B effective) model is capable of real work on real tasks.

The bottleneck is developer experience.

The toolchain gap will slow adoption more than any model limitation. Consider the developer journey for on-device AI today:

  1. Pick a model on HuggingFace. (Easy.)
  2. Fine-tune it in Colab for your domain. (Straightforward, well-documented.)
  3. Export to .litertlm. (Works, with undocumented gotchas like the chat template trap.)
  4. Run on GPU. (Works.)
  5. Run on NPU for production performance. (Blocked without vendor toolchain access.)

Steps 1 through 4 can be done in a weekend. Step 5 has no public path. And step 5 is the difference between a demo and a product - because the NPU is architecturally designed for lower power draw per inference operation than the GPU, and mobile apps live and die by battery life.

Developer experience is an engineering problem, not a documentation problem. The issues I described - silent dlopen failures, DSP path ordering, chat template parser subsets, vision encoder config mismatches, Protection Domain one-shot constraints - these are not things that better documentation would fix. They are architectural gaps between the training ecosystem (Python, Colab, HuggingFace) and the deployment ecosystem (C++, Android NDK, Qualcomm QNN, chip-specific firmware). Closing them requires toolchain engineering, not more README files.

The edge AI space needs people who understand both the model and the metal. Most ML engineers have never dlopened a shared library. Most embedded engineers have never fine-tuned a model. The deployment boundary sits at the intersection of these two skill sets. The developers who understand why ADSP_LIBRARY_PATH must be set before any LiteRT library loads, and also understand why INT4 quantization changes entity recall differently across domains - those are the ones who will ship production edge AI.

Right now, that intersection is almost empty. The tools assume you are either an ML engineer who exports models or a platform engineer who integrates vendor SDKs. Nobody is building for the person who does both.

That is changing. On-device AI is shipping in production apps, processing real user data with real privacy requirements. The "last mile" between Colab and the device is where the value is created - and where the work is hardest.

How to Work With the Boundary Today

You cannot dissolve the deployment boundary on your own, but you can stop it from stalling your project. Three moves that follow directly from everything above:

Ship on GPU, treat NPU as a partnership, not a task. The public path takes you through step 4: fine-tune, export, run on the GPU. That is a real, shippable product. NPU compilation of a custom model is not something you will unblock by trying harder - it needs the vendor toolchain (AIMET, the QNN-AOT compiler, chip-specific calibration). Plan for it as a conversation with Qualcomm or Google, on their timeline, not as a sprint task on yours.

Design so the backend is swappable from day one. Every failure in the first half of this post - the silent dlopen, the one-shot Protection Domain, the vision-encoder crash - is a reason your app must degrade gracefully across NPU, GPU, and CPU. Build the cascade first: try the fastest backend, fall back cleanly, never assume any one of them initializes. The app that survives the deployment boundary is the one that never depended on a single backend succeeding.

Staff for the intersection, or grow into it. The problems here are not solved by an ML engineer or a platform engineer working alone; they are solved by someone who can read a litert::ml_drift source trace and reason about how INT4 quantization shifts entity recall. If that person is not on your team, the deployment boundary is where you will lose the most time. It is also the skill set that is about to be worth the most.

The model is not the blind spot. The deployment boundary is.

Related in this series of "Edge AI from the Trenches"

Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware. Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.

Sources:

Test environment:

All technical details and performance numbers come from my own testing of the Redacto project, in a single directional session (May 2026), on the configuration below.

  • Hardware: Samsung Galaxy S25 Ultra, Snapdragon 8 Elite (SM8750-AC), Hexagon V79 NPU, Adreno 830 GPU, Android 16 (API 36)
  • Runtime: LiteRT-LM 0.11.0-rc1, QAIRT 2.42 QNN libraries
  • Model: Gemma 4 E2B (5.1B total / 2.3B effective parameters), INT4 quantized (dynamic_wi4_afp32); standard model file 2.59 GB (GPU/generic), 3.02 GB (NPU)

Last updated: July 2026
19th of 23 posts in the "Edge AI from the Trenches" series

Top comments (0)