Most discussions about NPU inference are either vendor press releases or synthetic benchmarks on models nobody ships. I wanted something different: real numbers, from a real app, doing real work, on a phone you can actually buy.
This post breaks down what happened when I ran the same PII-redaction pipeline on the NPU, GPU, and CPU of a Samsung Galaxy S25 Ultra, using Gemma 4 E2B INT4 via Google's LiteRT-LM runtime. The results were not what I expected.
The Setup
Device: Samsung Galaxy S25 Ultra
Chipset: Snapdragon 8 Elite (SM8750-AC)
NPU: Hexagon V79, reached via QNN (Qualcomm AI Engine Direct), which targets the Hexagon Tensor Processor (HTP)
Android: 16 (API 36)
Runtime: LiteRT-LM 0.11.0-rc1
Model: Gemma 4 E2B Instruct INT4 (2.59 GB GPU/generic build, 3.02 GB NPU-compiled build)
The NPU path uses ahead-of-time (AoT) compilation, while the CPU and GPU paths use just-in-time (JIT) partitioning with CPU fallback.
App: Redacto, a privacy-redaction tool that takes sensitive text (medical records, financial documents, police reports) and replaces PII with category-coded placeholders like [NAME_1] or [SSN_1].
Pipeline: Four steps, executed sequentially. The first three are the LLM calls these benchmarks measure:
Step 1: CLASSIFY - determine document type (medical, financial, tactical, etc.)
Step 2: DETECT - identify all PII entities in the text
Step 3: REDACT - rewrite the text with placeholders replacing PII
Step 4: VALIDATE - check the redacted output for leaks (not benchmarked here)
Each step creates a fresh conversation (no shared context) and generates free-form text output. This is not a single-token classification task: Steps 2 and 3 produce substantial multi-token output.
Dataset: redacto_bench.jsonl, 30 entries across 5 redaction modes (HIPAA, Financial, Field Service, Tactical, Journalism), average input length 208 characters, all "easy" difficulty. Benchmarks run Steps 1-3 only (no validation pass), triggered via ADB broadcast receiver so the already-initialized engine is reused.
Metrics methodology: Wall-clock latency via System.currentTimeMillis(), TTFT via first onMessage callback timestamp, decode speed computed as (tokenCount - 1) * 1000 / (lastTokenTime - firstTokenTime), peak RSS from /proc/self/status. I only report metrics I can measure reliably: no battery draw estimates, no GPU/NPU utilization percentages, nothing derived from unreliable APIs.
One caveat before the numbers: everything below comes from a single benchmarking session on 2026-05-01. No raw logs were retained, and that device is no longer available to me. Treat these as directional findings from one run on one phone, not a rigorous multi-run study.
The Headline Numbers
Here is the per-step average across all 30 entries:
| Metric | NPU | GPU | NPU vs GPU |
|---|---|---|---|
| Step 1 (Classify) latency | 345 ms | 773 ms | 2.2x faster |
| Step 2 (Detect) latency | 624 ms | 1,586 ms | 2.5x faster |
| Step 3 (Redact) latency | 4,060 ms | 2,475 ms | 0.6x (GPU wins) |
| Step 1 TTFT | 99 ms | 381 ms | 3.8x faster |
| Step 2 TTFT | 104 ms | 375 ms | 3.6x faster |
| Step 3 TTFT | 92 ms | 366 ms | 4.0x faster |
| Decode speed (Step 1) | ~42 tok/s | ~25 tok/s | ~1.7x faster |
| Decode speed (Step 2) | ~42 tok/s | ~25 tok/s | ~1.7x faster |
| Decode speed (Step 3) | ~42 tok/s | ~25 tok/s | ~1.7x faster |
NPU decode sits at roughly 42 tok/s and stays essentially flat across the three steps. That flat rate is expected here: NPU decode is memory-bandwidth-bound rather than compute-bound, so the per-token throughput does not vary much with the step. NPU dominates TTFT (sub-100ms vs 366-381ms) and decode throughput (~42 vs ~25 tok/s). If the story ended here, this would be a straightforward "use the NPU" recommendation.
It does not end here.
The Paradox: Faster Per-Token, Slower Overall
| Metric | NPU | GPU |
|---|---|---|
| Avg total latency | 5,062 ms | 4,855 ms |
| Avg total tokens generated | 195 | 91 |
| Avg input length | 208 chars | 208 chars |
| Peak RSS | 1,934 MB | 1,375 MB |
Despite being ~1.7x faster per token, the NPU's average total latency is slightly higher than GPU: 5.1 seconds vs 4.9 seconds. And it uses 560 MB more memory.
The root cause is in the token counts. Look at the per-step breakdown:
| Step | NPU avg tokens | GPU avg tokens | Ratio |
|---|---|---|---|
| Step 1 (Classify) | 10 | 10 | 1.0x |
| Step 2 (Detect) | 22 | 30 | 0.7x |
| Step 3 (Redact) | 163 | 51 | 3.2x |
Step 3 is the problem. The NPU generates 3.2x more tokens on average for the redaction step.
Why? The NPU backend uses samplerConfig = null because constrained decoding is unsupported on the Hexagon Tensor Processor (it returns error 12: "not supported"). The GPU uses topK=64, topP=0.95, temperature=1.0, which constrains the output distribution and keeps responses concise. Without any sampling constraints, the NPU's default behavior is verbose: it rambles, adds explanations, repeats content. This is not a vendor failing so much as two independent design choices colliding: the AoT-compiled NPU path and the app's sampling expectations were built against different assumptions about what the runtime supports.
This is a runtime limitation, not a hardware limitation. The Hexagon V79 is doing its job perfectly, decoding tokens at roughly 42 tok/s consistently across all three steps. But because the software stack cannot apply sampling constraints on-NPU, the model generates far more tokens than necessary to complete the task.
Per-Step Latency Flow
The following diagram shows how latency distributes across the three benchmarked pipeline steps, and why the NPU's Step 3 erases its earlier advantage:
The NPU enters Step 3 with a 1,390 ms lead (969 ms vs 2,359 ms cumulative through Steps 1-2). But Step 3's verbosity costs 4,060 ms vs 2,475 ms, turning a 1.4-second advantage into a 200 ms deficit.
The Mode Breakdown: Where NPU Wins and Loses
Not all redaction modes behave the same. Here is the per-mode average total latency:
| Mode | Entries | NPU avg | GPU avg | Winner |
|---|---|---|---|---|
| HIPAA | 4 | 1,977 ms | 4,160 ms | NPU by 2.1x |
| FINANCIAL | 4 | 2,360 ms | 4,734 ms | NPU by 2.0x |
| FIELD_SERVICE | 5 | 2,325 ms | 5,284 ms | NPU by 2.3x |
| JOURNALISM | 10 | 2,348 ms | 4,564 ms | NPU by 1.9x |
| TACTICAL | 7 | 14,201 ms | 5,430 ms | GPU by 2.6x |
Four out of five modes: NPU wins decisively, often by 2x or more. These modes produce relatively short redaction outputs, so the verbosity penalty stays manageable.
TACTICAL mode is the outlier. NPU averaged 14.2 seconds, nearly 3x slower than GPU. One or more tactical entries triggered extremely long LLM output on the NPU due to unconstrained sampling. This is the worst-case scenario for the samplerConfig = null limitation: when the model decides to be verbose on already-complex inputs (police reports, incident descriptions with many entities), there is no mechanism to rein it in. There is a hard ceiling in play - the deployed build was compiled with a 1024-token KV cache (the app requested maxNumTokens=4000, but the compiled 1024 cache wins) - so verbosity is bounded, but 1024 tokens of unconstrained output is still enough to blow past GPU's total time.
The takeaway: NPU verbosity is not uniformly distributed. It is input-dependent and can spike dramatically on certain content types.
Single-Entry Deep Dive: hipaa_001
To isolate from averaging effects, here is one specific entry, hipaa_001, a 229-character medical text:
| Metric | NPU | GPU | NPU vs GPU |
|---|---|---|---|
| Total latency | 2,781 ms | 5,646 ms | 2.0x faster |
| Step 1 (Classify) | 757 ms | 955 ms | 1.3x faster |
| Step 2 (Detect) | 694 ms | 2,258 ms | 3.3x faster |
| Step 3 (Redact) | 1,309 ms | 2,413 ms | 1.8x faster |
| TTFT (Step 2) | 159 ms | 424 ms | 2.7x faster |
| Decode tok/s (Step 3) | 44.9 | 24.3 | 1.8x faster |
| Total tokens | 83 | 122 | NPU fewer |
| Peak RSS | 1,684 MB | 1,134 MB | GPU uses 550 MB less |
On this particular entry, the NPU wins across the board, 2.0x faster overall. Notably, the NPU generated fewer total tokens (83 vs 122) on this input. The verbosity problem is statistical, not deterministic. On short medical texts, the unconstrained sampler happens to behave well.
This is what NPU performance looks like when the token count cooperates: sub-3-second end-to-end for a three-step LLM pipeline. That is genuinely impressive for on-device inference, with the usual caveat that this is one entry from one session.
Memory: The Hidden Cost
| Metric | NPU | GPU |
|---|---|---|
| Peak RSS (30-entry bench) | 1,934 MB | 1,375 MB |
| Peak RSS (single entry) | 1,684 MB | 1,134 MB |
The NPU consistently uses ~560 MB more memory. Two factors contribute: the NPU-compiled model file is larger (3.02 GB vs 2.59 GB), and the QNN runtime allocates additional buffers for the Hexagon Tensor Processor communication pathway.
On a flagship with 12+ GB RAM, this is manageable. On a mid-range device with 6-8 GB, that extra 560 MB could be the difference between running and OOM-killing.
CPU: The Reliable Fallback
I did not run the full 30-entry benchmark on CPU for this comparison (CPU is the default fallback, not the focus). From the engineering data: CPU inference works on every device, requires no special libraries or DSP path configuration, uses the smallest memory footprint, and runs at roughly 4-6 tok/s, well under a quarter of the GPU's decode speed. It is the universal compatibility option: slowest, but it always works.
For reference, the previous single-pass pipeline on GPU averaged 12.8 tok/s with the standard model. This pipeline's GPU decode speed of ~25 tok/s reflects both the model upgrade and pipeline architecture changes, so these are not directly comparable.
What This Means for Developers
If you are building an on-device LLM feature and choosing between NPU, GPU, and CPU backends, here is what this data suggests:
Choose NPU when:
- Time-to-first-token matters for UX (sub-100ms vs 350ms+ is perceptible)
- Your outputs are short (classification, yes/no, short extractions)
- You can tolerate occasional verbosity spikes
- You are targeting flagship hardware with ample RAM
Choose GPU when:
- Output length consistency matters (sampling constraints available)
- Total wall-clock time matters more than TTFT
- Memory pressure is a concern (550 MB savings)
- You need constrained decoding (structured JSON, tool calls)
Choose CPU when:
- You need universal device compatibility
- NPU/GPU initialization fails (QNN library issues, driver mismatches)
- You are building a fallback path (and you should always build a fallback path)
The honest answer is that NPU vs GPU is not a clear winner today. The NPU hardware is genuinely faster: ~1.7x decode speed and 3.6-4.0x TTFT were real advantages in this session. But the software stack has a gap: no sampling control on-NPU means you cannot control output length, and that single limitation can erase the hardware advantage on certain workloads.
If constrained sampling support arrives for the Hexagon Tensor Processor, this calculus changes. A ~42 tok/s NPU with proper top-k/top-p sampling would likely win across the workloads I tested against a ~25 tok/s GPU. That is the future. Today, you need to test on your specific prompts and decide.
Methodology Note
All numbers in this post come from Redacto's benchmark suite, measured in a single session on 2026-05-01 on a Samsung Galaxy S25 Ultra running Android 16. No raw logs were retained from that run, and the device is no longer available to me, so these are directional results from one session, not a reproducible multi-run study. The benchmark runs 30 entries through the first three pipeline steps (Classify, Detect, Redact) with no validation pass, triggered via ADB broadcast receiver. Latency is wall-clock System.currentTimeMillis(), TTFT is first callback timestamp, decode speed excludes prefill, and memory is kernel-reported VmRSS. Full methodology is documented in the project's benchmark-results.md.
This work was part of Redacto, which my team Edge Artists built for the Qualcomm x Google LiteRT Hackathon 2026.
Related in this series of "Edge AI from the Trenches"
- Why My LLM Runs 4x Faster on Hardware I Had Never Heard Of: foundational context on the hardware being compared
- What On-Device AI Benchmarks Actually Feel Like: definitions and human-perception thresholds for the metrics used here
- One Model, Three Chips, Two Files: How LiteRT Delegates Really Work: how LiteRT routes inference to different backends
Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware. Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.
Last updated: July 2026
17th of 23 posts in the "Edge AI from the Trenches" series
Top comments (0)