When I started benchmarking Redacto - my on-device PII redaction app running Gemma 4 E2B on a Samsung Galaxy S25 Ultra - I wanted to measure everything. Latency, throughput, memory, battery drain, GPU utilization, NPU load, energy consumption in joules. The full picture.
I ended up measuring five things.
Not because I got lazy. Because I adopted a principle early in the project and refused to violate it: if I can't measure it properly, I don't show it. This post is about what that principle looks like in practice: which metrics survive contact with the Android runtime, which ones don't, and how to build a benchmarking system that tells you the truth instead of telling you what you want to hear.
A note on the numbers in this post. Every precise benchmark figure below comes from a single directional benchmarking session on a Galaxy S25 Ultra in May 2026. I did not save raw logs, and the device is no longer available to me. Treat these numbers as directional evidence of what the metrics reveal, not as a rigorous multi-run study. The point of the post is the measurement methodology, not the specific values.
The honesty problem in mobile AI benchmarking
The cloud benchmarking ecosystem is mature. You load a model, run an eval harness, and get structured results. Tools like EleutherAI's lm-eval-harness, Stanford's HELM, and MLPerf Inference provide standardized frameworks with reproducible methodologies. Profiling tools (PyTorch Profiler, NVIDIA Nsight) give you per-layer timing, memory allocation traces, and GPU utilization down to the individual streaming multiprocessor (SM), the cluster of cores that is the basic compute unit inside an NVIDIA GPU. You can see whether each one is saturated or sitting idle.
None of this maturity carries over cleanly to Android. Some vendor and platform tooling does exist (Perfetto for system tracing, Android GPU Inspector, Qualcomm's Snapdragon Profiler), but there is no single, reproducible, app-level harness the way there is in the cloud. And honestly, inside a hackathon I did not have the time to hunt down, install, and wire up every vendor tool and figure out which one could give me clean per-inference numbers. So my natural instinct kicked in: build what I needed myself, out of the raw primitives I was already comfortable with.
On-device benchmarking is a different discipline. You are working inside a constrained operating system that was designed for user-facing apps, not scientific measurement. The general-purpose APIs that are easy to reach are coarse-grained, system-wide, and often misleading. The measurements I actually needed, per-process power draw, accelerator utilization, structured inference metrics, are either absent from the public SDK or locked behind tethered vendor profilers that do not fit into an automated, on-device run.
MLPerf Mobile is the closest thing to a standard, but it measures single-model inference latency on standardized tasks (image classification, object detection, language models). It does not address application-level pipelines where a single user action triggers three or four sequential LLM calls, each with different prompts, different expected output lengths, and different hardware characteristics. MLPerf gives you component-level numbers. Building an app requires system-level numbers.
So I had to build my own system, and I had to be honest about what it could and could not tell me.
Metrics that work
These are the five metrics I kept in the app. Each one has a clear measurement methodology, a known confidence level, and a direct connection to something the user or developer cares about.
1. Wall-clock latency (HIGH confidence)
The simplest and most reliable metric. Wrap System.currentTimeMillis() around the full engine.infer() call:
val startMs = System.currentTimeMillis()
engine.infer(prompt, callback)
val latencyMs = System.currentTimeMillis() - startMs
This captures everything: tokenization, prefill, decode, and any runtime overhead. It is the number the user actually experiences: the time between pressing "Redact" and seeing the result.
Real numbers from 30 easy entries, 3-step pipeline (Classify, Detect, Redact):
| Backend | Avg Total Latency | Step 1 | Step 2 | Step 3 |
|---|---|---|---|---|
| GPU | 4,855ms | 773ms | 1,586ms | 2,475ms |
| NPU | 5,062ms | 345ms | 624ms | 4,060ms |
The surprise: NPU total latency is slightly worse than GPU despite faster per-token speed. The reason is that the NPU path cannot use constrained decoding (SamplerConfig must be null), so it generates 3.2x more tokens on Step 3 (163 avg vs GPU's 51). Faster per-token speed multiplied by more tokens equals roughly the same wall-clock time. Without measuring both wall-clock and per-step token counts, you would never see this.
2. Time to first token - TTFT (HIGH confidence)
TTFT is the time between starting inference and receiving the first token via the streaming callback. This is a UX metric: it is how long the user stares at a blank screen.
var firstTokenTime: Long? = null
val startMs = System.currentTimeMillis()
engine.infer(prompt) { token ->
if (firstTokenTime == null) {
firstTokenTime = System.currentTimeMillis()
}
// accumulate token...
}
val ttftMs = (firstTokenTime ?: startMs) - startMs
Real numbers:
| Backend | Step 1 TTFT | Step 2 TTFT | Step 3 TTFT |
|---|---|---|---|
| GPU | 381ms | 375ms | 366ms |
| NPU | 99ms | 104ms | 92ms |
NPU TTFT is 3.6-4.0x faster than GPU. This is NPU's clearest advantage: the user sees response text appearing almost instantly (sub-100ms feels immediate). GPU's ~370ms is noticeable but not painful. Anything above 500ms starts to feel slow.
TTFT measures the prefill phase: the time the model spends processing the input prompt before generating the first output token. It scales with prompt length, which matters for Redacto because the system prompts are substantial (category-specific prompt sets, each several hundred tokens).
3. Decode throughput - tok/s (HIGH confidence)
How fast tokens arrive after the first one. The formula excludes the prefill phase:
decode_tok_s = (tokenCount - 1) * 1000 / (lastTokenTime - firstTokenTime)
The - 1 is important: it counts inter-token intervals, not tokens. If you have 10 tokens and measure from first to last, there are 9 intervals. Getting this wrong inflates your numbers by 10-20% depending on token count.
Real numbers (averaged across 30 entries):
| Backend | Step 1 tok/s | Step 2 tok/s | Step 3 tok/s |
|---|---|---|---|
| GPU | ~25 | ~25 | ~25 |
| NPU | ~42 | ~42 | ~42 |
NPU is consistently around 1.6-1.7x faster on raw token throughput. GPU decode speed drifts slightly downward across steps while NPU stays essentially flat. The flat NPU rate is expected rather than surprising: NPU decode is memory-bandwidth-bound, so once the model weights are streaming through at the memory ceiling, each step decodes at roughly the same rate regardless of how many tokens it produces. The small GPU drift is harder to explain with confidence - it could be mild thermal throttling on the Adreno GPU during a sustained multi-step run, but I cannot measure GPU temperature or clock frequency via public APIs, so that is an inference from observed behavior, not a measured cause.
4. Output token count (MEDIUM confidence)
Increment a counter each time the onMessage callback fires:
var tokenCount = 0
engine.infer(prompt) { token ->
tokenCount++
}
Confidence is MEDIUM because I am assuming a 1:1 mapping between callbacks and tokens. The LiteRT-LM streaming API does not guarantee this: a single callback could theoretically deliver multiple tokens, or a partial token. In this session the counts were consistent with expected output lengths, so I trust the number but flag the assumption.
Why token count matters: It exposed the NPU verbosity problem. Without this metric, NPU would look strictly superior (faster TTFT, higher tok/s). Token counts revealed that NPU Step 3 generates 163 tokens on average versus GPU's 51: a 3.2x ratio that explains why NPU total latency is not 1.7x better despite 1.7x faster decoding.
5. Peak RSS memory (HIGH confidence)
Read directly from the kernel:
fun getPeakRssKb(): Long {
val status = File("/proc/self/status").readText()
val vmRssLine = status.lines().find { it.startsWith("VmRSS:") }
return vmRssLine?.split("\\s+".toRegex())?.get(1)?.toLong() ?: -1
}
/proc/self/status is a kernel-provided pseudo-file. VmRSS (Virtual Memory Resident Set Size) is the actual physical memory the process is using, not virtual address space. This is reliable because it is what the kernel uses internally for memory accounting and OOM decisions.
Real numbers:
| Backend | Peak RSS |
|---|---|
| GPU | 1,375 MB |
| NPU | 1,934 MB |
NPU uses 559 MB more. The NPU model file itself is larger (3.02 GB vs 2.59 GB, because the NPU build is compiled ahead-of-time through QNN), and the Qualcomm QNN runtime allocates additional buffers for the DSP communication path. This is not a measurement artifact: it is a real constraint. On a device with 12 GB RAM, NPU inference leaves less headroom for the OS, background apps, and your own app's non-inference workload.
Metrics that don't work (and why we removed them)
These are the metrics I wanted to include but couldn't, because the available Android APIs do not produce per-process, per-inference numbers.
Battery current draw: system-wide, not per-process
The obvious API is BatteryManager.BATTERY_PROPERTY_CURRENT_NOW:
val batteryManager = getSystemService(Context.BATTERY_SERVICE) as BatteryManager
val currentMicroAmps = batteryManager.getIntProperty(
BatteryManager.BATTERY_PROPERTY_CURRENT_NOW
)
This returns the instantaneous current draw of the entire device. Not per-process. Not per-component. The screen, the cellular radio, background services, the OS itself: all included. You cannot attribute any portion of this reading to your model inference.
I tried the naive approach: sample before inference, sample after, compute the delta. The problems are fundamental:
- It is instantaneous, not averaged. A single sample before and after tells you the current at two points in time. Inference changes current draw continuously over 2-8 seconds. Two point samples cannot characterize a curve.
- System noise dominates. A notification arriving, the display auto-brightness adjusting, or a background sync starting during your inference window can swing the reading by hundreds of milliamps.
- Current direction varies by charging state. Some devices report negative values when charging. Some report unsigned values. The sign convention is not standardized across OEMs.
Early benchmark runs reported numbers like "101 mA for standard model, 301 mA for fine-tuned model." These numbers were plausible: the fine-tuned model is larger and should draw more power. But plausible is not the same as correct. A plausible-but-unreliable metric is worse than no metric, because people will cite it.
We removed it.
Energy in joules: derived from garbage
If current draw is unreliable, then energy (which is derived from current draw) is doubly unreliable:
energy_joules = current_amps * voltage * duration_seconds
We used an assumed 3.7V (nominal lithium battery voltage). This is wrong on multiple levels:
- Actual battery voltage ranges from ~3.3V (nearly dead) to ~4.4V (fully charged). Using a constant 3.7V introduces up to 19% error before you even consider the current measurement problems.
- The current measurement is system-wide (see above), so the energy calculation attributes the entire device's power consumption to inference.
- Duration is the only reliable term in the equation. Multiplying one good number by two bad numbers gives you a bad number.
We removed it.
GPU/NPU utilization: no public API
There is no public Android API to query GPU or NPU utilization. On desktop Linux, you can read /sys/class/drm/card0/device/gpu_busy_percent or use vendor tools (nvidia-smi, rocm-smi). On Android with Qualcomm silicon:
- GPU: Qualcomm's GPU profiling tools (Snapdragon Profiler) require a USB-tethered connection and root-level access on many devices. There is no runtime API an app can call.
- NPU: The Hexagon DSP has no utilization API whatsoever in the public SDK. You cannot query whether the NPU is 10% busy or 100% busy during your inference.
This means you cannot answer basic questions like "is my inference compute-bound or memory-bound?" or "is the NPU fully utilized or is my model too small to saturate it?" These are questions that a desktop ML engineer answers in seconds with nvidia-smi. On Android, they are unanswerable without vendor tooling and physical device access.
I did not try to estimate utilization from inference timing. An estimate based on "the NPU should be able to do X tok/s at peak, I measured Y, so utilization is Y/X" would require knowing the theoretical peak, which depends on the model architecture, quantization scheme, and memory bandwidth, none of which are exposed.
I left it out entirely.
Building the benchmark system
Knowing what to measure is half the problem. The other half is running the measurements in a way that is automated, reproducible, and does not interfere with the thing being measured.
The ADB-triggered approach
Our benchmark system is triggered via Android's BroadcastReceiver, invoked from ADB:
# Run text benchmark, 30 easy entries, GPU backend:
adb shell am broadcast \
-n com.example.redacto/com.example.redacto.benchmark.BenchmarkReceiver \
-a com.example.redacto.BENCHMARK \
--es type text \
--ei count 30 \
--es difficulty easy
# Same benchmark on NPU:
adb shell am broadcast \
-n com.example.redacto/com.example.redacto.benchmark.BenchmarkReceiver \
-a com.example.redacto.BENCHMARK \
--es type text \
--ei count 30 \
--es difficulty easy \
--es backend NPU
# Monitor results in real time:
adb logcat -s RedactoBenchmark:I
Example logcat output, formatted the way the runner prints it. The per-step values shown here are the NPU averages across the 30-entry run rather than any single captured entry (individual entries vary; I did not retain per-entry logs), so read this as an illustration of the log format and the summary line, not as one real entry:
RedactoBenchmark: [n/30] <entry_id> | Step1: 345ms (TTFT 99ms, 10 tok, ~42 tok/s)
RedactoBenchmark: [n/30] <entry_id> | Step2: 624ms (TTFT 104ms, 22 tok, ~42 tok/s)
RedactoBenchmark: [n/30] <entry_id> | Step3: 4060ms (TTFT 92ms, 163 tok, ~42 tok/s)
RedactoBenchmark: [n/30] <entry_id> | TOTAL: ~5062ms | Tokens: ~195 | RSS: 1934MB
...
RedactoBenchmark: === SUMMARY ===
RedactoBenchmark: Entries: 30 | Avg latency: 5062ms | Avg tokens: 195 | Peak RSS: 1934MB
Why BroadcastReceiver instead of a separate test app? This is the critical design decision. A separate benchmarking app would need to create its own InferenceEngine instance and load the 2.59 GB model into memory. With the main app already running and holding a loaded model, that means two models in RAM simultaneously: roughly 5.2 GB just for inference weights. On a device with 12 GB total RAM, that is an OOM crash waiting to happen.
Instead, the BroadcastReceiver invokes a callback on the running app's ViewModel, which shares the already-initialized engine. One model in memory, zero duplication, no interference with the existing app state.
The tradeoff: the app must be in the foreground. The callback registration happens in Compose's lifecycle, so if the app is backgrounded, the receiver fires but the callback is not registered, and nothing happens. This means you cannot run benchmarks in a pure headless mode. For a hackathon project, this is acceptable. For a production benchmark suite, you would want to decouple the engine lifecycle from the UI framework.
Benchmark scope: Steps 1-3 only
The performance benchmark runs Classify, Detect, and Redact: the first three steps of the four-step pipeline. It does not run Step 4 (Validate).
This is deliberate. The benchmark is a performance test, not a quality test. I want to measure inference speed across backends, modes, and difficulty levels. Validation adds variable retry rounds (up to 2 retries) that make latency comparisons noisy. An entry that passes validation in round 1 takes ~300ms for Step 4. An entry that fails twice and succeeds on round 3 adds ~900ms of variable overhead. Including this in performance numbers would obscure the signal I care about: how fast is the inference engine itself?
Quality is measured separately, using the scoring methodology described below.
Dataset design
A benchmark is only as good as its dataset. Garbage in, garbage out, but also easy in, flattering out. If your test entries are trivially simple, your benchmark will produce impressive numbers that do not reflect real-world performance.
Structure
The redacto_bench.jsonl dataset contains 85 entries:
| Mode | Total | Easy | Medium | Hard |
|---|---|---|---|---|
| HIPAA | 17 | 4 | 4 | 4+5 |
| FINANCIAL | 17 | 4 | 4 | 4+5 |
| FIELD_SERVICE | 17 | 5 | 5 | 2+5 |
| TACTICAL | 17 | 7 | 5 | 0+5 |
| JOURNALISM | 17 | 10 | 2 | 0+5 |
(Plain numbers are the automated curation. The +5 in the hard column is the hand-crafted contextual entries: 5 per mode, all rated hard, which is why they land only in that column. That accounts for the full 25 hand-crafted entries; the other 60 are automated. Totals: 30 easy, 20 medium, 35 hard.)
Why text-only?
The dataset is pure text: no images, no OCR. This is an isolation decision. OCR (via ML Kit) adds variable latency of 200-1500ms depending on image complexity, resolution, and text density. Including OCR in the benchmark would make it impossible to distinguish "the model is slow" from "the image was hard to OCR." I measure the pipeline, not ML Kit.
Two-phase curation
Phase 1 - Automated (60 entries): I started with the ai4privacy/pii-masking-400k dataset (17,046 English entries with labeled PII spans). Quality filtering reduced this to roughly 2,000 candidates using criteria: longer than 80 characters, shorter than 600, at least 3 PII entities, no synthetic-looking names, PII density below 60%. Mode assignment used keyword matching (medical terminology maps to HIPAA, financial terms to FINANCIAL) and entity composition (entries with CREDITCARDNUMBER or TAXNUM map to FINANCIAL). Final sampling: 12 per mode, balanced across difficulty levels.
Phase 2 - Hand-crafted (25 entries): Five per mode, targeting contextual reasoning that generic PII data cannot test:
- HIPAA: Relational identifiers ("the patient's daughter Lisa"), contextual PHI (a depression diagnosis linked to an identified patient), medication dosages tied to named patients.
- TACTICAL: Distinguishing what to keep (suspect descriptions: race, height, clothing, vehicle, license plate) from what to redact (victim and witness names). This is the hardest category because the model must understand roles, not just entity types.
- JOURNALISM: Keep public officials (real names like Lloyd Austin, Elizabeth Warren) while redacting confidential source identities. Tests whether the model understands the difference between public and private individuals.
- FIELD_SERVICE: Contextual security information beyond passwords: "key is under the mat," "back door is usually unlocked." These are natural-language security risks that regex will never catch.
- FINANCIAL: Preserve dollar amounts and institution names while redacting account numbers, SSNs, and routing numbers. Tests entity boundary precision: the model must know where the institution name ends and the account number begins.
Why 85 entries?
On-device GPU inference takes 5-10 seconds per entry for the 3-step pipeline. 85 entries at ~7 seconds average is roughly 10 minutes per backend: 20 minutes total for GPU vs NPU. Long enough to cover 5 modes and 3 difficulties in one run. Short enough that you can iterate within a hackathon session. I also added a slider (1-85) to run quick 3-5 entry sanity checks during development.
Scoring methodology
Scoring measures redaction quality, which is separate from performance. The two are complementary: performance tells you how fast the model runs, scoring tells you how well it redacts.
Three metrics, weighted
Entity recall (weight: 50%): For each ground-truth PII entity, check whether the original value is absent from the model's output. This is intentionally lenient: it does not require exact placeholder format, just that the PII was removed in some form. If the ground truth says "Jane Smith" is PII and the model's output contains neither "Jane" nor "Smith," entity recall counts that as a hit.
Why 50% weight: missing a PII entity is the worst failure mode. An app that leaves a Social Security number visible has failed at its core purpose.
Format compliance (weight: 25%): Fraction of bracket placeholders in the output that match the [CATEGORY_N] pattern (e.g., [NAME_1], [SSN_2]). This distinguishes models that use Redacto's structured format from those that output generic [REDACTED] tags. Format matters because downstream consumers (audit logs, compliance reports) need to know what type of PII was removed, not just that something was removed.
Text preservation (weight: 25%): Fraction of non-PII words from the input that are retained in the output. This catches over-redaction: a model that replaces the entire document with [REDACTED] would score 100% on entity recall and 0% on preservation. "Left heel wound not improving" should survive redaction intact. The diagnosis is medical context, not PII.
Overall score: entityRecall * 0.5 + formatScore * 0.25 + preservationScore * 0.25
Why not exact string match?
The naive approach is to compare the model's output character-by-character against the expected output. This fails on minor formatting differences: an extra space after a comma, slightly different placeholder numbering ([NAME_2] vs [NAME_1] for the same entity detected in different order), line breaks versus spaces. Exact match produces false negatives that penalize correct redactions.
Entity-level scoring is more robust because it measures what actually matters: was the PII removed? It accepts any method of removal: the right placeholder, a different placeholder, or even deletion of the surrounding sentence. The format compliance metric separately penalizes non-standard placeholders, so sloppy redaction that removes the PII but uses wrong formatting still loses points, just not on the recall dimension.
What the numbers actually told us
With honest metrics and a carefully built dataset, even a single directional session revealed things that informal testing never would have:
NPU is not strictly faster than GPU. On a per-token basis, yes: roughly 42 vs 25 tok/s. But because the NPU path cannot use constrained decoding, it generates so many more tokens that total latency is roughly equal (5,062ms vs 4,855ms averaged across 30 entries). The TACTICAL mode is the extreme case: NPU averaged 14,201ms versus GPU's 5,430ms, because one or more entries triggered runaway token generation.
TTFT is NPU's real advantage. Sub-100ms time to first token versus ~370ms on GPU. For a user-facing app, this is the difference between "the app responded instantly" and "there was a noticeable pause." If your app streams output tokens to the UI, NPU TTFT alone may justify the higher memory cost.
A good prompt beat an under-resourced fine-tune. The prompt-driven standard model scored 80.5% overall versus 70.3% for the fine-tuned model. But that comparison is not fair to fine-tuning: the fine-tune was deliberately under-resourced (roughly 3,000 of the 400,000 ai4privacy samples, one epoch, a few minutes of training) and it learned the wrong output format ([REDACTED] instead of Redacto's [CATEGORY_N]), which cost it on format compliance directly. Even so, it won on FIELD_SERVICE entity recall (+13.2%) and TACTICAL entity recall (+13.1%), while the standard model won decisively on HIPAA (+55.8%) and text preservation (+17.8%). The overall score hides these per-mode stories. Without per-mode breakdowns, you would conclude "fine-tuning did not help," which is wrong: both the quantity and the alignment of the training data were off, and with the full dataset and the right target format this result could easily have flipped.
Memory is a hard constraint, not a preference. NPU uses 1,934 MB. GPU uses 1,375 MB. On a 12 GB device with the OS and background apps consuming 4-5 GB, NPU leaves roughly 5-6 GB free. That sounds comfortable until you remember that Android's Low Memory Killer starts getting aggressive well before you hit zero. In practice, NPU inference causes more frequent background app kills, which degrades the user experience in ways that a benchmark does not capture.
What I wish existed
Building this system ate up a large chunk of a hackathon where every hour counted. Here are the walls I hit, and what I wish had been within easy reach for an app developer. I am framing these as gaps I ran into, not as proof the ecosystem lacks them everywhere, because part of the point below is that I did not have time to check exhaustively.
Per-process power attribution. iOS exposes per-process energy metrics via MetricKit. I could not find an equivalent an app can call on Android. Without it, I do not see how an on-device AI benchmark on Android makes credible power-efficiency claims.
Accelerator utilization an app can read. Even a simple "NPU is X% busy" would let developers answer basic optimization questions. From inside my app, the NPU was effectively a black box: I found no runtime API to query it.
Structured inference metrics from LiteRT-LM. Currently, timing data comes from wrapping calls in
System.currentTimeMillis()and counting callbacks. The runtime knows its own internal timing (tokenization duration, prefill duration, KV cache hits) and could expose it. I would rather read that directly than parse logcat output for benchmark data.A standard app-level, on-device eval framework. An Android equivalent of lm-eval-harness that handles engine lifecycle, dataset loading, metric computation, and result formatting for a multi-step pipeline, so that every team does not reinvent this infrastructure from scratch. Single-model benchmark tools exist; what I wanted was something pitched at the application pipeline, not one model invocation.
CI/CD integration for device testing. Run benchmarks on device farms (Firebase Test Lab, Samsung Remote Test Lab) as part of a pull request pipeline. For me, on-device benchmarking stayed a manual, one-device-at-a-time process.
A fair caveat on this wishlist: some of these gaps may already be partly filled by tools I simply did not have time to evaluate during the hackathon. LiteRT itself ships a command-line benchmark tool, and there are vendor profilers I never wired up. I do not yet know how much of the multi-step, app-level, on-device measurement problem they actually solve. That is a project in itself, and it is the one I am planning next: to survey the existing on-device benchmarking tools properly, map what they can and cannot do for an application pipeline like this one, and see whether there is a real, unmet gap worth building for. If there is, that will be its own standalone post in this series.
The honesty principle, applied
The principle - "if I can't measure it properly, I don't show it" - sounds obvious. In practice, it means throwing away metrics that would make your demo more impressive. Battery current draw of "101 mA for standard, 301 mA for fine-tuned" is a great story for a hackathon presentation. It suggests the standard model is 3x more power-efficient. Judges would be impressed. And it would be misleading.
The discipline is not in measuring accurately. The discipline is in admitting what you cannot measure and choosing not to display it. Every on-device AI project I have seen, including published papers, reports at least one metric that does not survive this scrutiny. Usually it is power. Sometimes it is "NPU utilization." Occasionally it is throughput numbers that include prefill in the tok/s calculation (inflating the number by 20-30%).
If you are building on-device AI benchmarks, start with the question: "What can I actually measure, and at what confidence level?" Build your metrics table from the answers. Leave the empty cells empty.
The numbers you do not report are as important as the numbers you do.
All benchmark data here comes from a single directional session on the Redacto project running on a Samsung Galaxy S25 Ultra (Snapdragon 8 Elite, SM8750) with Gemma 4 E2B in May 2026. No raw logs were retained and the device is no longer available, so the numbers are directional, not a rigorous multi-run study. For the foundations behind the metrics mentioned here (TTFT, tok/s, quantization), see What On-Device AI Benchmarks Actually Feel Like.
Related in this series of "Edge AI from the Trenches"
- What On-Device AI Benchmarks Actually Feel Like - foundational definitions of the metrics measured here (TTFT, tok/s)
- Prompt Engineering Beat My Fine-Tuned Model. Here Is Why. - the decision framework behind model selection for benchmarks
- How I Benchmarked an LLM Running Entirely on a Phone (No Cloud, No API) - the dataset curation and scoring methodology in full detail (upcoming in this series)
Device: Samsung Galaxy S25 Ultra (Snapdragon 8 Elite SM8750, 12 GB RAM)
Model: Gemma 4 E2B Instruct (GPU: 2.59 GB, NPU: 3.02 GB)
Runtime: LiteRT-LM
Pipeline: 3-step (Classify, Detect, Redact); no validation step for performance runs
Dataset: 85 entries (60 curated + 25 hand-crafted), 5 modes, 3 difficulty levels
Benchmark session: single directional run, May 2026; no raw logs retained
Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware. Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.
Last updated: July 2026
14th of 23 posts in the "Edge AI from the Trenches" series
Top comments (0)