DEV Community

Jaydeep Shah (JD)
Jaydeep Shah (JD)

Posted on

How I Benchmarked an LLM Running Entirely on a Phone (No Cloud, No API)

#edgeai #android #litertlm #benchmarking

"It works on my test input" is the most dangerous sentence in on-device AI development.

I typed that sentence - or some version of it - a dozen times while building Redacto, our on-device PII redaction app running Gemma 4 E2B on a Samsung Galaxy S25 Ultra. The model would redact a patient name from a clinical note, I would nod, and I would move on. Then I would hand the phone to a teammate, they would type a police report, and the model would redact the suspect description instead of the victim name.

The problem is not the model. The problem is that manual spot-checking is not validation. You are testing a single input against your own expectations, with all the confirmation bias that entails. When you have five domain modes (HIPAA, Financial, Tactical, Journalism, Field Service), three difficulty levels, and two candidate models, you need something systematic. You need a benchmark suite.

This post covers how I built one - from dataset curation to scoring methodology to on-device infrastructure - for a hackathon app running entirely on a phone. No cloud. No API calls. No data leaving the device.

Why Not Use an Existing Framework?

The LLM evaluation space has mature tools. EleutherAI's lm-eval-harness is the community standard for evaluating language models against academic benchmarks like MMLU, HellaSwag, and ARC. Stanford's HELM (Holistic Evaluation of Language Models) provides a multi-metric evaluation framework with standardized scenarios. Google's BIG-bench offers hundreds of tasks for probing specific capabilities.

These frameworks are excellent for what they do. They are also completely wrong for this problem, for three reasons.

First, they assume server-side inference. lm-eval-harness expects to call a model through an API or load it in PyTorch on a GPU server. Redacto's model runs on a Qualcomm Hexagon NPU inside a phone. There is no Python runtime, no HuggingFace tokenizer at evaluation time, no way to hook into the framework's inference loop.

Second, their task formats do not match. Standard benchmarks test general capabilities: multiple choice, short answer, classification. Redacto's task is structured text transformation: take an input containing PII, produce an output with [CATEGORY_N] placeholders in the right positions while preserving all non-PII text. No existing benchmark tests this.

Third, they do not capture what matters. Even if you could somehow run MMLU on-device, a high MMLU score tells you nothing about whether the model will correctly identify that "key is under the mat" is a security risk in a field service context but not in a journalism article. Domain-specific behavior requires domain-specific evaluation.

So I built one from scratch. The entire system - dataset, scorer, runner, UI - came together inside the hackathon and fits in a handful of Kotlin files plus one JSONL asset.

Phase 1: Dataset Curation from ai4privacy

The foundation is data. I needed text samples containing realistic PII, with ground-truth labels identifying exactly which spans are personally identifiable and which category each belongs to.

Source Selection

I used the ai4privacy/pii-masking-400k dataset from HuggingFace, which contains 17,046 English entries with labeled PII spans and masked outputs. Each entry includes the original text, the masked version, and a list of entities with their positions, values, and categories.

This dataset has a critical advantage: it was created from real-world text patterns (job applications, medical forms, financial documents), not synthetically generated. The PII values are synthetic, but the surrounding context is natural. A benchmark needs to test whether the model understands context, not just whether it can regex-match a Social Security Number format.

Quality Filtering

Not all 17,046 entries are usable. I applied five quality filters, reducing the pool to roughly 2,000 candidates:

  1. Length bounds (>80 chars, <600 chars). Entries shorter than 80 characters do not provide enough context for mode assignment. Entries longer than 600 characters exceed what a user would realistically type or photograph on a phone, and they consume excessive inference tokens.

  2. Minimum entity count (3+). Single-entity entries ("John Smith lives here") are trivial. A useful benchmark entry should require the model to identify and track multiple PII elements across a passage.

  3. No synthetic-looking names. The ai4privacy dataset generates some names that read as obviously fake (e.g., "Xylophonia Quartzmaven"). These are easy for any model to flag as names, which defeats the purpose of testing entity detection. I filtered entries where names had character patterns inconsistent with real-world naming conventions.

  4. PII density below 60%. Some entries are almost entirely PII: a list of names and addresses with no surrounding context. These test nothing about context preservation. If 60% or more of the text is PII, the model cannot meaningfully fail at preservation because there is almost nothing to preserve.

  5. Parseable entity annotations. Entries with malformed or overlapping span annotations were excluded (a small number of entries).

Mode Assignment

The ~2,000 quality candidates need to be assigned to one of Redacto's five domain modes. This is not labeled in the source dataset, so I built a two-signal classifier:

Signal 1: Keyword matching. Medical terms (diagnosis, patient, prescription, medication) map to HIPAA. Financial terms (account, routing, balance, transaction) map to FINANCIAL. Security terms (password, gate code, access code, WiFi) map to FIELD_SERVICE. Law enforcement terms (suspect, witness, victim, incident) map to TACTICAL. Source/reporter/official terms map to JOURNALISM.

Signal 2: Label composition. The entity categories present in an entry provide a strong signal. If an entry contains CREDITCARDNUMBER or TAXNUM entities, it is likely FINANCIAL. If it contains MRN (Medical Record Number) entities, it is likely HIPAA.

When both signals agree, assignment is high confidence. When they disagree, keyword matching wins - the text context is more reliable than entity type alone, because a credit card number can appear in a medical billing context.

Label Remapping

This is the step most people would skip, and it is the step that matters most.

The ai4privacy dataset uses generic entity categories: GIVENNAME, SURNAME, EMAIL, PHONE, etc. Redacto's domain modes use different category names depending on context. The same person's name is [NAME_1] in a medical record, [CUSTOMER_1] in a field service report, [SOURCE_1] in a journalism context, and [VICTIM_1] in a police report.

The remapping rules per mode:

Source Category HIPAA FINANCIAL FIELD_SERVICE TACTICAL JOURNALISM
GIVENNAME+SURNAME NAME NAME CUSTOMER VICTIM SOURCE
EMAIL EMAIL EMAIL CONTACT CONTACT CONTACT
PHONE PHONE PHONE CONTACT CONTACT CONTACT
SSN SSN SSN ID ID ID
CREDITCARDNUMBER ACCOUNT CARD ACCOUNT ID ID
TAXNUM ID TAXID ID ID ID

I also merged adjacent GIVENNAME and SURNAME entities into single FULLNAME entities. The dataset annotates "Jane" and "Smith" as separate spans, but Redacto expects [NAME_1] to cover "Jane Smith" as a unit. Without this merge, the scorer would penalize correct behavior.

Difficulty Classification

Each entry gets a difficulty level based on entity count and complexity:

  • Easy: 3-4 entities, single category type
  • Medium: 5-7 entities, mixed categories
  • Hard: 8+ entities, or entities requiring contextual reasoning (e.g., a name that is also a place name)

Stratified Sampling

From the ~2,000 candidates, I sampled 12 entries per mode, balanced across difficulty levels. The curation report shows the actual distribution:

HIPAA          : 12 total  (easy=4, medium=4, hard=4)
FINANCIAL      : 12 total  (easy=4, medium=4, hard=4)
FIELD_SERVICE  : 12 total  (easy=5, medium=5, hard=2)
TACTICAL       : 12 total  (easy=7, medium=5, hard=0)
JOURNALISM     : 12 total  (easy=10, medium=2, hard=0)
Enter fullscreen mode Exit fullscreen mode

The imbalance in TACTICAL and JOURNALISM is real: the ai4privacy dataset contains fewer entries that naturally fit these domains. This is a limitation of automated curation and exactly why Phase 2 exists.

Phase 2: Hand-Crafted Contextual Entries

Automated curation gives you quantity and coverage. It does not give you the entries that actually break models.

I wrote 25 additional entries by hand - 5 per mode - targeting contextual reasoning challenges that generic PII datasets cannot test. These are the entries that separate a model that pattern-matches from one that understands.

HIPAA (5 entries): Relational identifiers. "The patient's daughter Lisa called to ask about the prescription." Lisa is PII because her relationship to the patient makes her identifiable. The model must understand that "daughter" creates a linkage. I also included entries with depression diagnoses tied to named patients (the diagnosis itself is PHI when linked to a person), and medication dosages that are clinical data in isolation but become PHI when attached to a name.

TACTICAL (5 entries): Selective redaction. "Suspect is a white male, approximately 6'2", wearing a red jacket, driving a blue 2019 Honda Civic with plate ABC-1234. Victim Jane Doe, age 34, was interviewed at the scene." The model must redact "Jane Doe" but preserve every detail of the suspect description: race, height, clothing, vehicle, plate number. This is the opposite of what most PII models are trained to do.

JOURNALISM (5 entries): Role-based entity handling. "Secretary of Defense Lloyd Austin confirmed the deployment. A senior Pentagon official, speaking on condition of anonymity, said the timeline was accelerated." Lloyd Austin must be preserved (public official, on the record). The anonymous source must be protected. The model needs to understand attribution and source protection conventions.

FIELD_SERVICE (5 entries): Natural-language security information. "Gate code is 4521. Key is under the mat by the side door. WiFi password is TechHouse2024. Back door is usually unlocked." These are not structured PII: there is no SSN or credit card. But they are security-critical information that a field service technician's notes should not expose.

FINANCIAL (5 entries): Entity boundary precision. "Transfer $45,000 from account 7823-4561-9900 (routing 021000021) to the client's trust fund." Dollar amounts and institution names must be preserved. Account numbers, routing numbers, and client names must be redacted. The model needs to distinguish between numeric values that are financial identifiers and numeric values that are transaction details.

Final Dataset

60 automated entries + 25 hand-crafted entries = 85 total entries, stored as redacto_bench.jsonl in the app's Android assets directory.

At 5-10 seconds per entry on GPU inference, 85 entries across 2 models runs in approximately 20 minutes. (These timing figures come from a single directional session on one device; treat them as ballpark, not a rigorous multi-run measurement - see the caveat below.) That is acceptable for a hackathon iteration cycle. I also added a slider (1-85) in the benchmark UI so I could run quick 3-entry sanity checks during development.

The JSONL Schema

Each entry in redacto_bench.jsonl follows this schema:

{
  "id": "hipaa_ctx_001",
  "mode": "HIPAA",
  "difficulty": "hard",
  "input": "The patient's daughter Lisa called to ask about the Metformin prescription for Margaret Chen, DOB 03/15/1952, MRN 4478291.",
  "expected_output": "The patient's daughter [NAME_1] called to ask about the Metformin prescription for [NAME_2], DOB [DATE_1], MRN [MRN_1].",
  "entities": [
    {"value": "Lisa", "category": "NAME", "placeholder": "[NAME_1]"},
    {"value": "Margaret Chen", "category": "NAME", "placeholder": "[NAME_2]"},
    {"value": "03/15/1952", "category": "DATE", "placeholder": "[DATE_1]"},
    {"value": "4478291", "category": "MRN", "placeholder": "[MRN_1]"}
  ],
  "entity_count": 4,
  "source_uid": "handcrafted"
}
Enter fullscreen mode Exit fullscreen mode

Key design choices in this schema:

  • expected_output is a reference, not an oracle. The scorer does not do exact string matching against this field. It exists for human review and debugging.
  • entities is the ground truth. Each entity lists its original value, its domain-specific category, and the placeholder it should become. The scorer uses this list.
  • source_uid distinguishes provenance. Automated entries carry the original ai4privacy UID. Hand-crafted entries are marked "handcrafted". This lets me segment results by provenance and check whether automated entries are too easy.
  • difficulty enables stratified analysis. I can answer "does the model handle hard HIPAA entries?" without writing a separate query.

The entity categories used per mode, as reported by the curation pipeline:

HIPAA:         DATE, EMAIL, ID, LOCATION, MRN, NAME, PHONE, SSN
FINANCIAL:     ACCOUNT, ADDRESS, CARD, DATE, EMAIL, ID, NAME, PHONE, SSN, TAXID
FIELD_SERVICE: ADDRESS, CONTACT, CUSTOMER, DATE, ID, SECURE
TACTICAL:      ADDRESS, CONTACT, DATE, ID, VICTIM
JOURNALISM:    CONTACT, DATE, ID, LOCATION, SOURCE
Enter fullscreen mode Exit fullscreen mode

Scoring Methodology: Why Not Exact Match?

The first scorer I wrote used exact string comparison between the model output and expected_output. It failed on the first test.

The model produced [NAME_1] where I expected [NAME_1], but with an extra space before the period. Score: 0%. The redaction was perfect. The formatting was trivially different. Exact match cannot distinguish between "wrong" and "not quite how I would have written it."

This is a known problem in NLP evaluation. BLEU scores for machine translation penalize valid paraphrases. ROUGE scores for summarization reward extractive copying over abstractive understanding. For structured text transformation, what matters is not the exact string: it is whether the PII was removed, the format was followed, and the context was preserved.

Three-Metric Scoring

I use three metrics, weighted into an overall score:

Entity Recall (weight: 50%). For each ground-truth PII entity in the entities list, check whether the original value is absent from the model's output. If the model output does not contain "Lisa" or "Margaret Chen" or "03/15/1952", those entities were successfully redacted regardless of what placeholder the model used. This is intentionally lenient: it rewards redaction in any form, not just the exact placeholder format.

entityRecall = (entities where value is absent from output) / (total entities)
Enter fullscreen mode Exit fullscreen mode

Format Compliance (weight: 25%). Count the bracket placeholders in the model output that match the [CATEGORY_N] pattern (e.g., [NAME_1], [SSN_2], [DATE_3]). Divide by the total number of bracket-enclosed tokens in the output. This distinguishes models that use Redacto's structured format from those that output generic [REDACTED] or [PII REMOVED].

formatScore = (placeholders matching [A-Z_]+_[0-9]+) / (total bracket tokens)
Enter fullscreen mode Exit fullscreen mode

Text Preservation (weight: 25%). Compute the fraction of non-PII words from the input that appear in the output. "The patient's daughter called to ask about the Metformin prescription" should survive redaction intact. A model that strips context while redacting PII is not useful: the whole point is to produce a readable, de-identified document.

preservationScore = (non-PII input words found in output) / (total non-PII input words)
Enter fullscreen mode Exit fullscreen mode

Overall Score:

overall = entityRecall * 0.50 + formatScore * 0.25 + preservationScore * 0.25
Enter fullscreen mode Exit fullscreen mode

Why These Weights?

Entity recall gets 50% because it is the primary safety metric. A model that leaks PII has failed its core job regardless of format or preservation. Format compliance and text preservation split the remaining 50% equally because both matter for usability: a perfectly redacted document is useless if it is unreadable (poor preservation) or uses non-standard placeholders that downstream systems cannot parse (poor format compliance).

On-Device Infrastructure

Running 85 benchmark entries on a phone is not the same as running them on a server. Three engineering decisions shaped the infrastructure.

Decision: Text-Only Dataset (D15)

The benchmark dataset is pure text: no images, no OCR. This is deliberate. Redacto supports both text and image input, but the image pipeline adds ML Kit OCR latency that varies with image quality, resolution, lighting, and text density. If I benchmark the full image pipeline, I am measuring OCR performance, not inference performance.

By isolating text input, every millisecond of latency and every scoring delta is attributable to the LLM pipeline. When the fine-tuned model came in roughly 1.9x slower than the standard model in that session, I could attribute it to the model, not the camera. (As with every figure here, that comes from one directional session; see the caveat further down.)

Decision: Steps 1-3 Only, No Validation (D16)

Redacto's production pipeline has four steps: Classify, Detect, Redact, Validate. The benchmark runs only Steps 1-3.

Step 4 (Validate) is a quality gate that reruns Steps 3-4 if it finds missed entities, up to 3 rounds. This is valuable in production but catastrophic for benchmarking. A model that needs 3 validation rounds to redact an entry correctly would show 3x the latency of one that gets it right on the first pass. The latency comparison would conflate model quality with model speed.

For the benchmark, I want to measure: given one pass through the pipeline, how good is the output? The validation step answers a different question (can iterative refinement fix mistakes?) which is worth testing separately.

Decision: ADB-Triggered Benchmarks via BroadcastReceiver (D17)

The benchmark can be triggered programmatically via ADB, without touching the UI:

# Run text benchmark, 5 entries, GPU backend:
adb shell am broadcast \
  -n com.example.redacto/com.example.redacto.benchmark.BenchmarkReceiver \
  -a com.example.redacto.BENCHMARK \
  --es type text \
  --ei count 5 \
  --es backend GPU

# Monitor results in real time:
adb logcat -s RedactoBenchmark:I
Enter fullscreen mode Exit fullscreen mode

This design has a critical architectural detail: the BroadcastReceiver invokes a callback on the running app's ViewModel, which shares the already-initialized inference engine. It does not create a new engine instance.

Why this matters: the Gemma 4 E2B model file is 2.59 GB (the NPU build is 3.02 GB), and a loaded engine's runtime footprint is on that order. Creating a second engine instance for benchmarking would roughly double that (to ~5.18 GB for the generic build), which exceeds available RAM on most devices and causes OOM crashes. By sharing the engine, the benchmark runs against the exact same inference context the user would experience in production: same memory state, same GPU/NPU allocation, same thermal conditions.

The tradeoff is that the app must be open (the callback registration happens in Compose). You cannot run benchmarks against a cold app. For automated CI testing, this would be a limitation. For hackathon iteration, it is fine.

Engine Isolation (v1 Approach)

The initial benchmarking system (v1, in the original Redacto codebase) took a different approach. The BenchmarkRunner created its own InferenceEngine instances, separate from the main app. Each model got a clean init/close cycle, and the main app's engine was untouched.

This worked for comparing two models head-to-head but required enough RAM for two full engine loads. The v2 approach (shared engine via BroadcastReceiver) was adopted when we hit OOM on the fine-tuned model, which is 4.7 GB.

Standard vs Fine-Tuned: An Unfair Fight

With the benchmark suite in place, I ran the first comparison: the standard litert-community Gemma 4 E2B against our QLoRA fine-tuned variant. The results were surprising.

A caveat before the numbers: every figure in the tables below (and the latency and power comparisons that follow) comes from a single directional benchmarking session on one Galaxy S25 Ultra. No raw logs were retained, and the device is no longer available to me, so treat these as one non-reproducible run that points in a direction, not a rigorous multi-run study.

Metric Standard Fine-Tuned Delta
Overall Score 80.5% 70.3% -10.2%
Entity Recall 79.3% 71.7% -7.6%
Format Score 79.8% 71.7% -8.1%
Preservation 83.7% 65.9% -17.8%

The standard model won overall. But the per-mode entity-recall breakdown told a different story:

Mode Standard Fine-Tuned Winner
FIELD_SERVICE 82.1% 95.3% Fine-tuned (+13.2%)
FINANCIAL 83.8% 85.5% Fine-tuned (+1.7%)
HIPAA 95.7% 39.9% Standard (+55.8%)
JOURNALISM 71.1% 61.3% Standard (+9.8%)
TACTICAL 63.7% 76.8% Fine-tuned (+13.1%)

(These are entity-recall scores by mode, the 50%-weighted component of the overall score, not the blended per-mode totals.)

The fine-tuned model came out clearly ahead on FIELD_SERVICE and TACTICAL - the two modes where contextual reasoning matters most. It correctly identified "key is under the mat" as a security risk. It correctly preserved suspect descriptions while redacting victim names.

But it lost badly on HIPAA (-55.8%) and overall preservation (-17.8%). Why?

The Output Format Mismatch

It is worth being honest about how this fine-tune came together, because it frames the whole result: this was an under-resourced attempt. We trained on 3,000 of the 400,000 samples in ai4privacy/pii-masking-400k, for a single epoch, in only a few minutes (about 217 seconds). And it was trained with a generic instruction prompt ("Mask all Personally Identifiable Information"), so it learned to output [REDACTED] or [REDACTED NAME] style placeholders. Redacto's benchmark expects the structured [CATEGORY_N] format ([NAME_1], [SSN_2]).

That mismatch penalizes the fine-tuned model on both Format Compliance (its [REDACTED] placeholders do not match the [CATEGORY_N] pattern) and Entity Recall (some entities are partially redacted with free-form text that still contains fragments of the original value). So the honest read is not "fine-tuning failed" - it is that a good prompt on the standard model beat an under-resourced fine-tune. Both the quantity of training data (3,000 vs 400,000 samples) and its alignment ([REDACTED] vs [CATEGORY_N]) mattered, and with the full dataset and the right output format the result could easily have flipped. The comparison is not measuring model capability: it is measuring how much training investment and format alignment each side had.

What Would Make It Fair

Four changes would make this a valid comparison:

  1. Train on the full dataset. 3,000 of 400,000 samples for one epoch is not a real training run. Using the full corpus (and more than one epoch) is the single biggest thing that could flip the result.
  2. Re-train with Redacto's prompts. Use the exact system prompts and [CATEGORY_N] output format as training targets, rather than the generic [REDACTED] format the fine-tune learned.
  3. Comparable footprint and backend. The fine-tuned model is 4.7 GB versus 2.59 GB for the standard build, and critically it is GPU-only: at that size and quantization it cannot be compiled to the NPU at all. That footprint-and-backend gap, not just quantization granularity, is what drives the 1.9x latency gap and higher power draw (roughly 3x in this one session) - so it is not an apples-to-apples model comparison.
  4. Same chat template. The fine-tuned model originally shipped with a HuggingFace Jinja template that LiteRT-LM could not parse (it used map.get(), which the on-device template parser does not support). This required a re-export with a compatible Gemma 3 template.

The benchmark did exactly what it was supposed to do: it surfaced a training data problem that manual testing would never have caught. I would have looked at three FIELD_SERVICE examples, seen improvement, and shipped the fine-tuned model - not knowing it dragged down HIPAA performance.

What I Would Do Differently

More entries per mode. Seventeen entries per mode (12 automated + 5 hand-crafted) is enough to see trends but not enough for statistical confidence intervals. For production, I would target 50+ per mode.

Automated regression. The current system is manual: I run the benchmark, read the results table, compare to last run. A CI-integrated version would store results in a database and flag regressions automatically.

Latency percentiles. The session doc records average latency, but averages hide outliers. A single 30-second entry in an otherwise fast run skews the mean. P50, P95, and P99 latency would be more informative.

Cross-device testing. All measurements are from a single Samsung Galaxy S25 Ultra. Different Snapdragon variants, MediaTek chips, and Tensor processors will behave differently.

The Minimum Viable Benchmark

If you are building an on-device LLM app and want to start benchmarking today, here is the minimum:

  1. Curate 30-50 entries that represent your actual use cases, not generic NLP benchmarks. Include edge cases you have seen fail in manual testing.

  2. Define 2-3 metrics that measure what matters for your specific task. For redaction, that is entity recall, format compliance, and preservation. For summarization, it might be factual accuracy, compression ratio, and fluency. For classification, it might be accuracy, calibration, and latency.

  3. Store them as a flat file (JSONL, CSV) in your app's assets. No database, no API, no build-time code generation.

  4. Build a scorer that does not use exact match. Unless your task has exactly one correct output, exact match will generate false failures that erode trust in the benchmark.

  5. Measure on the device, not the emulator. Emulator performance numbers are meaningless for on-device inference. Thermal throttling, memory pressure, and accelerator behavior differ fundamentally.

  6. Log everything. Per-entry latency, token counts, backend (GPU/NPU/CPU), and raw model output. You will need it when debugging a score drop.

The goal is not perfection. The goal is replacing "it works on my test input" with "it works on 85 representative inputs across 5 domains and 3 difficulty levels, with quantified scores on the metrics that matter." That is a different kind of confidence.

Related in this series of "Edge AI from the Trenches"

Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware. Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.

Sources:

Last updated: July 2026
18th of 23 posts in the "Edge AI from the Trenches" series

Top comments (0)