I built Redacto with my team in a single day at a hackathon: an on-device PII redaction app that runs a quantized small model entirely on a phone. No cloud calls, no API keys, no data leaving the device. The model is Gemma 4 E2B Instruct (roughly 2.3B effective parameters, 5.1B total, using Per-Layer Embeddings, where the E stands for effective), quantized to INT4 and running inside LiteRT-LM on a Snapdragon 8 Elite.
Before this project, I thought I understood prompt engineering. I had written system prompts for GPT-4 and Claude. I had done few-shot formatting, chain-of-thought reasoning, role injection. All the standard moves.
None of it prepared me for what happens when the model is small, the context window is tight, and the hardware has opinions about how your output gets generated.
This post is about what I learned. Not abstract advice: concrete failures and the fixes that emerged from one directional benchmarking session of 30 entries across five modes on real hardware. That session ran once on a Galaxy S25 Ultra in May 2026. I did not save the raw logs and no longer have the device, so treat every precise number here as directional, not as a rigorous multi-run study.
1. Token budget is a real constraint
When you prompt GPT-4 or Claude, your system prompt lives inside a 128K or 200K context window. A 500-token system prompt is a rounding error. You can be verbose. You can add examples. You can explain edge cases at length.
On device, the picture is very different. Gemma 4 E2B is trained for a 128K context, but that is not what I deployed. The model is compiled with a fixed KV cache of 1024 tokens (a 256-token prefill), so the deployed working set is 1024 tokens, not 128K. The app requested maxNumTokens=4000, but the compiled 1024-token cache is what actually wins at runtime. That same 500-token system prompt consumes roughly half of the 1024-token cache before the user's input text arrives, and before the model has generated a single output token.
The math is unforgiving. Redacto runs a 4-step pipeline: Classify, Detect, Redact, Validate. Each step gets its own system prompt. Each prompt had to be compressed ruthlessly. Seven category-specific prompt sets, one each for Medical, Financial, Legal, Tactical, Journalism, Field Service, and General, all had to fit within budgets that would be laughable in cloud prompting.
On a cloud model, you might write:
You are a HIPAA-compliant redaction assistant. Your task is to carefully
analyze the following medical document and identify all Protected Health
Information (PHI) as defined under the HIPAA Privacy Rule, including but
not limited to: patient names, dates of birth, Social Security numbers,
medical record numbers, health plan beneficiary numbers, account numbers,
certificate/license numbers, vehicle identifiers and serial numbers,
device identifiers, web URLs, IP addresses, biometric identifiers,
full-face photographs and comparable images, and any other unique
identifying number, characteristic, or code...
On a small model with a 1024-token deployed cache, that prompt is a luxury you cannot afford. Every word of instruction competes with the input text and the output tokens for the same finite space. The prompt I actually ship looks more like this:
You are a medical document redactor.
Detect these PHI types: names, DOB, SSN, MRN, addresses, phone, fax, email,
account numbers, dates, provider IDs.
Output format: CATEGORY: detected_value (one per line)
Keep: diagnoses, medications, vitals, body locations.
The cloud version teaches. The on-device version commands. Both work, but only one of them fits.
2. Format enforcement vs. format suggestion
Here is something that surprised me early. On cloud models, you can say "please output in JSON format" and get well-formed JSON 95%+ of the time. The model has enough parameters and enough context to track bracket nesting, key-value structure, and quoting rules.
A small quantized model does not have that luxury. In my experience, Gemma 4 E2B running at INT4 precision sometimes struggled with bracket matching. I recall responses where the opening brace appeared but the closing brace was missing, or where keys looked hallucinated, or where the JSON structure would collapse into free text partway through. I did not keep saved samples of these failures, so treat the specifics as recollection rather than logged evidence, but the pattern was consistent enough that I stopped trusting JSON output from the model.
This led to Decision D8 in our engineering log: we abandoned JSON entirely. The output format became line-based, CATEGORY: value, one detection per line. It is less elegant than JSON. It is also dramatically more reliable, because the model only needs to produce a newline and a colon to maintain format compliance, not matched brackets and quoted strings.
The benchmark numbers point the same direction. Across the single session, the standard model using carefully engineered line-based prompts scored higher on overall quality than the fine-tuned model, which had learned a different output format during training (an [REDACTED] style that does not match Redacto's [CATEGORY_N] convention). The format itself is a prompt engineering decision, and it is one of the most consequential decisions you will make on a small model. The format-compliance sub-score bears this out: 79.8% for the standard model versus 71.7% for the fine-tune (the overall scores were 80.5% and 70.3%). As with every figure here, these come from the one directional session, so treat them as directional rather than a multi-run result.
The lesson: on cloud models, you suggest a format and the model follows it. On small models, you must design a format the model can reliably produce. The format has to be simpler than you think. If your model cannot count brackets, do not ask it to count brackets.
3. Conversation architecture replaces conversation length
Cloud prompt engineering has a well-known pattern: stuff everything into one long conversation. Give the model the full context. Let it reason across the entire input. The context window is large enough that you can do classification, detection, transformation, and verification in a single call.
On a small model, this pattern is a trap. Redacto's pipeline has four steps, Classify, Detect, Redact, and Validate, and each step creates and destroys its own conversation object (Decision D5). No shared context. No memory between steps. Four completely independent calls. (The benchmark numbers below cover only the first three steps; the measured session ran without the Validate step.)
Why? Because context pollution in a small model is far more damaging than in a large model. If the Detect step and the Validate step share a conversation, the validator has already seen the detection list. It is no longer an independent auditor: it is biased by its own prior work. On GPT-4, the model is large enough to reason past this bias. On Gemma 4 E2B, it cannot.
This is a shift in thinking. On cloud, you optimize conversation content, what goes into the prompt. On device, you optimize conversation architecture: how many conversations exist, what each one sees, and what is deliberately hidden from each step.
Four fresh conversations cost more total inference time than one long conversation would. But the quality difference is not close. The multi-pass architecture with isolated conversations produces consistently better results because each step does exactly one thing with maximum focus.
4. The prompt IS the specialization layer
Large language models can generalize. You can tell Claude "keep suspect descriptions but redact victim names" in a general instruction, and it will usually reason through the distinction correctly. The model has enough capacity to understand the intent behind your instruction.
A small model needs you to be explicit. Not "keep relevant medical information" but "Keep: diagnoses, medications, vitals, body locations." Not "redact financial identifiers" but "Detect: account numbers, routing numbers, credit card numbers, SSNs, Tax IDs." The model cannot infer what you mean from a general instruction: you must enumerate.
This is Decision D12: seven category-specific prompt sets, each with explicit preserve lists. The preserve list is not a nice-to-have. It is the mechanism that prevents over-redaction.
The benchmark data points the same way, though I want to be careful about the framing. In this single directional session, the standard model with explicit preserve lists scored higher on text preservation (83.7%) than the fine-tuned model (65.9%). That gap is not evidence that "fine-tuning failed." It is evidence that a good prompt beat an under-resourced fine-tune. The fine-tune was deliberately small: about 3,000 of the 400,000 available ai4privacy/pii-masking-400k samples, one epoch, only a few minutes of training (about 217 seconds), and it learned a generic mask-all-PII behavior with an [REDACTED] output format that does not match Redacto's [CATEGORY_N] convention. Both the quantity of data and its alignment to the task were off, so the model over-redacted, removing diagnoses from medical notes and stripping dollar amounts from financial documents, because nothing in its limited training told it what to keep. With the full dataset and an aligned output format, that result could easily flip. There was also a hard deployment constraint: the fine-tuned .litertlm was 4.7 GB and GPU-only, so it could not be compiled to the NPU at all.
Here is what the HIPAA detection prompt's preserve section looks like:
DO NOT flag as PHI:
- Diagnoses and conditions (Type 2 diabetes, hypertension)
- Medications and dosages (Metformin 500mg)
- Vital signs (BP 120/80, HR 72)
- Body locations (left heel, right shoulder)
- Procedures (MRI, CBC, A1C)
And the Tactical (law enforcement) prompt has a completely different preserve list:
DO NOT flag as PII:
- Suspect physical descriptions (race, height, weight, clothing)
- Vehicle descriptions (make, model, color, plate number)
- Officer names and badge numbers
- Crime scene locations and cross streets
- Weapon descriptions
On a cloud model, you might get away with a single general-purpose prompt. On a small on-device model, the prompt is the specialization layer. It compensates for what the small model cannot infer on its own. Each category needs its own carefully curated set of instructions, and the difference between "good enough" and "actually works" is in the preserve lists.
5. Sampling config is intertwined with prompting
This is the one I had not paid enough attention to. In cloud prompt engineering, you rarely think about sampling parameters. Temperature, top-K, top-P: these are set once and forgotten. The model self-regulates output length. Your prompt controls what the model says; the sampling config is background plumbing.
On device, prompting and sampling are inseparable.
Redacto runs on both GPU and NPU backends. The GPU backend uses a constrained sampling configuration: topK=64, topP=0.95, temperature=1.0. The NPU backend cannot use constrained decoding at all: it errors with "not supported, error 12", so it runs with samplerConfig=null, meaning the NPU uses whatever default sampling the runtime provides (Decision D26).
A caveat before the numbers: everything in this section comes from the single directional session described earlier (Galaxy S25 Ultra, May 2026, no raw logs saved, device no longer available). The figures are consistent enough to reason about, but they are directional, not a rigorous multi-run study.
The consequence is dramatic. On Step 3 (Redact), the GPU backend produces an average of 51 tokens per call. The NPU backend, with the same prompt and the same model architecture, produces an average of 163 tokens, roughly 3.2x more output.
The NPU is faster per token: about 42 tok/s versus about 25 tok/s on GPU. That flat NPU rate is expected rather than fabricated precision, because NPU decode here is memory-bandwidth-bound, so it tends to sit at a roughly constant tokens-per-second regardless of the step. But that speed advantage is entirely consumed by the verbosity. Total wall-clock latency for the pipeline averages about 5,062ms on NPU versus about 4,855ms on GPU. The NPU's raw speed advantage is nearly zeroed out because the model generates so much more text without constrained sampling to rein it in.
The TACTICAL mode is the worst case. NPU averaged about 14,201ms per entry on Tactical documents versus GPU's 5,430ms. The unconstrained NPU was generating enormous outputs for complex law enforcement scenarios, sometimes producing narrative explanations and commentary that the GPU's constrained sampling would have truncated.
This means your prompt cannot be evaluated in isolation. The same prompt produces different behavior on different hardware backends because the sampling configuration changes what the model is allowed to generate. When I tune a prompt, I have to test it on both GPU and NPU, because a prompt that produces tight, focused output on GPU might produce runaway verbosity on NPU.
6. The meta-lesson
After benchmarking 30 entries across five modes on two hardware backends in a single directional session, testing both a standard and an under-resourced fine-tuned model, and iterating through dozens of prompt revisions, I have arrived at a framing that I think captures the core difference. (Again: one session, no saved logs, directional rather than rigorous.)
On cloud models, prompt engineering is about getting the best answer. You have abundant context, powerful reasoning, and flexible output. You are optimizing for quality. The ceiling is high and the constraints are loose.
On a small on-device model, prompt engineering is about getting a usable answer within severe constraints. You are optimizing against a budget: token budget, context budget, format reliability budget, and hardware capability budget. Every prompt decision is a tradeoff. More instruction means less room for input. Richer format means more failure modes. Longer context means slower inference.
The skill set is different. Cloud prompt engineering rewards creativity, expressiveness, and thorough instruction. On-device prompt engineering rewards compression, discipline, and architectural thinking. You spend less time on what to say to the model and more time on how to structure the system around the model's limitations.
If I had to place it on a spectrum, on-device prompt engineering is closer to embedded systems programming than it is to chatting with ChatGPT. You are working within hard resource constraints, designing around hardware limitations, and making tradeoffs that would be invisible in an unconstrained environment.
What I would tell someone starting out
If you are moving from cloud prompt engineering to on-device work, here is what I wish someone had told me:
Measure format compliance, not just accuracy. Your model can find the right entities and still produce output your parser cannot read. Track format compliance as a separate metric from the start.
Design your output format for the model, not for your code. Your parsing code can handle complexity. Your small model cannot. Make the format as simple as the model needs it to be, then write a parser that handles the rest.
Isolate your conversations. Do not let steps share context. Each call should see only what it needs. The overhead of multiple conversations is less expensive than the quality loss from context pollution.
Write preserve lists, not just detection lists. Telling the model what to redact is half the problem. Telling it what NOT to redact, explicitly, per category, is the other half. Without preserve lists, small models over-redact aggressively.
Test on every backend. A prompt that works on GPU may fail on NPU because the sampling configuration is different. Your prompt and your sampling config are one system, not two.
Budget your tokens like you budget memory in embedded code. Every token in your system prompt is a token that is not available for input or output. Count them. Compress them. Cut the ones that do not earn their keep.
The model is small. The deployed cache is tight. The hardware has constraints you did not choose. But within those constraints, there is a craft to this work, and it is a craft worth learning, because on-device AI is where the next decade of applications will be built.
Related in this series of "Edge AI from the Trenches"
- What Is a Chat Template and Why Does It Matter?: the formatting layer that sits between your prompt and the model's tokenizer
- TTFT, tok/s, and What "Fast" Means: the metrics that prompt length and output format directly affect
- Fine-Tuning vs Quantization vs Prompt Engineering: when to stay with prompts and when to escalate to fine-tuning
Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware. Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.
Last updated: July 2026
15th of 23 posts in the "Edge AI from the Trenches" series
Top comments (0)