---
title: "On-Device LLM Function Calling on Android: Schema Injection, Token Budgets, and Fallback Parsing"
published: true
description: "Wire a quantized on-device LLM to Android's NPU for real-time function calling. Covers schema injection, token budgets, fallback parsing, and 3B vs 7B tradeoffs."
tags: [kotlin, android, mobile, architecture]
canonical_url: https://mvpfactory.co/blog/android-nnapi-on-device-llm-function-calling
---
## What We Are Building
Let me show you a pattern I use in every on-device agent project: structured output generation — function calling — from a quantized LLM running directly on Android's NPU.
You will wire together three layers that must all work: a schema-injected prompt prefix, a hard token budget that enforces sub-300ms tool-call latency, and a two-pass fallback parser for when the model drifts. Get all three right and you have a viable on-device agent tier that keeps sensitive user data off the wire.
## Prerequisites
- Android project targeting API 26+
- LiteRT (Google's rebrand of TensorFlow Lite) or MediaPipe LLM Inference API — **do not build on the raw `ANeuralNetworks*` NDK APIs**, which were deprecated in Android 13 (API 33). The patterns here apply to either runtime; where I say "NPU-delegated inference," I mean the hardware path, not the deprecated surface.
- A quantized 3B or 7B model (INT4 or INT8) — Llama 2, Gemma, Phi, or Mistral all work
- Baseline familiarity with Android inference sessions
## The Architecture
Prompt Builder → [Schema Prefix + User Turn]
↓
NPU-Delegated Inference (INT4/INT8 quantized weights, via LiteRT)
↓
Token Budget Gate (hard stop at N tokens)
↓
Structured Output Parser → JSON tool call
↓
Fallback Heuristic Parser (regex + key extraction)
### Step 1 — Inject the Schema at the Prompt Prefix
The model does not know your tool schema unless you tell it — every time. Inject it as a system-level prefix before the user turn, serialized as compact JSON. The docs do not mention this, but verbose descriptions will eat your latency ceiling before the model generates a single output token. Keep the schema under 80 tokens.
The example below uses Llama 2 chat format. If you are on Gemma, Phi, or Mistral, swap in the correct special tokens for your model family — the structural principle is identical, the delimiters are not interchangeable.
kotlin
// Llama 2 chat format — replace [INST]/[/INST] with your model's special tokens
val schemaPrefix = """
[TOOLS]
{"name":"search_products","params":{"query":"string","category":"string|null"}}
[/TOOLS]
[INST]
""".trimIndent()
val fullPrompt = schemaPrefix + userMessage + "\n[/INST]"
### Step 2 — Enforce a Token Budget as a Latency Contract
Here is the gotcha that will save you hours. Most teams treat `maxNewTokens` as a model configuration detail. It is a latency contract. Measure tokens-per-second on your target NPU tier, then compute the budget directly:
| Model Size | Quantization | NPU Tier | Tokens/sec | Budget for <300ms |
|---|---|---|---|---|
| 3B | INT4 | Mid-range NPU | ~40–60 | ~12–18 tokens |
| 3B | INT8 | High-end NPU | ~55–80 | ~16–24 tokens |
| 7B | INT4 | Flagship NPU | ~25–40 | ~7–12 tokens |
| 7B | INT8 | Flagship NPU | ~30–50 | ~9–15 tokens |
kotlin
// Inline the budget math directly — no abstraction needed
val tokensPerSec = 50.0 // measure on your target device
val maxTokens = (targetLatencyMs / 1000.0 * tokensPerSec).toInt()
inferenceSession.generate(prompt, maxNewTokens = maxTokens)
Your tool call output — the JSON your agent acts on — must fit within that budget. Strip descriptions, abbreviate field names where safe, no prose, no explanation.
### Step 3 — Ship a Two-Pass Fallback Parser from Day One
Quantized models drift from their schema. This is not a flaw you eliminate — it is a distribution property you engineer around.
1. **Pass 1:** Strict JSON deserialization. If it succeeds, you are done.
2. **Pass 2:** Regex extraction of known field names and values. Reconstruct a best-effort tool call from what you can recover.
If both passes fail, surface a graceful degradation — fall back to keyword-based routing or escalate to a cloud call. Do not silently swallow parse failures.
## Gotchas
**Cold-start latency is invisible in benchmarks.** A 3B INT4 model loading from eMMC adds 800ms–2s on first inference. For infrequent agent invocations, this initialization cost can dominate total response time. Keep the model warm in memory, or reconsider cloud fallback for low-frequency use cases.
**3B vs 7B is a device-tier decision, not a quality decision.** A 3B INT4 model fits in 2–2.5GB of RAM — comfortable on mid-range devices from the last three years. A 7B model needs 4–5GB, which limits you to flagship hardware from the past 18 months. For function calling specifically, where output length is intentionally short, the 3B model's latency advantage compounds across multiple agent turns.
**Why bother at all?** A single round-trip to a hosted model — network, inference, deserialization — lands between 800ms and 2,500ms on a mid-tier connection. NPU-delegated inference with a 3B quantized model runs a forward pass in 150–250ms. That delta is the difference between a responsive agent and a loading spinner, and it keeps sensitive user context (location, purchase history, sensor data) off your servers entirely.
## Conclusion
The path is narrower than most teams assume, but it is shippable. Enforce the token budget first, design schemas for token efficiency second, and build the two-pass fallback parser before you go to production — not after your first drift incident.
**Resources:**
- [LiteRT documentation](https://ai.google.dev/edge/litert)
- [MediaPipe LLM Inference API](https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/android)
- [Android Neural Networks API deprecation notes](https://developer.android.com/ndk/guides/neuralnetworks)
Top comments (0)