Run an On-Device LLM in Your Capacitor App Without an API Key

Adding an AI feature to a mobile app usually means sending the user's text to a server. You pick a provider, find somewhere for the API key that isn't the app bundle (in practice, a proxy you now run), and accept a bill that grows with every token.

Modern phones offer another route. iOS 26 ships Apple Intelligence behind the Foundation Models framework, and recent Android devices run Gemini Nano through ML Kit and AICore. Both models belong to the operating system, so the prompt, the conversation history and the answer stay on the device. There is no key to protect and no per-request cost.

We built the Capacitor LLM plugin to put one TypeScript API over both. To be upfront, it is our plugin and it ships as part of Capawesome Insiders, a paid subscription. This post condenses the full announcement on our blog.

What On-Device Inference Buys You

Privacy is the headline, but three other things follow from keeping the model on the phone:

• No per-token cost. Inference runs on hardware the user already paid for.
• Offline generation. Responses come back in airplane mode or with no reception at all.
• No app size increase. The model is part of the OS, so the feature adds nothing to your download.

A system model is small next to a frontier model behind an HTTP endpoint, the usable context is roughly 4,000 tokens on both platforms, and the hardware requirements exclude most devices in use today. That fits summarizing a note or rewriting a paragraph, not reasoning across a long document.

One API Over Two System Models

The plugin ships neither a model nor a runtime. It calls each vendor's on-device API and normalizes the result:

Platform	Model	Underlying API	Requirements
Android	Gemini Nano	ML Kit GenAI Prompt via AICore	API level 26+, Gemini Nano-capable device
iOS	Apple Intelligence	Foundation Models	iOS 26+, iPhone 15 Pro or later, Xcode 26 to build
Web	none	none	getAvailability() resolves unavailable

Two details shape a release plan. Google's ML Kit GenAI Prompt SDK is still in beta, so the plugin pins com.google.mlkit:genai-prompt at 1.0.0-beta2 and lets you override it with the $mlkitGenaiPromptVersion Gradle variable. And Gemini Nano runs on few devices today, roughly the Pixel 9 and Galaxy S25 series, which makes the availability check mandatory.

If you only ship Android, the free Capacitor ML Kit GenAI Prompt plugin wraps the same Google API directly. For the iOS side, our guide on using Apple Intelligence in a Capacitor app walks through the Foundation Models path on a real device, so I won't repeat it here.

Availability Is the First Call You Make

getAvailability() never rejects. On a platform or OS version with no system model it resolves with unavailable, so you can call it at startup and branch on it. Each of the seven statuses maps to a different UI action:

Status	Reported on	What you do
available	Android, iOS	Start generating.
device-not-eligible	iOS	Fall back to a cloud model.
downloadable	Android	Call downloadModel().
downloading	Android	Show progress and wait.
not-enabled	iOS	Point the user to Settings.
not-ready	iOS	Try again later.
unavailable	Android, iOS, Web	Hide the feature.

The status changes while your app runs. A user can switch on Apple Intelligence in Settings, or an Android download can finish, and the availabilityChange event covers both cases:

import { Llm } from '@capawesome-team/capacitor-llm';

const watchAvailability = async () => {
  const { status } = await Llm.getAvailability();
  updateUi(status);
  await Llm.addListener('availabilityChange', event => {
    updateUi(event.status);
  });
};

Switch over all seven statuses, not the four Android reports or the five iOS reports.

Downloading Gemini Nano on Android

When the status is downloadable, the model has to reach the device first, and Android is the only platform where your app starts it. downloadModel() resolves on completion, and downloadProgress reports a value between 0 and 1. Show that progress, because the download runs long enough that a silent button reads as broken:

import { Llm } from '@capawesome-team/capacitor-llm';

const downloadModel = async () => {
  await Llm.addListener('downloadProgress', event => {
    setProgress(event.progress);
  });
  await Llm.downloadModel();
};

Chats Hold the Conversation Context

A chat is the unit of context. createChat(...) returns an identifier you pass to every generation, and optionally takes instructions, the system prompt that shapes the model's role. Pass your own id to map chats onto your data model, or leave it out and get a UUID.

The implementation differs underneath. On iOS each chat is backed by a native language model session. On Android the system API has no multi-turn concept, so the plugin keeps the history in memory and includes it in each prompt. Either way, history does not survive an app restart, and every chat holds native resources until you release it:

import { Llm } from '@capawesome-team/capacitor-llm';

const startChat = async () => {
  const { id } = await Llm.createChat({
    instructions: 'You are a helpful assistant that answers briefly.',
  });
  return id;
};

const endChat = async (chatId: string) => {
  await Llm.deleteChat({ id: chatId });
};

deleteChat(...) also cancels an in-flight generation for that chat, so tearing down a chat screen mid-response is safe.

Generating and Streaming Text

There are two ways to get text out. generateText(...) resolves with the complete response and suits short outputs where a spinner is fine. streamText(...) emits chunks through the textChunk event and resolves with the full text at the end. Every event carries the chatId it belongs to, so one listener can serve several chats:

import { Llm } from '@capawesome-team/capacitor-llm';

const streamAnswer = async (chatId: string) => {
  await Llm.addListener('textChunk', event => {
    if (event.chatId === chatId) {
      appendToUi(event.text);
    }
  });
  const { text } = await Llm.streamText({
    chatId,
    prompt: 'Summarize this note in three bullet points.',
  });
  return text;
};

Only one generation runs per chat at a time. A second prompt sent into a busy chat rejects with GENERATION_IN_PROGRESS instead of queueing, so disable the send button while a response is in flight.

Canceling a Generation

A small model can spend several seconds on an answer nobody wants any more. cancelGeneration(...) stops the current generation, and the pending promise rejects with GENERATION_CANCELED, a normal outcome rather than an error to report. On iOS it stops immediately. On Android cancellation is best-effort, so a few more textChunk events may arrive afterwards. Track a canceled flag per chat and drop those:

import { Llm } from '@capawesome-team/capacitor-llm';

const canceledChatIds = new Set();

const stopGeneration = async (chatId: string) => {
  canceledChatIds.add(chatId);
  await Llm.cancelGeneration({ chatId });
};

Where the Platform Limits Bite

Two generation parameters are exposed. maxOutputTokens caps the length of a response and temperature controls how deterministic it is. Both work as chat defaults in createChat(...) and as per-request overrides. The ranges come from the platforms:

Parameter	Android (Gemini Nano)	iOS (Apple Intelligence)
maxOutputTokens	Maximum of 4096.	No documented limit.
temperature	Between 0.0 and 1.0.	Values above 1.0 allowed.
Context size	Input under ~4,000 tokens.	~4,096 tokens per session.

Stay inside the Android ranges if you want one code path across both platforms. The context limit is the one to design around: a long conversation eventually overruns it, the generation rejects with GENERATION_FAILED, and the fix is a new chat seeded with a summary generated while there was still room.

Wrapping Up

Start with getAvailability() and a fallback path, because on most devices today the answer is still unavailable. Once that gate is in place, wire up generateText(...) first, then streamText(...) and cancelGeneration(...) when the UI calls for them.

Every method and error code lives in the plugin documentation, and the longer write-up is in the announcement post. If you have shipped an on-device feature already, tell me in the comments which platform limit you hit first.