AI features in mobile apps: 4 patterns that ship

TL;DR

• Almost every AI feature shipping in mobile apps is one of four patterns: text-in/text-out, voice-in/text-out, vision-in/structured-out, or retrieval over the user's own content.
• Everything else (multi-agent orchestration, autonomous browsing, on-device frontier models, real-time voice) is still a science project on mobile in 2026.
• If a feature request doesn't map to one of the four, it's probably not ready to ship.
• Ship one pattern deeply instead of four badly.

Most AI features in mobile apps collapse into four patterns. Not forty. Four. Strip away the marketing pages and the LinkedIn threads about "AI-first paradigm shifts," and what's actually shipping in production React Native, Swift, and Kotlin apps is a very small, very boring set of shapes. Every one of them has been solved. None of them require an autonomous multi-agent framework.

Why the taxonomy matters

Founders and PMs describe AI features in terms of what the user sees: "the app understands my meals," "the app writes my notes for me." Engineers have to build in terms of what the model actually does. It takes input tokens of some modality, emits output tokens of some modality, sometimes conditioned on retrieved context.

When you translate a product ask into one of the four patterns below, three good things happen:

• You know which API to call (transcription, chat, embeddings, vision).
• You know the latency budget you're working with.
• You know whether it'll actually work at all with today's models.

If you can't map a feature request onto one of these four, that's usually a signal the feature isn't ready to ship, not a signal you need a more sophisticated architecture.

Pattern 1: Text-in, text-out

What it is: the user types (or the app produces) text; the model returns text. Chat, summarization, rewriting, translation, tone-adjustment, "explain this like I'm five."

Lowest engineering cost, highest UX ceiling. Which is why every consumer app now has some flavor of it, and why most of them feel forgettable. The wins live in the small stuff: streaming so the first token appears in under 400ms, keeping the input responsive during generation, and handling network drops without losing the partial response.

React Native hazards specifically:

• Streaming over fetch on iOS is painful. The default fetch doesn't stream response bodies on Hermes reliably. Use expo-fetch or an SSE library.
• Keyboard behavior. KeyboardAvoidingView fights with a growing message list. Pin the input to the bottom of the safe area and animate the message list instead.
• Cost per DAU adds up fast. A chatty user at Claude Sonnet or GPT-4o quality is $0.10–$0.40 per day. Cache aggressively; use a cheaper model for the first turn.

Pattern 2: Voice-in, text-out

What it is: the user speaks; the app returns a transcript, and (usually) a summary, action items, or a structured extract.

The transcription half is solved. Whisper on OpenAI, Deepgram Nova, AssemblyAI Universal: all three produce production-quality transcripts of 30–60 second clips for well under a cent. The interesting engineering is the second half. Taking a 4,000-word verbatim transcript and returning something a human actually wants to look at.

The pipeline that ships:

1. Record audio locally with expo-av (or react-native-audio-recorder-player on bare RN).
2. On stop, POST the file to your backend.
3. Backend calls the transcription API, gets back the transcript.
4. Backend passes the transcript to an LLM with a structured-output prompt: { "summary": string, "action_items": string[], "key_decisions": string[] }.
5. Backend saves both the raw transcript and the structured extract.
6. Client fetches the structured extract when the recording detail screen opens.

That's the entire pipeline. Boring on purpose. Every "smart voice notes" app in the store is doing this. The differentiation is in the summary prompt, the UI for surfacing action items, and how graceful the recording UX feels during long meetings. If you want the client side already wired up (recording screen, waveform, processing states, transcript/summary/actions tabs), the AI Voice Notes template ships exactly that with a pluggable transcription seam.

Pattern 3: Vision-in, structured-out

What it is: the user takes a photo; the app extracts structured data from it. Meal → macros. Receipt → line items. Whiteboard → text. Business card → contact.

This is the under-appreciated one. GPT-4o and Claude Sonnet both accept image inputs and can be prompted to return strict JSON. A single API call replaces what used to require a tesseract OCR pipeline plus a NER model plus a lot of regex.

The shape of the call:

const response = await claude.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 1024,
  messages: [{
    role: "user",
    content: [
      { type: "image", source: { type: "base64", media_type: "image/jpeg", data: base64 } },
      { type: "text", text: "Extract every line item from this receipt. Return strict JSON matching the schema: { items: [{ name: string, price_cents: integer, quantity: integer }] }" },
    ],
  }],
});

The gotchas are all on the input side:

• Compress before upload. A raw iPhone photo is 4MB. The model doesn't need it. Resize to 1568px longest side, JPEG quality 0.7. That's ~200KB and the vision quality is indistinguishable.
• Validate the JSON. Sonnet and GPT-4o will occasionally emit valid-looking JSON with the wrong keys. Parse with Zod and retry once if it fails.
• Latency budget is real. A single vision call is 3–8 seconds. Show a "reading your photo" progress state; don't lie with a fake spinner.

This is the pattern that most obviously doesn't need on-device inference. Frontier vision quality beats every on-device option by so much that the round-trip is worth it.

Pattern 4: Retrieval-augmented anything

What it is: the user asks a question; the app finds the semantically relevant chunks of their own content and passes them to a model along with the query.

Most often mis-implemented, because two very different things get called "RAG":

1. Full-doc-in-context: shove a 20-page PDF into a 200K-context Claude call along with the question. Zero infrastructure. This is what most "chat with PDF" apps actually do.
2. True retrieval: hundreds or thousands of documents, so you chunk, embed, store vectors, and retrieve top-k at query time.

You want #1 until the user's corpus outgrows a single context window. Don't build pgvector-backed retrieval on day one for an app whose users have three documents.

When you do need true retrieval:

• Embeddings: OpenAI text-embedding-3-small (1536 dims, ~$0.02 per million tokens) or Voyage voyage-3 for higher quality.
• Storage: Supabase's pgvector extension, a single Postgres column, indexed with HNSW, queryable with cosine similarity in normal SQL.
• Chunking: 500–800 token chunks with 100 token overlap, one embedding per chunk plus a foreign key to the source document.
• Retrieval: top-8, then rerank with a cross-encoder if quality matters.

The mobile client doesn't need to know any of this. From RN's perspective it's still POST /api/chat with a message.

The four patterns compared

Pattern	Typical latency	Cost per request	On-device viable?	Where it breaks
Text-in / text-out	200ms–3s (streaming)	$0.001–$0.05	Only tiny 1–3B models on flagships	Streaming on Hermes; keyboard UX
Voice-in / text-out	2s–15s	$0.006/min + LLM	Whisper Tiny for short clips	Long-audio memory; iOS background recording
Vision-in / structured-out	3s–8s	$0.01–$0.05 per image	Not at frontier quality	JSON schema drift; slow uploads
Retrieval	500ms–4s	$0.001 per query	Embeddings yes; retrieval cloud	Chunking; stale embeddings

What doesn't ship (yet)

An honest list of patterns that keep showing up in pitch decks and don't survive contact with a shipping mobile app in 2026:

• Multi-agent orchestration inside the app. A LangGraph-style DAG of five agents debating each other is fun in a notebook and a debugging nightmare on a phone. Ship a single call with a good prompt first.
• Autonomous browsing agents. "Book me a flight" works in demos and falls apart the moment a real airline site loads a captcha. Mobile isn't the platform to prove this out.
• On-device frontier-quality models. Gemini Nano and Apple Intelligence exist. They don't match Sonnet or GPT-4o. Ship cloud, add on-device later as a latency/privacy optimization.
• Real-time interruptible voice conversation. The APIs are technically usable, but the mobile plumbing (WebRTC, background audio, Bluetooth switching) is where 80% of the work lives.

None of these are bad ideas. But if your roadmap depends on any of them working right now in an app launching next quarter, you're setting engineering up to fail.

FAQ

Q: Do I need to fine-tune?
Almost never in 2026. A well-written system prompt plus structured output constraints (JSON schema, Zod validation, retry on failure) gets you to production quality for all four patterns. Fine-tune only when you have thousands of high-quality examples and frontier models still fail on your task.

Q: Client or server inference?
Server, by default. On-device is worth it only for very latency- or privacy-sensitive flows, usually a subset of pattern 1.

Q: What do these cost per user?
Rough 2026 numbers per DAU: chat $0.05–$0.40; voice notes $0.02–$0.15; vision extraction $0.01–$0.10 per photo; retrieval $0.001–$0.01 per query. Cache aggressively and pick the smallest model that clears your quality bar.

Q: What about TensorFlow Lite and friends?
Useful for non-LLM tasks: image classification, pose estimation, wake-word detection, offline OCR. They complement the four cloud-LLM patterns; they don't replace them.

Ship one pattern well, not four badly

Every pattern above has a full quarter of UX work behind it: good recording state, good streaming, good empty states, good failure modes. Pick the one that actually solves your user's problem. Ship it deeply. Add the second in v2.

Which of the four are you building right now, and what's the ugliest part so far? Drop a comment.