DEV Community

pielouNW
pielouNW

Posted on

On-Device AI in Flutter

#ai #flutter #llm #rag

In this Flutter tutorial, you'll learn how to run a large language model (LLM) directly on a user's device: no server, no API key needed. We'll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.

Each concept is explained before the code, so you can follow along whether you're new to on-device AI.

Why run AI On-Device?

Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.

Running the model directly on the device avoids all of them:

  • Works offline — no internet connection required
  • Privacy by design — user data never leaves the device
  • Low latency — no network round-trip
  • No cloud costs — inference is free

The tradeoff is raw capability: on-device models are smaller and less powerful than frontier cloud models. But for many use cases like summarization, chatbots, or local search, they're more than good enough.

About NobodyWho

We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and ships bindings for several languages and frameworks: Kotlin, Python, Expo/React Native, Swift, Flutter, and Godot. It exposes a clean Dart API for running any model locally in .gguf format, on iOS and Android.

Add it with:

flutter pub add nobodywho
Enter fullscreen mode Exit fullscreen mode

Import it under the nobodywho namespace, since the package uses common names like Model and Chat that could otherwise collide with your own code:

import 'package:nobodywho/nobodywho.dart' as nobodywho;
Enter fullscreen mode Exit fullscreen mode

Before calling anything else, initialize the native bindings exactly once in main before runApp():

await nobodywho.NobodyWho.init();
Enter fullscreen mode Exit fullscreen mode

Loading a Model

NobodyWho can download a GGUF model for you directly from Hugging Face, cache it, and reuse it on every subsequent launch. That means you don't need to bundle anything into your app or manage downloads yourself:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
);
Enter fullscreen mode Exit fullscreen mode

The first time this runs, the model is downloaded to the platform cache directory. Every call after that loads the model directly.

modelPath accepts a few different forms:

Form Example Notes
HuggingFace reference hf:owner/repo/file.gguf Downloaded and cached on first use
HTTPS URL https://example.com/model.gguf Downloaded and cached on first use
Local path ./model.gguf Used as-is, no download

The HuggingFace prefix is case-insensitive and the // is optional, so hf:, hf://, huggingface:, and huggingface:// are all equivalent. You can also pass "auto" to let NobodyWho pick a chat model based on the device's available memory, which is a handy default if you don't want to think about model selection at all.

You can track a remote download by passing onDownloadProgress to Chat.fromPath:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
  onDownloadProgress: (downloaded, total) {
    print('$downloaded / $total bytes');
  },
);
Enter fullscreen mode Exit fullscreen mode

You can find thousands of LLMs in .gguf format on Hugging Face here.

Basic Chat

With a model loaded, you're ready to start a conversation:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
);
final response = await chat.ask('Is water wet?').completed();
print(response); // Yes, indeed, water is wet!
Enter fullscreen mode Exit fullscreen mode

chat.ask() sends your message and returns a TokenStream. Calling .completed() waits for the whole response and gives you back the final string, which is fine for a one-off question. But a real chat interface needs to stream tokens as they arrive, otherwise users stare at a blank screen until generation finishes.

Streaming Tokens 

final response = chat.ask('What is the capital of Denmark?');

await for (final token in response) {
  print(token);
}
Enter fullscreen mode Exit fullscreen mode

A token is the smallest unit a model generates, typically a word, or a fragment of a word.

Multimodal Models

Some models can natively ingest images and audio. To use them, you need two things: a multimodal LLM, and its projection model that converts images and/or audio into tokens the LLM can consume (usually named with mmproj in it). A solid default that handles both image and audio is Gemma 4 with its BF16 projection model.

final model = await nobodywho.Model.load(
  modelPath: "./vision-model.gguf",
  projectionModelPath: "./mmproj.gguf",
);
final chat = nobodywho.Chat(model: model);
Enter fullscreen mode Exit fullscreen mode

To actually send image or audio content, build a Prompt mixing text, images, and audio, and pass it to chat.askWithPrompt() instead of a plain string:

final response = await chat.askWithPrompt(nobodywho.Prompt([
  nobodywho.TextPart("Tell me what you see in the image and what you hear in the audio."),
  nobodywho.ImagePart("./dog.png"),
  nobodywho.AudioPart("./sound.mp3"),
])).completed();
Enter fullscreen mode Exit fullscreen mode

Keep in mind that images and audio consume context fast, so you'll likely want a bigger contextSize than you'd use for text-only chat. Also note that the language model and its projection model have to be trained together — you can't mix an LLM and a projection model you happen to like and expect them to work.

Speech to Text

If you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through SpeechToText.

final stt = await nobodywho.SpeechToText.load(source: 'hf://onnx-community/whisper-base');

final text = await stt.transcribeFile('recording.mp3').completed();
print(text);
Enter fullscreen mode Exit fullscreen mode

source is a Hugging Face repo (hf://owner/repo) or a local directory laid out the same way. Browse the Whisper ONNX models on Hugging Face to find one that fits your accuracy and speed needs.

If your audio comes from a buffer rather than a file, use transcribePcm:

final text = await stt.transcribePcm(samples, 16000).completed();
Enter fullscreen mode Exit fullscreen mode

The buffer needs to be mono i16 PCM samples. The sample rate can be anything, NobodyWho resamples internally to what Whisper expects. And just like chat, transcription can be streamed piece by piece instead of waiting for the full result:

import 'dart:io';

await for (final piece in stt.transcribeFile('recording.mp3')) {
  stdout.write(piece);
}
Enter fullscreen mode Exit fullscreen mode

Text to Speech

Going the other direction, TextToSpeech turns text into WAV audio you can play back or save.

import 'dart:io';

final tts = await nobodywho.TextToSpeech.load(
  source: 'hf://NobodyWho/Kokoro-82M',
  voice: 'bf_emma',
  language: 'en-gb',
);

final wav = await tts.synthesize(text: 'Hello from NobodyWho!');
await File('out.wav').writeAsBytes(wav);
Enter fullscreen mode Exit fullscreen mode

Three architectures are supported, all ONNX-based: Kokoro, Pocket TTS, and Supertonic. NobodyWho infers which one you're using from the source string, so you only need to set architecture explicitly when loading from a custom local folder.

Each architecture has its own voice and language options that need to agree with what the model supports.

Voice Activity Detection

Before transcribing audio, it helps to know when someone is actually speaking rather than relying on a fixed silence timeout. VoiceActivityDetection uses a small model to reliably tell speech and silence apart, and pairs naturally with SpeechToText.

For streaming microphone input, push chunks in as they arrive:

final vad = await nobodywho.VoiceActivityDetection.load(
  sampleRate: 16000,
  source: 'hf://onnx-community/silero-vad',
);
final stt = await nobodywho.SpeechToText.load(source: 'hf://onnx-community/whisper-base');

while (true) {
  final chunk = readMic();
  if (vad.push(chunk) == nobodywho.VoiceActivityDetectionEvent.speechEnded) break;
}

final speech = vad.finish();
final transcription = await stt.transcribePcm(speech, 16000).completed();
print(transcription);
Enter fullscreen mode Exit fullscreen mode

Each push() call reports the current state (speechStarted, speechEnded, speech, or silence), and finish() hands you back the buffered speech segment while resetting internal state for the next turn.

If you already have a full recording and just want to pull out the speech segments from it, segment() does that in one pass:

final audio = readWavPcm('recording.wav');

for (final speech in vad.segment(audio)) {
  final transcription = await stt.transcribePcm(speech, 16000).completed();
  print(transcription);
}
Enter fullscreen mode Exit fullscreen mode

Sensitivity is tunable via threshold, minSpeechDurationMs, minSilenceDurationMs, and prerollDurationMs (how much audio to keep before the detected start, so you don't clip the beginning of a sentence). The defaults are a reasonable starting point, but VAD is one of those things that usually benefits from tuning to your actual environment.

Tool Calling

Tools let the model call out to real functions in your app rather than just generating text. Any Dart function that returns a String or Future<String> becomes a tool by wrapping it in Tool() with a name and description:

import 'dart:math' as math;

final circleAreaTool = nobodywho.Tool(
  name: "circle_area",
  description: "Calculates the area of a circle given its radius",
  function: ({ required double radius }) {
    final area = math.pi * radius * radius;
    return "Circle with radius $radius has area ${area.toStringAsFixed(2)}";
  },
);

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
  tools: [circleAreaTool],
);
Enter fullscreen mode Exit fullscreen mode

NobodyWho also ships two general-purpose tools out of the box, a Python interpreter and a Bash interpreter, for models that need to reason precisely or compute something:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
  tools: [nobodywho.Tool.python(), nobodywho.Tool.bash()],
);
Enter fullscreen mode Exit fullscreen mode

Not every model supports tool calling well, the Qwen family is a solid choice if you need it to be reliable. See the Tool Calling documentation for more.

RAG

Retrieval-Augmented Generation combines document search with LLM generation, so the model grounds its answers in your own knowledge base instead of what it happened to learn during training. NobodyWho provides an Encoder for embeddings and a CrossEncoder for reranking:

final encoder = await nobodywho.Encoder.fromPath(modelPath: './embedding-model.gguf');

final queryEmbedding = await encoder.encode(text: "What is the return policy?");
final docEmbeddings = await encoder.encodeBatch(texts: knowledge);
final similarities = docEmbeddings.map((doc) => nobodywho.cosineSimilarity(
  a: queryEmbedding.toList(),
  b: doc.toList(),
)).toList();
Enter fullscreen mode Exit fullscreen mode

For better precision, rerank the top candidates with a CrossEncoder before handing them to the model:

final crossencoder = await nobodywho.CrossEncoder.fromPath(modelPath: './reranker-model.gguf');
final ranked = await crossencoder.rankAndSort(query: "What is the return policy?", documents: topDocs);
Enter fullscreen mode Exit fullscreen mode

See the Embeddings & RAG documentation for the full walkthrough, including how to wire this up as a tool the model calls automatically.

What's Next?

You now have a complete foundation for building on-device AI features in Flutter:

  • Download and run a GGUF model
  • Send messages and get streamed tokens back
  • Feed images and audio directly into a multimodal model
  • Transcribe speech, synthesize it back, and detect when someone's actually talking
  • Extend the model with tool calling and perform search with RAG

You can also have a look at the Flutter starter example to see a full implementation.

Links

Github - pub.dev - Docs

Top comments (0)