DEV Community: pielouNW

NobodyWho vs Cactus compared on engine design, model format, hardware, platforms, cloud, and licensing.

pielouNW — Wed, 16 Sep 2026 18:35:15 +0000

Choosing an on-device inference engine sets where the model runs and under what terms. NobodyWho and Cactus both run models on the user's device, with no API key, no per-request cost, and no data leaving the hardware once the model is downloaded. Underneath, they are different engines. NobodyWho Edge is built on llama.cpp and runs GGUF models. Cactus is a from-scratch engine with its own quantization format, and positions itself as a llama.cpp alternative. This is a technical comparison of NobodyWho vs Cactus across engine and model format, hardware, installation, platform coverage, cloud behaviour, and licensing. The better fit depends on the target device and the commercial model of the product.

Engine and model format

NobodyWho runs GGUF models through llama.cpp. It loads any GGUF file from Hugging Face or a URL directly, with no conversion step, so any model already published in GGUF, at any of the quantization levels the ecosystem provides, runs as-is.

Cactus runs its own format. Cactus Quants (CQ) is a rotation-and-codebook quantization applied to every weight tensor, from 4-bit down to 1-bit, and the engine runs from CQ bundles. You get a bundle by downloading one Cactus has pre-built for its own model catalog (cactus download) or by converting a source model yourself (cactus convert), which Cactus documents as experimental for models it has not pre-built.

NobodyWho runs the existing GGUF catalog with no conversion step. Cactus's CQ is tuned in-house for on-device size and quality, with published accuracy tables per bit-width, at the cost of depending on its own bundles or an experimental conversion for anything outside its catalog.

Both also cover the parts of an app beyond chat: embeddings, speech-to-text, text-to-speech, and tool calling. NobodyWho generates the tool-calling grammar from your function signatures and constrains generation to it, so you pass plain functions and the output conforms to the expected structure without writing schemas by hand. Cactus takes OpenAI and MCP-style tool definitions, ships a vector index for retrieval, and provides Needle, a 26M-parameter model dedicated to tool calling.

Hardware acceleration

Cactus is built around the mobile processor. Its kernels are hand-written in ARM NEON SIMD for the CPU, with Metal on Apple GPUs and the Apple Neural Engine for some vision models. Qualcomm, MediaTek, and Exynos NPU support is on its roadmap, not shipped. Cactus publishes per-device benchmarks in its README, listing tokens per second and peak RAM for named iPhone and Mac hardware.

NobodyWho takes a hardware-aware approach to acceleration, using Vulkan and Metal for GPU execution rather than targeting the NPU. This is the sharpest hardware split between them. On older or low-end phones that lean on the CPU, Cactus's hand-written kernels are the advantage. On devices with a capable GPU, NobodyWho runs on the accelerator most platforms already expose.

Installation and first call

NobodyWho installs through each ecosystem's own package manager: pip install nobodywho for Python, flutter pub add nobodywho for Flutter, npm install react-native-nobodywho for React Native, ai.nobodywho:nobodywho from Maven Central for Kotlin, Swift Package Manager for Swift, and the in-editor asset library for Godot. Each binding is a thin wrapper over one shared Rust core.

Cactus installs its engine by cloning the repository and running a setup script, then building the binding and downloading a model bundle:

git clone https://github.com/cactus-compute/cactus && cd cactus && source ./setup 
cactus build --python 
cactus download LiquidAI/LFM2-VL-450M

Cactus also publishes per-platform packages on pip, Maven, pub, and npm.

The difference carries into the first call. NobodyWho's Python API handles model loading and lifecycle for you, and returns the reply as a string:

from nobodywho import Chat

chat = Chat("hf:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf") 
response = chat.ask("What is the capital of Denmark?").completed() 
print(response)

Cactus's Python binding is a ctypes FFI over its C engine, with explicit model lifecycle and JSON message payloads:

from cactus import ensure_model, cactus_init, cactus_complete, cactus_destroy
import json

bundle = ensure_model("LiquidAI/LFM2-VL-450M")
model = cactus_init(str(bundle), None, False)
messages = json.dumps([{"role": "user", "content": "What is the capital of Denmark?"}])
result = cactus_complete(model, messages, None, None, None)
print(result["response"])
cactus_destroy(model)

Both snippets are each project's own documented quick-start.

Platform support

Cactus targets mobile and embedded. Its C and C++ core runs on wearables, smart-home devices, robots, and Raspberry Pi, with bindings for Flutter, React Native, Kotlin Multiplatform, Swift, Python, and Rust.

NobodyWho targets mobile, desktop (Linux, macOS, Windows), Python, and the JVM, and ships a binding for the Godot game engine.

Both engines cover phones and wearables, and Cactus reaches further into embedded hardware like robots and Raspberry Pi. Cactus also supports desktop deployment on macOS and ARM Linux. NobodyWho ships a Godot binding and targets desktop app runtimes across Linux, macOS, and Windows through its JVM and Python bindings. Cactus has no game-engine binding. Neither ships a browser or WebAssembly target. NobodyWho has an open GitHub issue tracking WASM export, and Cactus does not target the browser.

Cloud behaviour

Both run inference locally by default. Cactus adds an optional cloud handoff that routes low-confidence queries to a hosted model. NobodyWho has no cloud router in the engine. In deployments that prohibit outbound network calls, NobodyWho has nothing to disable, and Cactus's handoff must be turned off and verified.

Licensing

NobodyWho uses EUPL-1.2, an OSI-approved open-source licence. It permits proprietary and commercial use with no revenue limit. The only obligation is that redistributing a modified version of the engine requires publishing those engine changes.

Cactus is source-available under its own licence. That licence grants free use to individuals, students, non-profits, and organizations with "Less than $2,000,000 USD in total funding" and "Less than $2,000,000 USD in gross annual revenue." An organization that does not meet those criteria "must obtain a separate commercial license," and a qualifying organization that later crosses either threshold has "thirty (30) days" to obtain one. Both licences were verified on 9 September 2026. Cactus has revised its terms before, so check the current file before shipping a commercial product.

Both engines are free under those thresholds. Once a company crosses either one, Cactus requires a paid commercial licence and NobodyWho stays free.

Choosing between NobodyWho and Cactus

The decision follows target hardware and commercial model.

Requirement	Engine
Run any existing GGUF model with no conversion step	NobodyWho
In-house quantization tuned for on-device size and quality	Cactus
Hand-optimized CPU performance on ARM without a capable GPU	Cactus
Robotics, smart-home, or Raspberry Pi	Cactus
Desktop applications alongside mobile	NobodyWho
A model running locally inside a Godot game	NobodyWho
OSI open-source licence with no revenue ceiling	NobodyWho

The two are different engines with different model formats, so the choice is set by where the model has to run and how the licence treats the product, not by a shared core. The NobodyWho source and per-binding docs list current platform and format support.

On-Device AI in Flutter

pielouNW — Wed, 02 Sep 2026 13:50:47 +0000

In this Flutter tutorial, you'll learn how to run a large language model (LLM) directly on a user's device: no server, no API key needed. We'll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.

Each concept is explained before the code, so you can follow along whether you're new to on-device AI.

Why run AI On-Device?

Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.

Running the model directly on the device avoids all of them:

Works offline — no internet connection required
Privacy by design — user data never leaves the device
Low latency — no network round-trip
No cloud costs — inference is free

The tradeoff is raw capability: on-device models are smaller and less powerful than frontier cloud models. But for many use cases like summarization, chatbots, or local search, they're more than good enough.

About NobodyWho

We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and ships bindings for several languages and frameworks: Kotlin, Python, Expo/React Native, Swift, Flutter, and Godot. It exposes a clean Dart API for running any model locally in .gguf format, on iOS and Android.

Add it with:

flutter pub add nobodywho

Import it under the nobodywho namespace, since the package uses common names like Model and Chat that could otherwise collide with your own code:

import 'package:nobodywho/nobodywho.dart' as nobodywho;

Before calling anything else, initialize the native bindings exactly once in main before runApp():

await nobodywho.NobodyWho.init();

Loading a Model

NobodyWho can download a GGUF model for you directly from Hugging Face, cache it, and reuse it on every subsequent launch. That means you don't need to bundle anything into your app or manage downloads yourself:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
);

The first time this runs, the model is downloaded to the platform cache directory. Every call after that loads the model directly.

modelPath accepts a few different forms:

Form	Example	Notes
HuggingFace reference	`hf:owner/repo/file.gguf`	Downloaded and cached on first use
HTTPS URL	`https://example.com/model.gguf`	Downloaded and cached on first use
Local path	`./model.gguf`	Used as-is, no download

The HuggingFace prefix is case-insensitive and the // is optional, so hf:, hf://, huggingface:, and huggingface:// are all equivalent. You can also pass "auto" to let NobodyWho pick a chat model based on the device's available memory, which is a handy default if you don't want to think about model selection at all.

You can track a remote download by passing onDownloadProgress to Chat.fromPath:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
  onDownloadProgress: (downloaded, total) {
    print('$downloaded / $total bytes');
  },
);

You can find thousands of LLMs in .gguf format on Hugging Face here.

Basic Chat

With a model loaded, you're ready to start a conversation:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
);
final response = await chat.ask('Is water wet?').completed();
print(response); // Yes, indeed, water is wet!

chat.ask() sends your message and returns a TokenStream. Calling .completed() waits for the whole response and gives you back the final string, which is fine for a one-off question. But a real chat interface needs to stream tokens as they arrive, otherwise users stare at a blank screen until generation finishes.

Streaming Tokens

final response = chat.ask('What is the capital of Denmark?');

await for (final token in response) {
  print(token);
}

A token is the smallest unit a model generates, typically a word, or a fragment of a word.

Multimodal Models

Some models can natively ingest images and audio. To use them, you need two things: a multimodal LLM, and its projection model that converts images and/or audio into tokens the LLM can consume (usually named with mmproj in it). A solid default that handles both image and audio is Gemma 4 with its BF16 projection model.

final model = await nobodywho.Model.load(
  modelPath: "./vision-model.gguf",
  projectionModelPath: "./mmproj.gguf",
);
final chat = nobodywho.Chat(model: model);

To actually send image or audio content, build a Prompt mixing text, images, and audio, and pass it to chat.askWithPrompt() instead of a plain string:

final response = await chat.askWithPrompt(nobodywho.Prompt([
  nobodywho.TextPart("Tell me what you see in the image and what you hear in the audio."),
  nobodywho.ImagePart("./dog.png"),
  nobodywho.AudioPart("./sound.mp3"),
])).completed();

Keep in mind that images and audio consume context fast, so you'll likely want a bigger contextSize than you'd use for text-only chat. Also note that the language model and its projection model have to be trained together — you can't mix an LLM and a projection model you happen to like and expect them to work.

Speech to Text

If you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through SpeechToText.

final stt = await nobodywho.SpeechToText.load(source: 'hf://onnx-community/whisper-base');

final text = await stt.transcribeFile('recording.mp3').completed();
print(text);

source is a Hugging Face repo (hf://owner/repo) or a local directory laid out the same way. Browse the Whisper ONNX models on Hugging Face to find one that fits your accuracy and speed needs.

If your audio comes from a buffer rather than a file, use transcribePcm:

final text = await stt.transcribePcm(samples, 16000).completed();

The buffer needs to be mono i16 PCM samples. The sample rate can be anything, NobodyWho resamples internally to what Whisper expects. And just like chat, transcription can be streamed piece by piece instead of waiting for the full result:

import 'dart:io';

await for (final piece in stt.transcribeFile('recording.mp3')) {
  stdout.write(piece);
}

Text to Speech

Going the other direction, TextToSpeech turns text into WAV audio you can play back or save.

import 'dart:io';

final tts = await nobodywho.TextToSpeech.load(
  source: 'hf://NobodyWho/Kokoro-82M',
  voice: 'bf_emma',
  language: 'en-gb',
);

final wav = await tts.synthesize(text: 'Hello from NobodyWho!');
await File('out.wav').writeAsBytes(wav);

Three architectures are supported, all ONNX-based: Kokoro, Pocket TTS, and Supertonic. NobodyWho infers which one you're using from the source string, so you only need to set architecture explicitly when loading from a custom local folder.

Each architecture has its own voice and language options that need to agree with what the model supports.

Voice Activity Detection

Before transcribing audio, it helps to know when someone is actually speaking rather than relying on a fixed silence timeout. VoiceActivityDetection uses a small model to reliably tell speech and silence apart, and pairs naturally with SpeechToText.

For streaming microphone input, push chunks in as they arrive:

final vad = await nobodywho.VoiceActivityDetection.load(
  sampleRate: 16000,
  source: 'hf://onnx-community/silero-vad',
);
final stt = await nobodywho.SpeechToText.load(source: 'hf://onnx-community/whisper-base');

while (true) {
  final chunk = readMic();
  if (vad.push(chunk) == nobodywho.VoiceActivityDetectionEvent.speechEnded) break;
}

final speech = vad.finish();
final transcription = await stt.transcribePcm(speech, 16000).completed();
print(transcription);

Each push() call reports the current state (speechStarted, speechEnded, speech, or silence), and finish() hands you back the buffered speech segment while resetting internal state for the next turn.

If you already have a full recording and just want to pull out the speech segments from it, segment() does that in one pass:

final audio = readWavPcm('recording.wav');

for (final speech in vad.segment(audio)) {
  final transcription = await stt.transcribePcm(speech, 16000).completed();
  print(transcription);
}

Sensitivity is tunable via threshold, minSpeechDurationMs, minSilenceDurationMs, and prerollDurationMs (how much audio to keep before the detected start, so you don't clip the beginning of a sentence). The defaults are a reasonable starting point, but VAD is one of those things that usually benefits from tuning to your actual environment.

Tool Calling

Tools let the model call out to real functions in your app rather than just generating text. Any Dart function that returns a String or Future<String> becomes a tool by wrapping it in Tool() with a name and description:

import 'dart:math' as math;

final circleAreaTool = nobodywho.Tool(
  name: "circle_area",
  description: "Calculates the area of a circle given its radius",
  function: ({ required double radius }) {
    final area = math.pi * radius * radius;
    return "Circle with radius $radius has area ${area.toStringAsFixed(2)}";
  },
);

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
  tools: [circleAreaTool],
);

NobodyWho also ships two general-purpose tools out of the box, a Python interpreter and a Bash interpreter, for models that need to reason precisely or compute something:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
  tools: [nobodywho.Tool.python(), nobodywho.Tool.bash()],
);

Not every model supports tool calling well, the Qwen family is a solid choice if you need it to be reliable. See the Tool Calling documentation for more.

RAG

Retrieval-Augmented Generation combines document search with LLM generation, so the model grounds its answers in your own knowledge base instead of what it happened to learn during training. NobodyWho provides an Encoder for embeddings and a CrossEncoder for reranking:

final encoder = await nobodywho.Encoder.fromPath(modelPath: './embedding-model.gguf');

final queryEmbedding = await encoder.encode(text: "What is the return policy?");
final docEmbeddings = await encoder.encodeBatch(texts: knowledge);
final similarities = docEmbeddings.map((doc) => nobodywho.cosineSimilarity(
  a: queryEmbedding.toList(),
  b: doc.toList(),
)).toList();

For better precision, rerank the top candidates with a CrossEncoder before handing them to the model:

final crossencoder = await nobodywho.CrossEncoder.fromPath(modelPath: './reranker-model.gguf');
final ranked = await crossencoder.rankAndSort(query: "What is the return policy?", documents: topDocs);

See the Embeddings & RAG documentation for the full walkthrough, including how to wire this up as a tool the model calls automatically.

What's Next?

You now have a complete foundation for building on-device AI features in Flutter:

Download and run a GGUF model
Send messages and get streamed tokens back
Feed images and audio directly into a multimodal model
Transcribe speech, synthesize it back, and detect when someone's actually talking
Extend the model with tool calling and perform search with RAG

You can also have a look at the Flutter starter example to see a full implementation.

Links

Github - pub.dev - Docs

On-Device AI in Python

pielouNW — Wed, 02 Sep 2026 11:26:56 +0000

In this Python tutorial, you'll learn how to run a large language model (LLM) directly on a user's device: no server, no API key needed. We'll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.

Each concept is explained before the code, so you can follow along whether you're new to on-device AI.

Why run AI On-Device?

Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.

Running the model directly on the device avoids all of them:

Works offline — no internet connection required
Privacy by design — user data never leaves the device
Low latency — no network round-trip
No cloud costs — inference is free

About NobodyWho

We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and ships bindings for several languages and frameworks: Kotlin, Python, Expo/React Native, Swift, Flutter & Godot. It exposes a clean API for running any model locally in .gguf format.

Install it with:

pip install nobodywho

Or, preferably:

uv add nobodywho

Loading a Model

NobodyWho can download a GGUF model for you directly from Hugging Face, cache it, and reuse it on every subsequent launch. That means you don't need to manage downloads yourself:

from nobodywho import Chat

chat = Chat('huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf')

The first time this runs, the model is downloaded to the platform cache directory. Every call after that loads the model directly.

model_path accepts a few different forms:

Form	Example	Notes
HuggingFace reference	`hf:owner/repo/file.gguf`	Downloaded and cached on first use
HTTPS URL	`https://example.com/model.gguf`	Downloaded and cached on first use
Local path	`./model.gguf`	Used as-is, no download

The HuggingFace prefix is case-insensitive and the // is optional, so hf:, hf://, huggingface:, and huggingface:// are all equivalent.

You can track a remote download by calling download_model directly and passing an on_download_progress callback. It receives (downloaded_bytes, total_bytes) and is skipped for cached or local files — if you don't pass anything, NobodyWho prints a default terminal progress bar:

from nobodywho import download_model, Chat

model_path = download_model(
    'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
    on_download_progress=lambda downloaded, total: print(f"{downloaded}/{total} bytes"),
)
chat = Chat(model_path)

You can find thousands of LLMs in .gguf format on Hugging Face here.

Basic Chat

With a model loaded, you're ready to start a conversation:

from nobodywho import Chat

chat = Chat('huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf')
response = chat.ask('Is water wet?').completed()
print(response)  # Yes, indeed, water is wet!

chat.ask() sends your message and returns a TokenStream. Calling .completed() blocks until the model is done generating and gives you back the final string, which is fine for a one-off question. But a real chat interface needs to stream tokens as they arrive, otherwise users stare at a blank screen until generation finishes.

Streaming Tokens

response = chat.ask('What is the capital of Denmark?')

for token in response:
    print(token, end="", flush=True)

A token is the smallest unit a model generates, typically a word, or a fragment of a word.

If you'd rather not block synchronously while a model loads or generates, swap Chat for ChatAsync — the API stays the same, so await chat.ask(...).completed() or async for token in chat.ask(...) both work.

Multimodal Models

from nobodywho import Model, Chat

model = Model("./vision-model.gguf", projection_model_path="./projection_model.gguf")
chat = Chat(model, system_prompt="You are a helpful assistant, that can hear and see stuff!")

To actually send image or audio content, build a Prompt mixing text, images, and audio, and pass it to chat.ask() instead of a plain string:

from nobodywho import Audio, Image, Prompt, Text

prompt = Prompt([
    Text("Tell me what you see in the image and what you hear in the audio."),
    Image("./dog.png"),
    Audio("./sound.mp3"),
])
response = chat.ask(prompt).completed()

Keep in mind that images and audio consume context fast, so you'll likely want a bigger n_ctx than you'd use for text-only chat. Also note that the language model and its projection model have to be trained together — you can't mix an LLM and a projection model you happen to like and expect them to work.

Speech to Text

If you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through SpeechToText.

from nobodywho import SpeechToText

stt = SpeechToText(source="hf://onnx-community/whisper-base")

text = stt.transcribe_file("recording.mp3").completed()
print(text)

source is a Hugging Face repo (hf://owner/repo) or a local directory laid out the same way. Browse the Whisper ONNX models on Hugging Face to find one that fits your accuracy and speed needs.

If your audio comes from a buffer rather than a file, use transcribe_pcm:

text = stt.transcribe_pcm(samples, sample_rate=16000).completed()

for piece in stt.transcribe_file("recording.mp3"):
    print(piece, end="", flush=True)

Text to Speech

Going the other direction, TextToSpeech turns text into WAV audio you can play back or save.

from pathlib import Path
from nobodywho import TextToSpeech

tts = TextToSpeech(
    source="hf://NobodyWho/Kokoro-82M",
    voice="bf_emma",
    language="en-gb",
)

wav = tts.synthesize("Hello from NobodyWho!")
Path("out.wav").write_bytes(wav)

Each architecture has its own voice and language options that need to agree with what the model supports.

Voice Activity Detection

For streaming microphone input, push chunks in as they arrive:

from nobodywho import VoiceActivityDetection, VoiceActivityDetectionEvent, SpeechToText

vad = VoiceActivityDetection(source="hf://onnx-community/silero-vad", sample_rate=16000)
stt = SpeechToText(source="hf://onnx-community/whisper-base")

while chunk := read_mic():
    if vad.push(chunk) == VoiceActivityDetectionEvent.SpeechEnded:
        break

speech = vad.finish()
transcription = stt.transcribe_pcm(speech, sample_rate=16000).completed()
print(transcription)

Each .push() call reports the current state (SpeechStarted, SpeechEnded, Speech, or Silence), and .finish() hands you back the buffered speech segment while resetting internal state for the next turn.

If you already have a full recording and just want to pull out the speech segments from it, .segment() does that in one pass:

audio = read_wav_pcm("recording.wav")

for speech in vad.segment(audio):
    transcription = stt.transcribe_pcm(speech, sample_rate=16000).completed()
    print(transcription)

Sensitivity is tunable via threshold, min_speech_duration_ms, min_silence_duration_ms, and preroll_duration_ms (how much audio to keep before the detected start, so you don't clip the beginning of a sentence). The defaults are a reasonable starting point, but VAD is one of those things that usually benefits from tuning to your actual environment.

Tool Calling

Tools let the model call out to real functions in your app rather than just generating text. Any synchronous function that returns a string becomes a tool with the @tool decorator:

import math
from nobodywho import tool, Chat

@tool(description="Calculates the area of a circle given its radius")
def circle_area(radius: float) -> str:
    area = math.pi * radius ** 2
    return f"Circle with radius {radius} has area {area:.2f}"

chat = Chat('./model.gguf', tools=[circle_area])

NobodyWho inspects the function's name, parameter names, and types to describe the tool to the model — add a params dict to @tool if a parameter needs more explanation than its name alone gives.

NobodyWho also ships two general-purpose tools out of the box, a Python interpreter and a Bash interpreter, for models that need to reason precisely or compute something:

from nobodywho import python_tool, bash_tool

chat = Chat('./model.gguf', tools=[python_tool(), bash_tool()])

Not every model supports tool calling well, the Qwen family is a solid choice if you need it to be reliable. See the Tool Calling documentation for more.

RAG

from nobodywho import Encoder, cosine_similarity

encoder = Encoder('./embedding-model.gguf')

query_embedding = encoder.encode("What is the return policy?")
doc_embeddings = encoder.encode_batch(knowledge)
similarities = [cosine_similarity(query_embedding, doc_emb) for doc_emb in doc_embeddings]

For better precision, rerank the top candidates with a CrossEncoder before handing them to the model:

from nobodywho import CrossEncoder

crossencoder = CrossEncoder('./reranker-model.gguf')
ranked = crossencoder.rank_and_sort("What is the return policy?", top_docs)

See the Embeddings & RAG documentation for the full walkthrough, including how to wire this up as a tool the model calls automatically.

What's Next?

You now have a complete foundation for building on-device AI features in Python:

Download and run a GGUF model
Send messages and get streamed tokens back, synchronously or with ChatAsync
Feed images and audio directly into a multimodal model
Transcribe speech, synthesize it back, and detect when someone's actually talking
Extend the model with tool calling and perform search with RAG

Links

Github - PyPI - Docs

On-Device AI in Kotlin

pielouNW — Wed, 02 Sep 2026 11:03:27 +0000

In this Kotlin tutorial, you'll learn how to run a large language model (LLM) directly on a user's device: no server, no API key needed. We'll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.

Each concept is explained before the code, so you can follow along whether you're new to on-device AI.

Why run AI On-Device?

Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.

Running the model directly on the device avoids all of them:

Works offline — no internet connection required
Privacy by design — user data never leaves the device
Low latency — no network round-trip
No cloud costs — inference is free

About NobodyWho

We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and ships bindings for several languages and frameworks: Kotlin, Python, Expo/React Native, Swift, Flutter & Godot. It exposes a clean API for running any model locally in .gguf format, on Android and desktop JVM (Linux, macOS, Windows).

Add it to your build.gradle.kts:

// Android
implementation("ai.nobodywho:nobodywho-android:2.2.0")

// Desktop JVM (Linux, macOS, Windows)
implementation("ai.nobodywho:nobodywho:2.2.0")

Loading a Model

import ai.nobodywho.Chat

val chat = Chat.fromPath(
    modelPath = "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)

The first time this runs, the model is downloaded to the platform cache directory. Every call after that loads the model directly.

modelPath accepts a few different forms:

Form	Example	Notes
HuggingFace reference	`hf:owner/repo/file.gguf`	Downloaded and cached on first use
HTTPS URL	`https://example.com/model.gguf`	Downloaded and cached on first use
Local path	`./model.gguf`	Used as-is, no download

You can track a remote download by passing a callback as the trailing lambda to Chat.fromPath. It receives (downloadedBytes, totalBytes) and is skipped for cached or local files:

val chat = Chat.fromPath(
    modelPath = "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
) { downloaded, total ->
    println("$downloaded / $total bytes")
}

You can find thousands of LLMs in .gguf format on Hugging Face here.

Basic Chat

With a model loaded, you're ready to start a conversation:

import ai.nobodywho.Chat
import kotlinx.coroutines.runBlocking

fun main() = runBlocking {
    val chat = Chat.fromPath(
        modelPath = "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
    )
    val response = chat.ask("Is water wet?").completed()
    println(response) // Yes, indeed, water is wet!
}

On Android, use lifecycleScope or viewModelScope instead of runBlocking.

Streaming Tokens

chat.ask("What is the capital of Denmark?").asFlow().collect { token ->
    print(token)
}

A token is the smallest unit a model generates, typically a word, or a fragment of a word.

Multimodal Models

import ai.nobodywho.Model
import ai.nobodywho.Chat

val model = Model.load(
    modelPath = "./vision-model.gguf",
    projectionModelPath = "./mmproj.gguf"
)
val chat = Chat(model = model)

To actually send image or audio content, build a Prompt mixing text, images, and audio, and pass it to chat.ask() instead of a plain string:

import ai.nobodywho.Prompt

val response = chat.ask(Prompt(
    Prompt.Text("Tell me what you see in the image and what you hear in the audio."),
    Prompt.Image("./dog.png"),
    Prompt.Audio("./sound.mp3"),
)).completed()
println(response)

Speech to Text

If you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through SpeechToText.

import ai.nobodywho.SpeechToText

val stt = SpeechToText.load(source = "hf://onnx-community/whisper-base")

val text = stt.transcribeFile("recording.mp3").completed()
println(text)

source is a Hugging Face repo (hf://owner/repo) or a local directory laid out the same way. Browse the Whisper ONNX models on Hugging Face to find one that fits your accuracy and speed needs.

If your audio comes from a buffer rather than a file, use transcribePcm:

val text = stt.transcribePcm(samples, sampleRate = 16000u).completed()

stt.transcribeFile("recording.mp3").asFlow().collect { piece ->
    print(piece)
}

Text to Speech

Going the other direction, TextToSpeech turns text into WAV audio you can play back or save.

import ai.nobodywho.TextToSpeech
import java.io.File
import kotlinx.coroutines.runBlocking

fun main() = runBlocking {
    val tts = TextToSpeech.load(
        source = "hf://NobodyWho/Kokoro-82M",
        voice = "bf_emma",
        language = "en-gb",
    )
    val wav = tts.synthesize(text = "Hello from NobodyWho!")
    File("out.wav").writeBytes(wav)
}

Each architecture has its own voice and language options that need to agree with what the model supports.

Voice Activity Detection

For streaming microphone input, push chunks in as they arrive:

import ai.nobodywho.VoiceActivityDetection
import ai.nobodywho.VoiceActivityDetectionEvent
import ai.nobodywho.SpeechToText

val vad = VoiceActivityDetection.load(sampleRate = 16000u, source = "hf://onnx-community/silero-vad")
val stt = SpeechToText.load(source = "hf://onnx-community/whisper-base")

while (true) {
    val chunk = readMic()
    if (vad.push(chunk) == VoiceActivityDetectionEvent.SPEECH_ENDED) break
}

val speech = vad.finish()
val transcription = stt.transcribePcm(speech, sampleRate = 16000u).completed()
println(transcription)

Each push() call reports the current state (SPEECH_STARTED, SPEECH_ENDED, SPEECH, or SILENCE), and finish() hands you back the buffered speech segment while resetting internal state for the next turn.

If you already have a full recording and just want to pull out the speech segments from it, segment() does that in one pass:

val audio = readWavPcm("recording.wav")

for (speech in vad.segment(audio)) {
    val transcription = stt.transcribePcm(speech, sampleRate = 16000u).completed()
    println(transcription)
}

Tool Calling

Tools let the model call out to real functions in your app rather than just generating text. NobodyWho uses Kotlin reflection to inspect a function's parameter names and types, so declaring a tool is as simple as passing a function reference:

import ai.nobodywho.Tool

fun getWeather(city: String, unit: String): String {
    return """{"temp": 22, "unit": "$unit"}"""
}

val weatherTool = Tool(
    name = "get_weather",
    description = "Get the current weather for a city",
    function = ::getWeather
)

val chat = Chat.fromPath(
    modelPath = "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
    tools = listOf(weatherTool)
)

Suspend functions work the same way, so a tool can make a network call without blocking the caller. Because it relies on reflection, the function has to be a top-level function, a class method, or a companion object method — local functions defined inside another function or coroutine aren't supported.

Not every model supports tool calling well, the Qwen family is a solid choice if you need it to be reliable. See the Tool Calling documentation for more.

RAG

import ai.nobodywho.Encoder
import ai.nobodywho.cosineSimilarity

val encoder = Encoder.fromPath(modelPath = "./embedding-model.gguf")

val queryEmbedding = encoder.encode("What is the return policy?")
val docEmbeddings = encoder.encodeBatch(knowledge)
val similarities = docEmbeddings.map { cosineSimilarity(queryEmbedding, it) }

For better precision, rerank the top candidates with a CrossEncoder before handing them to the model:

import ai.nobodywho.CrossEncoder

val crossEncoder = CrossEncoder.fromPath(modelPath = "./reranker-model.gguf")
val ranked = crossEncoder.rankAndSort("What is the return policy?", topDocs)

See the Embeddings & RAG documentation for the full walkthrough, including how to pass the reranked documents into the chat's system prompt.

What's Next?

You now have a complete foundation for building on-device AI features in Kotlin:

Download and run a GGUF model
Send messages and get streamed tokens back
Feed images and audio directly into a multimodal model
Transcribe speech, synthesize it back, and detect when someone's actually talking
Extend the model with tool calling and perform search with RAG

Links

Github - Docs - Maven Central

The best AI SDKs for on-device inference in 2026

pielouNW — Wed, 02 Sep 2026 07:37:03 +0000

Local AI stopped being a research demo. Phones, laptops and headsets now have enough compute to run capable language, speech, and vision models locally. It means you can build AI features that work offline, keep user data on the device, and cost nothing per request.

The hard part is no longer whether you can run a model on-device. It's picking the library that gets you from a downloaded model file to a working feature in the language you actually ship in, without giving up privacy, speed, or the features you need.

This guide compares the best AI SDKs for on-device inference in 2026, with a side-by-side competitor table and an honest look at what each one is good at.

Why run AI on-device?

Most AI features today route every request through a hosted API: you send data to a remote server, it runs the model, and sends a response back. That works, but it comes with tradeoffs that get worse as you scale.

Running the model directly on the device avoids them:

Privacy by design — user data never leaves the device
Works offline — no internet connection required, anywhere
Low latency — no network round-trip on every interaction
No cloud costs — inference is free, with no per-token bill
No infrastructure — no servers to provision, scale, or babysit

The tradeoff is raw capability, a local model is smaller than a frontier cloud model, but for chat, summarization, classification, transcription, structured extraction, and many agentic workflows, a good 1B–8B model running locally is more than enough.

What to look for in an on-device AI SDK

Not every SDK that can run a model on a device is a good fit for shipping a real product. When you're evaluating one, look for:

Cross-platform bindings — does it support the language and framework you actually build in (Swift, Kotlin, Flutter, React Native, Python, Godot)?
A fast, portable runtime — GPU acceleration (Metal, Vulkan) and CPU fallback so it runs well across the whole device fleet.
Model format support — broad compatibility (e.g. GGUF) so you can use thousands of open-weight models instead of a walled garden.
More than just LLMs — text-to-speech, speech-to-text, voice activity detection, and multimodal input, so a voice assistant doesn't require five separate libraries.
Developer ergonomics — streaming, tool calling, structured output, embeddings/RAG, and model downloading built in.
A sane license and open source — so you can audit it, trust it, and ship it without surprises.

Competitor comparison

SDK	How it runs on-device	Languages / platforms	Beyond LLM	License
NobodyWho	Rust core over llama.cpp; GPU via Metal/Vulkan, CPU fallback; GGUF	Swift, Kotlin, Flutter, React Native / Expo, Python, Godot	TTS, STT, VAD, multimodal, tool calling, RAG	EUPL v1.2 (open source)
llama.cpp	Reference C/C++ engine; Metal/Vulkan/CUDA; GGUF	C/C++ (community bindings vary)	LLM + some multimodal; you assemble the rest	MIT (open source)
Google AI Edge / LiteRT (MediaPipe)	LiteRT runtime; NNAPI/GPU delegates	Android, iOS, web	LLM + vision + classic ML	Apache 2.0 (open source)
ONNX Runtime Mobile	ONNX graph execution; execution providers	C/C++, Swift, Kotlin, Python	CV, audio, NLP building blocks	MIT (open source)
Apple MLX	Apple-silicon array framework	Swift, Python (Apple only)	LLM + more, research-oriented	MIT (open source)
RunAnywhere	On-device runtime + hybrid cloud routing; OTA models	React Native, Flutter, Kotlin, Swift, Web	LLM, STT, TTS, VAD, vision	Source-available (commercial cap)
Cactus	On-device engine (proprietary `.cact` format) + cloud fallback	React Native, Flutter, Kotlin, Swift, Python, Rust	LLM, vision, STT, VAD, embeddings	Source-available (commercial cap)

The best AI SDKs for on-device inference

1. NobodyWho

NobodyWho is an inference engine for running AI models locally and efficiently. A single, well-maintained Rust core, wrapping the excellent llama.cpp. It supports several languages and frameworks Swift, Kotlin, Flutter, React Native / Expo, Python, and Godot with a consistant API naming.

What makes it stand out for on-device work specifically:

Runs fully offline and free — no API keys, no servers, no hidden fees
A complete on-device stack, not just an LLM — text-to-speech (Kokoro, Pocket-tts, Supertonic), speech-to-text (Whisper), and voice activity detection (Silero VAD) are first-class, so you can build a full local voice assistant without gluing five libraries together.
Multimodal input — feed images and audio to your model.
Type-safe tool calling — it generates structured grammars from your function signatures automatically, so you never hand-write a JSON schema.
Fast and efficient — GPU-accelerated inference via Metal and Vulkan, with conversation-aware preemptive context shifting so you keep full conversation memory without message-length limits.
Any GGUF model — Gemma, Qwen, Mistral, Llama, and thousands more, loaded directly from Hugging Face or any URL.
Genuinely open source — EUPL v1.2, a good open-source citizen with 500+ closed PRs.

It's also proven in production: the team ships free, open-source apps built entirely on NobodyWho, including NobodyWho Chat (iOS/Android), NobodyWho Wrist the first app to run an LLM entirely on an Apple Watch, and NobodyWho Eyes for Vision Pro.

Best for: developers who want efficient, private, cross-platform on-device & complete AI stack.

2. llama.cpp

The reference C/C++ engine that most of the on-device world is built on, including NobodyWho. It defines the GGUF format, supports Metal/Vulkan/CUDA, and runs an enormous range of quantized models efficiently on CPUs and GPUs.

Strengths: unmatched model coverage, excellent performance, active community.

Limitations: it's a low-level engine, not an app SDK. You get the runtime; you build the bindings, the voice/vision stack, tool calling, and the ergonomics yourself. Great as a foundation, a lot of work as a product SDK.

3. Google AI Edge / LiteRT (MediaPipe)

Google's on-device stack — LiteRT (formerly TensorFlow Lite) plus MediaPipe's LLM Inference API — with hardware delegates (NNAPI, GPU) and a large ecosystem for vision and classic ML.

Strengths: mature tooling, strong quantization and hardware acceleration, excellent for computer-vision and classical-ML workloads.

Limitations: the LLM side is newer and narrower, model conversion can be fiddly, and it doesn't offer a unified LLM+voice+vision developer experience across every framework.

4. ONNX Runtime Mobile

A compact, portable runtime that executes ONNX graphs with pluggable execution providers across CPU, GPU, and NPUs.

Strengths: framework-agnostic, small footprint, solid building blocks for CV, audio, and NLP.

Limitations: it's infrastructure, not an on-device LLM/voice product. Getting good mobile LLM ergonomics — streaming, tool calling, context management — requires meaningful extra work.

5. Apple MLX

Apple's array framework for machine learning on Apple silicon, with Swift and Python APIs and a growing collection of ported models.

Strengths: beautifully tuned for Apple hardware, great for experimentation and research.

Limitations: Apple-only by design — no Android, no cross-platform story — and lower-level than a batteries-included app SDK.

6. RunAnywhere

An SDK that pairs an on-device runtime with hybrid cloud routing, OTA model distribution, and fleet management for LLM, speech (STT/TTS/VAD), and vision models, with bindings for React Native, Flutter, Kotlin, Swift, and Web.

Strengths: cross-platform SDKs, hybrid on-device/cloud routing, and operational controls like model rollouts and observability.

Limitations: cloud routing, over-the-air model updates, and telemetry run through RunAnywhere's own hosted backend, with no self-hosting option, so you're tied to their cloud for the operational side. Its licence isn't truly open source either: it's Apache 2.0 with a commercial-use cap, free only for individuals, non-profits, and organisations under $1M in both funding and annual revenue.

7. Cactus

Cactus is a hybrid inference engine for mobile and edge, with cross-platform SDKs covering LLMs, vision, speech-to-text, VAD, and embeddings, plus optional cloud fallback.

Strengths: low setup, straightforward integration, and real on-device NPU acceleration across major mobile chipsets (Apple, Snapdragon, Google, Exynos, MediaTek).

Limitations: it uses its own proprietary .cact model format rather than GGUF, so models have to be converted with cactus convert first — and while any Hugging Face model can in theory go through it, that path is still very experimental, so getting the model you actually want running on-device is far from guaranteed. Its hybrid cloud routing also runs through Cactus's own hosted backend with no self-hosting option, and the hybrid features sit behind a paid tier. Its licence isn't truly open source either: it's source-available with a commercial-use cap, free only for individuals, non-profits, and organisations under $2M in both funding and annual revenue.

Conclusion: NobodyWho is the best AI SDK for on-device inference

Most of the field is either a low-level engine (llama.cpp, ONNX Runtime, MLX) that leaves you to build the app layer yourself, a single-platform solution (MLX, LiteRT's strengths), or a commercial stack tied to a vendor-hosted backend you can't self-host (RunAnywhere, Cactus).

NobodyWho is the one option that puts it all together: a fast, GPU-accelerated runtime on top of the best portable inference engine, a complete on-device stack (LLM + TTS + STT + VAD + multimodal), type-safe tool calling and RAG, use of any GGUF model, and support for every major framework while working fully offline, private by design, and already shipping in real apps on phones, watches, and headsets.

If you're building an AI feature that has to be private, work offline, cost nothing per request, and ship across platforms, NobodyWho is the best AI SDK for on-device inference in 2026.

Get started

GitHub — star the repo, open an issue, or start a discussion.
Documentation — pick your framework and ship your first on-device model.
Discord — ask questions, share what you're building, and chat with the team and other NobodyWho users.

Happy hacking!

On-Device AI for iOS & macOS

pielouNW — Tue, 01 Sep 2026 13:42:20 +0000

In this Swift tutorial, you'll learn how to run a large language model (LLM) directly on a user's device: no server, no API key needed. We'll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.

Each concept is explained before the code, so you can follow along whether you're new to on-device AI.

Why run AI On-Device?

Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.

Running the model directly on the device avoids all of them:

Works offline — no internet connection required
Privacy by design — user data never leaves the device
Low latency — no network round-trip
No cloud costs — inference is free

About NobodyWho

We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and ships bindings for several languages and frameworks: Kotlin, Python, Expo/React Native, Swift, Flutter & Godot. It exposes a clean API for running any model locally in .gguf format, across iOS, macOS, visionOS, and watchOS.

Add it with Swift Package Manager. In Xcode, go to File → Add Package Dependencies and enter:

https://github.com/nobodywho-ooo/nobodywho-swift.git

Or add it to your Package.swift:

dependencies: [
    .package(url: "https://github.com/nobodywho-ooo/nobodywho-swift.git", from: "2.1.0")
]

Loading a Model

import NobodyWho

let chat = try await Chat.fromPath(
    modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)

The first time this runs, the model is downloaded to the platform cache directory. Every call after that loads the model directly.

modelPath accepts a few different forms:

Form	Example	Notes
HuggingFace reference	`hf://owner/repo/file.gguf`	Downloaded and cached on first use
HTTPS URL	`https://example.com/model.gguf`	Downloaded and cached on first use
Local path	`/path/to/model.gguf`	Used as-is, no download

The HuggingFace prefix is case-insensitive, so hf:// and huggingface:// are equivalent. You can also track a remote download by passing a progress closure to Chat.fromPath, which receives (downloaded, total) byte counts and is skipped for cached or local files:

let chat = try await Chat.fromPath(
    modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
) { downloaded, total in
    print("Downloaded \(downloaded)/\(total) bytes")
}

You can find thousands of LLM in .gguf format on Hugging Face here.

Basic Chat

With a model loaded, you're ready to start a conversation:

let chat = try await Chat.fromPath(
    modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)
let response = try await chat.ask("Is water wet?").completed()
print(response) // Yes, indeed, water is wet!

chat.ask() sends your message and returns a TokenStream, which conforms to AsyncSequence. Calling .completed() waits for the whole response and gives you back the final string, which is fine for a one-off question. But a real chat interface needs to stream tokens as they arrive, otherwise users stare at a blank screen until generation finishes.

Streaming Tokens

let response = chat.ask("What is the capital of Denmark?")

for await token in response {
    print(token)
}

A token is the smallest unit a model generates, typically a word, or a fragment of a word.

If you need to cancel a response mid-generation, for example when the user taps a "Stop" button, call chat.stopGeneration(). It's synchronous and safe to call from any thread; the tokens already produced stay in the stream and are kept in the chat history, so the conversation stays coherent.

Multimodal Models

import NobodyWho

let chat = try await Chat.fromPath(
    modelPath: "/path/to/vision-model.gguf",
    projectionModelPath: "/path/to/mmproj.gguf"
)

To actually send image or audio content, build a Prompt mixing text, images, and audio, and pass it to chat.ask() instead of a plain string:

let prompt = Prompt([
    Prompt.text("Tell me what you see in the image and what you hear in the audio."),
    Prompt.image("/path/to/dog.png"),
    Prompt.audio("/path/to/sound.mp3"),
])
let response = try await chat.ask(prompt).completed()

Speech to Text

If you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through SpeechToText.

import NobodyWho

let stt = try await SpeechToText.load(source: "hf://onnx-community/whisper-base")

let text = try await stt.transcribeFile(path: "recording.mp3").completed()
print(text)

source is a Hugging Face repo (hf://owner/repo) or a local directory laid out the same way. Browse the Whisper ONNX models on Hugging Face to find one that fits your accuracy and speed needs.

If your audio comes from a buffer rather than a file, use transcribePcm:

let text = try await stt.transcribePcm(samples: samples, sampleRate: 16000).completed()

for try await piece in stt.transcribeFile(path: "recording.mp3") {
    print(piece)
}

Text to Speech

Going the other direction, TextToSpeech turns text into WAV audio you can play back or save.

import Foundation
import NobodyWho

let tts = try await TextToSpeech.load(
    source: "hf://NobodyWho/Kokoro-82M",
    voice: "bf_emma",
    language: "en-gb"
)

let wav = try await tts.synthesize("Hello from NobodyWho!")
try wav.write(to: URL(fileURLWithPath: "out.wav"))

Each architecture has its own voice and language options that need to agree with what the model supports.

Voice Activity Detection

For streaming microphone input, push chunks in as they arrive:

import NobodyWho

let vad = try await VoiceActivityDetection.load(sampleRate: 16000, source: "hf://onnx-community/silero-vad")
let stt = try await SpeechToText.load(source: "hf://onnx-community/whisper-base")

while let chunk = readMic() {
    if try vad.push(chunk: chunk) == .speechEnded {
        break
    }
}

let speech = vad.finish()
let transcription = try await stt.transcribePcm(samples: speech, sampleRate: 16000).completed()
print(transcription)

Each push call reports the current state (.speechStarted, .speechEnded, .speech, or .silence), and finish() hands you back the buffered speech segment while resetting internal state for the next turn.

If you already have a full recording and just want to pull out the speech segments from it, segment does that in one pass:

let audio = readWavPcm(path: "recording.wav")

for speech in try vad.segment(samples: audio) {
    let transcription = try await stt.transcribePcm(samples: speech, sampleRate: 16000).completed()
    print(transcription)
}

Tool Calling

Tools let the model call out to real functions in your app rather than just generating text. The easiest way to create one is with the @DeclareTool macro: annotate any top-level or type-member function with a description, and NobodyWho generates the tool for you.

import NobodyWho

@DeclareTool("Calculates the area of a circle given its radius")
func circleArea(radius: Double) -> String {
    let area = Double.pi * radius * radius
    return "Circle with radius \(radius) has area \(String(format: "%.2f", area))"
}

let chat = try await Chat.fromPath(
    modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
    tools: [circleAreaTool]
)

The macro names the generated variable <functionName>Tool, so circleArea becomes circleAreaTool. It doesn't work inside function bodies though, for a tool that needs to capture local state, use the manual Tool initializer instead.

Not every model supports tool calling well, the Qwen family is a solid choice if you need it to be reliable. See the Tool Calling documentation for more.

RAG

import NobodyWho

let encoder = try await Encoder.fromPath(modelPath: "/path/to/embeddings.gguf", contextSize: 512, useGpu: true)

let queryEmbedding = try await encoder.encode("What is the return policy?")
let docEmbeddings = try await encoder.encodeBatch(knowledge)
let similarities = docEmbeddings.map { cosineSimilarity(a: queryEmbedding, b: $0) }

For better precision, rerank the top candidates with a CrossEncoder before handing them to the model:

let crossEncoder = try await CrossEncoder.fromPath(modelPath: "/path/to/reranker.gguf", contextSize: 512, useGpu: true)
let ranked = try await crossEncoder.rankAndSort(query: "What is the return policy?", documents: topDocs)

See the Embeddings & RAG documentation for the full walkthrough, including how to pass the reranked documents into the chat's system prompt.

What's Next?

You now have a complete foundation for building on-device AI features in Swift:

Download and run a GGUF model
Send messages and get streamed tokens back
Feed images and audio directly into a multimodal model
Transcribe speech, synthesize it back, and detect when someone's actually talking
Extend the model with tool calling and perform search with RAG

You can also have a look at the iOS & macOS starter examples to see a full implementation.

Links

Github - Docs - SPI

On-Device AI in React Native & Expo

pielouNW — Tue, 01 Sep 2026 12:52:40 +0000

In this Expo & React Native tutorial, you’ll learn how to run a large language model (LLM) directly on a user’s device: no server, no API key needed. We’ll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.

Each concept is explained before the code, so you can follow along whether you're new to on-device AI.

Why run AI On-Device?

Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.

Running the model directly on the device avoids all of them:

Works offline — no internet connection required
Privacy by design — user data never leaves the device
Low latency — no network round-trip
No cloud costs — inference is free

About NobodyWho

We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and ships bindings for several languages and frameworks: Kotlin, Python, Expo/React Native, Swift, Flutter & Godot. It exposes a clean API for running locally any model in .gguf format.

Install it with npm install react-native-nobodywho or npx expo install react-native-nobodywho for Expo.

Loading a Model

import { Chat } from "react-native-nobodywho";

const chat = await Chat.fromPath({
  modelPath: "huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
});

The first time this runs, the model is downloaded to the app’s cache directory. Every call after that loads the model directly.

modelPath accepts a few different forms:

Form	Example	Notes
HuggingFace reference	`hf:owner/repo/file.gguf`	Downloaded and cached on first use
HTTPS URL	`https://example.com/model.gguf`	Downloaded and cached on first use
Local path	`./model.gguf`	Used as-is, no download

You can track a remote download by passing onDownloadProgress to Chat.fromPath:

const chat = await Chat.fromPath({
  modelPath: "huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
  onDownloadProgress: (downloaded, total) => {
    console.log(`${downloaded} / ${total} bytes`);
  },
});

You can find thousands of LLMs in .gguf format on Hugging Face here.

Basic Chat

With a model loaded, you're ready to start a conversation:

const chat = await Chat.fromPath({
  modelPath: "huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
});
const response = await chat.ask("Is water wet?").completed();
console.log(response); // Yes, indeed, water is wet!

Streaming Tokens

const response = chat.ask("What is the capital of Denmark?");

for await (const token of response) {
  console.log(token); // Each token arrives as it's generated
}

A token is the smallest unit a model generates, typically a word, or a fragment of a word.

Multimodal Models

import { Chat } from "react-native-nobodywho";

const chat = await Chat.fromPath({
  modelPath: "/path/to/vision-model.gguf",
  projectionModelPath: "/path/to/mmproj.gguf",
});

To actually send image or audio content, build a Prompt mixing text, images, and audio, and pass it to chat.ask() instead of a plain string:

import { Chat, Prompt } from "react-native-nobodywho";

const response = await chat
  .ask(
    new Prompt([
      Prompt.Text("Tell me what you see in the image and what you hear in the audio."),
      Prompt.Image("/path/to/dog.png"),
      Prompt.Audio("/path/to/sound.mp3"),
    ]),
  )
  .completed();

Keep in mind that images and audio consume context fast, so you'll likely want a bigger contextSize than you'd use for text-only chat.

Speech to Text

If you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through SpeechToText.

import { SpeechToText } from "react-native-nobodywho";

const stt = await SpeechToText.load({
  source: "hf://onnx-community/whisper-base",
});

const text = await stt.transcribeFile("recording.mp3").completed();
console.log(text);

source is a Hugging Face repo (hf://owner/repo) or a local directory laid out the same way. Browse the Whisper ONNX models on Hugging Face to find one that fits your accuracy and speed needs.

If your audio comes from a microphone buffer rather than a file, use transcribePcm:

const text = await stt.transcribePcm(samples, 16000).completed();

Text to Speech

Going the other direction, TextToSpeech turns text into WAV audio you can play back or save.

import { TextToSpeech } from "react-native-nobodywho";

const tts = await TextToSpeech.load({
  source: "hf://NobodyWho/Kokoro-82M",
  voice: "bf_emma",
  language: "en-gb",
});

const wav = await tts.synthesize("Hello from NobodyWho!");
// wav is a Uint8Array containing WAV bytes.

Each architecture has its own voice and language options that need to agree with what the model supports.

Voice Activity Detection

For streaming microphone input, push chunks in as they arrive:

import {
  VoiceActivityDetection,
  VoiceActivityDetectionEvent,
  SpeechToText,
} from "react-native-nobodywho";

const vad = await VoiceActivityDetection.load({
  sampleRate: 16000,
  source: "hf://onnx-community/silero-vad",
});
const stt = await SpeechToText.load({ source: "hf://onnx-community/whisper-base" });

while (true) {
  const chunk = readMic();
  if (vad.push(chunk) === VoiceActivityDetectionEvent.SpeechEnded) break;
}

const speech = vad.finish();
const transcription = await stt.transcribePcm(speech, 16000).completed();
console.log(transcription);

Each push() call reports the current state (SpeechStarted, SpeechEnded, Speech, or Silence), and finish() hands you back the buffered speech segment while resetting internal state for the next turn.

If you already have a full recording and just want to pull out the speech segments from it, segment() does that in one pass:

const audio = readWavPcm("recording.wav");

for (const speech of vad.segment(audio)) {
  const transcription = await stt.transcribePcm(speech, 16000).completed();
  console.log(transcription);
}

Tool Calling

Tools let the model call out to real functions in your app rather than just generating text. You define a name, a description, and a set of typed parameters, and NobodyWho takes care of getting the model to call it correctly:

import { Chat, Tool } from "react-native-nobodywho";

const circleAreaTool = new Tool({
  name: "circle_area",
  description: "Calculates the area of a circle given its radius",
  parameters: [
    { name: "radius", type: "number", description: "The radius of the circle" },
  ],
  call: (radius: number) => `Area is ${(Math.PI * radius * radius).toFixed(2)}`,
});

const chat = await Chat.fromPath({
  modelPath: "huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
  tools: [circleAreaTool],
});

Not every model supports tool calling well, the Qwen family is a solid choice if you need it to be reliable. See the Tool Calling documentation for more.

RAG

const searchKnowledgeTool = new Tool({
  name: "search_knowledge",
  description: "Search the knowledge base for relevant information",
  parameters: [{ name: "query", type: "string", description: "The search query" }],
  call: async (query: string) => {
    const ranked = await crossencoder.rankAndSort(query, knowledge);
    return ranked.slice(0, 3).map(([doc]) => doc).join("\n");
  },
});

See the Embeddings & RAG documentation for the full walkthrough.

What's Next?

You now have a complete foundation for building on-device AI features in React Native:

Download and run a GGUF model
Send messages and get streamed tokens back
Feed images and audio directly into a multimodal model
Transcribe speech, synthesize it back, and detect when someone's actually talking
Extend the model with tool calling and perform search with RAG

You can also have a look at the Expo or the React Native starter examples to see full implementation.

Links

Github - NPM - Docs

Beginner's Guide to Essential Terms in Artificial Intelligence

pielouNW — Thu, 07 May 2026 12:14:58 +0000

AI has its own language, and if you're just getting started, it can feel like everyone else got the memo but you.

Terms like tokens, inference, and quantization get tossed around in articles, videos, and job descriptions as if they're common knowledge, but they're not.

This guide helps you to navigate in the AI jungle, it covers the core AI vocabulary you'll encounter and defines it simply. Whether you're building something, exploring the field, or just trying to follow the conversation, these are the terms worth knowing.

You can also find this article on NobodyWho website and learn how to integrate LLMs into your applications.

1. The AI Stack

The big picture.

Artificial Intelligence (AI)

The field of computer science focused on building systems that can perform tasks that normally require human intelligence, like understanding text or audio, recognizing images or making decisions.

The term "AI" is ultimately a moving target. As the border between machine tasks and human tasks moves, the definition shifts too.

Machine Learning (ML)

A subset of AI where systems learn from data instead of being programmed. Rather than writing rules by hand, you feed the system examples and it figures out the patterns on its own.

Deep Learning (DL)

A subset of machine learning that uses neural networks with many layers to learn from large amounts of data. It's the technology behind most modern AI breakthroughs. State-of-the-art systems for image recognition, speech synthesis, and large language models all rely on it.

Generative AI (Gen AI)

Generative AI refers to Artificial Intelligence systems that are capable of creating new content such as text, images, audio, video, or code.

2. How Models Are Built

Behind the scenes.

Dataset

A structured collection of data used to train, test, or evaluate a model. Datasets can contain text, images, audio, or any other form of information. The quality and size of a dataset directly affects how well a model performs.

Training

The process of exposing a model to data so it can learn patterns. During training, the model adjusts its internal parameters millions (or billions) of times to get better at its task.

Parameters / Weights

The internal numerical values a model learns during training. Parameters are what the model actually "knows", before being fed any prompts. They encode the patterns extracted from training data. A model with 70 billion parameters has 70 billion of these numbers, all tuned to make its outputs as accurate as possible. Weights is another term for the same thing, often used when referring to the files you download for open-weight models.

Fine-tuning

The process of taking a pre-trained model and continuing to train it on a smaller, specialized dataset to adapt it to a specific task or style. Fine-tuning is faster and cheaper than training from scratch, and it's how generic models get turned into domain-specific ones.

Distillation

A training technique where a smaller model (the student) is trained to mimic the behavior of a larger model (the teacher). The goal is to compress the capabilities of a large, expensive model into a smaller, faster one.

Quantization

A technique for reducing a model's size by lowering the precision of its weights like for example, going from 32-bit floats to 8-bit integers. Quantized models are faster and cheaper to run, with a trade-off in accuracy.

3. What a Model Is

The different shapes a model can take.

Model

The output of training, which consists of one or several files that have learned to map inputs to outputs. Models can range from a few megabytes to several terabytes.

LLM (Large Language Model)

A type of deep learning model trained on massive amounts of text data to understand and generate human language. LLMs like GPT, Claude, and Gemini predict the next most likely word/token given a context. They're the engine behind most modern AI chat and writing tools.

SLM (Small Language Model)

A language model trained with fewer parameters than a typical LLM, designed to run efficiently on limited hardware like laptops, phones, or even smartwatches. SLMs are not categorically different from LLMs, but simply smaller variants.

Mixture of Experts (MoE)

An architecture where only a fraction of the model's parameters are used for any given token, rather than all of them. This means MoE models still need a lot of memory to hold all the weights, but they spend less compute per token, so they run faster than a dense model of equivalent size. Recent examples include DeepSeek, Mixtral, and Qwen's MoE variants.

Open-weights Models

Models that are publicly released, allowing anyone to download, run, and fine-tune them. Popular examples include Llama 3, Mistral, Qwen, Gemma and DeepSeek.

The term "open-weights" is used rather than "open-source" to specify exactly what is being released. "Open-source" refers to the publishing of source code, which is human-readable code used to produce a non-human-readable binary artifact (the compiled program). The model itself is a non-human-readable binary artifact, so the term "open-weights" is used to specify that it's the weights of the model that are open, and not necessarily the training source code or dataset that was used to produce the model.

Vision Model

A model specialized in processing and understanding images. Vision models can classify what's in an image, detect objects, generate captions, or power visual search.

Multimodal Model

A model that can process or generate more than one type of data like text, images, audio or video. For example, GPT-4o and Gemini are multimodal: you can send them an image and ask a question about it, or have them describe what they hear in an audio file.

Multimodal models aren't necessarily capable of ingesting and outputting the same types of data. Many multimodal models can receive image or text inputs, and only generate text outputs.

Reasoning Model vs. Thinking Model

These terms are often used interchangeably, but there's a subtle distinction.
A reasoning model is explicitly trained or prompted to work through problems step by step before producing an answer, breaking complex tasks into logical stages.
A thinking model typically refers to models that have a dedicated internal "thinking" phase, where the model processes before responding.
In practice, both aim to improve accuracy on complex tasks by slowing down the output process.

4. Language & Text Processing

How models read and represent text under the hood.

Token

The basic unit an LLM processes. For text, a token is roughly a word fragment, "learning" might be one token, while "incredible" might be split into two tokens: "in" and "credible". Models don't read characters or full words, they read tokens.

Tokenization

The process of converting some kind of input (text, image, audio, etc.) into tokens. All model inputs are converted to tokens before being fed into the model.

Embeddings

A way of representing data (text, images, audio) as vectors (lists of numbers), in a high-dimensional space. Similar concepts end up close together in that space. Embeddings are what allow models to understand that "king" and "queen" are related, or that a photo of a cat is similar to the word "cat."

Embeddings are particularly useful in RAG systems, to identify relevant sources of information to include.

5. Using a Model

The controls and inputs that shape how a model behaves.

Prompt

The input you give to a model. For language models, a prompt is the text, like a question, instruction, or context that the model responds to. For multimodal models, the prompt could also contain an image or some audio. Prompt quality directly affects output quality. Small changes in wording can produce significantly different results.

System Prompt

A special prompt, invisible to the end user, that sets the model's behavior, tone, and constraints before the conversation begins. Developers use system prompts to give a model its "personality" or restrict what it can and can't do. It's a configuration layer on top of the model. Most models are trained to prioritize following instructions in the system prompt over any subsequent instructions.

Context Window

The maximum amount of text a model can process at once, both input and output combined. If a model has a 128k token context window, it can "see" roughly 100,000 words at a time, since there are roughly 0.75 words per token. Anything outside the context window is invisible to the model.

Latency

The time it takes for a model to respond after receiving input.
In AI products, latency matters for user experience. It's influenced by model size, what device it's running on, and whether the output is streamed token by token or returned all at once.
It's useful to measure both the time-to-first-token (TTFT) and the time to complete an entire response. In use-cases where you can stream the tokens to the user, displaying the very first token when it is generated, TTFT matters most.

Token throughput

Typically measured in tokens per second, this is a measure of how quickly the model can process and generate tokens.
The speed to read a bunch of tokens and the speed to write a bunch of tokens are very different. The token throughput of reading is often around 10x as fast as the token throughput of writing tokens.

6. Sampling

Controlling how tokens are selected.

Sampling

Under the hood, generative language models output a probability distribution for the next token in a sequence. If given the sequence of tokens ["Once ", "upon ", "a "], a model might output a distribution with a high probability for the token "time", a much lower probability for the token "hill", and an incredibly low probability for nonsense tokens like "13".

In order to actually generate a sequence, we must select one of these tokens to accept and output to the user. This process of selecting a token from the probability distribution is known as sampling.

While it's tempting to simply select the most probable token, it has been shown that language models generate much better outputs when some randomness is applied. The field of sampling in LLMs is about designing exactly how this random selection works.

The Sampler Chain

A sampler consists of two phases:

First, any number of transformations is applied to the probability distribution. These steps might zero the probability of a bunch of tokens, or shift the distribution of all tokens.

You can try to play with the token probabilities on this website. If you drag-and-drop the sampling steps, you may notice that the order of steps applied can make a difference on the result.

Once the sequence of transformations has been applied, the sampler chain finalizes by selecting a token from that distribution.

Greedy Sampling

Greedy sampling is the sampling technique where you always select the most probable token, sidestepping any randomness in the sampling process. Greedy sampling leads to very predictable and boring output.

Dist sampling

Dist sampling is the practice of selecting a token randomly, weighted by each token's probability.

Temperature

A transformation that can be applied to token probability distributions to shift it towards preferring the more probable or the less probable tokens.
If a temperature of greater than 1 is applied, the high-probability tokens are made less likely, and the low-probability tokens are made more likely.
If a temperature of less than 1 is applied, the high-probability tokens are made more likely, and the low-probability tokens are made less likely.
A temperature of exactly 1 has no effect.

Low temperature makes the model more focused and deterministic, making it feel measured and predictable.
High temperature introduces more randomness and variation, making it feel creative and spontaneous.

Top-k

Top-k limits the model to choosing from only the k most likely tokens at each step. For example, top-k of 40 means only the 40 most probable options are considered.

Top-p

Top-p (also called nucleus sampling) is more dynamic: it picks from the smallest group of tokens whose combined probability adds up to p, so at top-p of 0.9, the model considers just enough tokens to cover 90% of the probability mass.

Grammar

A formal grammar can be applied as a transformation on token probabilities. This will exclude tokens (by setting their probability to zero), if they can't possibly result in a valid completion of the grammar. This can be used to guarantee that the output will always be compatible with a certain well-defined language, so a certain parser will always work. E.g. you can apply a formal grammar to force the model to only output valid JSON.

DRY

A transformation that reduces the likelihood of tokens if they have been used recently. This is useful for preventing models from repeating themselves.

7. How Models Think & Respond

What's actually happening when a model generates an output.

Inference

The act of running a trained model on a new input to get an output. Training is when a model learns; inference is when it's actually used. Most of what happens when you use an AI product like chatting, generating images, transcribing audio is inference.

Chain-of-Thought (CoT)

A prompting technique where the model is encouraged to reason step by step before giving a final answer, rather than jumping straight to a conclusion. By writing out intermediate reasoning, like a person writing their thoughts on paper, the model tends to make fewer mistakes on complex tasks.

Hallucination

When a model generates information that sounds confident but is factually wrong or completely made up. Hallucinations happen because models predict plausible sounding text, not verified truth.

8. Advanced Techniques

Methods that extend or enhance what models can do.

RAG (Retrieval-Augmented Generation)

A technique where a model retrieves relevant external information before generating a response. Instead of relying solely on what it learned during training, the model pulls in fresh data from a database or document store at inference time. It's a practical way to keep responses accurate and up to date.

Tool Calling

The ability of a model to invoke external functions or APIs during a conversation, things like searching the web, running code, querying a database, or reading a file. Rather than generating a plain text answer, the model recognizes when a tool would help, calls it with the right inputs, receives the result, and incorporates it into its response. Tool calling is what bridges a language model and the real world, and it's the core mechanism behind most agentic systems.

9. AI Systems & Evaluation

How models are deployed, measured, and put to work.

Agent / Agentic

An AI system that can take actions, use tools, and pursue a goal across multiple steps, rather than just responding once to a single prompt. An agentic system consists of a model, a suite of tools, and some logic for when and for how long to run it. An agent will often run in several steps, until it reaches some well-defined result.

Guardrails

Rules and filters applied to a model's inputs or outputs to keep it within acceptable boundaries. Guardrails can block harmful content, enforce topic restrictions, prevent the model from impersonating real people, or ensure responses stay on-brand for a product.

Alignment

The challenge of making AI systems behave in ways that reflect human intentions, values, and goals. A misaligned model might be highly capable but pursue objectives in ways its creators didn't intend.

Eval Benchmark

A standardized test used to measure and compare response quality. Benchmarks like MMLU, HumanEval, or HellaSwag evaluate specific capabilities like reasoning, coding, language understanding, or maths. They're useful for comparing models, but a high benchmark score doesn't always translate to real-world usefulness.

Visit nobodywho.ai to start integrating AI into your applications!

Run LLMs locally in Flutter apps

pielouNW — Mon, 23 Mar 2026 14:16:40 +0000

In this tutorial, you'll learn how to run a large language model (LLM) directly on a user's device — no cloud, no server, no cost. We'll start from scratch, build a working chat interface, and progressively introduce more advanced features: tool calling, sampling, and RAG.

Each concept is explained before the code, so you can follow along whether you're new to on-device AI or just new to NobodyWho.

The example app for this article is available on GitHub if you want to jump straight to working code. It is kept up to date with the latest features — if you want the code that matches this tutorial exactly, check out this commit.

Why Run AI On-Device?

Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.

Running the model directly on the device avoids all of them:

Works offline — no internet connection required
Privacy by design — user data never leaves the device
Low latency — no network round-trip
No cloud costs — inference is free

The tradeoff is raw capability: on-device models are smaller and less powerful than frontier cloud models. But for many use cases — summarization, chatbots, or local search — they're more than good enough.

About NobodyWho

We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and exposes a clean Flutter API for running any model in .gguf format.

Install it with:

flutter pub add nobodywho

Then initialize the engine in main.dart before your app launches:

import 'package:nobodywho/nobodywho.dart' as nobodywho;

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await nobodywho.NobodyWho.init();
  runApp(const MyApp());
}

Choosing and Loading a Model

Picking a Model

We'll use LFM2, a new generation of hybrid models developed by Liquid AI, specifically designed for edge AI and on-device deployment. Models must be in .gguf format; most will work with NobodyWho, though some may fail due to chat template formatting issues. See the model selection guide for more details.

Getting the Model onto the Device

You have two options:

Approach	Pros	Cons
Bundle in assets	Simple setup, great for development	Increases app size significantly
Download on demand	Keeps app size small	Requires more implementation work

For this tutorial, we'll bundle the model in assets to keep things simple. In production, you'd want to use a download-on-demand approach with something like background_downloader.

Steps:

Create an assets/ folder at the root of your project (if it doesn't exist).
Register it in pubspec.yaml:

flutter:
  assets:
    - assets/

Download this GGUF model, rename it model.gguf, and place it in the assets/ folder.

Accessing the Model at Runtime

NobodyWho reads the model from the filesystem, so we copy it from Flutter's asset bundle to the app's documents directory on first launch. Add path_provider to handle this:

flutter pub add path_provider

import 'dart:io';
import 'package:flutter/services.dart';
import 'package:path_provider/path_provider.dart';

final dir = await getApplicationDocumentsDirectory();
final model = File('${dir.path}/model.gguf');

if (!await model.exists()) {
  final data = await rootBundle.load('assets/model.gguf');
  await model.writeAsBytes(data.buffer.asUint8List(), flush: true);
}

Basic Chat

With the model in place, you're ready to start a conversation. Here's the simplest possible usage — good for testing or when you don't need a full chat UI:

final chat = await nobodywho.Chat.fromPath(modelPath: model.path);
final msg = await chat.ask('Is water wet?').completed();
print(msg);

Putting It Together

Here's the complete minimal app so far:

import 'dart:io';
import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'package:nobodywho/nobodywho.dart' as nobodywho;
import 'package:path_provider/path_provider.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await nobodywho.NobodyWho.init();
  runApp(const MainApp());
}

class MainApp extends StatelessWidget {
  const MainApp({super.key});

  Future<void> _onPressed() async {
    try {
      final dir = await getApplicationDocumentsDirectory();
      final model = File('${dir.path}/model.gguf');
      if (!await model.exists()) {
        final data = await rootBundle.load('assets/model.gguf');
        await model.writeAsBytes(data.buffer.asUint8List(), flush: true);
      }
      final chat = await nobodywho.Chat.fromPath(modelPath: model.path);
      final msg = await chat.ask('How do I code a button in Flutter?').completed();
      debugPrint(msg);
    } catch (err) {
      debugPrint("Error: $err");
    }
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        body: Center(
          child: ElevatedButton(
            onPressed: _onPressed,
            child: const Text("Ask - How do I code a button in Flutter?"),
          ),
        ),
      ),
    );
  }
}

For customization options like system prompts and context size, see the Chat documentation.

Building a Chat Interface

A one-shot ask().completed() call is fine for single questions, but a real chat interface needs to stream tokens as they arrive — otherwise users stare at a blank screen until the full response is ready.

Streaming Tokens

final response = chat.ask('How do I code a button in Flutter?');

await for (final token in response) {
  print(token); // Each token arrives as it's generated
}

A token is the smallest unit a model generates — typically a word fragment, punctuation mark, or whitespace character.

Handling the Streaming Content

class _ChatScreenState extends State<ChatScreen> {
  final List<nobodywho.Message> _messages = [];
  final TextEditingController _textController = TextEditingController();

  String? _streamingContent;
  bool _responding = false;

  Future<void> _ask() async {
    final userInput = _textController.text.trim();
    if (userInput.isEmpty || _responding) return;

    setState(() {
      _responding = true;
      _streamingContent = null;
    });

    final tokenStream = chat.ask(userInput);

    await for (final token in tokenStream) {
      setState(() {
        _streamingContent = (_streamingContent ?? '') + token;
      });
    }

    // ...continued below
  }
}

Updating the Message List

Once the stream completes, fetch the full chat history and update your UI state:

final history = await chat.getChatHistory();
final List<nobodywho.Message> messages = [];

for (var message in history) {
  messages.add(
    message.copyWith(
      content: message.content,
    ),
  );
}

setState(() {
  _messages.clear();
  _messages.addAll(messages);
  _streamingContent = null;
  _responding = false;
});

Wiring Up the UI

Connect your TextField to call _ask() via onSubmitted
Render _messages in a ListView
Append _streamingContent at the bottom while streaming

Tool Calling

Tool calling lets the model interact with the outside world. You define a set of functions — each with a name, a description, and an implementation — and the model decides when and how to call them based on the user's request.

import 'dart:math' as math;
import 'package:nobodywho/nobodywho.dart' as nobodywho;

final circleAreaTool = nobodywho.Tool(
  name: "circle_area",
  description: "Calculates the area of a circle given its radius",
  function: ({required double radius}) {
    final area = math.pi * radius * radius;
    return "Circle with radius $radius has area ${area.toStringAsFixed(2)}";
  },
);

final getWeatherTool = nobodywho.Tool(
  name: "get_weather",
  description: "Get the current weather for a given city",
  function: ({required String city}) async {
    return await fetchWeather(city);
  },
);

final chat = await nobodywho.Chat.fromPath(
  modelPath: model.path,
  tools: [circleAreaTool, getWeatherTool],
);

final response = await chat
    .ask('What is the area of a circle with a radius of 2?')
    .completed();
print(response);

The model reads each tool's description to decide when to call it, so writing clear, specific descriptions matters.

See the Tool Calling documentation for more.

Sampling

When generating a token, the model produces a probability distribution over every possible next token. A sampler controls how the final token is chosen from that distribution.

The default behavior involves some randomness, which produces natural, varied output. But you can tune it:

Lower temperature → more deterministic, predictable output
Higher temperature → more creative, varied output
Constrained sampling → force output into a specific format, such as JSON

final chat = await nobodywho.Chat.fromPath(
  modelPath: model.path,
  sampler: nobodywho.SamplerPresets.temperature(temperature: 0.2),
);

See the Sampling documentation for more.

RAG

Retrieval-Augmented Generation (RAG) combines document search with LLM generation. The model uses retrieved documents to ground its responses in your knowledge base.

Example: A Customer Service Assistant

import 'package:nobodywho/nobodywho.dart' as nobodywho;

Future<void> main() async {
  // The cross-encoder re-ranks retrieved documents by relevance.
  // Recommended model:
  // https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF/resolve/main/bge-reranker-v2-m3-Q8_0.gguf
  // Follow the same approach as the chat model to import the reranker model.
  final crossencoder = await nobodywho.CrossEncoder.fromPath(
    modelPath: rerankerModel.path,
  );

  final knowledge = [
    "Our company offers a 30-day return policy for all products",
    "Free shipping is available on orders over \$50",
    "Customer support is available via email and phone",
    "We accept credit cards, PayPal, and bank transfers",
    "Order tracking is available through your account dashboard",
  ];

  final searchKnowledgeTool = nobodywho.Tool(
    name: "search_knowledge",
    description: "Search the knowledge base for relevant information",
    function: ({required String query}) async {
      final ranked = await crossencoder.rankAndSort(query: query, documents: knowledge);
      final topDocs = ranked.take(3).map((e) => e.$1).toList();
      return topDocs.join("\n");
    },
  );

  final chat = await nobodywho.Chat.fromPath(
    modelPath: model.path,
    systemPrompt:
        "You are a customer service assistant. Use the search_knowledge tool "
        "to find relevant information from our policies before answering.",
    tools: [searchKnowledgeTool],
  );

  final response = await chat.ask("What is your return policy?").completed();
  print(response);
}

See the Embeddings & RAG documentation for more.

What's Next?

You now have a complete foundation for building on-device AI features in Flutter:

Load and run a GGUF model
Build a streaming chat interface
Extend the model with tool calling
Control output style with sampling
Ground responses in a knowledge base with RAG

From here, you can explore the full NobodyWho documentation or dig into the example app to see everything working end to end.