In this Python tutorial, you'll learn how to run a large language model (LLM) directly on a user's device: no server, no API key needed. We'll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.
Each concept is explained before the code, so you can follow along whether you're new to on-device AI.
Why run AI On-Device?
Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.
Running the model directly on the device avoids all of them:
- Works offline — no internet connection required
- Privacy by design — user data never leaves the device
- Low latency — no network round-trip
- No cloud costs — inference is free
The tradeoff is raw capability: on-device models are smaller and less powerful than frontier cloud models. But for many use cases like summarization, chatbots, or local search, they're more than good enough.
About NobodyWho
We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and ships bindings for several languages and frameworks: Kotlin, Python, Expo/React Native, Swift, Flutter & Godot. It exposes a clean API for running any model locally in .gguf format.
Install it with:
pip install nobodywho
Or, preferably:
uv add nobodywho
Loading a Model
NobodyWho can download a GGUF model for you directly from Hugging Face, cache it, and reuse it on every subsequent launch. That means you don't need to manage downloads yourself:
from nobodywho import Chat
chat = Chat('huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf')
The first time this runs, the model is downloaded to the platform cache directory. Every call after that loads the model directly.
model_path accepts a few different forms:
| Form | Example | Notes |
|---|---|---|
| HuggingFace reference | hf:owner/repo/file.gguf |
Downloaded and cached on first use |
| HTTPS URL | https://example.com/model.gguf |
Downloaded and cached on first use |
| Local path | ./model.gguf |
Used as-is, no download |
The HuggingFace prefix is case-insensitive and the // is optional, so hf:, hf://, huggingface:, and huggingface:// are all equivalent.
You can track a remote download by calling download_model directly and passing an on_download_progress callback. It receives (downloaded_bytes, total_bytes) and is skipped for cached or local files — if you don't pass anything, NobodyWho prints a default terminal progress bar:
from nobodywho import download_model, Chat
model_path = download_model(
'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
on_download_progress=lambda downloaded, total: print(f"{downloaded}/{total} bytes"),
)
chat = Chat(model_path)
You can find thousands of LLMs in .gguf format on Hugging Face here.
Basic Chat
With a model loaded, you're ready to start a conversation:
from nobodywho import Chat
chat = Chat('huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf')
response = chat.ask('Is water wet?').completed()
print(response) # Yes, indeed, water is wet!
chat.ask() sends your message and returns a TokenStream. Calling .completed() blocks until the model is done generating and gives you back the final string, which is fine for a one-off question. But a real chat interface needs to stream tokens as they arrive, otherwise users stare at a blank screen until generation finishes.
Streaming Tokens
response = chat.ask('What is the capital of Denmark?')
for token in response:
print(token, end="", flush=True)
A token is the smallest unit a model generates, typically a word, or a fragment of a word.
If you'd rather not block synchronously while a model loads or generates, swap Chat for ChatAsync — the API stays the same, so await chat.ask(...).completed() or async for token in chat.ask(...) both work.
Multimodal Models
Some models can natively ingest images and audio. To use them, you need two things: a multimodal LLM, and its projection model that converts images and/or audio into tokens the LLM can consume (usually named with mmproj in it). A solid default that handles both image and audio is Gemma 4 with its BF16 projection model.
from nobodywho import Model, Chat
model = Model("./vision-model.gguf", projection_model_path="./projection_model.gguf")
chat = Chat(model, system_prompt="You are a helpful assistant, that can hear and see stuff!")
To actually send image or audio content, build a Prompt mixing text, images, and audio, and pass it to chat.ask() instead of a plain string:
from nobodywho import Audio, Image, Prompt, Text
prompt = Prompt([
Text("Tell me what you see in the image and what you hear in the audio."),
Image("./dog.png"),
Audio("./sound.mp3"),
])
response = chat.ask(prompt).completed()
Keep in mind that images and audio consume context fast, so you'll likely want a bigger n_ctx than you'd use for text-only chat. Also note that the language model and its projection model have to be trained together — you can't mix an LLM and a projection model you happen to like and expect them to work.
Speech to Text
If you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through SpeechToText.
from nobodywho import SpeechToText
stt = SpeechToText(source="hf://onnx-community/whisper-base")
text = stt.transcribe_file("recording.mp3").completed()
print(text)
source is a Hugging Face repo (hf://owner/repo) or a local directory laid out the same way. Browse the Whisper ONNX models on Hugging Face to find one that fits your accuracy and speed needs.
If your audio comes from a buffer rather than a file, use transcribe_pcm:
text = stt.transcribe_pcm(samples, sample_rate=16000).completed()
The buffer needs to be mono i16 PCM samples. The sample rate can be anything, NobodyWho resamples internally to what Whisper expects. And just like chat, transcription can be streamed piece by piece instead of waiting for the full result:
for piece in stt.transcribe_file("recording.mp3"):
print(piece, end="", flush=True)
Text to Speech
Going the other direction, TextToSpeech turns text into WAV audio you can play back or save.
from pathlib import Path
from nobodywho import TextToSpeech
tts = TextToSpeech(
source="hf://NobodyWho/Kokoro-82M",
voice="bf_emma",
language="en-gb",
)
wav = tts.synthesize("Hello from NobodyWho!")
Path("out.wav").write_bytes(wav)
Three architectures are supported, all ONNX-based: Kokoro, Pocket TTS, and Supertonic. NobodyWho infers which one you're using from the source string, so you only need to set architecture explicitly when loading from a custom local folder.
Each architecture has its own voice and language options that need to agree with what the model supports.
Voice Activity Detection
Before transcribing audio, it helps to know when someone is actually speaking rather than relying on a fixed silence timeout. VoiceActivityDetection uses a small model to reliably tell speech and silence apart, and pairs naturally with SpeechToText.
For streaming microphone input, push chunks in as they arrive:
from nobodywho import VoiceActivityDetection, VoiceActivityDetectionEvent, SpeechToText
vad = VoiceActivityDetection(source="hf://onnx-community/silero-vad", sample_rate=16000)
stt = SpeechToText(source="hf://onnx-community/whisper-base")
while chunk := read_mic():
if vad.push(chunk) == VoiceActivityDetectionEvent.SpeechEnded:
break
speech = vad.finish()
transcription = stt.transcribe_pcm(speech, sample_rate=16000).completed()
print(transcription)
Each .push() call reports the current state (SpeechStarted, SpeechEnded, Speech, or Silence), and .finish() hands you back the buffered speech segment while resetting internal state for the next turn.
If you already have a full recording and just want to pull out the speech segments from it, .segment() does that in one pass:
audio = read_wav_pcm("recording.wav")
for speech in vad.segment(audio):
transcription = stt.transcribe_pcm(speech, sample_rate=16000).completed()
print(transcription)
Sensitivity is tunable via threshold, min_speech_duration_ms, min_silence_duration_ms, and preroll_duration_ms (how much audio to keep before the detected start, so you don't clip the beginning of a sentence). The defaults are a reasonable starting point, but VAD is one of those things that usually benefits from tuning to your actual environment.
Tool Calling
Tools let the model call out to real functions in your app rather than just generating text. Any synchronous function that returns a string becomes a tool with the @tool decorator:
import math
from nobodywho import tool, Chat
@tool(description="Calculates the area of a circle given its radius")
def circle_area(radius: float) -> str:
area = math.pi * radius ** 2
return f"Circle with radius {radius} has area {area:.2f}"
chat = Chat('./model.gguf', tools=[circle_area])
NobodyWho inspects the function's name, parameter names, and types to describe the tool to the model — add a params dict to @tool if a parameter needs more explanation than its name alone gives.
NobodyWho also ships two general-purpose tools out of the box, a Python interpreter and a Bash interpreter, for models that need to reason precisely or compute something:
from nobodywho import python_tool, bash_tool
chat = Chat('./model.gguf', tools=[python_tool(), bash_tool()])
Not every model supports tool calling well, the Qwen family is a solid choice if you need it to be reliable. See the Tool Calling documentation for more.
RAG
Retrieval-Augmented Generation combines document search with LLM generation, so the model grounds its answers in your own knowledge base instead of what it happened to learn during training. NobodyWho provides an Encoder for embeddings and a CrossEncoder for reranking:
from nobodywho import Encoder, cosine_similarity
encoder = Encoder('./embedding-model.gguf')
query_embedding = encoder.encode("What is the return policy?")
doc_embeddings = encoder.encode_batch(knowledge)
similarities = [cosine_similarity(query_embedding, doc_emb) for doc_emb in doc_embeddings]
For better precision, rerank the top candidates with a CrossEncoder before handing them to the model:
from nobodywho import CrossEncoder
crossencoder = CrossEncoder('./reranker-model.gguf')
ranked = crossencoder.rank_and_sort("What is the return policy?", top_docs)
See the Embeddings & RAG documentation for the full walkthrough, including how to wire this up as a tool the model calls automatically.
What's Next?
You now have a complete foundation for building on-device AI features in Python:
- Download and run a GGUF model
- Send messages and get streamed tokens back, synchronously or with
ChatAsync - Feed images and audio directly into a multimodal model
- Transcribe speech, synthesize it back, and detect when someone's actually talking
- Extend the model with tool calling and perform search with RAG
Top comments (0)