Beyond chat with llamadart · Part 3
Pip’s scarf is missing. I wanted Dash to be able to fly over to Mo and ask about it out loud.
Dash: “Mo, have you seen Pip’s red scarf?”
Mo: “Yes, it's in the reeds beside the pond.”
I tried that exchange with local models. For now, it runs in a small command-line program outside Llama Village. The question is a synthetic recording, and Mo gets the scarf’s location in his prompt. The game does not have voice controls yet.
The existing village. I’d like to let the player speak to these characters.
Two ways for Mo to listen
ASR is speech recognition. TTS is text to speech. Both routes finish with TTS. Concept illustration.
Via text: Qwen3-ASR transcribes the recording, a dialogue model answers, and Qwen3-TTS speaks. Having a transcript helps when “Pip” is misheard.
Direct audio: Qwen2.5-Omni receives the recording and writes an answer. This example has no separate transcript. The GGUF conversion accepts audio but does not generate it, so I still use TTS for Mo’s reply.
For the comparison, both routes use the same 3B dialogue model. In the first route it receives text; in the second it also loads its audio projector. This is one tested multimodal route, not a claim that all multimodal models need separate TTS.
I also tried a smaller pair for the text route: Qwen3.5 0.8B for dialogue and Qwen3-TTS 0.6B Base for speech. Qwen3-ASR 0.6B still handles recognition. The smaller pair is the default below; staged keeps the original 3B/1.7B pair for comparison. Qwen3.5 is used for text here, not audio input.
Try it on your Mac
The complete example is below. You need macOS, Dart 3.10.7 or newer, and about 2.31 GB for the five small-model files, plus space for Dart packages and the native runtime. I tested on an M4 Max with Dart 3.13.5 and llamadart 0.11.0. The commands use macOS say, afconvert, and afplay; they are not a phone or browser demo.
1. Create the project
Run this in a terminal. The Dart program uses pinned ModelSource.huggingFace references. Each engine’s load() uses llamadart’s download manager to fetch missing files and reuse its cache. There is no separate curl script. The first run needs a network connection and takes longer; subsequent generation uses the downloaded models locally.
Copy the setup commands
mkdir -p llama-voice-demo/bin
cd llama-voice-demo
cat > pubspec.yaml <<'YAML'
name: llama_voice_demo
publish_to: none
environment:
sdk: ^3.10.7
dependencies:
llamadart: 0.11.0
YAML
dart pub get
2. Add the complete program
Stay in llama-voice-demo and paste this block. It creates bin/voice.dart. The program loads one engine at a time and disposes it before loading the next.
Copy the full Dart program
cat > bin/voice.dart <<'DART'
import 'dart:io';
import 'package:llamadart/llamadart.dart';
const params = ModelParams(
contextSize: 4096,
numberOfThreads: 4,
numberOfThreadsBatch: 4,
);
const sources = {
'asr': (
'ggml-org/Qwen3-ASR-0.6B-GGUF',
'928ab958557df9aa2ef1c93e0e83c7ad0933fae2',
),
'omni': (
'ggml-org/Qwen2.5-Omni-3B-GGUF',
'75f1b73b657a50f5092502799457ccb4a4a1f9df',
),
'dialogue': (
'unsloth/Qwen3.5-0.8B-GGUF',
'6ab461498e2023f6e3c1baea90a8f0fe38ab64d0',
),
'tts': (
'ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF',
'ca27d74bc954b73dadab5b71ca265d87fc861a7c',
),
'smallTts': (
'mradermacher/Qwen3-TTS-12Hz-0.6B-Base-GGUF',
'0762b31e9cd99274d49dca4e895bd81863134f08',
),
};
ModelSource model(String family, String file) {
final (repo, revision) = sources[family]!;
return ModelSource.huggingFace(
repoId: repo,
revision: revision,
filePath: file,
);
}
Future<void> main(List<String> args) async {
if (args.isEmpty ||
args.length > 3 ||
(args.length > 1 && !['small', 'staged', 'audio'].contains(args[1]))) {
stderr.writeln(
'Usage: dart run bin/voice.dart question.wav '
'[small|staged|audio] [voice-reference.wav]',
);
exitCode = 64;
return;
}
final question = args[0];
final small = args.length == 1 || args[1] == 'small';
final directAudio = args.length > 1 && args[1] == 'audio';
final reference = args.length == 3 ? args[2] : null;
for (final path in [question, if (reference != null) reference]) {
if (!await File(path).exists()) {
throw ArgumentError('Audio file not found: $path');
}
}
String? transcript;
if (!directAudio) {
final asr = await SpeechToTextEngine.load(
SpeechToTextModel(
model('asr', 'Qwen3-ASR-0.6B-Q8_0.gguf'),
projector: model('asr', 'mmproj-Qwen3-ASR-0.6B-Q8_0.gguf'),
adapter: const Qwen3AsrAdapter(),
),
params: params,
);
try {
final result = await asr.transcribeOnce(
SpeechToTextRequest(
audio: SpeechAudioFileInput(question),
contextPrompt: 'Mo, Pip, Dash, Llama Village',
),
);
transcript = result.text.trim();
if (transcript.isEmpty) throw StateError('No speech recognized');
stdout.writeln('Dash: $transcript');
} finally {
await asr.dispose();
}
}
final llm = await LlamaEngine.load(
LlamaModel(
small
? model('dialogue', 'Qwen3.5-0.8B-Q4_K_M.gguf')
: model('omni', 'Qwen2.5-Omni-3B-Q4_K_M.gguf'),
projector: directAudio
? model('omni', 'mmproj-Qwen2.5-Omni-3B-Q8_0.gguf')
: null,
),
params: params,
);
final reply = StringBuffer();
try {
if (directAudio && !(await llm.capabilities).supportsAudio) {
throw StateError('This model/runtime cannot accept audio');
}
await for (final chunk in llm.create(
[
const LlamaChatMessage.fromText(
role: LlamaChatRole.system,
text:
'You are Mo, a friendly baker llama. Dash is visiting you. '
"Known fact: Pip's red scarf is in the reeds beside the pond. "
'Answer the visitor in one short sentence in English. '
'No stage directions. Do not invent facts you do not know.',
),
if (transcript != null)
LlamaChatMessage.fromText(role: LlamaChatRole.user, text: transcript)
else
LlamaChatMessage.withContent(
role: LlamaChatRole.user,
content: [LlamaAudioContent(path: question)],
),
],
enableThinking: false,
params: const GenerationParams(maxTokens: 100, temp: 0, seed: 42),
)) {
reply.write(chunk.text);
}
} finally {
await llm.dispose();
}
final text = reply.toString().trim();
if (text.isEmpty) throw StateError('The model returned no reply');
stdout.writeln('Mo: $text');
final tts = await TextToSpeechEngine.load(
TextToSpeechModel(
small
? model('smallTts', 'Qwen3-TTS-12Hz-0.6B-Base.Q4_K_M.gguf')
: model('tts', 'Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf'),
projector: small
? model('smallTts', 'Qwen3-TTS-12Hz-0.6B-Base.mmproj-Q8_0.gguf')
: model('tts', 'mmproj-Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf'),
adapter: const Qwen3TtsAdapter(),
),
params: params,
);
try {
final audio = await tts.synthesizeOnce(
TextToSpeechRequest(
text: text,
language: 'en',
seed: 42,
speakerReference: reference == null
? null
: SpeechAudioFileInput(reference),
),
);
await File('reply.wav').writeAsBytes(audio.toWavBytes());
stdout.writeln('Saved reply.wav');
} finally {
await tts.dispose();
}
}
DART
3. Ask Mo a question
say -o question.aiff "Mo, have you seen Pip's red scarf?"
afconvert -f WAVE -d LEI16@16000 -c 1 question.aiff question.wav
dart run bin/voice.dart question.wav small
afplay reply.wav
This creates a synthetic question so you can try the whole path without a microphone package or a sample download. You can replace question.wav with your own short WAV. The program prints the transcript and answer, then saves reply.wav in the project directory. It does not record a microphone.
The main calls inside the program are:
// ASR: a recording becomes text.
final result = await asr.transcribeOnce(
SpeechToTextRequest(
audio: SpeechAudioFileInput(question),
contextPrompt: 'Mo, Pip, Dash, Llama Village',
),
);
// TTS: the generated answer becomes a WAV file.
final audio = await tts.synthesizeOnce(
TextToSpeechRequest(text: text, language: 'en', seed: 42),
);
await File('reply.wav').writeAsBytes(audio.toWavBytes());
These excerpts sit inside the loaded engines’ try/finally blocks in the full program. Between them, llm.create answers using the transcript and Mo’s supplied fact.
Try the same question as audio
Change the mode to download and load Omni’s audio-capable model and projector, plus the original 1.7B TTS pair:
dart run bin/voice.dart question.wav audio
afplay reply.wav
That route skips ASR and sends LlamaAudioContent(path: question) to the dialogue model. The program checks supportsAudio before asking it to answer. The files used by this route total about 5.13 GB; keeping all three modes cached takes about 7.44 GB.
In the game, Mo should only receive facts he is allowed to reveal. DecisionEngine could choose an allowed intention before dialogue generation. A friendly persona prompt does not enforce those rules.
Make it sound like Mo
A possible production workflow: prepare a voice, save a reference, and reuse it for new dialogue. Concept illustration.
I’d like Mo to sound warm and unhurried. Dash could be brighter and quicker. A short reference clip lets several characters share one TTS model while sounding different. When the player speaks as Dash, they already provide his voice.
The full program accepts an optional third argument. Use small for the 0.8B/0.6B pair, or staged for the original 3B/1.7B pair:
dart run bin/voice.dart question.wav small mo-reference.wav
afplay reply.wav
Use a short, clear recording you have permission to use. For my test I made a synthetic reference with the macOS Daniel voice. Inside TextToSpeechRequest, the program passes it as speakerReference: SpeechAudioFileInput(reference).
Qwen’s voice-design workflow also lets you create a voice during development, then use the resulting clip with a Base model. Players would download the synthesis model and reference, without needing the voice-design model. I have not tested that workflow here.
A reference gives Mo a voice, but the tested API has no instruction such as “say this nervously.” That is not just a missing Dart parameter. Qwen’s 1.7B CustomVoice and VoiceDesign models support instruction control; the Base model used here does not offer that same mode. Exposing it would need a compatible model, native runtime support, and a Dart API. Changing the dialogue prompt only changes the words.
Smaller models, smaller downloads. Faster conversations?
I tested the smaller pair on the same M4 Max (64 GB), with llamadart 0.11.0. Both routes kept Qwen3-ASR 0.6B and used the same synthetic question and Daniel voice reference.
| Text route | Model files | Complete turn, median | Peak process RSS |
|---|---|---|---|
| Omni 3B + TTS 1.7B | 4.61 GB | 4.27 s | 3.86 GiB |
| Qwen3.5 0.8B + TTS 0.6B | 2.31 GB | 5.83 s | 2.51 GiB |
Half the download, about 35% less peak RSS, but a slower turn in this pilot. The smaller dialogue model gave a longer reply:
3B: “Yes, Pip's red scarf is in the reeds beside the pond.”
0.8B: “Yes, I saw it! It was tucked inside the reeds by the pond.”
The 0.8B model also ignored the one-sentence instruction. More words mean more speech to generate. These are complete application turns with different replies, so I also tested the components separately:
| Loaded component | Generation time, median | Peak process RSS |
|---|---|---|
| Omni 3B dialogue | 171 ms | 2.57 GiB |
| Qwen3.5 0.8B dialogue | 96 ms | 1.08 GiB |
| Qwen3-TTS 1.7B | 1.78 s | 3.85 GiB |
| Qwen3-TTS 0.6B | 1.78 s | 2.48 GiB |
For TTS, both models spoke exactly “Yes, Pip's red scarf is in the reeds beside the pond.” The 1.7B clips lasted 4.16–4.56 seconds; the 0.6B clips lasted 4.80–6.48 seconds. The smaller model’s synthesis times ranged from 1.78 to 2.53 seconds. Smaller did not mean a consistently shorter wait.
Did Mo say the right thing?
Both dialogue models supplied the location and colour correctly when asked. But when I asked “Who put the scarf in the pond?”, both blamed Pip. That fact was never supplied, even with “Do not invent facts you do not know” in the prompt. The smaller model’s “I saw it” also adds an unsupported claim about how Mo knows.
All six measured TTS clips transcribed back to the intended sentence with Qwen3-ASR. That is an intelligibility check, not a listening score for naturalness, emotion, or similarity to Daniel. I would still listen to more lines before choosing a character voice.
I would use the smaller pair for this prototype’s storage budget. I would keep the game’s knowledge rules outside the prompt, and measure real conversations before claiming a latency win.
This is a small development-machine pilot, not a device benchmark. Other development workloads were active; the village renderer was not. The complete-turn test alternated the two setups, discarded the first turn for each, then measured three fresh-process turns per setup. Baseline turns ranged from 3.76 to 4.41 seconds; smaller-pair turns ranged from 5.82 to 5.91 seconds. Timing includes model loading and disposal, but excludes process startup, downloads, recording, and playback. Peak RSS was sampled every 100 ms and is the largest observed peak across the three runs. It is process memory, not total GPU or device memory. Component tests loaded each model in its own process and discarded one warm-up. Dialogue used three questions: the scarf’s location, who put it there, and its location plus colour. TTS repeated the same sentence three times with the same reference. Component times exclude loading and disposal; memory uses the process high-water RSS. Both dialogue models used temperature 0, seed 42, a 100-token limit, thinking disabled, a 4096-token context, and four threads. TTS used seed 42; its waveforms still varied between calls. The dialogue weights are Q4_K_M; audio projectors are Q8_0. The new files are Unsloth’s Qwen3.5 0.8B conversion and mradermacher’s Qwen3-TTS 0.6B Base conversion. The exact revisions and matching projector filenames are pinned in the program above. A GGUF file made for a different TTS runtime is not automatically interchangeable.Test conditions and model files
Let the player read while Mo gets ready
This TTS API returns the complete audio before playback. Its progress events are not playable audio chunks. I’d show Mo’s reply as text immediately, then play it when synthesis finishes.
For a first game version, I’d use push-to-talk, stop recording while Mo speaks, and discard an old reply if the player starts another turn. Text should remain available when voice is switched off or a model cannot load. I still need to measure speech alongside rendering before adding this to the game.
For model pairs and platform requirements, see the speech recognition guide and speech synthesis guide. Partial live captions use a different recognition path from this file-based example.
Top comments (0)