We’ve spent the past year helping teams like Volkswagen, Verbali and LunaFit ship voice agents that run entirely on the phone. This is what we learned building a full voice-to-voice agent in React Native, from the wake word to the spoken reply.
I’m the founder of DaVoice, and I drafted this guide with AI help, which fits the subject. Our example app ships how-to files (englishHowTo.md and AI_AGENT_INTEGRATION_GUIDE.md) written for AI coding agents. Hand them to Claude Code, Cursor or Copilot along with your codebase, and the agent can add the wake word, speech-to-text, local LLM and on-device voice to your app step by step, in the order that works. I checked every detail here against the repo.
Short answer: To build an on-device Voice-to-Voice AI Agent in React Native, chain five stages that all run on the phone: a wake word detector, optional speaker verification, speech-to-text (STT), a small local LLM that calls your app's tools, and on-device text-to-speech (TTS). Stream the LLM's reply into TTS sentence by sentence, and hand the microphone carefully between stages. This tutorial does exactly that with the open-source DaVoice React Native example app and three npm packages from the DaVoice Voice AI Agent SDK: react-native-wakeword, react-native-davoice and react-native-davoice-llm.
The result is an agent that listens for "Hey Coach", understands "open the rear right window", runs a real function in your app, and answers out loud: "I opened the rear right window." No cloud LLM, no per-token bill, and no audio leaving the device during the voice loop.
What "token-free" means here. A local LLM still processes tokens. "Token-free" means you pay no cloud LLM inference token fees, because inference runs on the user's phone. DaVoice SDKs are licensed separately; evaluation and non-commercial proofs of concept are free, and production use needs a commercial license.
Used in production
The same on-device voice stack already runs in shipped products:
- Verbali (MaTalk AI, an AAC app for non-verbal children) moved its wake word, speech-to-text and text-to-speech fully on-device with DaVoice, so no audio leaves the tablet and voice keeps working without internet. In Verbali's own write-up, DaVoice reached 97.65% wake word accuracy with zero false positives observed on their test setup, versus about 75% for the other engines they evaluated. In their words: "We have no affiliation with DaVoice beyond being a happy customer."
- Volkswagen chose DaVoice for OTTO, its generative AI car assistant launched in Brazil (customer story · video).
- LunaFit built its entire voice-to-voice AI agent on DaVoice, including speaker-aware speech-to-text that hears its user in a noisy gym (video). The interruption pattern in section 8 comes from LunaFit's workout mode.
Table of contents
- What you will build
- Architecture: wake word → STT → local LLM → TTS
- What is really offline (and what is not)
- Prerequisites
- Run the DaVoice example app
- Implementation, stage by stage
- Add the agent to your existing React Native app
- Advanced: speaker verification, voice cloning, interruptions
- Testing and troubleshooting
- FAQ
- Resources
1. What you will build
A hands-free assistant inside a React Native app (iOS and Android) that:
- listens continuously for a custom wake word ("Hey Coach" in the example);
- optionally checks that the speaker is the enrolled owner;
- transcribes the request in real time;
- lets a 0.6B-parameter on-device model decide whether to call one of your app's tools, ask a clarifying question, decline, or hand the request to an (optional) remote model;
- speaks the result with an on-device neural voice, which can be a cloned voice.
The example ships two demos you can try immediately:
| Demo | Model (downloaded on first use) | Tools |
|---|---|---|
| Car demo | DaVoice Automotive Agent 0.6B (~380 MB GGUF) | 22 simulated car tools: windows, doors, locks, climate, navigation, radio |
| Calories demo | DaVoice Personal Agent 0.6B (~380 MB GGUF) | calorie tracker, personal memory, web search |
Honest status from the repo: the automotive model is labeled a test model, still in training. On DaVoice's frozen automotive test set it gets about 82% of Brazilian Portuguese commands exactly right (tool + arguments) and picks the right tool about 90% of the time. Every car action is simulated and checked by app-side guards. Treat it as a preview of the voice pipeline, not final accuracy.
2. Architecture: how a Voice-to-Voice AI Agent works on-device
┌────────────────────────── on the phone ──────────────────────────┐
microphone ──► │ Wake word ──► Speaker ──► Speech-to- ──► Local LLM │
│ "Hey Coach" verification text (STT) Agent │
│ (react-native- (optional, (react-native- (react-native-│
│ wakeword) owner only) davoice) davoice-llm) │
│ │ ▲ │
│ tool call ──►│ │result│
│ App tools (JS) │
│ │ │
speaker ◄──── │ Text-to-speech (react-native-davoice) ◄── sentence-by-sentence ─┘ │
└──────────────────────────────────────────────────────────────────┘
optional, needs network: remote model hand-over, web search, MCP
The loop, step by step:
- Wake word runs continuously at low power. When it fires, detection is paused (not stopped) and the mic is handed to STT.
-
Speaker verification (optional) tags every STT result with
speakerId/speakerIds, and can restrict the wake word to the enrolled voice. - STT streams partial results. The end of the user's turn is a silence timeout (2,000 ms in the example).
-
The local agent (
Agent.turn) gets the transcript plus history. The fine-tuned model decides: call a tool, ask, decline, or hand over. -
Tool calling: your JavaScript handler runs and returns
toolReply(result, sentence). The app, not the model, says what actually happened, and the turn ends without a second model round. - TTS speaks each sentence as soon as it is complete. STT is paused while the agent speaks, then the mic reopens.
Why a pipeline instead of one speech-to-speech model? Each stage can be tuned, tested and replaced on its own; tools can be gated by speaker identity; and a 0.6B action model stays small enough for a phone.
3. What is really offline (and what is not)
| Stage | Runs on device | Notes |
|---|---|---|
| Wake word | Yes | model shipped in the app bundle |
| Speaker verification | Yes |
sv.dm shipped in the app bundle |
| Speech-to-text | Yes, for installed on-device languages | DaVoice's guide notes that languages without an installed on-device model fall back to cloud recognition |
| Local LLM + tool calling | Yes, after a one-time model download | GGUF downloaded on first use, checked by sha256 |
| Text-to-speech + voice cloning | Yes | English and Brazilian Portuguese voices are bundled; other languages download once |
| Remote model hand-over | No (optional, off by default) |
LOCAL_LLM_LAYER2 = 'none' in the example |
| Web search / MCP tools | No (optional) | Wikipedia by default in the calories demo |
| "Full AI Chat" mode (Gemini) | No (optional cloud demo) | needs your own Gemini key, not part of the offline pipeline |
So: once models are downloaded, the core loop (wake word → STT → local agent → app tools → TTS) works in airplane mode for supported languages. Anything labeled optional above is cloud and is not part of the offline pipeline.
4. Prerequisites
-
Real devices. Wake word, speaker verification, STT and the LLM need a real microphone and real memory. Use simulators for UI only.
- iPhone: iPhone 11 or newer per DaVoice's guide (Metal GPU on A14+; CPU on A13).
- Android: 64-bit ARM (
arm64-v8a),minSdkVersion 29.
-
React Native CLI app. The example uses React Native 0.82.0 and React 19.1.1. (An Expo version lives in the same repo under
expo/; it needs a development build, not Expo Go.) - Node 18+, Xcode + CocoaPods, Android Studio.
- Git LFS, because model files are stored with LFS.
-
A DaVoice license key for your own app (the example contains a time-limited demo key; don't copy it). Contact
info@davoice.io. - Disk/network for the first run: the car demo downloads about 780 MB (model + pre-warmed state) once.
Versions used by the example (example/package.json) - use these together, since all three DaVoice packages ship native binaries:
"react-native": "0.82.0",
"react-native-davoice": "1.0.140",
"react-native-davoice-llm": "1.0.13",
"react-native-fs": "^2.20.0",
"react-native-wakeword": "1.1.171"
5. Run the DaVoice example app
Want to hear it before you build? The DaVoice Playground runs in your browser: try the wake word and listen to the text-to-speech voices without installing anything.
git clone https://github.com/frymanofer/ReactNative_DaVoice.git
cd ReactNative_DaVoice
git lfs pull # model files are LFS objects
cd example
npm install # the project uses npm / package-lock.json
cd ios && pod install && cd ..
npx react-native run-android # or: npx react-native run-ios
Useful extras from the repo's how-to:
adb reverse tcp:8081 tcp:8081 # Android debug build over USB (Metro)
adb logcat | grep ReactNativeJS # JS logs on Android
cd android && ./gradlew :app:assembleRelease # Android release build
First run walkthrough:
- Pick a language (English and Brazilian Portuguese are built in; others download once).
- Accept the microphone permission.
- Pick a voice, or clone a new one from a recording.
- Speaker verification: create a new voice signature (5 samples), reuse a saved one, or skip.
- Say "Hey Coach", then pick Local AI · Car demo.
- Wait for the loading screen (download → load into memory → warm up). The car demo ships a pre-warmed prompt state, so after the download it is ready in about a second.
Then try: "open the rear right window", "set the temperature to 22 degrees", "lock the car", "how much range do I have?". Incomplete requests should trigger a question: "open a window" → "Which window?".
A general question like "tell me a fun fact about Brazil" returns "I need to send this to a remote Model…". That is expected: no remote model is configured, and the 0.6B model is trained to act, not to chat.
6. Implementation, stage by stage
The example's voice logic is split into copyable modules under example/src/. App.tsx is the orchestration layer. The snippets below are the minimal APIs documented in the repo, plus excerpts from the real modules.
6.1 Install and native setup
npm install react-native-wakeword react-native-davoice react-native-davoice-llm react-native-fs
cd ios && pod install && cd ..
Android - each DaVoice package ships its native library as a local Maven repo. From example/android/build.gradle:
// minSdkVersion = 29
allprojects {
repositories {
maven { url "${project(":react-native-wakeword").projectDir}/libs" }
maven { url "${project(":react-native-davoice").projectDir}/libs" }
maven { url "${project(":react-native-davoice-llm").projectDir}/libs" }
}
}
Add RECORD_AUDIO and INTERNET (for the one-time model download) to AndroidManifest.xml.
iOS - in Info.plist: NSMicrophoneUsageDescription, NSSpeechRecognitionUsageDescription, and UIBackgroundModes → audio if the wake word should listen in the background. For the LLM, add the entitlement the example uses:
<key>com.apple.developer.kernel.increased-memory-limit</key>
<true/>
Metro - let require() resolve model files (metro.config.js):
resolver: {
assetExts: [...defaultConfig.resolver.assetExts, 'dm', 'onnx', 'wav', 'mp3'],
},
Model files loaded by name (wake word hey_coach_model_28_22012026b.dm, speaker verification sv.dm) go into android/app/src/main/assets/ and into the Xcode target's Copy Bundle Resources.
6.2 Wake word (react-native-wakeword)
From the repo's integration guide:
import { createKeyWordRNBridgeInstance, setWakewordAudioRoutingConfig } from 'react-native-wakeword';
await setWakewordAudioRoutingConfig(defaultAudioRoutingConfig); // iOS routing, from src/wakeword/index.ts
const inst = await createKeyWordRNBridgeInstance('main', false);
// One or more models: names, thresholds, buffer counts, ms between callbacks.
await inst.createInstanceMulti(['your_phrase_model.dm'], [0.999], [3], [1000]);
const sub = inst.onKeywordDetectionEvent((phrase: string) => onWakeWord(phrase));
if (!(await inst.setKeywordDetectionLicense(DAVOICE_LICENSE_KEY))) { /* show license error */ }
await inst.startKeywordDetection(0.999, true);
In the callback, pause detection before giving the mic to STT, and un-pause afterwards:
await inst.pauseDetection(false); // false: keep the mic session
// ... STT / LLM / TTS ...
await inst.unPauseDetection();
Tuning knobs: threshold (0-1) and bufferCnt trade false triggers against misses (the example uses 0.999 / 3); msBetweenCallbacks debounces repeats. Custom wake phrases are separate models you get from DaVoice. To see how detection behaves before you write code, test it in the wake word playground.
6.3 Speech-to-text and text-to-speech (react-native-davoice)
STT and TTS share one Speech object because they share one audio session:
import Speech from 'react-native-davoice/speech';
await Speech.setAudioRoutingConfig(defaultAudioRoutingConfig); // iOS; same object as the wake word
if (!(await Speech.setLicense(DAVOICE_LICENSE_KEY))) { /* show license error */ }
await Speech.initAll({ locale: 'en-US', model: require('./assets/models/model_ex2_rich_kate.dm') });
await Speech.pauseSpeechRecognition(); // start paused; un-pause when it's the user's turn
Speech.onSpeechPartialResults = (e) => { /* e.value[0]: text so far */ };
Speech.onSpeechResults = (e) => { /* e.value[0]: final text; e.speakerId when verification is on */ };
Speech.onSpeechError = (e) => { /* log; most codes are recoverable */ };
await Speech.unPauseSpeechRecognition(-1); // listen
await Speech.pauseSpeechRecognition(); // stop listening (e.g. before TTS)
await Speech.speak('I opened the window.', 0, 1.0); // text, speaker index, speed
To pick a voice, listen to them in the TTS playground.
initAll can take many seconds the first time, so the example wraps it in a timeout and keeps the wake word running if it stalls (initializeWakewordBootstrap in src/wakeword/index.ts):
await withTimeout(
initializeSpeechLibrary(enrollmentJsonPath ?? null),
speechInitTimeoutMs,
'Speech.initAll',
);
6.4 The local LLM and the Agent (react-native-davoice-llm)
The package has two layers: an engine (setLicense, loadModel, generateStream, prefill, cancel, saveState, loadState) and an Agent that runs a whole assistant turn - prompt assembly, tool calls, guards, memory and streaming - configured by an agent pack of JSON files instead of code. DaVoice's on-device LLM guide covers model choice in more depth.
Models. The GGUF files are downloaded at runtime and checked by sha256 - never bundled in the binary. From src/local_llm/constants.ts:
{
id: LOCAL_LLM_DAVOICE_AUTOMOTIVE_R4B,
displayName: 'DaVoice Automotive Agent 0.6B',
filename: 'davoice-automotive-qwen3-0.6b-r4b-it100-q4_0.gguf',
url: 'https://pub-27f94b1602c647ab95a601086cf52820.r2.dev/agent-llm/auto5_r4b_it100-q4_0.gguf',
sizeBytes: 381_565_920,
sha256: '1c078e961c5f4e16ae0f667d213fa06de53f3f4bc92c9248427bdb37237bd205',
contextSize: 8192,
chatTemplate: CHAT_TEMPLATE_PRESETS.qwen3.template,
// ...
}
The picker also lists general models (Qwen2.5 1.5B/3B with chatml, Gemma 3n E2B/E4B with gemma). They are larger and slower; the 0.6B agents are the defaults.
Minimal agent setup (from the repo's englishHowTo.md):
import { setLicense, loadModel, Agent, toolReply } from 'react-native-davoice-llm';
await setLicense(LICENSE);
await loadModel(pathToGguf, { contextSize: 8192 });
const agent = await Agent.create({
pack: { agent: require('./agent_automotive/agent.json'), skills: [] },
databasePath: `${RNFS.DocumentDirectoryPath}/davoice_agent_car.db`,
chatTemplate: 'qwen3',
}, { onRequest });
await agent.configure({ intentRouter: false, webSearchTool: true, timeTool: true, guards: { /* see provider.ts */ } });
await agent.setTools(myTools); // a handler returns toolReply(result, 'I opened the window.') to say it itself
const r = await agent.turn({ text, history }, (ev) => { if (ev.type === 'text') speak(ev.text); });
// agent.cancel() stops a running turn
The example's real configuration for the fine-tuned models (src/aichat/provider.ts, condensed - the original reads temperature and memory options from the model entry) shows the settings that matter:
agent.configure({
chatTemplate: model.chatTemplate,
remote: !handsOver ? false
: LOCAL_LLM_LAYER2 === 'device' ? { enabled: true, answer: 'layer2' }
: { enabled: true, timeoutMs: 15000 },
intentRouter: model.usesRouter !== false, // false: the model decides with every tool
webSearchTool: model.modelTools === true,
timeTool: model.modelTools === true,
memory: { profile: 'small' },
sampling: { temperature: 0 }, // greedy for tool calling
guards: callGuardSettings(model.callGuards ?? 'off', model.historyAsRun === true),
historyAsRun: model.historyAsRun === true,
guidedToolCalls: model.guidedToolCalls === true,
promptOrder: model.promptOrder ?? 'trained',
});
Create the agent once and answer its requests (src/agent/LocalAgent.ts). onRequest is where the app decides about remote hand-over and sensitive tools:
onRequest: async (request) => {
if (request.type === 'remote') return answerRemoteRequest(request as AgentRemoteRequest);
if (request.type === 'authorizeTool') return { authorized: isToolAuthorized(request as AgentApprovalRequest) };
if (request.type === 'confirmTool') return { confirmed: await askToConfirm(request as AgentApprovalRequest) };
throw new Error(`The app does not answer "${request.type}" requests.`);
},
6.5 Tool calling: define tools in MCP shape, let the app confirm
Tools are plain JSON in MCP shape (name, description, inputSchema, optional execution). From src/car/carToolDefinitions.h3.json:
{
"name": "set_window",
"description": "Opens or closes windows, fully, halfway or just a crack.",
"inputSchema": {
"type": "object",
"properties": {
"window": {
"type": "string",
"enum": ["all", "front_left", "front_right", "rear_left", "rear_right",
"driver", "passenger", "front", "rear", "left", "right"],
"description": "Which window: ... Ask if the user did not say which"
},
"position": {
"type": "string",
"enum": ["open", "close", "half", "crack"],
"description": "open (lower it), close (raise it), half, crack (a little)"
}
},
"required": ["window", "position"]
}
}
A sensitive tool declares an authorization scope, and the agent core never runs it without a "yes" from the app:
"execution": { "authorization": { "scope": "door_locks", "message": "Unlock the car?" } }
The JS handler does the work and returns the sentence to speak. This is the pattern in src/car/carTools.ts (trimmed):
import { toolReply } from 'react-native-davoice-llm';
function toTool(t: any): ToolDefinition {
return {
name: t.name,
description: t.description,
inputSchema: t.inputSchema,
...(t.execution ? { execution: t.execution } : {}),
handler: (rawArgs: Args) => {
const args = rawArgs ?? {};
const run = HANDLERS[t.name];
try {
if (!run) throw new Refused("Sorry, this car doesn't have that function.");
const { summary, result, say } = run(args);
// The app says what it did and the turn ends there: no second model round.
if (say) return toolReply({ ok: true, ...result }, say);
return { ok: true, ...result };
} catch (error) {
const reason = error instanceof Error ? error.message : String(error);
if (error instanceof Refused) return toolReply({ ok: false, refused: reason }, reason);
throw new Error(`Not done: ${reason}`);
}
},
};
}
Why toolReply? Two reasons from the repo's field testing: the spoken confirmation is built from the real result ("The front right door is already locked."), so the model can't announce something that didn't happen; and skipping the second model round saves seconds on a phone CPU. More patterns are in the tool calling guide.
Tip for small models: a 0.6B model depends heavily on tool descriptions. Keep them short, say explicitly "absolute vs. relative" and which side is which, and keep exactly the wording the model was trained on (the car model expects
carToolDefinitions.h3.json).
6.6 The voice loop: one turn, streamed into TTS
This is the order of calls that keeps mic hand-offs stable (from docs/AI_AGENT_INTEGRATION_GUIDE.md):
startup:
mic permission
setWakewordAudioRoutingConfig + Speech.setAudioRoutingConfig (iOS, same object)
create wake-word instance, attach listener, license
startKeywordDetection
pauseDetection(false)
Speech.setLicense, Speech.initAll (with timeout), Speech.pauseSpeechRecognition
unPauseDetection -> listening for the wake word
wake word detected:
pauseDetection(false)
Speech.unPauseSpeechRecognition(-1) -> user talks; partials -> agent.prefill
silence timeout:
Speech.pauseSpeechRecognition
agent.turn -> fillers and sentences go to Speech.speak as they arrive
all sentences spoken:
Speech.unPauseSpeechRecognition(-1) (follow-up turn)
or unPauseDetection (back to the wake word)
The turn itself (src/aichat/provider.ts):
const result = await agent.turn(
{ text: userText, history: toAgentHistory(history), profile, longTermFacts, localPassages, date: currentDateText() },
(event) => {
if (event.type === 'text') onDelta(event.text, false);
else onEvent?.(event);
},
);
onDelta('', true);
return { toolMessages: result.toolMessages, webSearch: result.webSearch };
Keep result.toolMessages in the history so the next turn knows what ran ("no, the other one at the back" depends on it).
Speak sentence by sentence, without awaiting each one. The example buffers streamed text and cuts complete sentences (splitCompletedSentences in src/aichat/index.ts), carefully not splitting "DaVoice.io" or "12.5":
// A period ends a sentence only before a space (or a closing quote): not inside "DaVoice.io", "12.5" or "e.g".
if (text[index] === '.') {
const next = text[index + 1];
if (next === undefined || !/[\s"')\]]/.test(next)) continue;
}
Each complete sentence is handed to Speech.speak immediately; the native library queues them, so there are no gaps. The reply is finished when every queued sentence has resolved - not on the first onFinishedSpeaking.
Latency tricks used by the example:
- Warm up when the chat screen opens: download, load, and process the system prompt (~3,100 tokens for the car demo).
-
Save and restore the processed prompt with
saveState/loadState. Measured on a Galaxy S20 (CPU): restore 0.7-0.9 s instead of a 30-40 s first warm-up. The car model ships a pre-warmed state (restore 0.23 s, first answer 0.34 s). -
Prefill while the user talks: call
agent.prefill(history, partialText)on partial transcripts so only the last words remain when the silence timeout fires. - Measured decision time on a Galaxy S20, CPU only: about 1.3-2 s per command; iPhones with Metal are faster.
7. Add the agent to your existing React Native app
The repo contains two documents written specifically for this: example/englishHowTo.md and docs/AI_AGENT_INTEGRATION_GUIDE.md. They're written so you can hand them to an AI coding agent (Claude Code, Cursor, Copilot) along with your codebase. The core rule: copy the module from src/ plus its assets, install the package, use your own license.
| You want | Copy | Assets / setup |
|---|---|---|
| Wake word |
src/wakeword/index.ts (instanceConfigs, initializeWakewordBootstrap, attachKeywordListenerOnce) |
wake word .dm in native bundle |
| Speaker verification |
src/speaker_verification/ (onboarding.ts, verification.ts, currentSpeaker.ts), thresholds in src/appflow.ts
|
sv.dm in native bundle |
| STT + TTS |
src/initialization/, src/stt/, src/tts/ (+ voiceClone.ts) |
TTS model + voice-cloning encoder per language |
| Local agent + tools |
src/local_llm/, src/agent/LocalAgent.ts, src/agent/AgentPack.ts, src/aichat/provider.ts, src/aichat/fragments.ts, src/aichat/remote.ts
|
agent pack JSON (agent/ or agent_automotive/), your tool JSON |
A safe integration order:
- Install only the packages you need and add the Android Maven lines / iOS plist entries. Build.
- Wake word: ship your
.dm, copydefaultAudioRoutingConfigandinitializeWakewordBootstrap. Build and test on a device. - STT/TTS:
Speech.initAllwith a timeout; register handlers fromsrc/stt/index.ts(keepmergeSmartKeepPunct- Android restarts partials per segment). Build and test. - Agent: copy
src/local_llm/for download/sha256/load-once,getLocalAgent()for a single agent, andprovider.tsfor configure + warm-up + streaming. - Replace the car tools with your own tool JSON and handlers that return
toolReply(...). - Edit the agent pack (persona, fixed replies, messages) - behavior changes without code changes.
Rules worth following from the guide:
- Never copy the example's demo license key. Put a placeholder like
DAVOICE_LICENSE_KEYin a git-ignoredlocal.config.ts(template:local.config.example.ts). - Keep one wake-word instance, one listener and one agent.
- Keep the order of native calls the same as the example; that order is what makes audio hand-offs stable.
- Don't
require()a.dmyou don't use - everyrequire()ships the file. - Don't give a
.tsmodule and a.jsonfile the same base name; Metro resolves the.jsonfirst. - Change one feature at a time and build after each.
8. Advanced capabilities
Speaker verification: an owner-only agent
Enroll once (5 samples in the example), save the enrollment JSON, then gate both the wake word and STT live:
import { createSpeakerVerificationMicController } from 'react-native-wakeword';
const ctrl = await createSpeakerVerificationMicController('svMic1');
await ctrl.create(JSON.stringify({ modelPath: 'sv.dm', options: { decisionThreshold: 0.35 } }));
await ctrl.beginOnboarding('davoice', 5, true);
// 5x: await ctrl.getNextEmbeddingFromMic(); -> progress events; onboardingDone carries enrollmentJson
// save the JSON (react-native-fs), destroy the controller, then enroll:
await wakewordInstance.addSpeaker('owner', enrollmentJsonPath);
await Speech.addSpeaker('owner', enrollmentJsonPath);
// turn the filter off again:
await wakewordInstance.setSpeakers([]);
await Speech.setSpeakers([]);
Each STT result then carries speakerIds / speakerId. The example uses it to authorize the door_locks scope (src/speaker_verification/currentSpeaker.ts):
export function ownerMayAct(): { allowed: boolean; reason: string } {
if (!gateOn) return { allowed: true, reason: 'speaker verification is off (demo allows it)' };
if (ownerId && heard.has(ownerId)) return { allowed: true, reason: `owner "${ownerId}" verified` };
return { allowed: false, reason: heard.size ? `heard ${[...heard].join(', ')}, not the owner` : 'no verified speaker in this request' };
}
In a car, that means a passenger or the radio can't unlock the doors. The enrollment JSON is personal voice data: keep it in the app's documents folder, never in logs. See also the owner-only agent guide.
Voice cloning
await Speech.setExternalVoicesPath(clonedVoicesPath());
const result = await Speech.createVoiceClone(
audioPath,
`${clonedVoicesPath()}/${name.trim()}.emd`,
getVoiceEmbedder(), // e.g. require('./assets/models/embedder_en.dm')
);
// every .emd in the folder becomes a voice: Speech.changeVoice(name)
If the result has warnings, ask the user to record again. Switching voices inside one model is instant (changeVoice); switching to another model file needs Speech.destroyAll() and initAll again.
Turn-taking and interruptions (barge-in)
The example is turn-based by design: STT is paused while the agent thinks and speaks, so the agent never transcribes its own voice. If you want "Hey Coach" to interrupt the reply, the repo documents the change (based on a production app):
- Keep the wake word listening while TTS plays: call
unPauseDetection()right afterpauseSpeechRecognition(). - Turn echo cancellation on: replace
WakewordAEC: { regular: false, duringTTS: false }withAEC: { mode: 'always' }indefaultAudioRoutingConfig. - In the keyword callback, if the agent is speaking, stop everything and open a new turn:
if (aiChatInFlightRef.current) { // the AI is thinking or speaking
if (interruptInFlightRef.current) return; // the same "Hey Coach" detected twice
interruptInFlightRef.current = true;
try {
aiChatRequestIdRef.current += 1; // late text and events of the old turn are ignored
(await getLocalAgent()).cancel(); // stops the LLM turn
aiChatSpeechQueueRef.current = []; // drops the sentences not spoken yet
ignoreNextFinishedSpeakingRef.current = true;
await Speech.stopSpeaking(); // silences the voice now
resetSpeechTranscriptState(); // discards whatever STT caught meanwhile
aiChatInFlightRef.current = false;
await myInstanceRef.current?.pauseDetection(false);
await Speech.unPauseSpeechRecognition(-1, 0); // new user turn; preFetchMs 0
} finally {
interruptInFlightRef.current = false;
}
return;
}
Open STT with preFetchMs: 0, otherwise it replays the previous ~2 s of audio and transcribes "Hey Coach" as the new message. Combined with speaker verification, only the owner can interrupt. Details: turn-taking and interruptions guide.
Optional cloud layer (clearly not offline)
Set LOCAL_LLM_LAYER2 in src/local_llm/constants.ts to 'cloud' to hand general-knowledge questions to Gemini (src/aichat/remote.ts), or implement answerRemoteRequest for your own backend. This is an optional cloud integration and incurs that provider's token costs. Keep keys out of the client in production (use a backend proxy).
9. Testing and troubleshooting
Run the app tests:
npx jest __tests__/carTools.test.ts __tests__/fragments.test.ts
Read the conversation log (user words, model output, guard decisions, tool calls, timings):
adb shell run-as com.exampleapp cat files/davoice_chat.log # debug build
adb logcat | grep DavoiceChat # release build
For every command check: the right tool, exact arguments, a spoken sentence that matches what happened, a question when info is missing, a decline when unsupported, and no action on small talk or speech noise.
| Symptom | Likely cause |
|---|---|
Gradle: Could not find com.davoice:...
|
Missing maven { url ".../libs" } line for that package |
| Model "not found" at runtime | File not in the bundle/assets under that exact name, or it's a Git LFS pointer (run git lfs pull) |
setLicense returns false
|
Expired or wrong key for this bundle id / package name |
| The app transcribes its own voice | STT not paused during TTS, or AEC off |
| Wake word fires twice | Listener attached twice; use attachKeywordListenerOnce
|
| STT dead after the wake word | Detection was stopped instead of paused, or STT never un-paused |
| Gaps between spoken sentences | Awaiting each speak; enqueue without awaiting |
| iOS app killed while loading the LLM | Missing increased-memory-limit entitlement |
| First reply very slow | No warm-up/prefill, or no saved prompt state |
| "I need to send this to a remote Model" | Expected: a general question with LOCAL_LLM_LAYER2 = 'none'
|
| "Sorry, I didn't catch that" on short phrases | Speech-noise filter (src/aichat/fragments.ts) - expected for filler like "The." |
10. FAQ
What is a Voice-to-Voice AI Agent?
An assistant you talk to and that talks back, while taking actions through tools. Here it's a pipeline - wake word, STT, a local LLM with tool calling, TTS - not a single speech-to-speech model.
Can it run fully offline?
The core loop can, once models are downloaded and for languages with an on-device STT model. Remote model hand-over, web search, MCP tools and the Gemini demo mode need a network and are optional.
Is it really token-free?
There are no cloud LLM inference token charges, because the model runs on the phone. The local model still processes tokens, and DaVoice SDKs require a commercial license for production.
How big is the model and how fast is it?
The DaVoice Agent 0.6B GGUF is about 380 MB, downloaded once. On a Galaxy S20 (CPU only) the decision takes about 1.3-2 s per command; iPhones with A14+ use the GPU.
Can I use my own LLM?
Yes - the engine runs GGUF models with several chat templates (qwen3, chatml, gemma, llama3, phi3). Larger general models are slower on phones.
Does it work with Expo?
Yes, with a development build or EAS Build (not Expo Go). See expo/, docs/EXPO_AI_AGENT_INTEGRATION_GUIDE.md and the Expo guide. A Flutter version exists too.
Can users interrupt the agent?
The example is turn-based. Barge-in with the wake word plus echo cancellation is documented in section 8.
How do I stop the model from doing dangerous things?
Declare execution.authorization or execution.confirmation on the tool, gate it in onRequest, enable call guards (callGuards: 'correct'), and let the app - not the model - confirm results via toolReply.
11. Resources
- Example app (React Native + Expo): github.com/frymanofer/ReactNative_DaVoice
- Integration how-to used in this article:
example/englishHowTo.md - Guide for AI coding agents:
docs/AI_AGENT_INTEGRATION_GUIDE.md - Voice-loop and tool code:
src/aichat/provider.ts,src/agent/LocalAgent.ts,src/car/carTools.ts - DaVoice Voice Agent Guide: davoice.io/voice-agent-guide and the React Native tutorial
- Voice AI Agent SDK · Wake word · Speaker recognition
- Try it in your browser: DaVoice Playground · Text-to-speech playground · Wake word playground
- Customer stories: Verbali on going fully on-device · Volkswagen OTTO · DaVoice on YouTube
- Licenses and custom wake words:
info@davoice.io
Top comments (3)
Oh, very impressive . I tested on personal app and worked very well.
We embraced this solution
Thank you very much. I am curious which project is it?
Interesting work! I love the fact that it's all 'on device' !