DEV Community

Ofer Fryman
Ofer Fryman

Posted on Edited on AI-assisted

How to Build an Offline, On-Device Voice-to-Voice AI Agent in React Native

We’ve spent the past year helping teams like Volkswagen, Verbali and LunaFit ship voice agents that run entirely on the phone. This is what we learned building a full voice-to-voice agent in React Native, from the wake word to the spoken reply.

I’m the founder of DaVoice, and I drafted this guide with AI help, which fits the subject. Our example app ships how-to files (englishHowTo.md and AI_AGENT_INTEGRATION_GUIDE.md) written for AI coding agents. Hand them to Claude Code, Cursor or Copilot along with your codebase, and the agent can add the wake word, speech-to-text, local LLM and on-device voice to your app step by step, in the order that works. I checked every detail here against the repo.

Short answer: To build an on-device Voice-to-Voice AI Agent in React Native, chain five stages that all run on the phone: a wake word detector, optional speaker verification, speech-to-text (STT), a small local LLM that calls your app's tools, and on-device text-to-speech (TTS). Stream the LLM's reply into TTS sentence by sentence, and hand the microphone carefully between stages. This tutorial does exactly that with the open-source DaVoice React Native example app and three npm packages from the DaVoice Voice AI Agent SDK: react-native-wakeword, react-native-davoice and react-native-davoice-llm.

The result is an agent that listens for "Hey Coach", understands "open the rear right window", runs a real function in your app, and answers out loud: "I opened the rear right window." No cloud LLM, no per-token bill, and no audio leaving the device during the voice loop.

What "token-free" means here. A local LLM still processes tokens. "Token-free" means you pay no cloud LLM inference token fees, because inference runs on the user's phone. DaVoice SDKs are licensed separately; evaluation and non-commercial proofs of concept are free, and production use needs a commercial license.

Used in production

The same on-device voice stack already runs in shipped products:

  • Verbali (MaTalk AI, an AAC app for non-verbal children) moved its wake word, speech-to-text and text-to-speech fully on-device with DaVoice, so no audio leaves the tablet and voice keeps working without internet. In Verbali's own write-up, DaVoice reached 97.65% wake word accuracy with zero false positives observed on their test setup, versus about 75% for the other engines they evaluated. In their words: "We have no affiliation with DaVoice beyond being a happy customer."
  • Volkswagen chose DaVoice for OTTO, its generative AI car assistant launched in Brazil (customer story · video).
  • LunaFit built its entire voice-to-voice AI agent on DaVoice, including speaker-aware speech-to-text that hears its user in a noisy gym (video). The interruption pattern in section 8 comes from LunaFit's workout mode.


Table of contents

  1. What you will build
  2. Architecture: wake word → STT → local LLM → TTS
  3. What is really offline (and what is not)
  4. Prerequisites
  5. Run the DaVoice example app
  6. Implementation, stage by stage
  7. Add the agent to your existing React Native app
  8. Advanced: speaker verification, voice cloning, interruptions
  9. Testing and troubleshooting
  10. FAQ
  11. Resources

1. What you will build

A hands-free assistant inside a React Native app (iOS and Android) that:

  • listens continuously for a custom wake word ("Hey Coach" in the example);
  • optionally checks that the speaker is the enrolled owner;
  • transcribes the request in real time;
  • lets a 0.6B-parameter on-device model decide whether to call one of your app's tools, ask a clarifying question, decline, or hand the request to an (optional) remote model;
  • speaks the result with an on-device neural voice, which can be a cloned voice.

The example ships two demos you can try immediately:

Demo Model (downloaded on first use) Tools
Car demo DaVoice Automotive Agent 0.6B (~380 MB GGUF) 22 simulated car tools: windows, doors, locks, climate, navigation, radio
Calories demo DaVoice Personal Agent 0.6B (~380 MB GGUF) calorie tracker, personal memory, web search

Honest status from the repo: the automotive model is labeled a test model, still in training. On DaVoice's frozen automotive test set it gets about 82% of Brazilian Portuguese commands exactly right (tool + arguments) and picks the right tool about 90% of the time. Every car action is simulated and checked by app-side guards. Treat it as a preview of the voice pipeline, not final accuracy.


2. Architecture: how a Voice-to-Voice AI Agent works on-device

                 ┌────────────────────────── on the phone ──────────────────────────┐
  microphone ──► │ Wake word  ──►  Speaker      ──►  Speech-to-  ──►  Local LLM    │
                 │ "Hey Coach"     verification      text (STT)       Agent        │
                 │ (react-native-  (optional,        (react-native-   (react-native-│
                 │  wakeword)       owner only)       davoice)         davoice-llm) │
                 │                                                       │   ▲      │
                 │                                          tool call ──►│   │result│
                 │                                                  App tools (JS)  │
                 │                                                       │          │
  speaker  ◄──── │ Text-to-speech (react-native-davoice) ◄── sentence-by-sentence ─┘ │
                 └──────────────────────────────────────────────────────────────────┘
                     optional, needs network: remote model hand-over, web search, MCP
Enter fullscreen mode Exit fullscreen mode

The loop, step by step:

  1. Wake word runs continuously at low power. When it fires, detection is paused (not stopped) and the mic is handed to STT.
  2. Speaker verification (optional) tags every STT result with speakerId / speakerIds, and can restrict the wake word to the enrolled voice.
  3. STT streams partial results. The end of the user's turn is a silence timeout (2,000 ms in the example).
  4. The local agent (Agent.turn) gets the transcript plus history. The fine-tuned model decides: call a tool, ask, decline, or hand over.
  5. Tool calling: your JavaScript handler runs and returns toolReply(result, sentence). The app, not the model, says what actually happened, and the turn ends without a second model round.
  6. TTS speaks each sentence as soon as it is complete. STT is paused while the agent speaks, then the mic reopens.

Why a pipeline instead of one speech-to-speech model? Each stage can be tuned, tested and replaced on its own; tools can be gated by speaker identity; and a 0.6B action model stays small enough for a phone.


3. What is really offline (and what is not)

Stage Runs on device Notes
Wake word Yes model shipped in the app bundle
Speaker verification Yes sv.dm shipped in the app bundle
Speech-to-text Yes, for installed on-device languages DaVoice's guide notes that languages without an installed on-device model fall back to cloud recognition
Local LLM + tool calling Yes, after a one-time model download GGUF downloaded on first use, checked by sha256
Text-to-speech + voice cloning Yes English and Brazilian Portuguese voices are bundled; other languages download once
Remote model hand-over No (optional, off by default) LOCAL_LLM_LAYER2 = 'none' in the example
Web search / MCP tools No (optional) Wikipedia by default in the calories demo
"Full AI Chat" mode (Gemini) No (optional cloud demo) needs your own Gemini key, not part of the offline pipeline

So: once models are downloaded, the core loop (wake word → STT → local agent → app tools → TTS) works in airplane mode for supported languages. Anything labeled optional above is cloud and is not part of the offline pipeline.


4. Prerequisites

  • Real devices. Wake word, speaker verification, STT and the LLM need a real microphone and real memory. Use simulators for UI only.
    • iPhone: iPhone 11 or newer per DaVoice's guide (Metal GPU on A14+; CPU on A13).
    • Android: 64-bit ARM (arm64-v8a), minSdkVersion 29.
  • React Native CLI app. The example uses React Native 0.82.0 and React 19.1.1. (An Expo version lives in the same repo under expo/; it needs a development build, not Expo Go.)
  • Node 18+, Xcode + CocoaPods, Android Studio.
  • Git LFS, because model files are stored with LFS.
  • A DaVoice license key for your own app (the example contains a time-limited demo key; don't copy it). Contact info@davoice.io.
  • Disk/network for the first run: the car demo downloads about 780 MB (model + pre-warmed state) once.

Versions used by the example (example/package.json) - use these together, since all three DaVoice packages ship native binaries:

"react-native": "0.82.0",
"react-native-davoice": "1.0.140",
"react-native-davoice-llm": "1.0.13",
"react-native-fs": "^2.20.0",
"react-native-wakeword": "1.1.171"
Enter fullscreen mode Exit fullscreen mode

5. Run the DaVoice example app

Want to hear it before you build? The DaVoice Playground runs in your browser: try the wake word and listen to the text-to-speech voices without installing anything.

git clone https://github.com/frymanofer/ReactNative_DaVoice.git
cd ReactNative_DaVoice
git lfs pull                 # model files are LFS objects
cd example
npm install                  # the project uses npm / package-lock.json
cd ios && pod install && cd ..
npx react-native run-android # or: npx react-native run-ios
Enter fullscreen mode Exit fullscreen mode

Useful extras from the repo's how-to:

adb reverse tcp:8081 tcp:8081                 # Android debug build over USB (Metro)
adb logcat | grep ReactNativeJS               # JS logs on Android
cd android && ./gradlew :app:assembleRelease  # Android release build
Enter fullscreen mode Exit fullscreen mode

First run walkthrough:

  1. Pick a language (English and Brazilian Portuguese are built in; others download once).
  2. Accept the microphone permission.
  3. Pick a voice, or clone a new one from a recording.
  4. Speaker verification: create a new voice signature (5 samples), reuse a saved one, or skip.
  5. Say "Hey Coach", then pick Local AI · Car demo.
  6. Wait for the loading screen (download → load into memory → warm up). The car demo ships a pre-warmed prompt state, so after the download it is ready in about a second.

Then try: "open the rear right window", "set the temperature to 22 degrees", "lock the car", "how much range do I have?". Incomplete requests should trigger a question: "open a window" → "Which window?".

A general question like "tell me a fun fact about Brazil" returns "I need to send this to a remote Model…". That is expected: no remote model is configured, and the 0.6B model is trained to act, not to chat.


6. Implementation, stage by stage

The example's voice logic is split into copyable modules under example/src/. App.tsx is the orchestration layer. The snippets below are the minimal APIs documented in the repo, plus excerpts from the real modules.

6.1 Install and native setup

npm install react-native-wakeword react-native-davoice react-native-davoice-llm react-native-fs
cd ios && pod install && cd ..
Enter fullscreen mode Exit fullscreen mode

Android - each DaVoice package ships its native library as a local Maven repo. From example/android/build.gradle:

// minSdkVersion = 29
allprojects {
  repositories {
    maven { url "${project(":react-native-wakeword").projectDir}/libs" }
    maven { url "${project(":react-native-davoice").projectDir}/libs" }
    maven { url "${project(":react-native-davoice-llm").projectDir}/libs" }
  }
}
Enter fullscreen mode Exit fullscreen mode

Add RECORD_AUDIO and INTERNET (for the one-time model download) to AndroidManifest.xml.

iOS - in Info.plist: NSMicrophoneUsageDescription, NSSpeechRecognitionUsageDescription, and UIBackgroundModes → audio if the wake word should listen in the background. For the LLM, add the entitlement the example uses:

<key>com.apple.developer.kernel.increased-memory-limit</key>
<true/>
Enter fullscreen mode Exit fullscreen mode

Metro - let require() resolve model files (metro.config.js):

resolver: {
  assetExts: [...defaultConfig.resolver.assetExts, 'dm', 'onnx', 'wav', 'mp3'],
},
Enter fullscreen mode Exit fullscreen mode

Model files loaded by name (wake word hey_coach_model_28_22012026b.dm, speaker verification sv.dm) go into android/app/src/main/assets/ and into the Xcode target's Copy Bundle Resources.

6.2 Wake word (react-native-wakeword)

From the repo's integration guide:

import { createKeyWordRNBridgeInstance, setWakewordAudioRoutingConfig } from 'react-native-wakeword';

await setWakewordAudioRoutingConfig(defaultAudioRoutingConfig); // iOS routing, from src/wakeword/index.ts
const inst = await createKeyWordRNBridgeInstance('main', false);
// One or more models: names, thresholds, buffer counts, ms between callbacks.
await inst.createInstanceMulti(['your_phrase_model.dm'], [0.999], [3], [1000]);
const sub = inst.onKeywordDetectionEvent((phrase: string) => onWakeWord(phrase));
if (!(await inst.setKeywordDetectionLicense(DAVOICE_LICENSE_KEY))) { /* show license error */ }
await inst.startKeywordDetection(0.999, true);
Enter fullscreen mode Exit fullscreen mode

In the callback, pause detection before giving the mic to STT, and un-pause afterwards:

await inst.pauseDetection(false);   // false: keep the mic session
// ... STT / LLM / TTS ...
await inst.unPauseDetection();
Enter fullscreen mode Exit fullscreen mode

Tuning knobs: threshold (0-1) and bufferCnt trade false triggers against misses (the example uses 0.999 / 3); msBetweenCallbacks debounces repeats. Custom wake phrases are separate models you get from DaVoice. To see how detection behaves before you write code, test it in the wake word playground.

6.3 Speech-to-text and text-to-speech (react-native-davoice)

STT and TTS share one Speech object because they share one audio session:

import Speech from 'react-native-davoice/speech';

await Speech.setAudioRoutingConfig(defaultAudioRoutingConfig); // iOS; same object as the wake word
if (!(await Speech.setLicense(DAVOICE_LICENSE_KEY))) { /* show license error */ }
await Speech.initAll({ locale: 'en-US', model: require('./assets/models/model_ex2_rich_kate.dm') });
await Speech.pauseSpeechRecognition(); // start paused; un-pause when it's the user's turn

Speech.onSpeechPartialResults = (e) => { /* e.value[0]: text so far */ };
Speech.onSpeechResults        = (e) => { /* e.value[0]: final text; e.speakerId when verification is on */ };
Speech.onSpeechError          = (e) => { /* log; most codes are recoverable */ };

await Speech.unPauseSpeechRecognition(-1); // listen
await Speech.pauseSpeechRecognition();     // stop listening (e.g. before TTS)
await Speech.speak('I opened the window.', 0, 1.0); // text, speaker index, speed
Enter fullscreen mode Exit fullscreen mode

To pick a voice, listen to them in the TTS playground.

initAll can take many seconds the first time, so the example wraps it in a timeout and keeps the wake word running if it stalls (initializeWakewordBootstrap in src/wakeword/index.ts):

await withTimeout(
  initializeSpeechLibrary(enrollmentJsonPath ?? null),
  speechInitTimeoutMs,
  'Speech.initAll',
);
Enter fullscreen mode Exit fullscreen mode

6.4 The local LLM and the Agent (react-native-davoice-llm)

The package has two layers: an engine (setLicense, loadModel, generateStream, prefill, cancel, saveState, loadState) and an Agent that runs a whole assistant turn - prompt assembly, tool calls, guards, memory and streaming - configured by an agent pack of JSON files instead of code. DaVoice's on-device LLM guide covers model choice in more depth.

Models. The GGUF files are downloaded at runtime and checked by sha256 - never bundled in the binary. From src/local_llm/constants.ts:

{
  id: LOCAL_LLM_DAVOICE_AUTOMOTIVE_R4B,
  displayName: 'DaVoice Automotive Agent 0.6B',
  filename: 'davoice-automotive-qwen3-0.6b-r4b-it100-q4_0.gguf',
  url: 'https://pub-27f94b1602c647ab95a601086cf52820.r2.dev/agent-llm/auto5_r4b_it100-q4_0.gguf',
  sizeBytes: 381_565_920,
  sha256: '1c078e961c5f4e16ae0f667d213fa06de53f3f4bc92c9248427bdb37237bd205',
  contextSize: 8192,
  chatTemplate: CHAT_TEMPLATE_PRESETS.qwen3.template,
  // ...
}
Enter fullscreen mode Exit fullscreen mode

The picker also lists general models (Qwen2.5 1.5B/3B with chatml, Gemma 3n E2B/E4B with gemma). They are larger and slower; the 0.6B agents are the defaults.

Minimal agent setup (from the repo's englishHowTo.md):

import { setLicense, loadModel, Agent, toolReply } from 'react-native-davoice-llm';

await setLicense(LICENSE);
await loadModel(pathToGguf, { contextSize: 8192 });
const agent = await Agent.create({
  pack: { agent: require('./agent_automotive/agent.json'), skills: [] },
  databasePath: `${RNFS.DocumentDirectoryPath}/davoice_agent_car.db`,
  chatTemplate: 'qwen3',
}, { onRequest });
await agent.configure({ intentRouter: false, webSearchTool: true, timeTool: true, guards: { /* see provider.ts */ } });
await agent.setTools(myTools);   // a handler returns toolReply(result, 'I opened the window.') to say it itself
const r = await agent.turn({ text, history }, (ev) => { if (ev.type === 'text') speak(ev.text); });
// agent.cancel() stops a running turn
Enter fullscreen mode Exit fullscreen mode

The example's real configuration for the fine-tuned models (src/aichat/provider.ts, condensed - the original reads temperature and memory options from the model entry) shows the settings that matter:

agent.configure({
  chatTemplate: model.chatTemplate,
  remote: !handsOver ? false
    : LOCAL_LLM_LAYER2 === 'device' ? { enabled: true, answer: 'layer2' }
    : { enabled: true, timeoutMs: 15000 },
  intentRouter: model.usesRouter !== false,   // false: the model decides with every tool
  webSearchTool: model.modelTools === true,
  timeTool: model.modelTools === true,
  memory: { profile: 'small' },
  sampling: { temperature: 0 },               // greedy for tool calling
  guards: callGuardSettings(model.callGuards ?? 'off', model.historyAsRun === true),
  historyAsRun: model.historyAsRun === true,
  guidedToolCalls: model.guidedToolCalls === true,
  promptOrder: model.promptOrder ?? 'trained',
});
Enter fullscreen mode Exit fullscreen mode

Create the agent once and answer its requests (src/agent/LocalAgent.ts). onRequest is where the app decides about remote hand-over and sensitive tools:

onRequest: async (request) => {
  if (request.type === 'remote') return answerRemoteRequest(request as AgentRemoteRequest);
  if (request.type === 'authorizeTool') return { authorized: isToolAuthorized(request as AgentApprovalRequest) };
  if (request.type === 'confirmTool') return { confirmed: await askToConfirm(request as AgentApprovalRequest) };
  throw new Error(`The app does not answer "${request.type}" requests.`);
},
Enter fullscreen mode Exit fullscreen mode

6.5 Tool calling: define tools in MCP shape, let the app confirm

Tools are plain JSON in MCP shape (name, description, inputSchema, optional execution). From src/car/carToolDefinitions.h3.json:

{
  "name": "set_window",
  "description": "Opens or closes windows, fully, halfway or just a crack.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "window": {
        "type": "string",
        "enum": ["all", "front_left", "front_right", "rear_left", "rear_right",
                 "driver", "passenger", "front", "rear", "left", "right"],
        "description": "Which window: ... Ask if the user did not say which"
      },
      "position": {
        "type": "string",
        "enum": ["open", "close", "half", "crack"],
        "description": "open (lower it), close (raise it), half, crack (a little)"
      }
    },
    "required": ["window", "position"]
  }
}
Enter fullscreen mode Exit fullscreen mode

A sensitive tool declares an authorization scope, and the agent core never runs it without a "yes" from the app:

"execution": { "authorization": { "scope": "door_locks", "message": "Unlock the car?" } }
Enter fullscreen mode Exit fullscreen mode

The JS handler does the work and returns the sentence to speak. This is the pattern in src/car/carTools.ts (trimmed):

import { toolReply } from 'react-native-davoice-llm';

function toTool(t: any): ToolDefinition {
  return {
    name: t.name,
    description: t.description,
    inputSchema: t.inputSchema,
    ...(t.execution ? { execution: t.execution } : {}),
    handler: (rawArgs: Args) => {
      const args = rawArgs ?? {};
      const run = HANDLERS[t.name];
      try {
        if (!run) throw new Refused("Sorry, this car doesn't have that function.");
        const { summary, result, say } = run(args);
        // The app says what it did and the turn ends there: no second model round.
        if (say) return toolReply({ ok: true, ...result }, say);
        return { ok: true, ...result };
      } catch (error) {
        const reason = error instanceof Error ? error.message : String(error);
        if (error instanceof Refused) return toolReply({ ok: false, refused: reason }, reason);
        throw new Error(`Not done: ${reason}`);
      }
    },
  };
}
Enter fullscreen mode Exit fullscreen mode

Why toolReply? Two reasons from the repo's field testing: the spoken confirmation is built from the real result ("The front right door is already locked."), so the model can't announce something that didn't happen; and skipping the second model round saves seconds on a phone CPU. More patterns are in the tool calling guide.

Tip for small models: a 0.6B model depends heavily on tool descriptions. Keep them short, say explicitly "absolute vs. relative" and which side is which, and keep exactly the wording the model was trained on (the car model expects carToolDefinitions.h3.json).

6.6 The voice loop: one turn, streamed into TTS

This is the order of calls that keeps mic hand-offs stable (from docs/AI_AGENT_INTEGRATION_GUIDE.md):

startup:
  mic permission
  setWakewordAudioRoutingConfig + Speech.setAudioRoutingConfig   (iOS, same object)
  create wake-word instance, attach listener, license
  startKeywordDetection
  pauseDetection(false)
  Speech.setLicense, Speech.initAll (with timeout), Speech.pauseSpeechRecognition
  unPauseDetection                    -> listening for the wake word

wake word detected:
  pauseDetection(false)
  Speech.unPauseSpeechRecognition(-1) -> user talks; partials -> agent.prefill
  silence timeout:
    Speech.pauseSpeechRecognition
    agent.turn                        -> fillers and sentences go to Speech.speak as they arrive
  all sentences spoken:
    Speech.unPauseSpeechRecognition(-1)  (follow-up turn)
    or unPauseDetection                  (back to the wake word)
Enter fullscreen mode Exit fullscreen mode

The turn itself (src/aichat/provider.ts):

const result = await agent.turn(
  { text: userText, history: toAgentHistory(history), profile, longTermFacts, localPassages, date: currentDateText() },
  (event) => {
    if (event.type === 'text') onDelta(event.text, false);
    else onEvent?.(event);
  },
);
onDelta('', true);
return { toolMessages: result.toolMessages, webSearch: result.webSearch };
Enter fullscreen mode Exit fullscreen mode

Keep result.toolMessages in the history so the next turn knows what ran ("no, the other one at the back" depends on it).

Speak sentence by sentence, without awaiting each one. The example buffers streamed text and cuts complete sentences (splitCompletedSentences in src/aichat/index.ts), carefully not splitting "DaVoice.io" or "12.5":

// A period ends a sentence only before a space (or a closing quote): not inside "DaVoice.io", "12.5" or "e.g".
if (text[index] === '.') {
  const next = text[index + 1];
  if (next === undefined || !/[\s"')\]]/.test(next)) continue;
}
Enter fullscreen mode Exit fullscreen mode

Each complete sentence is handed to Speech.speak immediately; the native library queues them, so there are no gaps. The reply is finished when every queued sentence has resolved - not on the first onFinishedSpeaking.

Latency tricks used by the example:

  • Warm up when the chat screen opens: download, load, and process the system prompt (~3,100 tokens for the car demo).
  • Save and restore the processed prompt with saveState / loadState. Measured on a Galaxy S20 (CPU): restore 0.7-0.9 s instead of a 30-40 s first warm-up. The car model ships a pre-warmed state (restore 0.23 s, first answer 0.34 s).
  • Prefill while the user talks: call agent.prefill(history, partialText) on partial transcripts so only the last words remain when the silence timeout fires.
  • Measured decision time on a Galaxy S20, CPU only: about 1.3-2 s per command; iPhones with Metal are faster.

7. Add the agent to your existing React Native app

The repo contains two documents written specifically for this: example/englishHowTo.md and docs/AI_AGENT_INTEGRATION_GUIDE.md. They're written so you can hand them to an AI coding agent (Claude Code, Cursor, Copilot) along with your codebase. The core rule: copy the module from src/ plus its assets, install the package, use your own license.

You want Copy Assets / setup
Wake word src/wakeword/index.ts (instanceConfigs, initializeWakewordBootstrap, attachKeywordListenerOnce) wake word .dm in native bundle
Speaker verification src/speaker_verification/ (onboarding.ts, verification.ts, currentSpeaker.ts), thresholds in src/appflow.ts sv.dm in native bundle
STT + TTS src/initialization/, src/stt/, src/tts/ (+ voiceClone.ts) TTS model + voice-cloning encoder per language
Local agent + tools src/local_llm/, src/agent/LocalAgent.ts, src/agent/AgentPack.ts, src/aichat/provider.ts, src/aichat/fragments.ts, src/aichat/remote.ts agent pack JSON (agent/ or agent_automotive/), your tool JSON

A safe integration order:

  1. Install only the packages you need and add the Android Maven lines / iOS plist entries. Build.
  2. Wake word: ship your .dm, copy defaultAudioRoutingConfig and initializeWakewordBootstrap. Build and test on a device.
  3. STT/TTS: Speech.initAll with a timeout; register handlers from src/stt/index.ts (keep mergeSmartKeepPunct - Android restarts partials per segment). Build and test.
  4. Agent: copy src/local_llm/ for download/sha256/load-once, getLocalAgent() for a single agent, and provider.ts for configure + warm-up + streaming.
  5. Replace the car tools with your own tool JSON and handlers that return toolReply(...).
  6. Edit the agent pack (persona, fixed replies, messages) - behavior changes without code changes.

Rules worth following from the guide:

  • Never copy the example's demo license key. Put a placeholder like DAVOICE_LICENSE_KEY in a git-ignored local.config.ts (template: local.config.example.ts).
  • Keep one wake-word instance, one listener and one agent.
  • Keep the order of native calls the same as the example; that order is what makes audio hand-offs stable.
  • Don't require() a .dm you don't use - every require() ships the file.
  • Don't give a .ts module and a .json file the same base name; Metro resolves the .json first.
  • Change one feature at a time and build after each.

8. Advanced capabilities

Speaker verification: an owner-only agent

Enroll once (5 samples in the example), save the enrollment JSON, then gate both the wake word and STT live:

import { createSpeakerVerificationMicController } from 'react-native-wakeword';

const ctrl = await createSpeakerVerificationMicController('svMic1');
await ctrl.create(JSON.stringify({ modelPath: 'sv.dm', options: { decisionThreshold: 0.35 } }));
await ctrl.beginOnboarding('davoice', 5, true);
// 5x: await ctrl.getNextEmbeddingFromMic(); -> progress events; onboardingDone carries enrollmentJson
// save the JSON (react-native-fs), destroy the controller, then enroll:
await wakewordInstance.addSpeaker('owner', enrollmentJsonPath);
await Speech.addSpeaker('owner', enrollmentJsonPath);
// turn the filter off again:
await wakewordInstance.setSpeakers([]);
await Speech.setSpeakers([]);
Enter fullscreen mode Exit fullscreen mode

Each STT result then carries speakerIds / speakerId. The example uses it to authorize the door_locks scope (src/speaker_verification/currentSpeaker.ts):

export function ownerMayAct(): { allowed: boolean; reason: string } {
  if (!gateOn) return { allowed: true, reason: 'speaker verification is off (demo allows it)' };
  if (ownerId && heard.has(ownerId)) return { allowed: true, reason: `owner "${ownerId}" verified` };
  return { allowed: false, reason: heard.size ? `heard ${[...heard].join(', ')}, not the owner` : 'no verified speaker in this request' };
}
Enter fullscreen mode Exit fullscreen mode

In a car, that means a passenger or the radio can't unlock the doors. The enrollment JSON is personal voice data: keep it in the app's documents folder, never in logs. See also the owner-only agent guide.

Voice cloning

await Speech.setExternalVoicesPath(clonedVoicesPath());
const result = await Speech.createVoiceClone(
  audioPath,
  `${clonedVoicesPath()}/${name.trim()}.emd`,
  getVoiceEmbedder(),   // e.g. require('./assets/models/embedder_en.dm')
);
// every .emd in the folder becomes a voice: Speech.changeVoice(name)
Enter fullscreen mode Exit fullscreen mode

If the result has warnings, ask the user to record again. Switching voices inside one model is instant (changeVoice); switching to another model file needs Speech.destroyAll() and initAll again.

Turn-taking and interruptions (barge-in)

The example is turn-based by design: STT is paused while the agent thinks and speaks, so the agent never transcribes its own voice. If you want "Hey Coach" to interrupt the reply, the repo documents the change (based on a production app):

  1. Keep the wake word listening while TTS plays: call unPauseDetection() right after pauseSpeechRecognition().
  2. Turn echo cancellation on: replace WakewordAEC: { regular: false, duringTTS: false } with AEC: { mode: 'always' } in defaultAudioRoutingConfig.
  3. In the keyword callback, if the agent is speaking, stop everything and open a new turn:
if (aiChatInFlightRef.current) {             // the AI is thinking or speaking
  if (interruptInFlightRef.current) return;  // the same "Hey Coach" detected twice
  interruptInFlightRef.current = true;
  try {
    aiChatRequestIdRef.current += 1;         // late text and events of the old turn are ignored
    (await getLocalAgent()).cancel();        // stops the LLM turn
    aiChatSpeechQueueRef.current = [];       // drops the sentences not spoken yet
    ignoreNextFinishedSpeakingRef.current = true;
    await Speech.stopSpeaking();             // silences the voice now
    resetSpeechTranscriptState();            // discards whatever STT caught meanwhile
    aiChatInFlightRef.current = false;
    await myInstanceRef.current?.pauseDetection(false);
    await Speech.unPauseSpeechRecognition(-1, 0); // new user turn; preFetchMs 0
  } finally {
    interruptInFlightRef.current = false;
  }
  return;
}
Enter fullscreen mode Exit fullscreen mode

Open STT with preFetchMs: 0, otherwise it replays the previous ~2 s of audio and transcribes "Hey Coach" as the new message. Combined with speaker verification, only the owner can interrupt. Details: turn-taking and interruptions guide.

Optional cloud layer (clearly not offline)

Set LOCAL_LLM_LAYER2 in src/local_llm/constants.ts to 'cloud' to hand general-knowledge questions to Gemini (src/aichat/remote.ts), or implement answerRemoteRequest for your own backend. This is an optional cloud integration and incurs that provider's token costs. Keep keys out of the client in production (use a backend proxy).


9. Testing and troubleshooting

Run the app tests:

npx jest __tests__/carTools.test.ts __tests__/fragments.test.ts
Enter fullscreen mode Exit fullscreen mode

Read the conversation log (user words, model output, guard decisions, tool calls, timings):

adb shell run-as com.exampleapp cat files/davoice_chat.log   # debug build
adb logcat | grep DavoiceChat                                  # release build
Enter fullscreen mode Exit fullscreen mode

For every command check: the right tool, exact arguments, a spoken sentence that matches what happened, a question when info is missing, a decline when unsupported, and no action on small talk or speech noise.

Symptom Likely cause
Gradle: Could not find com.davoice:... Missing maven { url ".../libs" } line for that package
Model "not found" at runtime File not in the bundle/assets under that exact name, or it's a Git LFS pointer (run git lfs pull)
setLicense returns false Expired or wrong key for this bundle id / package name
The app transcribes its own voice STT not paused during TTS, or AEC off
Wake word fires twice Listener attached twice; use attachKeywordListenerOnce
STT dead after the wake word Detection was stopped instead of paused, or STT never un-paused
Gaps between spoken sentences Awaiting each speak; enqueue without awaiting
iOS app killed while loading the LLM Missing increased-memory-limit entitlement
First reply very slow No warm-up/prefill, or no saved prompt state
"I need to send this to a remote Model" Expected: a general question with LOCAL_LLM_LAYER2 = 'none'
"Sorry, I didn't catch that" on short phrases Speech-noise filter (src/aichat/fragments.ts) - expected for filler like "The."

10. FAQ

What is a Voice-to-Voice AI Agent?
An assistant you talk to and that talks back, while taking actions through tools. Here it's a pipeline - wake word, STT, a local LLM with tool calling, TTS - not a single speech-to-speech model.

Can it run fully offline?
The core loop can, once models are downloaded and for languages with an on-device STT model. Remote model hand-over, web search, MCP tools and the Gemini demo mode need a network and are optional.

Is it really token-free?
There are no cloud LLM inference token charges, because the model runs on the phone. The local model still processes tokens, and DaVoice SDKs require a commercial license for production.

How big is the model and how fast is it?
The DaVoice Agent 0.6B GGUF is about 380 MB, downloaded once. On a Galaxy S20 (CPU only) the decision takes about 1.3-2 s per command; iPhones with A14+ use the GPU.

Can I use my own LLM?
Yes - the engine runs GGUF models with several chat templates (qwen3, chatml, gemma, llama3, phi3). Larger general models are slower on phones.

Does it work with Expo?
Yes, with a development build or EAS Build (not Expo Go). See expo/, docs/EXPO_AI_AGENT_INTEGRATION_GUIDE.md and the Expo guide. A Flutter version exists too.

Can users interrupt the agent?
The example is turn-based. Barge-in with the wake word plus echo cancellation is documented in section 8.

How do I stop the model from doing dangerous things?
Declare execution.authorization or execution.confirmation on the tool, gate it in onRequest, enable call guards (callGuards: 'correct'), and let the app - not the model - confirm results via toolReply.


11. Resources

Top comments (3)

Collapse
 
danny_green_2d7412a66ffb4 profile image
Danny Green •

Oh, very impressive . I tested on personal app and worked very well.
We embraced this solution

Collapse
 
ofer_fryman_69213ebc646a7 profile image
Ofer Fryman •

Thank you very much. I am curious which project is it?

Collapse
 
hvoss profile image
Voss •

Interesting work! I love the fact that it's all 'on device' !