DEV Community: Sanchita Sunil

I Gave OpenClaw a Voice and It Ordered Me Dinner

Sanchita Sunil — Wed, 03 Jun 2026 09:09:15 +0000

Quick links.
Code: https://github.com/murf-ai/murf-cookbook/tree/main/examples/openclaw/food_ordering_agent
Video Walkthrough: https://www.youtube.com/watch?v=ypqzB093VLc
Configuration Deep Dive: https://dev.to/sanchita_sunil/notes-from-the-openclaw-voice-tutorial-4ngg

Building a working voice agent usually means stitching state across speech, logic, and external APIs by hand. OpenClaw gives you a runtime that handles most of that for you. To see how far that gets you in practice, I wired OpenClaw up to a microphone, a Murf Falcon voice, and a Swiggy account. In about 800 lines of TypeScript, I had an agent that could search restaurants, take add-to-cart instructions, and place a real order end to end.

This post is an architecture walkthrough. I'll explain what OpenClaw is doing under the hood, where I had to fight its defaults, and how the same blueprint applies to any voice agent you want to build.

Why OpenClaw

There are several agent frameworks out there. Most of them treat an agent as a function: input goes in, tool calls happen, output comes out. OpenClaw is different — it treats the agent as a runtime, more like a long-running server than a single call. Sessions can be paused and resumed, state is keyed and persisted, tool calls go through a typed MCP (Model Context Protocol) interface. And critically for what we are building, OpenClaw exposes block-level streaming hooks that let you intercept the model's output as it arrives.

A voice agent is the hardest case any agent framework will face, because the user can hear every millisecond of latency. If your framework only hands you the full reply at the end, you cannot stream audio to the speakers. The user is left in silence while the model generates 300 characters, which can take 2 to 4 seconds, which feels like forever in conversation time. OpenClaw hands you each block, a sentence or two, the moment it arrives. You turn that block into audio and play it while the model keeps generating the next one.

Three things in particular made this build feel small once I understood them:

Skills are markdown, not function definitions. The Swiggy integration is a SKILL.md file the model reads. No JSON schemas, no function-calling boilerplate. To swap Swiggy for GitHub or Notion later, I would install a different skill and change one config line.
MCP is built in. OpenClaw treats MCP servers as first-class. The Swiggy MCP plugs in through mcporter. Adding a new tool surface means adding a new skill, not writing glue code.
Streaming hooks are real. onBlockReply fires as the model writes. You drive synthesis from inside the callback.

Once those three things are in place, the rest of the build is mostly wiring the audio loop around them.

The pipeline

A microphone library captures audio and a streaming STT turns it into transcripts. Those transcripts go into OpenClaw, which decides what to do, calls skills, and streams text out. A streaming TTS turns each block into audio as it arrives, and a speaker library plays it back.

The audio loop is the same regardless of what the agent is doing. Plug a calendar skill into OpenClaw and you have a voice scheduling assistant. Plug in a GitHub skill and you have a voice PR reviewer. The loop does not change, only the skill and the system prompt do.

The stack

Layer	Tool	Why this one
Agent runtime	OpenClaw	Skill registry, MCP integration, block-level streaming hooks. The framework this post is built around.
Tool surface	Swiggy skill via ClawHub	Vendored MCP skill. Documents the API in markdown the model can read.
Microphone and speaker	Decibri	Native WASAPI on Windows, CoreAudio on Mac, ALSA on Linux. No browser layer.
Speech to text	Deepgram Flux	Streaming STT with end-of-turn detection inside the model. No separate VAD to wire up.
Text to speech	Murf Falcon	Low time-to-first-audio, and conversational voice styles that sound right in back-and-forth dialogue.
Language model	Gemini	Free tier, supports tool calling, fast on first token. Substitutable with any tool-calling LLM.

What I deliberately left out

Before the build, here is what is not in this version:

Wake word detection. The microphone is always on while the agent is not speaking. No "Hey Claw" trigger.
Cross-session memory. Every restart starts fresh. The session key is per-process.
Order cancellation. Swiggy's MCP does not expose it, so the skill routes the user to customer care.
Production hardening. This is a single-user CLI. No auth, no rate limiting, no observability. Don't ship it as is.

A note on latency worth setting expectations on now. Streaming TTS plays each sentence as soon as it is ready, which makes the agent feel responsive on most turns. But tool calls still take as long as tool calls take. When the agent is hitting Swiggy's API for restaurant search, there is real waiting that streaming cannot hide. I cover this in detail in the latency section.

Requirements

Set these up before continuing.

Node and package manager

Node 22.16 or newer. The repo is ESM-only and breaks on earlier versions.
pnpm 9 or newer. The lockfile is pnpm. npm and yarn will resolve different versions.

Platform audio dependencies

Decibri uses the native audio stack on each operating system, so the install steps differ.

Linux: apt install libasound2-dev on Debian-family, or alsa-lib-devel on Fedora-family.
Windows: WASAPI is built in. You need a C++ build toolchain for the Decibri binary. Install "Desktop development with C++" through Visual Studio Installer.
macOS: CoreAudio is built in. You need Xcode Command Line Tools: xcode-select --install.

External CLIs

npm install -g clawhub

clawhub is OpenClaw's skill registry.

API keys

Deepgram for the Flux STT key. New accounts get $200 in starter credit, no card required.
Murf for the Falcon TTS key. Created on the API tab of your Murf account, separate from a regular Murf Studio account.
An LLM provider of your choice. Most have a free tier sufficient for development.

Swiggy account

A Swiggy account with at least one saved delivery address. The agent orders to saved addresses, not live GPS, because the MCP surface exposes addresses, not coordinates.

Step 1: Clone, install, env

git clone --filter=blob:none --sparse https://github.com/murf-ai/murf-cookbook.git
cd murf-cookbook
git sparse-checkout set examples/openclaw/food_ordering_agent
cd examples/openclaw/food_ordering_agent
pnpm install
cp .env.example .env

Open .env and add:
DEEPGRAM_API_KEY=...
MURF_API_KEY=...
GEMINI_API_KEY=...

If you would rather use OpenAI or Anthropic instead of Gemini, change one line in openclaw.json and the env variable name. Tool-calling support is the only requirement.

Step 2: Authenticate Swiggy

Swiggy's MCP needs OAuth. Run this once, the browser opens, you log in, you approve.

node scripts/swiggy-auth.mjs

This opens a browser and signs you into Swiggy via PKCE OAuth, and writes the token as a static Authorization header into ~/.mcporter/mcporter.json. You won't need to do this again unless the token expires.

If the browser doesn't open automatically, the script prints the full auth URL that you can copy and paste manually.

Confirm that the skill can actually reach Swiggy:

node skills/swiggy/swiggy-cli.js food addresses

This should print your saved addresses. If the list is empty, save one in the Swiggy app before moving on.

Note: In the video I use mcporter auth swiggy-food — that no longer works. See the repo README for current auth steps.

Step 3: The four files you write

src/
ear.ts ~110 lines microphone capture and Deepgram WebSocket
brain.ts ~500 lines streaming TTS pipeline, calls OpenClaw
voice.ts ~140 lines speaker output, two channels
index.ts ~140 lines the event loop

The whole agent fits in 900 lines. Three of these files are pure adapter code: microphone in, speaker out. The interesting file is brain.ts, because that is where OpenClaw and Murf Falcon meet.

ear.ts: microphone in, transcript out

Decibri captures 16-bit PCM at 16 kHz in 100 ms chunks. Each chunk goes to a Deepgram Flux WebSocket on /v2/listen.

const params = new URLSearchParams();
params.append("model", "flux-general-en");
params.append("encoding", "linear16");
params.append("sample_rate", "16000");
for (const k of keyterms) params.append("keyterm", k);
const url = `wss://api.deepgram.com/v2/listen?${params.toString()}`;

Two things to know.

First, Flux has end-of-turn detection inside the transcription model. You don't need a separate Voice Activity Detector. You get one event called EndOfTurn and you respond to it.

if (data.type === "TurnInfo") {
  if (data.event === "EndOfTurn") {
    const transcript = data.transcript ?? "";
    if (transcript.trim().length > 0) {
      onTranscription(transcript.trim());
    }
  }
  return;
}

Second, there is a contextual keyterm trick that mattered a lot for Indian-English food vocabulary. After each agent reply, I extract the capitalised words ("Punjab Grill," "Paneer," "Meghana") and pass them as keyterms for the next turn. This is what fixes "Kadhai Paneer" being heard as "car die panel." Standard English ASR doesn't handle Indian food names well. Per-turn keyterm biasing gets it most of the way there.

I wired Deepgram in directly here, not through OpenClaw's STT plugin slot. OpenClaw's STT integration is built for telephony, not a local CLI microphone. 110 lines of WebSocket code was the right tool for this job.

brain.ts: where OpenClaw earns its keep

This is the file that uses every OpenClaw primitive worth using.

The basic flow:

Call OpenClaw's chat() with the user's transcript.
Subscribe to OpenClaw's onBlockReply hook.
Hand each block to Murf Falcon for synthesis as it arrives.
Stream audio back to voice.ts in order.

OpenClaw's defaults are tuned for chat, where each block can be a paragraph and the user is reading on a screen. For voice, three overrides matter.

Override 1: turn streaming on. OpenClaw has two switches that both have to allow streaming. The naming is confusing because one is called disable. So you want disableBlockStreaming: false, which means "do not disable," which means "do stream."

const llmCall = getReplyFromConfig(ctx, {
  disableBlockStreaming: false,
});

Override 2: fix the coalescer. OpenClaw has a coalescer that decides when to flush a buffered block to your code. Its default minChars is 800. A typical voice reply is 200 to 300 characters, so the coalescer waits for a block that never arrives, then dumps everything at end-of-reply. Streaming defeated.

blockStreamingCoalesce: {
  minChars: 1,
  maxChars: 200,
  idleMs: 0,
  flushOnEnqueue: true,
},

flushOnEnqueue: true is the line that makes the rest of this work. It tells OpenClaw to hand the block over the moment it arrives, instead of waiting for more.

Override 3: track deltas yourself. OpenClaw's onBlockReply callback gives you the full text so far, not just the new piece. You compute the delta yourself. Three cases: extension (new starts with old), duplicate (skip), and reset (fresh string after a tool call). The reset case is easy to miss and shows up after every tool call.

let delta: string;
if (currentBlockStream && text.startsWith(currentBlockStream)) {
  delta = text.slice(currentBlockStream.length);
  currentBlockStream = text;
} else if (currentBlockStream && currentBlockStream.includes(text)) {
  return;
} else {
  delta = text;
  currentBlockStream = text;
}

Once you have the delta, you call Murf's synthesize(). Synthesis runs in parallel across blocks, but playback runs in order, serialised through a Promise chain so that chunk 2 always plays after chunk 1 even if chunk 2's network call finishes first.

const synthP = synthesizeSpeech(trimmed).catch(() => null);
emitChain = emitChain.then(async () => {
  const audio = await synthP;
  if (audio) onAudioChunk(audio);
});

That is roughly 30 lines of streaming logic. The rest of brain.ts is the agent setup, the OpenClaw config, and a fallback path for when the model batches output after tool calls.

voice.ts: two speakers, not one

Falcon's synthesis is fast — Murf reports 130 ms time-to-first-audio, and that matches what I see in practice. So when there is dead air on the agent's first turn, it is not the TTS that is causing it. It is the cold-start cost of OpenClaw initialising, the Swiggy MCP handshake, the LLM doing its first call against a fresh tool chain. All of that has to finish before the model has produced its first block of text for Falcon to synthesise.

That is the gap pre-recorded filler audio is for. Short clips like "One moment please" or "let me check that for you" play 100 ms after the user stops talking, which is fast enough that the user does not perceive a delay.

The catch: the filler is a variable-length clip, and the first real audio chunk can arrive before the filler finishes. If both play through one audio output, one cuts off the other. The fix is two separate Decibri outputs.

let oneShotSpeaker: InstanceType<typeof DecibriOutput> | null = null;
let streamSpeaker: InstanceType<typeof DecibriOutput> | null = null;

oneShotSpeaker plays fillers. streamSpeaker plays the real reply. When the first reply chunk arrives, I stop the filler channel without touching the reply channel. Anything queued on the reply channel keeps playing.

This sounds like overkill until you hear the alternative. With one channel, the filler clips the agent saying "Sure" and the user only hears "...I'll add that."

index.ts: the loop

async function startSession() {
  renderBanner();
  setImmediate(() => warmup());      // amortise OpenClaw cold start
  await playIntro();
  await openMicrophone();
}

ear.on("transcript", async (text) => {
  closeMicrophone();
  await playFiller();                  // mask LLM latency
  await chat(text);                    // streams audio as it arrives
  await openMicrophone();
});

That is the whole loop. Render the banner, kick off OpenClaw warmup in the background, play the intro, open the microphone. On each transcript: stop the microphone, play a filler, run the agent, reopen the microphone.

The setImmediate(() => warmup()) line runs OpenClaw's initialisation and the Swiggy MCP handshake while the user is hearing the intro. By the time the user finishes their first sentence, both are warm. That shaves several seconds off turn 1.

How the skill actually works

This is the part that surprised me most when I first used OpenClaw.

The agent learns to use Swiggy by reading a markdown file. Not a JSON schema, not function definitions. A human-readable file called SKILL.md that documents the commands, the sequencing rules, and the things to never do. The model reads this, figures out what to call, and emits shell commands that run against a CLI wrapper.

The wrapper is small. node skills/swiggy/swiggy-cli.js food <command> is the shape of every call. The skill knows commands like search-restaurants, get-menu, add-to-cart, checkout. The model sequences them on its own, based on the markdown documentation.

Here is a snippet from SKILL.md (paraphrased):

search-restaurants: Find restaurants matching a cuisine or dish. Use this first whenever the user mentions a food. Example: search-restaurants --query "biryani". Always call get-addresses first if you have not yet, because results depend on delivery location.

The model reads it the same way a new developer would read documentation on day one.

The one tweak I made: every swiggy food <cmd> call in SKILL.md became node skills/swiggy/swiggy-cli.js food <cmd>. OpenClaw's shell executor doesn't have npm globals on PATH, so the swiggy binary from npm link is not reachable.

The implication for builders: writing a new skill is writing a markdown file and a thin CLI. There is no SDK to learn, no function-calling glue to debug. If you can document an API in English with examples, you can give an OpenClaw agent the ability to call it.

Latency

The first turn is the slowest. Before any audio plays, OpenClaw has to initialise, complete the Swiggy MCP handshake, and make its first LLM call against a fresh tool chain. On a typical machine that takes anywhere from 15 to 50 seconds, depending on your network and your LLM provider. Streaming TTS does not save you here — the model has not produced anything to synthesise yet.

What does help is the combination of filler audio (which plays 100 ms after the user stops talking) and the background warmup that runs during the intro. Together they keep the perceived gap small even when the actual cold start is not.

Turn 2 onwards is a different story. With the runtime warm and the MCP connection open, first audio arrives 5 to 10 seconds after the user stops talking, and most of that is the LLM's time to its first sentence. Falcon's 130 ms TTFA is what makes "first sentence" actually translate to "first audio you hear."

If you genuinely need to push first-turn latency below this on tool-heavy turns, the only real lever is to take OpenClaw out of the loop on those turns — wiring the tool calls in directly, parallelising what OpenClaw would have serialised. I haven't done that in this build.

Swap the skill

The voice loop in this post does not care what the agent does. The skill lives in two files:

agents.defaults.skills in openclaw.json. Replace swiggy with another MCP skill. Google Calendar. GitHub. Notion. Linear. Pick one from ClawHub or write your own.
workspace/IDENTITY.md. The system prompt that describes who the agent is and how it should talk. Rewrite it for the new domain.

That portability is the case I wanted to make for OpenClaw with this post. The framework is doing real work behind the scenes, hiding the runtime, the MCP integration, the streaming, and the skill format behind primitives that are small enough to use without ceremony.

What I learned

The skill format is the part I underestimated going in. The model was reading it the way a developer would read API docs on day one. There is no JSON schema to maintain, no function-calling boilerplate to update when the API changes. If your API is documentable in markdown, an OpenClaw agent can use it.

Voice agents are mostly a latency engineering problem. The transcription, the agent, the TTS are mostly solved. The work that made this build feel real was around the seams — two-channel playback, background warmup, per-turn keyterm bias, pre-baked fillers. You have to find these by listening to your own demo and noticing what sounds wrong.

The combination of streaming hooks and per-block synthesis is what made the conversational rhythm work. Falcon at 130 ms TTFA is fast on its own, OpenClaw handing off blocks the moment they arrive is fast on its own. Together, if the LLM produces text in roughly 200 ms chunks and the TTS adds 130 ms on top, the user hears a new sentence every ~330 ms. That is faster than most humans speak, and it is what makes the agent feel like it is actually thinking out loud rather than waiting to deliver a finished answer.

If this was useful, the code is at github.com/murf-ai/murf-cookbook. A star helps the project reach more builders. Clone it, swap the skill, and build something else tonight. The configuration deep dive, with the parameter tables and error mappings, is at dev.to/sanchita_sunil/notes-from-the-openclaw-voice-tutorial-4ngg.

I would love to hear what you build with it.

Notes from the Openclaw Voice Tutorial

Sanchita Sunil — Wed, 03 Jun 2026 08:29:41 +0000

This is a companion to the food-ordering agent tutorial video (You can find the video here: https://www.youtube.com/watch?v=ypqzB093VLc). The video walks you through cloning the repo and placing a real Swiggy order with your voice. This post fills in the parts the video pointed at but did not have time to cover:

Every Deepgram Flux parameter, what it does, and how the event model behaves
Why OpenClaw's block streaming defaults are wrong for voice, and which ones to flip
Falcon voice and locale compatibility, and how to swap voices without breaking things
Streaming-pipeline bugs that show up after setup, with their root causes

Repo: https://github.com/murf-ai/murf-cookbook/tree/main/examples/openclaw/food_ordering_agent
Video: https://www.youtube.com/watch?v=ypqzB093VLc

OpenClaw treats an agent as a runtime, not a prompt. A runtime is a program that runs continuously and remembers state between calls, like a server. A prompt, in contrast, is a single block of text sent to the model. The difference matters because OpenClaw can pause, resume, and track sessions across many turns.

That model works well for chat. Voice is where it starts to break down.

A microphone does not produce text. It produces audio frames (small chunks of raw sound data). A speaker cannot wait for the full reply before playing anything. The user will hear silence and assume the agent is broken. The same tool-call delay that is invisible in a chat UI becomes obvious dead air the moment the user can hear it.

Every piece of OpenClaw still works for voice. You just have to point each piece at the voice use case on purpose, instead of relying on the chat-friendly defaults. The next three sections walk through which defaults to change and why.

Requirements

If you do not have these, set them up before continuing.

Node and package manager

Node 22.16 or newer. The repo is ESM-only and breaks on earlier versions.
pnpm 9 or newer. The lockfile is pnpm. npm and yarn will resolve different versions.

Platform audio dependencies

Decibri uses the native audio stack on each operating system, so the install steps differ.

Linux: apt install libasound2-dev on Debian-family distros, or alsa-lib-devel on Fedora-family. Required at install time.
Windows: WASAPI is built in. You need a C++ build toolchain for the Decibri binary. Install "Desktop development with C++" through Visual Studio Installer.
macOS: CoreAudio is built in. You need Xcode Command Line Tools: xcode-select --install.

External CLIs

clawhub. OpenClaw's skill registry. The Swiggy skill in this repo is vendored, so you do not strictly need clawhub to run the agent, but you will need it if you want to fetch other skills later.

API keys

Deepgram for the Flux STT key. New accounts get $200 in starter credit, no card required.
Murf for the Falcon TTS key. This is created on the API tab of your Murf account, separate from a regular Murf Studio account.
An LLM provider of your choice. Most have a free tier sufficient for development.

Swiggy

A Swiggy account with at least one saved delivery address. The agent orders to saved addresses, not live GPS, because the MCP surface exposes addresses, not coordinates.

Update: The Swiggy auth flow has changed since the video was recorded. mcporter auth swiggy-food no longer works — Swiggy MCP now requires an approved client_id and uses a custom PKCE script instead. Run node scripts/swiggy-auth.mjs. See the repo README for current steps.

Deepgram Flux

Flux is the STT we use in this build. There are several streaming STTs that work for voice agents; Flux is the one wired up here, and the parts below are the configuration you need to get right regardless of which API you go with.

One concept worth covering before the parameters: turn-taking. This is the decision of when the user has stopped talking and the agent should respond. Many streaming STT APIs hand back partial transcripts and leave turn-taking to your code, which usually means adding a separate Voice Activity Detector (VAD) that listens for silence. Flux does turn-taking inside the transcription model and emits structured events for it, so for this build we do not need a separate VAD.

Endpoint

An endpoint is the URL path you connect to on a server. Flux only works on /v2/listen. The older /v1/listen endpoint will silently reject the model parameter. You will spend an hour wondering why nothing transcribes.

const params = new URLSearchParams();
params.append("model", "flux-general-en");
params.append("encoding", "linear16");
params.append("sample_rate", "16000");
for (const k of keyterms) params.append("keyterm", k);
const url = `wss://api.deepgram.com/v2/listen?${params.toString()}`;

Use URLSearchParams to build the URL. It encodes spaces in multi-word keyterms correctly (as +). If you build the query string by hand and use %20 instead, Deepgram will close the connection without telling you why. This is the most common setup bug.

Parameters

The audio format below uses the term PCM, which means pulse-code modulation. It is the standard way to represent raw audio as numbers. linear16 means each sample is a 16-bit number stored in little-endian byte order. Most audio libraries use this format by default.

Parameter	Value used	What it does
`model`	`flux-general-en`	Flux English. Use `flux-general-multi` for multilingual.
`encoding`	`linear16`	16-bit PCM audio. Must match what your microphone library outputs.
`sample_rate`	`16000`	16 kHz audio. Decibri captures at this rate by default.
`keyterm`	repeated	Vocabulary biasing. Up to 100 keyterms per connection.
`eager_eot_threshold`	not set	Enables EagerEndOfTurn events at this confidence. Off in this repo.

You can also pass eot_threshold to tune end-of-turn sensitivity. The default works well for short food-ordering sentences. If your agent handles longer thinking-out-loud utterances, raise it.

The Flux events we use

Flux sends five event types on its TurnInfo stream. The repo only consumes one of them, but the others are worth knowing because you will probably want some of them later.

Update. Partial transcript, updated as the user keeps talking. Useful if you want a live transcript display. Not used here.
StartOfTurn. The user just started speaking. This is where you would handle barge-in (cutting off the agent if it is still talking, so the user can interrupt). Not connected here.
EndOfTurn. High confidence the user is done. This is the only event the repo uses. When it fires, the transcript goes to the LLM and the agent starts generating a reply.
EagerEndOfTurn. Medium confidence the user is done. Off by default. If you turn it on (with eager_eot_threshold), the agent can start drafting a reply early. Saves some delay at the cost of more LLM calls because some drafts get thrown away.
TurnResumed. Only fires after an EagerEndOfTurn. Means the user was not actually done, and any draft you started should be discarded.

if (data.type === "TurnInfo") {
  if (data.event === "EndOfTurn") {
    const transcript: string = data.transcript ?? "";
    if (transcript.trim().length > 0) {
      onTranscription(transcript.trim());
    }
  }
  return;
}

Keyterm biasing for Indian-English food vocabulary

Deepgram lets you pass up to 100 keyterms per connection. Keyterms tell the model "if you hear something close to one of these words, lean toward this spelling." Most apps set keyterms once at connect time using a fixed vocabulary.

Flux's Configure control message lets you update keyterms on every turn. The repo uses this to bias the next turn on whatever proper nouns the agent just said.

function extractContextualKeyterms(text: string): string[] {
  const tokens = text
    .replace(/[.,!?;:()"']/g, " ")
    .split(/\s+/)
    .filter((w) => w.length >= 3 && /^[A-Z]/.test(w) && !KEYTERM_STOPWORDS.has(w));
  return [...new Set(tokens)];
}

The idea is simple. If the agent just said "Paneer Butter Masala from Punjab Grill," the user's reply is much more likely to contain those words than some random restaurant name. So we extract the capitalised words from the agent's last reply and use them as bias for the next turn.

For Indian-English food vocabulary, where standard English speech recognition struggles the most, this one feature is the difference between the agent hearing "Kadhai Paneer" and hearing "car die panel."

Cost

Deepgram bills Flux per second of streaming audio. As of early 2026, the pay-as-you-go rate sits in the range of $0.0077 to $0.015 per minute, depending on the plan and region. Check Deepgram's pricing page for current numbers. New accounts get $200 in starter credit.

A rough cost estimate for the food-ordering agent:

Average turn: 3 seconds of user speech, microphone open during user speech only
Per-turn STT cost: 3 seconds at the higher end of the range, roughly $0.00075
Ten-turn ordering session: under one cent for STT

You will run out of $200 of credit long before you run out of patience for testing.

Block streaming

OpenClaw was built for chat first. Its block streaming was tuned for long replies on a screen. In that setup, each block (a unit of text the model sends back) might be a whole paragraph. For voice, each block should be a sentence or two. Every millisecond between "LLM produced text" and "speaker plays sound" is silence the user can hear.

The defaults are wrong for voice. Until you change them, OpenClaw quietly holds onto your blocks instead of sending them to your code right away.

First, turn streaming on

OpenClaw has two settings that control block streaming:

blockStreamingDefault in the config (the channel-wide default)
disableBlockStreaming at the call site (the override for one call)

Both have to allow streaming, or it will not happen.

const llmCall = getReplyFromConfig(ctx, {
  disableBlockStreaming: false,
});

The naming is confusing. The option is called disable, so false means "do not disable." Which means "do stream." So you want disableBlockStreaming: false. Read it twice if needed.

Fix the coalescer

The coalescer is the component that decides when to send a buffered block to your code. To buffer means to hold onto something until enough has built up. To send the buffered content onward is to flush it.

The coalescer's default minChars setting is 800. A typical voice reply is 200 to 300 characters. So with the default, the coalescer waits for an 800-character block that will never arrive. It gives up at the end of the reply and dumps everything at once. Streaming defeated.

Override it like this (brain.ts lines 96 to 109):

blockStreamingChunk: {
  minChars: 1,
  maxChars: 200,
  breakPreference: "sentence",
},
blockStreamingCoalesce: {
  minChars: 1,
  maxChars: 200,
  idleMs: 0,
  flushOnEnqueue: true,
},

The line that matters most is flushOnEnqueue: true. It tells the coalescer to send the block to your code the moment it arrives, without waiting. Every other override is necessary, but useless without this one.

Track deltas yourself

A callback is a function that OpenClaw calls when something happens, like a new block arriving. OpenClaw's onBlockReply callback is given the full text so far, not just the new piece. So you have to figure out what is new yourself. The new piece is called the delta.

Here is how the repo computes it (brain.ts lines 486 to 501):

let delta: string;
if (currentBlockStream && text.startsWith(currentBlockStream)) {
  delta = text.slice(currentBlockStream.length);
  currentBlockStream = text;
} else if (currentBlockStream && currentBlockStream.includes(text)) {
  return; // already covered
} else {
  delta = text;
  currentBlockStream = text;
}

There are three cases here, and the third is the one that matters most:

Extension. The new text starts with the old text. The delta is just the part at the end. Easy.
Duplicate. The same block got reported twice. Skip it.
Reset. The new text has nothing to do with the old text. This happens after a tool call finishes. OpenClaw starts a fresh block stream, and the new text is a brand-new string. Without this branch, you would either lose the new block or join it incorrectly to the old one.

The empty payload.text quirk

When block streaming is actually working, payload.text in the final reply is an empty string. This is not a bug.

OpenClaw has a check called shouldDropFinalPayloads that removes the text from the final payload once it has already been streamed. This avoids sending the same text twice. The repo handles this by collecting text in its own buffer (canonicalText) as chunks arrive. It only falls back to payload.text if the buffer is empty:

if (!canonicalText && payloadText) canonicalText = payloadText;

Murf Falcon

Synthesis is the technical word for generating audio from text. Murf Falcon is the TTS model used in this build. Murf reports a model latency of 55 ms and a time-to-first-audio of 130 ms, at $0.01 per 1,000 characters — roughly 1 cent per minute of generated audio.

Turn off OpenClaw's built-in TTS

OpenClaw ships with its own TTS pipeline. By default it runs in auto: "on" mode, which produces one final audio file at the end of a reply. That mode is incompatible with per-block streaming, so we turn it off (openclaw.json lines 30 to 47):

"tts": {
  "provider": "murf",
  "auto": "off",
  "mode": "final",
  "providers": {
    "murf": {
      "voiceId": "en-IN-anusha",
      "model": "FALCON",
      "locale": "en-IN",
      "style": "Conversational"
    }
  }
}

With auto: "off", the Murf provider stays loaded and configured. But your code is now in charge of synthesis. You call murfProvider.synthesize() directly on each block.

Voice and locale compatibility

A locale is a code that identifies a language and region together, like en-IN for English in India or es-MX for Spanish in Mexico.

Falcon supports voices across many languages, but each voice is bound to its locale. If you set voiceId to an English voice and locale to hi-IN, the API rejects the request. If you change just one of the two when swapping voices, things silently break.

Voice ID prefix	Locale	Notes
`en-IN-*`	`en-IN`	Indian English. Used in this repo.
`en-US-*`	`en-US`	American English.
`en-UK-*`	`en-UK`	British English.
`hi-IN-*`	`hi-IN`	Hindi.
`es-ES-*`	`es-ES`	Spanish (Spain).
`es-MX-*`	`es-MX`	Spanish (Mexico). Different voices than Spain.

The full list is in Murf's API docs. Before you change voiceId in openclaw.json, query /v1/speech/voices?model=FALCON and pick a voice and its matching locale together.

Pick the right voice style

Falcon exposes a style parameter. Pick Conversational for agent work. A voice that sounds great reading an audiobook usually sounds wrong in a back-and-forth conversation. Promotional and Narration styles sound theatrical when the agent is saying short things like "Sure, anything else?"

Two speaker outputs

The pre-recorded filler audio masks the cold-start delay by playing while the LLM is still thinking. The problem is that the filler clip is a variable length, and the first real audio chunk can arrive before the filler finishes.

If you play both through the same audio output, one of two bad things happens:

The filler cuts off the start of the real reply, or
The reply cuts off the end of the filler.

The fix is two separate audio outputs (voice.ts lines 10 to 11):

let oneShotSpeaker: InstanceType<typeof DecibriOutput> | null = null;
let streamSpeaker: InstanceType<typeof DecibriOutput> | null = null;

oneShotSpeaker plays fillers. streamSpeaker plays the actual reply. When the first reply chunk arrives, stopOneShotPlayback() stops the filler channel without touching the reply channel. Anything already queued on the reply channel keeps playing.

Synthesise in parallel, play back in order

There are two layers of parallelism worth understanding.

Within a single block. Murf splits long input into chunks of up to 1500 characters and synthesises them at the same time on its own infrastructure. You do not have to do anything for this.

Across blocks. The repo starts synthesis calls the moment each block arrives. So multiple blocks can be synthesising at the same time. But the audio plays back in order through a Promise chain:

const dispatchChunk = (text: string) => {
  const trimmed = text.trim();
  if (!trimmed) return;
  if (!streamingEnabled) return;
  const synthP = synthesizeSpeech(trimmed).catch(() => null);
  emitChain = emitChain.then(async () => {
    const audio = await synthP;
    if (audio) {
      streamedAnyAudio = true;
      onAudioChunk!(audio);
    }
  });
};

synthesizeSpeech() starts the Murf network call right away. emitChain.then() waits for the previous chunk's synthesis to finish before playing the current one. So if chunk 1 and chunk 2 both take 400 ms to synthesise but chunk 1's network is slower, chunk 2 still plays second. Never first.

Streaming-pipeline bugs and their root causes

The video has a short error table for the bugs you hit during setup. This section covers the ones specific to the streaming pipeline that show up later, when the agent is mostly working.

WebSocket closes with code 1008 the moment audio starts

Code 1008 means "policy violation," which Deepgram uses for invalid API keys. Check DEEPGRAM_API_KEY in your environment, and check the Deepgram console for remaining credit.

WebSocket closes with code 1011 partway through a session

Code 1011 means "internal server error," but in practice the most common cause is running out of credit mid-session. Top up and retry.

Transcripts come back empty even though audio is sending

Three things to check, in order:

Sample rate. sample_rate in the URL must match your microphone's actual rate. The repo captures at 16000. If your system is recording at 44100 or 48000, you have to resample before sending.
Encoding. The encoding parameter and the audio format must match. linear16 expects 16-bit signed little-endian PCM.
Model. model must be flux-general-en or flux-general-multi. No other model name works on /v2/listen.

The agent's first sentence plays, then nothing

This is the coalescer holding onto your blocks. If you did not override flushOnEnqueue, the first block flushes but nothing after it streams. Check brain.ts for the coalesce override.

Audio plays out of order

The Promise chain in dispatchChunk is what keeps playback in order. If you removed the emitChain.then(...) wrapper or replaced it with Promise.all, chunks will play in synthesis-completion order instead of arrival order. Put the chain back.

The agent talks over itself

This means the filler kept playing after the real reply started. Check that stopOneShotPlayback() runs on the first chunk of the real reply, not at the end of the reply.

Voice cuts off mid-sentence

Falcon synthesis can fail silently for a single chunk. The .catch(() => null) in dispatchChunk protects you from one failed chunk crashing the whole reply. But if too many chunks fail, the user hears gaps. Log the failures and check Murf's status page.

ALSA errors on Linux

On minimal Linux installs the ALSA development headers have to be installed before the npm package will build. apt install libasound2-dev covers it on Debian-family. If install completes but the device is not found at runtime, the default ALSA device is probably pointing at an output that does not exist.

No audio on Windows

Decibri on Windows uses WASAPI. If your default output device is a Bluetooth headset that is not currently connected, the stream opens silently and no audio plays. Switch the default device in Sound settings, or set the output device explicitly in code.

Silent failure on macOS

The first run asks for microphone permission. If you deny it, subsequent runs fail silently. The agent will appear to start normally and the WebSocket will connect, but no audio frames reach Deepgram. Check microphone permissions in System Settings under Privacy and Security.

Extending the agent to something that is not Swiggy

It takes two changes.

Swap the skill. The agents.defaults.skills array in openclaw.json is the list of MCP skills the agent can call. Remove the Swiggy skill, add a different one. A calendar scheduler imports a Google Calendar MCP skill. A GitHub PR merger imports the GitHub MCP skill. A Notion assistant imports the Notion MCP skill. The runtime does not change.

Rewrite the identity. workspace/IDENTITY.md is the system prompt. It describes who the agent is, what it does, what it refuses to do, and how it should format replies. Rewriting this file changes the agent's personality and its understanding of the task.

For a calendar scheduler, you would describe an assistant that looks up free slots and confirms bookings. For a PR merger, you would describe a reviewer that summarises diffs and merges when checks pass.

Everything else stays. The audio pipeline, the streaming coalescer, the keyterm bias, the two-channel playback. That is the value of keeping the voice layer separate from the agent layer. The voice layer does not care what the agent is doing.

What this pipeline does not fix

Turn 1 latency is not solved. Time-to-first-audio on a cold start is mostly caused by tool chains and LLM time-to-first-token, not by synthesis. The slow path still includes OpenClaw's cold start, the Swiggy MCP setup, and the LLM's first-token delay. Streaming synthesis cannot hide that. The filler audio can. That is why it is there.

Getting to true sub-second first audio on turn 1 would require starting the OpenClaw runtime ahead of time, keeping the MCP connection alive across sessions, and starting tool calls before the user finishes speaking. None of those are in this repo. What is in this repo is the pattern that makes the problem manageable: split the audio pipeline from the agent pipeline, stream what can be streamed, mask the rest with fillers, and measure the result.

Turn 2 onwards is a different story. With the runtime warm and the MCP connection open, first audio arrives 5 to 10 seconds after the user stops talking. Falcon plus block streaming are why. That is the number that makes the agent usable in practice. The cold-start number is what makes every tutorial-shaped demo look slower than it will be in production.

Block streaming, Falcon, and contextual keyterm biasing are three improvements that build on each other. Each does less than a demo suggests. Together they do more than any one of them alone. That is usually how voice pipelines work.