You open a WebSocket to a speech-to-speech model, start streaming your microphone, and three things go wrong at once: the model answers itself because it hears its own voice through your speakers, your transcript prints the same sentence a dozen times, and when you ask "who are you?" it describes its origin wrong. All three have small, specific fixes, and this tutorial builds them in from the start.
Grok Voice 3.0 is xAI's speech-to-speech model for realtime voice agents, released on 14 October 2026: you stream 16-bit PCM audio over one WebSocket and the same model answers with audio, using events that xAI describes as compatible with the OpenAI Realtime API. By the end you will have a terminal voice agent in Node.js that listens to your mic, plays replies, prints live transcripts and calls a function, plus the browser variant that never exposes your API key.
Disclosure: I work on apimodels.app, the API gateway the examples connect to, and the relay behaviour described below comes from its source code.
What you need
- Node.js 18 or newer and the
wspackage 8.x (pnpm add wsornpm install ws) - SoX on your PATH for microphone and speaker I/O:
brew install soxon macOS,sudo apt install soxon Debian or Ubuntu. The script uses itsrecandplaycommands. - An apimodels.app API key in the
APIMODELS_API_KEYenvironment variable, with at least one minute of balance ($0.104), which the server checks before it accepts the connection - Headphones. With open speakers the model hears its own voice and keeps answering itself.
How a Grok Voice session flows
A Grok Voice 3.0 session is a single WebSocket that carries JSON events in both directions, with audio inside the events as base64. You connect, send one session.update to set the voice, instructions and turn detection, then keep appending microphone audio with input_audio_buffer.append. With turn_detection: {"type": "server_vad"} the server decides when you stopped talking and starts answering on its own; you never send a "done" signal.
The answer arrives as a stream: response.output_audio.delta events carry base64 PCM chunks to play in order, response.output_audio_transcript.delta events carry the text of what the model is saying, and response.done closes the turn. Your own speech comes back as conversation.item.input_audio_transcription.updated.
That last event is where the "same sentence a dozen times" bug comes from. xAI's input transcription is cumulative: each updated event carries the whole transcript so far, which may correct earlier words, not a delta to append. Code written for OpenAI-style incremental deltas prints every partial version.
The default audio format is 16-bit PCM, 24 kHz, mono, little-endian, in both directions, so 100 ms of audio is 2,400 samples or 4,800 bytes.
Step 1: Connect and surface handshake errors
Save the four blocks in Steps 1 to 4 into one file, voice-agent.mjs, in order. This first block opens the socket with your key in the Authorization header and prints the server's reason if the handshake is rejected.
// voice-agent.mjs (example). Node 18+, ws 8.x, SoX on PATH. Use headphones.
import { spawn } from "node:child_process";
import WebSocket from "ws";
const URL = "wss://api.apimodels.app/v1/realtime?model=grok-voice-3.0";
const SOX = ["-q", "-t", "raw", "-r", "24000", "-e", "signed-integer", "-b", "16", "-c", "1", "-"];
const ws = new WebSocket(URL, {
headers: { Authorization: `Bearer ${process.env.APIMODELS_API_KEY}` },
});
const send = (event) => ws.readyState === WebSocket.OPEN && ws.send(JSON.stringify(event));
// A rejected handshake is a plain HTTP response with a JSON body:
// 401 bad key, 400 wrong model, 402 less than one minute of balance, 429 more than 5 sessions.
ws.on("unexpected-response", (_req, res) => {
let body = "";
res.on("data", (c) => (body += c));
res.on("end", () => {
console.error(`handshake rejected: ${res.statusCode} ${body}`);
process.exit(1);
});
});
With a wrong or missing key you see handshake rejected: 401 {"error":{"type":"invalid_api_key","code":"invalid_api_key","message":"Invalid or missing API key / client secret."}} and nothing is charged.
Step 2: Configure the session, identity included
As soon as the socket opens, send session.update. The voice, instructions and turn_detection fields sit directly under session.
ws.on("open", () => {
send({
type: "session.update",
session: {
voice: "eve", // one of 28 built-in voices; each one speaks every supported language
instructions:
"You are Grok Voice 3.0, a realtime voice assistant built by xAI, " +
"answering for the Acme help desk. Keep answers under three sentences.",
turn_detection: { type: "server_vad" },
audio: {
input: { format: { type: "audio/pcm", rate: 24000 } },
output: { format: { type: "audio/pcm", rate: 24000 } },
},
tools: [
{
type: "function",
name: "get_order_status",
description: "Look up an order by its id",
parameters: {
type: "object",
properties: { order_id: { type: "string" } },
required: ["order_id"],
},
},
],
},
});
});
The first sentence of instructions is the fix for the third bug. Setting custom instructions replaces the model's default persona, and without an identity in your prompt the model may describe its origin incorrectly when a caller asks who it is. If your product has its own persona ("Mia from Acme"), write that instead; just write something.
Step 3: Stream the microphone in 100 ms frames
Start the mic only after the server confirms the session with session.updated, so your first words are not processed with default settings. SoX's rec writes raw PCM to stdout; the code cuts it into 4,800-byte frames so no 16-bit sample is ever split across two events.
let mic = null;
function startMic() {
mic = spawn("rec", SOX, { stdio: ["ignore", "pipe", "ignore"] }); // raw 16-bit PCM, 24 kHz, mono
let pending = Buffer.alloc(0);
mic.stdout.on("data", (chunk) => {
pending = Buffer.concat([pending, chunk]);
while (pending.length >= 4800) { // 100 ms = 2,400 samples x 2 bytes
send({ type: "input_audio_buffer.append", audio: pending.subarray(0, 4800).toString("base64") });
pending = pending.subarray(4800);
}
});
}
function newSpeaker() {
const p = spawn("play", SOX, { stdio: ["pipe", "ignore", "ignore"] }); // reads raw PCM on stdin
p.stdin.on("error", () => {}); // ignore EPIPE after we kill it for a barge-in
return p;
}
let speaker = newSpeaker();
Step 4: Play replies, print transcripts, answer function calls
One message handler does the rest. Note the transcription case: it overwrites the current line instead of appending, because each event already holds the full sentence.
ws.on("message", (raw) => {
const ev = JSON.parse(raw.toString());
switch (ev.type) {
case "session.updated":
if (!mic) { startMic(); console.log("listening... (Ctrl+C to hang up)"); }
break;
case "input_audio_buffer.speech_started":
// Caller started talking over the model: drop queued audio.
// (Our browser playground handles this event the same way.)
speaker.kill();
speaker = newSpeaker();
break;
case "conversation.item.input_audio_transcription.updated":
case "conversation.item.input_audio_transcription.completed":
// Cumulative: overwrite the line, never append.
process.stdout.write(`\r\x1b[2Kyou: ${ev.transcript ?? ""}`);
break;
case "response.created":
process.stdout.write("\ngrok: ");
break;
case "response.output_audio_transcript.delta":
process.stdout.write(ev.delta);
break;
case "response.output_audio.delta":
speaker.stdin.write(Buffer.from(ev.delta, "base64"));
break;
case "response.done":
process.stdout.write("\n");
break;
case "response.function_call_arguments.done": {
const args = JSON.parse(ev.arguments || "{}");
const result = { order_id: args.order_id, status: "shipped" }; // replace with your lookup
send({
type: "conversation.item.create",
item: { type: "function_call_output", call_id: ev.call_id, output: JSON.stringify(result) },
});
send({ type: "response.create" }); // ask the model to speak the result
break;
}
case "error":
console.error(`\n[error] ${ev.error?.code}: ${ev.error?.message}`);
break;
}
});
function hangUp() { mic?.kill(); speaker.kill(); ws.close(); }
process.on("SIGINT", hangUp);
ws.on("close", (code, reason) => {
console.log(`\nsession closed (${code} ${reason})`);
mic?.kill(); speaker.kill();
});
Closing the socket is not cleanup trivia: it is what stops the billing clock. Sessions are billed by connected time, and the relay only treats a session as idle after 10 minutes with no client events at all. A process that keeps streaming microphone audio, even silence, is never idle and runs until the 60-minute cap.
Verify it works
Run it with headphones on:
APIMODELS_API_KEY=your_key_here node voice-agent.mjs
You should see listening..., then your own words appear on a you: line that updates in place while you speak. Ask "Who are you, and where is order A-1001?" The reply should introduce itself as Grok Voice 3.0 built by xAI (from your instructions), and the function call should come back as a spoken "shipped". Press Ctrl+C; the session appears as one record in your console call history with its duration and cost.
Do the same from a browser
Browsers cannot set headers on a WebSocket, so never put the API key in front-end code. Your server mints a short-lived token and the page passes it as a subprotocol.
// On your server (example): exchange the key for a token valid 10 to 600 seconds (default 60)
export async function mintRealtimeToken() {
const res = await fetch("https://api.apimodels.app/v1/realtime/client_secrets", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.APIMODELS_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ expires_after: { seconds: 60 } }),
});
return res.json(); // { value, expires_at }
}
// In the browser (example), from a click handler so audio is allowed to start
const { value } = await (await fetch("/api/realtime-token", { method: "POST" })).json();
const ws = new WebSocket(
"wss://api.apimodels.app/v1/realtime?model=grok-voice-3.0",
["xai-client-secret." + value],
);
The token only governs when a connection can be opened; an open session keeps running after it expires. The rest of the event handling is the same as in Node. For microphone capture, use an AudioWorklet and resample to 24 kHz yourself: Firefox refuses to connect a microphone to an AudioContext created at a non-default sample rate, which is why our own browser playground captures at the device rate and resamples in JavaScript. The Grok Voice 3.0 API docs on apimodels.app have a complete browser example.
Where apimodels.app fits
apimodels.app is a multi-model API gateway: one API key and OpenAI-compatible endpoints for 140+ image, video, audio and language models. Most mainstream models there have multiple upstream channels with automatic failover, failed requests are not charged, long jobs run asynchronously with webhooks, and any request's charge can be looked up with GET /v1/records/{id}. Grok Voice 3.0 itself has a single upstream route, so there is no failover for it; a session that cannot reach the model is not charged.
For this tutorial it supplied the endpoint, the key and the billing: $0.104 per minute of connected session, billed per second (a 90-second call costs $0.156), $0.0052 per typed text message, and no extra fee for function calls. You can also talk to the model with your microphone on the Grok Voice 3.0 model page before writing any code.
Do not route through it if you only use xAI, already have direct xAI access and want the lowest per-minute price: connect to xAI directly, where the same events work. The same goes if you need sessions longer than 60 minutes, more than 5 concurrent sessions per account, or xAI's server-side tools (web_search, x_search, file_search, mcp) inside the voice session, which the relay rejects.
Common errors
-
handshake rejected: 401 ... invalid_api_key: missing or wrong key, or a browser trying to send the key as a header. Use theAuthorizationheader from Node, or a client secret from a browser. -
handshake rejected: 400 ... "Model gpt-realtime is not available for realtime. Use grok-voice-3.0.": themodelquery parameter is wrong. Usegrok-voice-3.0(orgrok-voice-latest). -
handshake rejected: 404 ... Unknown API endpoint: GET /api/v1/realtime: you connected toapimodels.app/api/v1/realtime. The endpoint only exists onapi.apimodels.app, because the main site's hosting layer cannot proxy WebSockets. -
402
insufficient_balanceat connect, or anerrorevent with that code mid-call: balance below one minute. Each minute is charged when it starts, so the session closes when the next minute cannot be covered. -
429
too_many_sessions: more than 5 open sessions on one account. Close sockets you no longer use. -
[error] unsupported_tool: yoursession.updateincludedweb_search,x_search,file_searchormcp. Onlyfunctiontools are accepted. - The model keeps answering itself: speaker echo. Use headphones, or echo cancellation in the browser.
- The same sentence printed many times: you appended cumulative transcription events. Overwrite instead.
Limits
- This script is an example built from the documented event protocol; run it once with headphones before you build on it. SoX device handling differs by OS, and the SoX arguments above are the only platform-specific part.
- Barge-in via
input_audio_buffer.speech_startedis what our browser playground uses; xAI's documentation page does not list that event, so treat it as best-effort. - Sessions end at 60 minutes with a
session_limiterror; reconnect to continue.
What did you use for audio I/O in Node: SoX, a native addon, or a browser-only setup? I'd like to know what held up in production.
This post was written with AI assistance from the apimodels.app source code and public docs, and reviewed before publishing.
Top comments (0)