DEV Community: Mart Schweiger

Transcription Webhooks & Callbacks: The Complete Guide

Mart Schweiger — Wed, 22 Jul 2026 14:56:03 +0000

If you're polling GET /v2/transcript/{id} every few seconds waiting for a transcript to finish, you're burning requests and adding latency for no reason. AssemblyAI can just call you back. Set one field, expose one endpoint, and you get a POST the moment transcription completes.

This guide covers the whole path—not just "set webhook_url," but the part most docs skip: what the payload actually contains, how to verify the request is really from us, and how retries behave when your server hiccups. If you're building anything production-grade on top of speech-to-text, read to the end.

Polling vs. webhooks: when to use which

Both approaches answer the same question—"is my transcript done yet?"—but they trade off differently.

Polling means repeatedly calling GET /v2/transcript/{id} (say, every 3 seconds) until status comes back completed or error. It's dead simple, needs no public endpoint, and works from a script on your laptop. The cost is wasted requests and a delay equal to your polling interval.

Webhooks flip the direction. You register a URL, AssemblyAI POSTs to it the instant the job finishes, and you do zero waiting. The tradeoff: you need a publicly reachable endpoint that returns a 2xx within 10 seconds, plus a little plumbing to secure and verify it.

Rule of thumb: prototypes and CLI tools poll; backend services that process audio at any real volume use webhooks. If you're running a queue of transcription jobs behind a web service, webhooks are the obvious choice.

Enabling a webhook

You enable a webhook by setting webhook_url in the body of your POST /v2/transcript request against the base URL https://api.assemblyai.com. Auth is a plain authorization header with your API key—no Bearer prefix.

Here it is with curl:

curl https://api.assemblyai.com/v2/transcript \
  --header "authorization: <YOUR_API_KEY>" \
  --header "content-type: application/json" \
  --data '{
    "audio_url": "https://example.com/audio.mp3",
    "webhook_url": "https://your-app.com/webhooks/transcript"
  }'

In Python, build a TranscriptionConfig, call set_webhook(url), and submit the job with submit()—not transcribe(). Using submit() matters: it returns immediately instead of blocking until the transcript is ready, which is the entire point of a webhook:

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

config = aai.TranscriptionConfig().set_webhook(
    "https://your-app.com/webhooks/transcript"
)

transcriber = aai.Transcriber(config=config)
transcript = transcriber.submit("https://example.com/audio.mp3")

print(f"Submitted {transcript.id}. We'll get a callback when it's done.")

And in JavaScript:

import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: "<YOUR_API_KEY>" });

const transcript = await client.transcripts.submit({
  audio: "https://example.com/audio.mp3",
  webhook_url: "https://your-app.com/webhooks/transcript",
});

console.log(`Submitted ${transcript.id}`);

That's the whole setup. Full field reference lives in the submit endpoint docs.

What the payload actually contains

Here's the gotcha that trips up almost everyone. When transcription finishes, AssemblyAI POSTs to your URL with a minimal payload:

{
  "transcript_id": "your-transcript-id",
  "status": "completed"
}

status is either completed or error. That's it.

The payload does not contain the transcript text. It does not contain the error details either. To get the words—or to read the error field when something failed—you have to call GET /v2/transcript/{transcript_id} yourself. The webhook is a notification, not a delivery of the result.

So your receiver's job is really two steps: catch the notification, then go fetch the actual transcript:

import requests

def fetch_transcript(transcript_id, api_key):
    response = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers={"authorization": api_key},
    )
    response.raise_for_status()
    return response.json()

If you internalize one thing from this article, make it this: the webhook tells you when, the GET tells you what.

Building a receiver endpoint

Let's put it together into a complete, working receiver. Here's a Python/Flask service that validates the custom auth header, checks the status, fetches the transcript on success, and reads the error field on failure:

import os
import requests
from flask import Flask, request, abort

app = Flask(__name__)

ASSEMBLYAI_API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
WEBHOOK_SECRET = os.environ["WEBHOOK_SECRET"]

def fetch_transcript(transcript_id):
    response = requests.get(
        f"https://api.assemblyai.com/v2/transcript/{transcript_id}",
        headers={"authorization": ASSEMBLYAI_API_KEY},
    )
    response.raise_for_status()
    return response.json()

@app.route("/webhooks/transcript", methods=["POST"])
def handle_webhook():
    # 1. Verify the request via your custom auth header.
    if request.headers.get("X-My-Webhook-Secret") != WEBHOOK_SECRET:
        abort(401)

    payload = request.get_json(silent=True) or {}
    transcript_id = payload.get("transcript_id")
    status = payload.get("status")

    if not transcript_id:
        # Malformed request—4xx tells AssemblyAI to stop retrying.
        abort(400)

    # 2. Branch on status. Fetch the real data with a GET.
    if status == "completed":
        transcript = fetch_transcript(transcript_id)
        print(f"Transcript {transcript_id} ready: {transcript['text'][:120]}...")
        # ... enqueue downstream work, write to your DB, etc.
    elif status == "error":
        transcript = fetch_transcript(transcript_id)
        print(f"Transcript {transcript_id} failed: {transcript['error']}")
        # ... alert, retry the job, flag the customer record, etc.

    # 3. Return 2xx fast so AssemblyAI marks delivery successful.
    return "", 200

Notice the shape of it. Verify first, respond fast, and keep heavy work out of the request path. If fetching and processing the transcript is slow, push it onto a background queue and return 200 immediately—you've only got a 10-second budget, which we'll get to.

The same structure works in Node/Express:

import express from "express";

const app = express();
app.use(express.json());

const API_KEY = process.env.ASSEMBLYAI_API_KEY;
const WEBHOOK_SECRET = process.env.WEBHOOK_SECRET;

app.post("/webhooks/transcript", async (req, res) => {
  if (req.header("X-My-Webhook-Secret") !== WEBHOOK_SECRET) {
    return res.sendStatus(401);
  }

  const { transcript_id, status } = req.body;
  if (!transcript_id) return res.sendStatus(400);

  // Respond first, then do the work off the request path.
  res.sendStatus(200);

  const result = await fetch(
    `https://api.assemblyai.com/v2/transcript/${transcript_id}`,
    { headers: { authorization: API_KEY } }
  ).then((r) => r.json());

  if (status === "completed") {
    console.log(`Ready: ${result.text.slice(0, 120)}...`);
  } else if (status === "error") {
    console.error(`Failed: ${result.error}`);
  }
});

app.listen(3000);

Securing your webhook

There's no HMAC or signature scheme here—AssemblyAI doesn't sign the payload. So don't go looking for a signature header to validate. Instead, you get three complementary controls, and you should use more than one.

Custom auth header. Attach a secret header when you submit the job, and check it on every incoming request. Set webhook_auth_header_name and webhook_auth_header_value in the submit body:

curl https://api.assemblyai.com/v2/transcript \
  --header "authorization: <YOUR_API_KEY>" \
  --header "content-type: application/json" \
  --data '{
    "audio_url": "https://example.com/audio.mp3",
    "webhook_url": "https://your-app.com/webhooks/transcript",
    "webhook_auth_header_name": "X-My-Webhook-Secret",
    "webhook_auth_header_value": "secret-value"
  }'

In Python, pass the extra arguments to set_webhook:

config = aai.TranscriptionConfig().set_webhook(
    "https://your-app.com/webhooks/transcript",
    "X-My-Webhook-Secret",
    "secret-value",
)

AssemblyAI will send that header back on the webhook request, and your receiver rejects anything without the right value—that's the abort(401) in the example above.

Source-IP allow-list. Webhook requests originate from fixed IPs: 44.238.19.20 in the US and 54.220.25.36 in the EU. Restrict your endpoint to those addresses at the firewall or load balancer for a second layer of defense.

webhook_status_code. After delivery, the transcript object carries a webhook_status_code field—the HTTP status code AssemblyAI received from your endpoint. Pull it with a GET to confirm your receiver actually returned what you think it did. It's the fastest way to debug a webhook that "isn't firing" (spoiler: it usually is, and your endpoint returned a 500).

Use the auth header and the IP allow-list together. Neither is a signature, but combined they're a solid perimeter.

Reliability and retries

Your endpoint has to return a 2xx status within 10 seconds. That's the contract.

If you don't—timeout, 5xx, connection refused—AssemblyAI retries. You get up to 10 total attempts, spaced 10 seconds apart. That gives your service a bit under two minutes to recover from a transient blip without losing the notification.

One sharp edge: a 4xx response marks the delivery as failed and stops retries immediately. The reasoning is that 4xx means "your request is wrong and will always be wrong"—retrying won't help. So return 4xx only for genuinely bad requests (a missing transcript_id, a failed auth check). If your database is momentarily down, return a 5xx so the retry machinery kicks in and you get another shot.

This is exactly why the receiver examples respond 200 before doing the heavy lifting. Fetching the transcript, running downstream analysis, writing to storage—none of that should happen inside the 10-second window. Acknowledge fast, process async.

Adding metadata via query params

The webhook payload only carries transcript_id and status, so how do you know which customer or order a given callback belongs to? Put it in the URL. AssemblyAI POSTs to your webhook_url exactly as you supplied it, query string and all:

config = aai.TranscriptionConfig().set_webhook(
    "https://your-app.com/webhooks/transcript?customer_id=1234&order_id=5678"
)

When the callback lands, read those params off the request:

customer_id = request.args.get("customer_id")
order_id = request.args.get("order_id")

This is the clean way to correlate deliveries and do things like per-customer usage tracking without a lookup table keyed on transcript_id. Just don't put secrets in the query string—keep those in the auth header.

Testing locally

Your receiver needs a public URL, which is awkward on localhost. Use a tunneling tool—ngrok, Cloudflare Tunnel, or similar—to expose your local port to the internet, then submit a job pointing webhook_url at the tunnel's HTTPS address.

ngrok http 3000
# Use the forwarding URL (e.g. https://abc123.ngrok.io/webhooks/transcript)
# as your webhook_url when submitting a transcript.

Submit a short audio file, watch the request hit your terminal, and confirm you return 200. If nothing arrives, GET the transcript and check webhook_status_code—it'll tell you whether AssemblyAI reached your endpoint and what it got back.

A note on streaming webhooks

Everything above is for pre-recorded (async) transcription. If you're working with real-time audio, streaming has its own webhook mechanism—see the streaming webhooks docs. Don't mix the two; the payloads and lifecycle differ.

Next steps

Webhooks turn transcription from something you wait on into something that notifies you. Set webhook_url, verify the request, respond fast, and fetch the transcript with a GET—that's a production-grade integration.

From here, dig into the pre-recorded audio webhooks guide for the canonical reference, or read up on what speech-to-text is and how accurate it is in 2026 if you're still evaluating models.

View the full API reference for every field on the transcript object.

Frequently asked questions

Does AssemblyAI support webhooks?

Yes. Set webhook_url in the body of your POST /v2/transcript request (or use set_webhook() in the Python SDK / webhook_url in the JS submit() call), and AssemblyAI will POST to that URL when transcription finishes. It's the recommended alternative to polling for backend services.

What's in the webhook payload?

Just two fields: transcript_id and status (completed or error). The payload does not include the transcript text or error details. To get the actual result—or the error field on failure—call GET /v2/transcript/{transcript_id}.

How do I secure and verify webhook requests?

There's no HMAC signature. Use a custom auth header (webhook_auth_header_name and webhook_auth_header_value) that your receiver checks on every request, restrict your endpoint to AssemblyAI's source IPs (44.238.19.20 in the US, 54.220.25.36 in the EU), and inspect the webhook_status_code field on the transcript to confirm what your endpoint returned.

What happens if my endpoint is down when a webhook fires?

Your endpoint must return a 2xx within 10 seconds. If it doesn't, AssemblyAI retries up to 10 total attempts, 10 seconds apart. A 4xx response marks the delivery failed and stops retries immediately, so reserve 4xx for genuinely bad requests and return 5xx for transient errors you want retried.

Should I use webhooks or polling?

Polling (GET /v2/transcript/{id} every few seconds) is simplest and needs no public endpoint—good for scripts and prototypes. Webhooks push a notification the instant a job finishes with no wasted requests, but require a publicly reachable endpoint that returns 2xx in 10 seconds. Use webhooks for production backend services.

Can I attach my own metadata to a webhook?

Yes. Add query parameters to your webhook_url, like ?customer_id=1234&order_id=5678. AssemblyAI POSTs to the URL exactly as you supplied it, so you read those params off the incoming request to correlate deliveries. Keep secrets in the auth header, not the query string.

How to Transcribe Audio From a Mobile App (iOS, Android)

Mart Schweiger — Wed, 22 Jul 2026 14:55:56 +0000

If you searched for an AssemblyAI mobile SDK, here's the short version: there isn't one. The official SDKs are Python (pip install assemblyai) and JavaScript/TypeScript (npm install assemblyai). No Swift, no Kotlin, no React Native package.

That's not the problem you think it is. The bigger constraint is security: don't ship your API key to client-side code. A mobile binary can be decompiled, and anyone who pulls your key out of it can spend your account balance. So even if a native SDK existed, calling AssemblyAI directly from a phone would be the wrong move.

The right pattern is a thin backend you control. Your mobile app records audio and sends it to your server. Your server holds the API key, talks to AssemblyAI over the REST API, and returns plain text. That's the whole lesson, and it's a good one—you get key security, request logging, rate limiting, and the freedom to swap models later without shipping a new app build.

Here's what you'll build:

A backend (Node or Python) that receives an audio file, uploads it to AssemblyAI, creates a transcript, polls for the result, and returns the text.
Three thin clients—Swift, Kotlin, and React Native—that record audio and call your backend, never AssemblyAI.

Prerequisites

An AssemblyAI account and API key. Get your free API key from the dashboard.
Node 18+ or Python 3.8+ for the backend.
Xcode for iOS, Android Studio for Android, or a React Native toolchain—whichever platforms you're targeting.

If you're new to the API, the transcribe an audio file quickstart covers the async flow end to end. This tutorial adapts it for mobile.

Architecture overview

One rule shapes everything: the API key lives on the server, and only the server talks to AssemblyAI.

┌──────────────┐        ┌──────────────────────┐        ┌─────────────────┐
│  Mobile app  │        │   Your backend       │        │   AssemblyAI    │
│  (Swift /    │        │   (holds API key)    │        │   REST API      │
│   Kotlin /   │        │                      │        │                 │
│   RN)        │        │                      │        │                 │
│              │──audio─▶│ POST /transcribe     │        │                 │
│              │  file  │   1. POST /v2/upload ─┼──bytes─▶ upload_url      │
│              │        │   2. POST /v2/transcript──JSON─▶ transcript id   │
│              │        │   3. GET  /v2/transcript/{id}──poll──▶ completed │
│              │◀─text──│   return { text }     │        │                 │
└──────────────┘        └──────────────────────┘        └─────────────────┘

Note what's not in the diagram: the mobile client never sees the API key and never has AssemblyAI's URL. It only knows about your endpoint. If you rotate the key or change models, the app doesn't change.

Why a backend proxy specifically? For pre-recorded audio there's no temporary-token mechanism—temporary tokens exist only for streaming. So for file transcription, the key has to live somewhere trusted, and that's your server.

The AssemblyAI async flow

Everything the backend does is three REST calls against https://api.assemblyai.com. Auth is your API key in the authorization header—no Bearer prefix.

Upload (optional): POST /v2/upload with the raw audio bytes as the body. Returns { "upload_url": "..." }. Skip this if your audio already lives at a public URL.
Create transcript: POST /v2/transcript with JSON { "audio_url": "..." }. Returns a transcript object with an id and a status.
Get result: GET /v2/transcript/{id}. Poll every few seconds until status is completed or error.

The status lifecycle is queued → processing → completed (or error).

One gotcha that bites people on the upload step: the body must be raw bytes, not JSON. If you send a JSON envelope, the upload appears to succeed but transcription fails later with "Transcoding failed." Send the file bytes directly.

Backend implementation

Pick your language. Both versions expose a single POST /transcribe endpoint that accepts a multipart file upload from the mobile client and returns { "text": "..." }. Both read the API key from an environment variable—never hardcode it.

Node (Express)

npm install express multer
export ASSEMBLYAI_API_KEY="your_key_here"

import express from "express";
import multer from "multer";

const app = express();
const upload = multer({ storage: multer.memoryStorage() });

const API_KEY = process.env.ASSEMBLYAI_API_KEY;
const BASE_URL = "https://api.assemblyai.com";

// Poll until the transcript is done or errors out.
async function pollTranscript(id) {
  while (true) {
    const res = await fetch(`${BASE_URL}/v2/transcript/${id}`, {
      headers: { authorization: API_KEY },
    });
    const transcript = await res.json();

    if (transcript.status === "completed") return transcript;
    if (transcript.status === "error") {
      throw new Error(transcript.error);
    }
    await new Promise((r) => setTimeout(r, 3000)); // poll every 3s
  }
}

app.post("/transcribe", upload.single("audio"), async (req, res) => {
  try {
    // 1. Upload the raw audio bytes. Body is the buffer, NOT JSON.
    const uploadRes = await fetch(`${BASE_URL}/v2/upload`, {
      method: "POST",
      headers: { authorization: API_KEY },
      body: req.file.buffer,
    });
    const { upload_url } = await uploadRes.json();

    // 2. Create the transcript.
    const createRes = await fetch(`${BASE_URL}/v2/transcript`, {
      method: "POST",
      headers: {
        authorization: API_KEY,
        "content-type": "application/json",
      },
      body: JSON.stringify({ audio_url: upload_url }),
    });
    const { id } = await createRes.json();

    // 3. Poll for the result and return just the text.
    const transcript = await pollTranscript(id);
    res.json({ text: transcript.text });
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.listen(8080, () => console.log("Listening on :8080"));

Python (FastAPI)

pip install fastapi "uvicorn[standard]" python-multipart requests
export ASSEMBLYAI_API_KEY="your_key_here"

import os
import time

import requests
from fastapi import FastAPI, UploadFile, HTTPException

app = FastAPI()

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
BASE_URL = "https://api.assemblyai.com"
HEADERS = {"authorization": API_KEY}

def poll_transcript(transcript_id: str) -> dict:
    while True:
        res = requests.get(f"{BASE_URL}/v2/transcript/{transcript_id}", headers=HEADERS)
        transcript = res.json()
        if transcript["status"] == "completed":
            return transcript
        if transcript["status"] == "error":
            raise RuntimeError(transcript["error"])
        time.sleep(3)  # poll every 3s

@app.post("/transcribe")
async def transcribe(audio: UploadFile):
    try:
        audio_bytes = await audio.read()

        # 1. Upload the raw bytes. data=audio_bytes sends them as-is, not JSON.
        upload = requests.post(
            f"{BASE_URL}/v2/upload", headers=HEADERS, data=audio_bytes
        )
        upload_url = upload.json()["upload_url"]

        # 2. Create the transcript.
        create = requests.post(
            f"{BASE_URL}/v2/transcript",
            headers={**HEADERS, "content-type": "application/json"},
            json={"audio_url": upload_url},
        )
        transcript_id = create.json()["id"]

        # 3. Poll and return the text.
        transcript = poll_transcript(transcript_id)
        return {"text": transcript["text"]}
    except Exception as err:
        raise HTTPException(status_code=500, detail=str(err))

Run it with uvicorn main:app --port 8080.

A few production notes that apply to both:

Polling vs. webhooks. Polling every 3 seconds is simple and fine for short clips. For longer audio, hold the HTTP connection open only if your infra tolerates it—otherwise return the transcript id immediately and let the client poll a GET /transcribe/{id} route, or register a webhook so AssemblyAI notifies you when the job finishes.
Limits. The upload endpoint accepts files up to 2.2 GB. The transcript request body caps at 5 GB, and audio duration must be between 160ms and 10 hours.
Add features here, not on the client. Want speaker labels or summarization? Add the options to the /v2/transcript JSON body server-side. The app doesn't change.

For every field you can pass, see the submit transcript reference, the upload reference, and the get transcript reference.

Client implementations

All three clients do the same three things: record audio to a file, POST that file to your backend as multipart form data, and show the returned text. None of them know AssemblyAI exists.

Set BACKEND_URL to wherever your server runs. Use http://localhost:8080 for a simulator, or your machine's LAN IP for a physical device.

iOS / Swift

Record with AVAudioRecorder, then upload with URLSession. Add NSMicrophoneUsageDescription to your Info.plist first.

import AVFoundation

final class Transcriber: NSObject, AVAudioRecorderDelegate {
    private var recorder: AVAudioRecorder?
    private let backendURL = URL(string: "http://localhost:8080/transcribe")!
    private lazy var fileURL: URL = {
        FileManager.default.temporaryDirectory.appendingPathComponent("recording.m4a")
    }()

    func startRecording() throws {
        let session = AVAudioSession.sharedInstance()
        try session.setCategory(.playAndRecord, mode: .default)
        try session.setActive(true)

        let settings: [String: Any] = [
            AVFormatIDKey: Int(kAudioFormatMPEG4AAC),
            AVSampleRateKey: 44100,
            AVNumberOfChannelsKey: 1,
        ]
        recorder = try AVAudioRecorder(url: fileURL, settings: settings)
        recorder?.record()
    }

    func stopAndTranscribe() async throws -> String {
        recorder?.stop()

        let audioData = try Data(contentsOf: fileURL)
        let boundary = "Boundary-\(UUID().uuidString)"

        var request = URLRequest(url: backendURL)
        request.httpMethod = "POST"
        request.setValue(
            "multipart/form-data; boundary=\(boundary)",
            forHTTPHeaderField: "Content-Type"
        )

        // Build the multipart body under the "audio" field name.
        var body = Data()
        body.append("--\(boundary)\r\n".data(using: .utf8)!)
        body.append(
            "Content-Disposition: form-data; name=\"audio\"; filename=\"recording.m4a\"\r\n"
                .data(using: .utf8)!
        )
        body.append("Content-Type: audio/m4a\r\n\r\n".data(using: .utf8)!)
        body.append(audioData)
        body.append("\r\n--\(boundary)--\r\n".data(using: .utf8)!)

        let (data, _) = try await URLSession.shared.upload(for: request, from: body)
        let result = try JSONDecoder().decode(TranscriptResponse.self, from: data)
        return result.text
    }
}

struct TranscriptResponse: Decodable {
    let text: String
}

No API key anywhere. The client's entire world is backendURL.

Android / Kotlin

Record with MediaRecorder, upload with OkHttp. Declare RECORD_AUDIO in your manifest and request it at runtime.

import android.media.MediaRecorder
import okhttp3.MediaType.Companion.toMediaType
import okhttp3.MultipartBody
import okhttp3.OkHttpClient
import okhttp3.Request
import okhttp3.RequestBody.Companion.asRequestBody
import org.json.JSONObject
import java.io.File

class Transcriber(private val cacheDir: File) {

    private val client = OkHttpClient()
    private val backendUrl = "http://10.0.2.2:8080/transcribe" // 10.0.2.2 = host from emulator
    private val outputFile = File(cacheDir, "recording.m4a")
    private var recorder: MediaRecorder? = null

    fun startRecording() {
        recorder = MediaRecorder().apply {
            setAudioSource(MediaRecorder.AudioSource.MIC)
            setOutputFormat(MediaRecorder.OutputFormat.MPEG_4)
            setAudioEncoder(MediaRecorder.AudioEncoder.AAC)
            setAudioChannels(1)
            setOutputFile(outputFile.absolutePath)
            prepare()
            start()
        }
    }

    fun stopAndTranscribe(): String {
        recorder?.apply { stop(); release() }
        recorder = null

        // Multipart upload under the "audio" field name.
        val body = MultipartBody.Builder()
            .setType(MultipartBody.FORM)
            .addFormDataPart(
                "audio",
                "recording.m4a",
                outputFile.asRequestBody("audio/m4a".toMediaType())
            )
            .build()

        val request = Request.Builder().url(backendUrl).post(body).build()

        client.newCall(request).execute().use { response ->
            val json = JSONObject(response.body!!.string())
            return json.getString("text")
        }
    }
}

Call stopAndTranscribe() off the main thread (a coroutine on Dispatchers.IO works well).

React Native

Record with a library like react-native-audio-recorder-player, then upload with the built-in fetch and FormData:

import AudioRecorderPlayer from "react-native-audio-recorder-player";

const recorder = new AudioRecorderPlayer();
const BACKEND_URL = "http://localhost:8080/transcribe";

export async function startRecording() {
  return recorder.startRecorder(); // returns the file path
}

export async function stopAndTranscribe() {
  const filePath = await recorder.stopRecorder();

  const form = new FormData();
  form.append("audio", {
    uri: filePath,
    name: "recording.m4a",
    type: "audio/m4a",
  });

  const response = await fetch(BACKEND_URL, {
    method: "POST",
    body: form, // don't set Content-Type; fetch sets the multipart boundary
  });

  const { text } = await response.json();
  return text;
}

Because JavaScript is a first-class citizen here, you might be tempted to reach for the assemblyai npm package inside your React Native app. Don't. The Node SDK is built for server environments, and using it on the client means embedding your API key in the bundle—the exact thing we're avoiding. The fetch-to-your-backend pattern above is the correct one for React Native.

Testing and validation

Test the backend on its own before wiring up a phone. Record or grab any short audio file and hit the endpoint with curl:

curl -X POST http://localhost:8080/transcribe \
  -F "audio=@sample.m4a"

You should get back {"text":"..."} within a few seconds for a short clip. If you don't:

"Transcoding failed" almost always means the upload body wasn't raw bytes. Confirm you're sending the buffer/bytes directly, not a JSON wrapper.
401 Unauthorized means the authorization header is missing or wrong. Check that ASSEMBLYAI_API_KEY is set in the server's environment—and remember, no Bearer prefix.
The request hangs on long audio because polling holds the connection open. Switch to the return-id-and-poll pattern or a webhook.

Once curl works, point the app at the backend. On a physical device, localhost won't resolve to your dev machine—use its LAN IP (for example http://192.168.1.20:8080/transcribe), and on Android the emulator reaches the host at 10.0.2.2.

Realtime from mobile (next step)

If you need live captions instead of file-at-a-time transcription, AssemblyAI has a streaming API over WebSocket at wss://streaming.assemblyai.com/v3/ws. There's still no mobile-native streaming SDK, but streaming has something file transcription doesn't: short-lived tokens.

Your backend calls GET /v3/token?expires_in_seconds=60 (tokens last 1–600s and are single-use), hands the token to the app, and the app opens the WebSocket with the token as a token query param. The key still never leaves your server; the token is safe to give the client because it expires fast and works once. Audio goes up as mono 16-bit PCM.

That flow is documented in authenticate with a temporary token. For a working streaming client to model your backend on, see real-time transcription in Python.

Next steps

You now have a mobile transcription pipeline that keeps your API key where it belongs. From here:

Add speech understanding. Speaker diarization, summarization, and more are options on the /v2/transcript body—turn them on server-side and your app gets them for free. Start with what is speech-to-text and the Speech-to-Text product page.
Harden the backend. Add authentication on your /transcribe route, rate limiting, and request logging—the proxy is the natural place for all of it.
Explore every parameter. The full API reference documents the complete transcript object.

Frequently asked questions

Is there an official AssemblyAI iOS or Android SDK?

No. The only official SDKs are Python (pip install assemblyai) and JavaScript/TypeScript (npm install assemblyai). For mobile, call the REST API at https://api.assemblyai.com from a backend you control.

Can I call the AssemblyAI API directly from my mobile app?

Technically the REST API works from anywhere, but you shouldn't. Calling it directly means embedding your API key in the app binary, where it can be extracted. Route every request through your own backend instead.

How do I keep my API key safe in a mobile app?

Never ship it to the client. Store it as an environment variable on your server, have the app send audio to your backend, and let the backend attach the authorization header when it calls AssemblyAI. That's the architecture in this tutorial.

Does AssemblyAI support React Native?

There's no React Native SDK. Use fetch with FormData to send audio to your backend, exactly like the web. Avoid the assemblyai npm package on the client—it's a server SDK and would expose your key.

Can I do realtime transcription on mobile?

Yes, over the streaming WebSocket at wss://streaming.assemblyai.com/v3/ws. There's no mobile streaming SDK, so your backend mints a one-time token via GET /v3/token?expires_in_seconds=60 and the app connects with it. Audio is mono 16-bit PCM. See the temporary token docs.

What are the file size and duration limits?

The upload endpoint accepts files up to 2.2 GB. The transcript request body caps at 5 GB, and audio must run between 160ms and 10 hours.

Why Realtime Is the Future of Speech-to-Text

Mart Schweiger — Wed, 22 Jul 2026 14:55:30 +0000

For most of the last decade, transcription was something you did after the fact.

You recorded a call, dumped the file into a queue, waited a few minutes (or a few hours), and got back a block of text. That was the deal. Batch, async, post-hoc—whatever you want to call it, the audio was already over by the time the model saw it. And for a long time, that was fine, because that's all the technology could reliably do.

Recently, the field has quietly crossed the line where the most interesting, highest-value speech-to-text work happens while people are still talking. Realtime isn't a niche feature you bolt on for a captions demo. It's becoming the default that everything else gets measured against.

Here's the argument.

The shift: the interesting STT workloads are now live

Look at where the energy is going in Voice AI right now.

Voice agents that answer the phone and actually resolve the issue. Agent-assist tools that whisper the next best action to a rep mid-call. Live captions that keep up with a fast talker. Ambient scribes that write the clinical note as the visit unfolds. Real-time conversation intelligence that flags a compliance risk or a churn signal before the call ends, not in a report the next morning.

Every one of those is realtime-first. Not realtime-optional—realtime-first. The value evaporates if you make the user wait.

Compare that to the classic batch workloads: transcribing a podcast back catalog, indexing a media archive, generating subtitles for content that already exists. Those are real, they matter, and they're not going away. But notice the pattern—they're all about audio that's already done. The frontier, the stuff people are building companies around in 2026, is about audio that's happening.

So the question flips. It used to be "when do I actually need realtime?" Now it's closer to "why am I making this a two-step process at all?"

What changed: latency and accuracy crossed the usability line at the same time

For years you could have fast transcription or accurate transcription in the streaming setting, but picking both felt like a trap. Realtime models were noticeably worse than their batch cousins. So teams defaulted to batch when quality mattered and only reached for streaming when they absolutely had to.

That tradeoff is mostly gone. And it went away because two curves crossed the usability threshold at roughly the same moment.

The first is latency. Universal-3.5 Pro Realtime, the current streaming flagship, returns partial transcripts in a few hundred milliseconds and finalizes in the same ballpark. The prior generation, Universal-3 Pro Streaming, already ran around 150ms P50 post-VAD, roughly 240ms at P90, with time-to-complete-transcript near 250ms. Those aren't "good for realtime" numbers. Those are numbers where a person on the other end of a phone call can't tell a machine is in the loop.

The second curve is accuracy. On Pipecat's open STT benchmark, run on real agent conversations, Universal-3.5 Pro Realtime posted a 6.99% pooled word error rate. For context, that same benchmark put Deepgram Flux at 15.58%, ElevenLabs Scribe v2 at 9.76%, and Google Chirp3 at 9.04%. The entity numbers are where it really shows up: 15.31% entity error rate versus 50.50% for the next-closest streaming competitor on that run, and 3.55% on phone numbers.

Streaming model	Pooled word error rate
Universal-3.5 Pro Realtime	6.99%
Google Chirp3	9.04%
ElevenLabs Scribe v2	9.76%
Deepgram Flux	15.58%

Sit with that for a second. A streaming model, transcribing live, is now clearing accuracy bars that batch models were bragging about a couple of years ago.

And it keeps getting smarter mid-conversation. Universal-3.5 Pro Realtime takes direction from the application through agent_context—you pass in what the agent just asked, so a mumbled reply, a spelled-out account ID, or a one-word "yep" resolves correctly instead of turning into garbage. Across a benchmark of 20,000 voice agent audio files, passing that context cut WER by 10.2%, with fabrications down 18.3% and hallucinations down 17.2%.

Then there's turn-taking. Old systems decided you were done talking by waiting for silence, which is why they interrupted you every time you paused to think. The end-of-turn detection here reads tonality, pacing, and rhythm—not just a silence timer—and fires in about 300ms. That's the difference between a conversation and a walkie-talkie.

When fast and accurate stop being a tradeoff, the reason to default to batch quietly disappears.

But here's the thing: realtime is genuinely harder

It may be easy to read the section above and conclude realtime is just batch with a lower latency setting. It's not. It's a different engineering problem, and pretending otherwise is how teams ship a demo that falls apart in production.

Batch has a luxury realtime never gets: the whole recording. A batch model can look ahead, reconsider, and clean up before it hands you a single finished answer. A streaming model has to commit to words it can't take back while the sentence is still being spoken.

That creates hard problems that simply don't exist in async:

Turn detection. Deciding when a person is done is its own discipline. Cut them off and the agent talks over them; wait too long and it feels laggy and dead. Getting this right is arguably harder than the transcription itself.

Partial versus final transcripts. Streaming emits guesses (partials) that it later confirms or revises (finals). Your application has to be built around that flicker—rendering something useful now, correcting it gracefully a moment later—without making the UI jump around like a broken slot machine.

Live context and memory. A batch model gets the full conversation for free. A streaming model has to carry context forward as it goes. Universal-3.5 Pro Realtime keeps a short rolling memory (Context Carryover, on by default) so it remembers what was said thirty seconds ago instead of treating every utterance as a cold start.

Diarization with revision. Figuring out who's speaking, live, is brutal—especially with overlap and short back-and-forth turns. The approach that actually works is to label speakers in real time and then send a single correction that re-clusters the whole session within about half a second of the stream ending. We wrote up the mechanics in streaming speaker diarization, and it's a good illustration of the general principle: realtime often means being provisionally right now and definitively right slightly later.

None of this is a reason to avoid realtime. It's a reason to take it seriously as its own category—and to be skeptical of any provider that treats streaming as an afterthought to their batch product.

The business case: latency is starting to look like a revenue metric

Here's where it gets interesting for the people who sign the invoices.

We've spent years talking about transcription accuracy as the thing that drives business outcomes, and it does—we've written a whole piece on the true cost of inaccurate transcription. But in realtime systems, latency starts behaving like its own outcome variable.

Think about a voice agent taking an inbound call. Every extra beat of dead air after the caller stops talking is a beat where the caller wonders if the line dropped, gets annoyed, or hangs up. I'd frame it as an open question the whole industry should be measuring more rigorously: does streaming latency move call conversion and hangup rates? My strong suspicion, from what we see across customers, is yes—and that the effect is larger than most teams assume. A voice agent that responds in about a second feels like a conversation. One that responds in three feels like a hold queue. Same transcript, wildly different business result.

That reframes the whole evaluation. If you're only benchmarking word error rate, you're measuring half the system. In a live setting, how fast the words arrive and how naturally the turns flow are load-bearing for the metric you actually care about, whether that's containment rate, CSAT, or conversion. This is part of why we keep arguing that word error rate alone is broken as a way to judge a model—it says nothing about whether the system feels alive.

Where batch still absolutely makes sense

I'd lose credibility fast if I told you batch is dead. It isn't, and I'd steer you toward it in plenty of cases.

	Realtime (streaming)	Batch (async)
Best for	Interactive work — voice agents, agent assist, live captions, ambient scribes	Already-recorded audio — archives, media libraries, overnight analytics
Model	Universal-3.5 Pro Realtime	Universal-3.5 Pro
Pricing	$0.45/hr base, billed on session duration	$0.21/hr, billed on audio duration
Key advantage	Acts on speech while the conversation is still happening	Look-ahead over the whole file; simpler and cheaper at volume

If the audio is already recorded and nobody's waiting on the result, batch is the right tool. Transcribing an archive of ten thousand old calls. Subtitling a media library. Running analytics over a quarter of sales conversations overnight. There's no live interaction to preserve, so paying the complexity tax of streaming buys you nothing.

Cost is real, too. Batch speech-to-text with Universal-3.5 Pro runs $0.21/hr of audio, billed on audio duration. Streaming with Universal-3.5 Pro Realtime is $0.45/hr base, billed on session duration—the whole time the WebSocket is open. For huge volumes of already-finished audio, the async path is simpler and cheaper, full stop.

And batch still gets that look-ahead luxury. When you can afford to wait, a model that sees the entire file can sometimes squeeze out accuracy that no streaming system will match on the hardest audio.

So the honest position isn't "realtime replaces batch." It's "realtime becomes the default for interactive work, and batch keeps owning the archive." Two tools, clearer boundaries.

What to look for in a realtime STT API

If you buy the argument that realtime is where the field is heading, here's what I'd actually evaluate—because "we support streaming" on a pricing page tells you almost nothing.

Ask about latency and accuracy together, on realistic audio. A model that's fast on clean read speech and falls apart on a noisy phone call in an accent isn't solving your problem. That's why we benchmark on real agent conversations, not lab recordings, and publish the numbers on our benchmarks page.

Ask how turn detection works. Silence-timer only? Or does it read tonality and pacing? This single choice determines whether your product feels human.

Ask whether the model can take live context. Can you feed it the agent's current question or a set of key terms mid-stream? agent_context and keyterm prompting are the difference between a model that guesses at "my ID is A-as-in-apple, 4, 4, 9" and one that gets it right.

Ask about diarization and whether it revises. Live speaker labels that never get corrected will drift. The revise-at-end-of-turn pattern is what keeps them honest.

And ask about the production stuff—concurrency limits, EU data residency, a signable BAA for healthcare workloads. Universal-3.5 Pro Realtime runs with unlimited concurrency and no rate limits, which matters the day your traffic spikes and you don't want to file a ticket to serve it.

For the full picture of how we think about the live path, the streaming speech-to-text product page is the place to start, and if you're weighing accuracy specifically, how accurate is speech-to-text in 2026 digs into the numbers.

The close: the transcript stops being the product

Here's the forward-looking part, and it's the thing I've been circling the whole time.

As realtime becomes the default, the transcript itself stops being the deliverable. For decades the output of speech-to-text was a document—text you'd read, search, or file. But a realtime transcript isn't something a human reads. It's something another system consumes, instantly, to make a decision: route the call, surface the answer, flag the risk, generate the next turn.

That changes what "good" means. The winning realtime models won't just be the ones with the lowest word error rate. They'll be the ones that emit the most machine-actionable stream—context-aware partials, confident finals, live speaker turns, end-of-turn signals—in a shape the next system can act on without waiting.

In other words, speech-to-text is quietly turning from a transcription tool into a real-time interface between human speech and software. The transcript is becoming plumbing. And once you see it that way, batch-versus-realtime stops being a feature comparison and starts looking like the difference between reading history and participating in the present.

That's the future I'd bet on. The rest of the field is already building for it.

Frequently asked questions

Is realtime speech-to-text replacing batch transcription?

No—and anyone who says so is oversimplifying. Realtime is becoming the default for interactive workloads like voice agents, agent assist, live captions, and ambient scribes, where making a user wait kills the value. Batch still wins for already-recorded audio: archives, bulk media, and overnight analytics, where it's simpler and cheaper. Batch async runs $0.21/hr with Universal-3.5 Pro; streaming runs $0.45/hr base. Think two tools with clearer boundaries, not a replacement.

What's the difference between realtime and streaming speech-to-text?

They're the same thing. "Streaming" describes the mechanism—audio flows over a WebSocket connection and the model returns partial and final transcripts as it listens. "Realtime" describes the outcome—results arrive fast enough to act on while the conversation is still happening. AssemblyAI's streaming API uses wss://streaming.assemblyai.com/v3/ws.

How low does latency need to be for a good voice experience?

For a natural back-and-forth, you want transcripts arriving in a few hundred milliseconds and turn detection firing quickly enough that the system doesn't talk over people or leave dead air. Universal-3.5 Pro Realtime returns partials in a few hundred ms, and its end-of-turn detection reads tonality and pacing to fire in about 300ms rather than waiting on a silence timer. End to end, a well-built voice agent lands around one second.

Do you sacrifice accuracy by going realtime?

Much less than you used to. On Pipecat's open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime posted a 6.99% pooled word error rate—ahead of Deepgram Flux (15.58%), ElevenLabs Scribe v2 (9.76%), and Google Chirp3 (9.04%). Passing agent context cut WER by a further 10.2% across 20,000 files. Batch can still edge ahead on the hardest audio because it sees the whole file, but the gap has narrowed dramatically.

What are the main realtime speech-to-text use cases?

Voice agents, live agent assist for contact center reps, real-time conversation intelligence, live captioning, and ambient clinical scribes. The common thread: another system consumes the transcript instantly to make a decision, so post-call transcription wouldn't work.

How do I get started with realtime speech-to-text?

Open a WebSocket to the streaming endpoint and start sending audio—you'll get partial and final transcripts back right away. The streaming speech-to-text product page has the details.

AI Medical Scribe: Build vs Buy Nuance DAX & Abridge

Mart Schweiger — Wed, 22 Jul 2026 14:55:23 +0000

Every healthcare AI team lands on the same question eventually. You want ambient clinical documentation in your product, and there are two obvious paths. Buy a finished scribe like Nuance DAX Copilot or Abridge and wire it in. Or build your own.

Here's the reframe most comparison posts miss. DAX and Abridge are finished products. They're good ones. If you're a health system that wants a scribe in clinicians' hands next quarter and you're never going to differentiate on documentation itself, buying is often the right call. Full stop.

But if you're a product team building a clinical application, a specialty EHR, or an ambient scribe you plan to sell, the "which scribe do I buy" question is the wrong one. You're not shopping for a scribe. You're deciding whether documentation is a feature you own or a feature you rent. And if you build, the question becomes: what do you build on?

That's the fork. Let's walk it honestly.

The build-vs-buy fork for ambient clinical documentation

When you buy DAX or Abridge, you get a lot the same day: a working scribe, a note format clinicians already recognize, EHR integrations, a compliance posture, and a vendor who owns the roadmap. That's real value, and pretending otherwise is a mistake.

What you give up is control. The note format is theirs. The specialties they support are theirs. The pace of improvement is theirs. And the economics are per-seat, per-provider, forever—which means your gross margin on documentation is capped by someone else's price list.

When you build, you flip every one of those. You own the note structure, the specialty coverage, the UX, the model choices underneath, and the margin. You also own the work: accurate medical transcription, speaker separation, note generation, and compliance. That's not a weekend project. But none of it is mysterious anymore, and most of the hard part isn't where teams expect.

What "buy" actually costs you

The sticker price is the easy part. The real costs show up later.

Margin. Per-seat pricing scales with your clinician count, not your efficiency. Get 10,000 providers on the platform and your documentation cost grows linearly right alongside. You can't engineer that number down because you don't own the stack.

Control over the note. Cardiology notes and behavioral health notes shouldn't look the same. If your differentiation is a note tuned to a specialty or a workflow, a general-purpose scribe fights you on it. You file feature requests and wait.

Lock-in. Once your workflows, integrations, and clinician muscle memory are built around a vendor's output format, switching is a migration project, not a config change. That's leverage—theirs, not yours.

Differentiation. If you and three competitors all ship the same underlying scribe, documentation stops being a reason anyone chooses you. It becomes table stakes you're paying a premium to rent.

None of this makes buying wrong. It makes buying a strategic decision about where your product's value actually lives. If documentation isn't your edge, rent it and move on. If it is—or could be—renting your core is a strange way to build a company.

What "build" actually requires

An ambient scribe is a pipeline, and it's shorter than it looks. Four layers.

Medical speech-to-text. The audio-to-text layer. Multi-speaker clinical audio, drug names, procedures, dosages, accents, background noise. This is the input to everything downstream.
Speaker diarization. Who said what. You need clinician and patient separated cleanly, including the rapid back-and-forth and interruptions of a real visit.
Note generation. An LLM turns the diarized transcript into a structured note—SOAP, H&P, whatever your specialty needs. This is where teams love to spend their time.
Compliance. A BAA, PHI handling, access controls, audit trails.

Here's the part that surprises people. The LLM layer, the part everyone's excited about, is the most commoditized. Strong models are a few API calls away, and prompt engineering a good SOAP note is tractable. The layer that quietly decides whether your scribe is trustworthy is the first one—the speech-to-text foundation. Garbage transcript in, confident-sounding garbage note out. An LLM will happily summarize a hallucinated dosage into a clean, plausible sentence a clinician might sign.

So the build decision is really a decision about your transcription layer. Get that right and the rest of the pipeline has something solid to stand on.

The speech-to-text layer is make-or-break

This is the input that determines everything, so it's worth being specific about what "good enough for clinical" means.

Medical accuracy that isn't generic. General speech-to-text mangles clinical vocabulary—that's the whole problem. AssemblyAI's Universal-3.5 Pro is the flagship async model at $0.21/hr, and you turn on medical accuracy by adding one parameter: "domain": "medical-v1". That's Medical Mode, +$0.15/hr, so the flagship plus Medical Mode runs $0.36/hr combined. It reduces the Missed Entity Rate on drugs, conditions, procedures, and clinical terms by roughly 20%. No model switch, no separate pipeline—one param on the model you're already calling.

Context that adapts to the patient. Medical Mode gets you baseline clinical accuracy. Contextual prompting gets you the last mile. Prime the model with a patient's prior-visit note and it knows what to listen for. In an internal healthcare test, feeding a prior-visit note cut missed medical terms by 31%—even when the note was from an earlier visit. That's the kind of gain you can't buy off a shelf because it depends on data only your application has.

Diarization built in. Universal-3.5 Pro produces the transcript and the speaker turns together—the most accurate diarization we've shipped, optimized for cpWER and tuned for the short turns, rapid exchanges, and overlapped speech of an actual exam room. You're not bolting on a separate speaker model and hoping the timestamps line up.

A streaming option when you need it. Most ambient scribes are async—record the visit, generate the note. But if you're building live documentation, a real-time assistant, or clinician-facing prompts mid-visit, Universal-3.5 Pro Realtime offers Medical Mode too, with live diarization. Same foundation, real-time shape. That flexibility matters when your roadmap outgrows batch processing.

Languages. Medical Mode supports English, Spanish, German, and French—so a Spanish-language visit gets clinical-grade transcription, not a downgrade.

Compare that to buying: with DAX or Abridge, the transcription layer is a black box. You can't tune it, you can't prompt it with your data, and you can't fix it when it's wrong on a term that matters to your specialty. Building on a foundation you control means the accuracy ceiling is yours to raise. If you want the deeper argument for why this input dominates outcomes, we made it in the true cost of inaccurate transcription and in why transcription accuracy and transcription quality aren't the same thing.

Compliance, done right

Compliance is where "build" teams get nervous, and fair enough—PHI is not the place to improvise. The good news is that the transcription layer of your stack can carry a clean posture.

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI.

Practically, that BAA is available to sign without booking a sales call—so a compliance review doesn't turn into a procurement saga before you've written a line of code. For teams with data-residency requirements, EU processing is available too. Compliance is table stakes, but it shouldn't be the thing that slows your build to a crawl.

Build vs buy: the decision at a glance

Here's the comparison that actually matters—not a spec smackdown, but the strategic trade-offs.

Decision criterion	Buy (DAX / Abridge)	Build (on AssemblyAI)
Control over note format & UX	Vendor-defined; feature requests	Fully yours — tune per specialty and workflow
Gross margin on documentation	Capped by per-seat pricing	Yours to engineer; usage-based inputs
Time-to-market	Fastest — finished product	Longer, but the STT layer is an API call
Differentiation	Shared with everyone on the same scribe	Documentation becomes a product edge
Compliance ownership	Vendor's posture	Your posture; BAA available on the STT layer
Cost model	Per-seat, per-provider, recurring	Pay-as-you-go on transcription ($0.36/hr with Medical Mode)
Roadmap & pace of improvement	Vendor's priorities	Yours
Best fit	Orgs that won't differentiate on docs	Product teams selling or owning the experience

Who each path is for

Buy DAX or Abridge if: you're a health system or clinic that needs a working scribe in clinicians' hands fast, documentation isn't part of your competitive story, and you'd rather pay per seat than staff an engineering team against it. That's a legitimate, common, correct choice. These are mature products for exactly this buyer.

Build if: documentation is part of what you sell, you need note formats or specialty coverage a general scribe won't give you, per-seat economics break your margin at scale, or you're already a software company and owning the stack is how you compete. In that case, the smart move isn't building transcription from scratch—it's building on a transcription layer that's already solved medical accuracy, diarization, and compliance, so your team spends its time on the note logic and product experience that actually differentiate you.

Healthcare teams building their own documentation experiences tend to land in the second camp for the same reason: they want the margin and the control, and they don't want to reinvent speech-to-text to get there.

A quick decision checklist

Run your project through these before committing:

Is documentation a feature we sell, or a chore we want handled? If we sell it, lean build.
Do we need note formats or specialties a general scribe won't support? If yes, lean build.
What does per-seat pricing do to our margin at 1,000 and 10,000 providers? If it breaks, lean build.
Do we have (or want) engineering ownership of this workflow? If not, lean buy.
How fast do we need to ship? If it's this quarter and docs aren't our edge, lean buy.
If we build, is our transcription layer clinically accurate, diarized, and BAA-backed on day one? If not, fix that first—it's the foundation.

If you're weighing the broader landscape, our guides on Voice AI for healthcare market research and conversation intelligence show how the same speech foundation extends well past the exam room. For the numbers behind the models, the benchmarks and our take on how accurate speech-to-text is in 2026 are the receipts.

Explore Voice AI solutions to see the full platform under a scribe you own—or dig into the medical solution directly.

The insight most teams learn too late

Here's the thing nobody tells you at the start. Build vs buy isn't a one-time decision—it's a decision that gets more expensive to reverse the longer you wait. The team that buys "to move fast" and plans to build later usually can't—by then their workflows, integrations, and clinician habits are welded to a vendor's format. And the team that builds on a weak transcription layer finds the ceiling only after shipping, when clinicians stop trusting the notes.

So the real move is to decouple the decision. Even if you buy a finished scribe today, own the layer that's hardest to swap later—your speech-to-text foundation—so building stays a live option instead of a fantasy. Accuracy compounds. Lock-in compounds. Pick the one you want on your side.

Frequently asked questions

Should I build my own AI medical scribe or buy Nuance DAX or Abridge?

Buy if you need a finished scribe fast and documentation isn't your competitive edge—DAX and Abridge are mature products built for that. Build if documentation is part of what you sell, you need custom note formats or specialty coverage, per-seat pricing breaks your margin at scale, or you want to own the experience. If you build, don't build transcription from scratch—build on a medical-grade speech-to-text layer and spend your effort on note logic and UX.

Can I use AssemblyAI with PHI under HIPAA, and is a BAA available?

How accurate is AssemblyAI on medical terminology?

Universal-3.5 Pro with Medical Mode ("domain": "medical-v1") reduces the Missed Entity Rate on drugs, conditions, procedures, and clinical terms by roughly 20%. Contextual prompting goes further—feeding a patient's prior-visit note cut missed medical terms by 31% in an internal healthcare test, even when the note came from an earlier visit.

What does it cost to build a scribe on AssemblyAI?

Transcription is pay-as-you-go, billed per second with no minimums. Universal-3.5 Pro async is $0.21/hr, and Medical Mode adds $0.15/hr—so the flagship plus Medical Mode is $0.36/hr combined. That's a usage-based input cost you control, versus the recurring per-seat pricing of a finished scribe.

Which languages does Medical Mode support?

English, Spanish, German, and French. Medical Mode is available on Universal-3.5 Pro (async) and on Universal-3.5 Pro Realtime for streaming use cases.

How do I get started?

Add "domain": "medical-v1" to a Universal-3.5 Pro request and run it against your own clinical audio. Sign the BAA when you're ready to process PHI, and check the medical solution page for the full picture.

Best Voice Agent API for Startups: A First-Build Guide

Mart Schweiger — Wed, 22 Jul 2026 14:54:57 +0000

You've decided to build a voice product. Maybe it's a support agent, maybe an AI companion, maybe a voice layer on the app you already shipped. Either way, you're staring at a blank editor and a very fast-moving landscape, and the question in front of you isn't "which model is most accurate." It's "how do I get a working demo by Friday?"

If that's you, this guide is for you. Most people building their first voice product aren't voice agent power users. Roughly 80% are net-new to this. You're demo-driven, you're code-forward, and you don't want to spend your first week learning someone's proprietary conversation-design UI. So let's skip the theory and talk about the decision you're actually making.

What a voice agent API bundles, and why the pipeline is the hard part

A voice agent is a loop. Someone talks, your system transcribes it (speech-to-text), an LLM decides what to say, and a text-to-speech engine says it back. STT to LLM to TTS. That's the whole pipeline.

Sounds simple. It isn't.

The hard part isn't any single box in that diagram. It's the seams between them. When does the user stop talking? (Harder than it sounds - people pause mid-thought all the time.) What happens when they interrupt the agent? How do you keep latency low enough that the conversation doesn't feel like a walkie-talkie? And here's the one that quietly sinks most first builds: if the transcription is wrong, the LLM responds to the wrong thing. Garbage in, confident garbage out.

That last point matters more than founders expect. Your LLM prompt can be perfect, your TTS voice can sound gorgeous, but if the STT hears "Karen" as "Aaron" or drops the last four digits of an account number, your agent is now confidently wrong. Accuracy at the input is the foundation everything else sits on. If you want the deeper version of this argument, we wrote about what actually makes a good voice agent - the short version is that the listening layer is where good agents are won or lost.

A voice agent API bundles that whole pipeline - plus turn detection, interruption handling, and voice activity detection - so you don't have to hand-build the seams. That's the category. The question is how you get one.

The three paths for a startup

There are basically three ways to ship your first voice product. Each has a real trade-off, and the "right" one depends less on your budget than on how much of the plumbing you want to own.

Path	What you own	Trade-off	Best for
DIY multi-vendor	Separate STT, LLM, and TTS providers wired together yourself	Maximum control, but three vendors, three invoices, and you build turn detection, barge-in, and streaming orchestration	Teams whose differentiator is the pipeline itself
Platform (Retell, Vapi, etc.)	A managed environment, often no-code, with built-in telephony	Fast to launch, but opinionated conversation design, awkward custom integrations, and a real ceiling	No-code builders and telephony-first products
One API	Your system prompt, tools, and conversation logic — over a single connection	More control than a platform, far less overhead than DIY; you write code	Code-forward startups shipping a first voice product

Path 1: DIY multi-vendor. You pick an STT provider, an LLM provider, and a TTS provider, and you wire them together yourself. Maximum control. But you're now managing three vendors, three invoices, three sets of rate limits, and three places to debug when a call goes sideways. You also own the hard middle - turn detection, barge-in, streaming orchestration. For a first product, that's a lot of undifferentiated engineering before you've validated that anyone wants the thing.

Path 2: A platform (Retell, Vapi, and similar). These give you a managed environment, often with a no-code builder and built-in telephony. Great for getting something live fast without much code. The catch is the ceiling. Platforms are opinionated about conversation design, custom integrations can get awkward, and - candidly - a lot of platform-built agents end up sounding the same because they're built on the same rails. Vapi in particular is orchestration middleware, which means your agent is only as good as the weakest third-party link in its chain. We wrote more about where voice agent stacks start showing their limits if you want the honest version.

Path 3: One API. A single API that handles STT to LLM to TTS through one connection. You still write your own system prompt, define your own tools, and control the conversation - but you don't manage three vendors or build the streaming plumbing. It's the middle path: more control than a platform, far less overhead than DIY.

For a first voice product, path 3 is usually the sweet spot. You get to spend your time on product logic - the thing that makes your agent yours - instead of on infrastructure nobody will ever thank you for.

What to actually evaluate

Before you pick, here's the short list of things that genuinely move the needle for a first build. Ignore the spec-sheet noise; these five are what you'll feel in production.

Criterion	Why it matters	AssemblyAI Voice Agent API
Accuracy	Wrong transcription means the LLM answers the wrong thing — names, numbers, and addresses break flows	Built on Universal-3.5 Pro Realtime: 6.99% pooled WER and 15.31% entity error rate on Pipecat's open STT benchmark
Latency	End-to-end round-trip is what makes a conversation feel human	Around one second end-to-end
Developer experience	Time from "read the docs" to "talking to an agent" is the whole game for a first build	Standard JSON over one WebSocket, no SDK required; readable in ~10 minutes and works natively with Claude Code
Pricing	Predictable pricing lets you model unit economics before the product exists	Flat $4.50/hr billed by the minute — STT, LLM, and TTS included; roughly 4x cheaper than OpenAI's Realtime API (~$18/hr)
Lock-in	If you outgrow the tool, how painful is the exit?	A raw API over a standard WebSocket — you keep full control of conversation design, tools, VAD, and turn timing

Speech accuracy. This is the garbage-in problem from earlier. Look at word error rate on real conversational audio, not clean read-aloud benchmarks, and pay attention to entity accuracy - names, numbers, addresses - because that's what breaks support and booking flows. If you want to go deeper on why the usual metric is shaky, word error rate is broken is a good read.

Latency. End-to-end round-trip time is what makes a conversation feel human or robotic. You want something around a second. Don't obsess over shaving milliseconds off one stage; obsess over the full loop.

Developer experience and time-to-ship. How long from "read the docs" to "talking to a working agent"? For a first-timer this is the whole ballgame. SDK sprawl, dozens of event types, and framework lock-in all add days. A clean JSON API you can read in ten minutes does not.

Pricing model. Is it flat and predictable, or are you doing token math across three invoices plus per-minute STT plus TTS characters? Unpredictable pricing is its own kind of tax when you're trying to model unit economics for a product that doesn't exist yet.

Lock-in. If you outgrow the tool, how painful is the exit? No-code platforms can be sticky in ways that hurt later. An API you call over a standard WebSocket is a lot easier to walk away from.

How AssemblyAI's Voice Agent API maps to that list

Full disclosure: this is our product. I'm going to be specific so you can check the claims, not just take my word.

The Voice Agent API is one WebSocket API that runs the entire STT to LLM to TTS pipeline. You connect to a socket, stream audio in, and get audio back. You write the system prompt. That's the mental model.

On the five criteria:

Accuracy. It's built on Universal-3.5 Pro Realtime, our flagship streaming model, which posts a 6.99% pooled word error rate on Pipecat's open STT benchmark of real agent conversations. For comparison on that same benchmark, entity error rate lands at 15.31% - names, places, phone numbers, the stuff that actually breaks agents. It also takes your agent's question as context, so a mumbled one-word reply or a spelled-out account ID resolves correctly instead of getting mangled.

Latency. Around one second end-to-end. Fast enough to feel like a conversation, not a transaction.

Developer experience. Standard JSON over a WebSocket. No SDK required, no framework to learn. You can read the full API reference in about ten minutes, and most developers have something working the same afternoon. It also works natively with Claude Code - copy the docs, paste them in, and build - which is exactly why we designed it for coding agents.

Here's roughly what connecting looks like - one socket, JSON messages, no ceremony:

const ws = new WebSocket("wss://agents.assemblyai.com/v1/ws", {
  headers: { Authorization: `Bearer ${process.env.ASSEMBLYAI_API_KEY}` }
});

ws.on("open", () => {
  ws.send(JSON.stringify({ type: "session.update", session: {
    system_prompt: "You are a friendly booking assistant for a dental clinic.",
    // stream your audio frames in from here
  } }));
});

You can also update the system prompt, tools, and settings mid-conversation without reconnecting - handy when the flow branches.

Pricing. Flat $4.50/hr, billed by the minute, with STT, LLM, and TTS all included. One number, one bill, one set of logs. No reconciling three invoices. For context, that's roughly 4x cheaper than OpenAI's Realtime API (around $18/hr) and comes with a cleaner developer experience - their API has 30-plus event types to reason about.

Lock-in. It's a raw API over a standard WebSocket. We call it "invisible infrastructure" for a reason - your customers should feel like you built the thing from scratch, and you keep full control over conversation design, tools, VAD, and turn timing. We're not the agent. We're what you build the agent on.

It currently supports six languages - English, Spanish, French, German, Italian, and Portuguese - and ships drop-in plugins for LiveKit and Pipecat if you're already using those. When you're ready to go from "hello world" to something real, how to build with the Voice Agent API walks the full pipeline.

When a platform is the better call

I said this would be an honest guide, so here's the honest part. A one-API approach is not always right, and sometimes a platform genuinely is the better choice.

Reach for a platform like Retell or Vapi if:

You want no-code. If you're a non-technical founder or you need business users to build and tweak agents without touching a WebSocket, a visual builder is worth a lot. Our API assumes you can write code.
You need built-in telephony today. If your product is fundamentally about placing and receiving phone calls and you want that handled out of the box right now, platforms have mature telephony baked in. We're shipping telephony as a fast-follow, but "today" matters when you're shipping.
Call routing and hosted orchestration are the product. If the routing, queueing, and call-center-style workflow is your value, a platform gives you that scaffolding for free.

There's no shame in starting on a platform to validate demand and moving to an API when you hit the ceiling. Just go in knowing where that ceiling is.

And if the real question is DIY versus one API? Unless stitching STT, LLM, and TTS together is itself your differentiator, one API replaces three vendors, three invoices, and three debugging surfaces. For a first product, that's almost always the right trade.

Quickstart and a live demo to close

Here's the thing about voice products: reading about them tells you very little. Talking to one tells you everything. Accuracy, latency, turn-taking, the feel of an interruption landing cleanly - you catch all of it in about thirty seconds of conversation.

So the move is simple. Grab a key, wire up the socket, point it at a system prompt, and talk to it. If you can talk to it and it feels right, you're most of the way to a demo.

The best voice agent API for your first product is the one that gets you to that working demo fastest without boxing you in later. For most code-forward startups, that's a single API with strong accuracy at the input, predictable pricing, and an exit that isn't a trap.

Frequently asked questions

What's the best voice agent API for a startup building its first voice product?

The best one is whatever gets you to a working demo fastest without locking you in. For code-forward startups, that usually means a single API that bundles STT, LLM, and TTS over one connection - so you write product logic instead of managing three vendors. AssemblyAI's Voice Agent API fits that profile: one WebSocket, standard JSON, flat pricing, and around one-second end-to-end latency.

What's the cheapest voice agent API?

Watch out for pricing that looks cheap per stage but adds up across three invoices (STT per minute, LLM per token, TTS per character). A flat rate is easier to model. AssemblyAI's Voice Agent API is $4.50/hr all-in - STT, LLM, and TTS included - which is roughly 4x cheaper than OpenAI's Realtime API at around $18/hr. Cheapest on paper isn't always cheapest in practice; predictability matters when you're modeling unit economics.

One API or a platform like Retell or Vapi - which should I choose?

Choose a platform if you want no-code building, need telephony out of the box today, or if hosted call routing is your actual product. Choose a single API if you want full control over conversation design and integrations without hitting a platform ceiling, and you're comfortable writing code. Platforms trade complexity for constraints; an API keeps you in control but expects you to build.

How good does the developer experience really need to be?

For a first voice product, it's the whole game. The gap between an API you can read in ten minutes and one with dozens of event types and required SDKs is measured in days of your life. AssemblyAI's Voice Agent API uses standard JSON over a WebSocket with no SDK required, works natively with Claude Code, and lets you update prompts and tools mid-conversation without reconnecting.

Which languages does the Voice Agent API support?

Six today: English, Spanish, French, German, Italian, and Portuguese. If you need broader coverage for pre-recorded workflows, Universal-3.5 Pro handles native code-switching across 18 languages on the async side.

How do I get started?

Open the API reference, connect to the WebSocket, and stream audio at a system prompt. Most developers have a working agent the same afternoon.

Speech-to-Text API Fundamentals: Auth, Polling & JSON

Mart Schweiger — Wed, 22 Jul 2026 14:54:50 +0000

Every integration with a speech-to-text API comes down to three questions. How do I authenticate? How do I get results out of an async job? And what does the response actually contain? Get those right and everything else — speaker labels, PII redaction, language detection — is just another request option.

This is the canonical walkthrough. We'll go end-to-end: your first authenticated request, the status lifecycle of a transcription job, a correct polling loop, and a field-by-field tour of the JSON that comes back. Code in curl, Python, and JavaScript throughout.

If you're new to the space, What is speech-to-text? and What is Automatic Speech Recognition (ASR)? give you the conceptual grounding. Here, we're shipping a first successful call.

Overview

AssemblyAI's REST API lives at https://api.assemblyai.com. You submit audio, the API transcribes it asynchronously, and you read the result as JSON. There's no streaming to manage and no SDK required for the core flow — though we ship official SDKs that make it a two-liner.

The whole loop:

Authenticate — send your API key in the authorization header.
Submit — POST /v2/transcript with an audio_url. You get back an id and a status.
Poll — GET /v2/transcript/{id} every few seconds until status is completed or error.
Parse — read text, confidence, words[], and friends from the JSON.

Let's build it.

Prerequisites

An AssemblyAI API key. Get your free API key from the dashboard.
A publicly reachable audio file URL (or one you upload — that's a separate endpoint; here we'll use a URL).
Optionally, one of the official SDKs:
Python: pip install assemblyai
JavaScript/TypeScript: npm install assemblyai

Set your key as an environment variable so it never lands in source control:

export ASSEMBLYAI_API_KEY="your_api_key_here"

Every example below reads the key from ASSEMBLYAI_API_KEY.

Quick start

The SDKs collapse submit-and-poll into a single call. Here's the entire flow in each.

import os, assemblyai as aai

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

transcript = aai.Transcriber().transcribe(
    "https://assembly.ai/wildfires.mp3"
)

print(transcript.text)

import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const transcript = await client.transcripts.transcribe({
  audio: "https://assembly.ai/wildfires.mp3",
});

console.log(transcript.text);

That's the destination. Now let's take apart what's happening underneath, because understanding it is what lets you debug, add features, and use webhooks later.

1. Authentication

AssemblyAI authenticates with a single header on every request:

authorization: <YOUR_API_KEY>

Two things trip people up here, so let's be explicit. There's no Bearer prefix — the header value is your raw key, nothing else. And the header goes on every request, both the POST that submits a job and the GETs that poll it.

Here's a first authenticated request that lists your account's transcripts:

curl https://api.assemblyai.com/v2/transcript \
  --header "authorization: $ASSEMBLYAI_API_KEY"

If your key is valid, you get a 200 and a JSON body. If it's missing or wrong, you get a 401. That's your authentication smoke test before you send any audio.

Never hardcode the key. Read it from the environment, a secrets manager, or your platform's config store.

2. Submitting a job and the status lifecycle

Transcription is asynchronous. You hand the API an audio URL, it returns immediately with a job id, and the work happens in the background. You don't get the transcript in the POST response — you get a receipt.

Submit with POST /v2/transcript:

curl https://api.assemblyai.com/v2/transcript \
  --header "authorization: $ASSEMBLYAI_API_KEY" \
  --header "content-type: application/json" \
  --data '{"audio_url": "https://assembly.ai/wildfires.mp3"}'

The response is a transcript object. The two fields that matter right now:

{
  "id": "106993b6-...",
  "status": "queued"
}

Persist that id. It's how you retrieve the result, and it's how the transcript is addressed for the rest of its life.

The status lifecycle

A job moves through a small, fixed set of status values. These are the exact strings the API returns:

queued — waiting to start. In practice you'll typically only see this when you're being rate-limited; otherwise jobs go straight to processing.
processing — the model is transcribing the audio.
completed — success. The text field and all the other output fields are now populated.
error — the job failed. The object carries an error string explaining why.

completed and error are the two terminal states. Your job is to poll until you hit one of them. Everything about handling a transcription job is really just handling this state machine correctly.

3. Polling correctly

Since the result isn't in the POST response, you fetch it with GET /v2/transcript/{id} and check the status. Loop, sleep, repeat until terminal.

The convention: poll every 3 seconds, break on completed (read text), break on error (read error). Three seconds is a sane interval — frequent enough to feel responsive, relaxed enough that you're not hammering the endpoint.

curl

TRANSCRIPT_ID="106993b6-..."

while true; do
  response=$(curl --silent https://api.assemblyai.com/v2/transcript/$TRANSCRIPT_ID \
    --header "authorization: $ASSEMBLYAI_API_KEY")
  status=$(echo "$response" | jq -r '.status')

  if [ "$status" = "completed" ]; then
    echo "$response" | jq -r '.text'
    break
  elif [ "$status" = "error" ]; then
    echo "$response" | jq -r '.error'
    break
  fi

  sleep 3
done

Python

import os
import time
import requests

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
BASE_URL = "https://api.assemblyai.com"
headers = {"authorization": API_KEY}

# Submit the job
resp = requests.post(
    f"{BASE_URL}/v2/transcript",
    json={"audio_url": "https://assembly.ai/wildfires.mp3"},
    headers=headers,
)
resp.raise_for_status()
transcript_id = resp.json()["id"]
print(f"Submitted: {transcript_id}")

# Poll until terminal
polling_url = f"{BASE_URL}/v2/transcript/{transcript_id}"
while True:
    transcript = requests.get(polling_url, headers=headers).json()
    status = transcript["status"]

    if status == "completed":
        print(transcript["text"])
        break
    elif status == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")

    time.sleep(3)

JavaScript

const API_KEY = process.env.ASSEMBLYAI_API_KEY;
const BASE_URL = "https://api.assemblyai.com";
const headers = { authorization: API_KEY };

// Submit the job
const submit = await fetch(`${BASE_URL}/v2/transcript`, {
  method: "POST",
  headers: { ...headers, "content-type": "application/json" },
  body: JSON.stringify({ audio_url: "https://assembly.ai/wildfires.mp3" }),
});
const { id } = await submit.json();
console.log(`Submitted: ${id}`);

// Poll until terminal
const pollingUrl = `${BASE_URL}/v2/transcript/${id}`;
while (true) {
  const transcript = await (await fetch(pollingUrl, { headers })).json();

  if (transcript.status === "completed") {
    console.log(transcript.text);
    break;
  } else if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  }

  await new Promise((r) => setTimeout(r, 3000));
}

Three implementations, one pattern: submit, grab the id, poll on a 3-second interval, branch on the terminal states. When you're ready to skip the loop entirely, webhooks push the result to you instead — more on that below.

For the full endpoint contracts, see Transcribe an audio file and Check transcript status.

4. Parsing the JSON response

Once status is completed, the transcript object is fully populated. Here's the canonical shape, trimmed to the essentials:

{
  "id": "106993b6-...",
  "status": "completed",
  "text": "Smoke from hundreds of wildfires...",
  "language_code": "en",
  "audio_duration": 282,
  "confidence": 0.95,
  "words": [
    { "text": "Smoke", "start": 100, "end": 640, "confidence": 0.9, "speaker": null }
  ]
}

Let's walk the fields you'll actually use.

id — the job identifier. Persist it; you can GET the transcript again anytime.
status — where the job is in its lifecycle. On a completed job, completed.
text — the full transcript as one string. null until the job completes, so always gate on status first.
confidence — an overall confidence score as a float from 0.0 to 1.0. In the example, 0.95.
audio_duration — the length of the audio in seconds (an integer). 282 here means the file is 282 seconds long. This is what you meter usage against.
language_code — the detected or specified language, e.g. "en". When you use language detection, you'll also get a language_confidence between 0 and 1.
words[] — every word as its own object: text, start, end, confidence, and speaker. The start and end timestamps are in milliseconds. So start: 100, end: 640 means that word runs from 0.1s to 0.64s. This is what you build captions, search, and clip-to-timestamp features on.

Watch the units — this is the single most common parsing bug. audio_duration is in seconds; word timestamps are in milliseconds. Mix them up and your captions drift by three orders of magnitude.

Utterances and speakers

When you enable speaker_labels or multichannel, the response gains an utterances[] array. Each utterance groups contiguous speech by a single speaker:

{
  "utterances": [
    {
      "speaker": "A",
      "text": "Smoke from hundreds of wildfires...",
      "confidence": 0.94,
      "start": 100,
      "end": 4200,
      "words": [ { "text": "Smoke", "start": 100, "end": 640, "confidence": 0.9, "speaker": "A" } ]
    }
  ]
}

Same millisecond timestamps, plus a speaker label and a nested words[]. If you didn't request speaker labels or multichannel, utterances won't be present — so check before you iterate.

5. Error handling

Two failure modes, two different responses. Handle them differently.

HTTP-level errors show up as status codes on the request itself. A 400 means the request was malformed — bad JSON, a missing required field, an invalid option. Fix the request; retrying the same payload won't help. A 401 means your authorization header is missing or wrong. Server-side (5xx) errors can be transient — those are safe to resubmit.

Job-level errors show up as status: "error" on a job you already submitted successfully. The request was accepted (200), but transcription failed. Read the error field for the reason:

{
  "id": "106993b6-...",
  "status": "error",
  "error": "Download error, unable to download the audio file"
}

Common causes of an error status: an unsupported audio format, missing audio, or a URL the API couldn't reach. These are usually input problems — check that your audio_url is public and points at valid audio before resubmitting.

A resilient loop distinguishes the two:

if status == "error":
    # Job-level failure — inspect .error, likely a bad input
    raise RuntimeError(transcript["error"])

Wrap the POST in retry-with-backoff for 5xxs, and never blind-retry a 400.

Next steps

You've got the fundamentals. The same submit-poll-parse flow carries every feature — you just add request options to the POST body:

speaker_labels — who spoke when, via the utterances[] array.
language_detection and language_code — auto-detect or pin a language.
punctuate and format_text — control formatting of the output text.
multichannel — transcribe each audio channel separately.
redact_pii and filter_profanity — clean the transcript for compliance.
webhook_url — skip polling entirely.

That last one is the natural upgrade. Instead of a GET loop, set webhook_url on submission and AssemblyAI POSTs the finished result to your endpoint when the job hits a terminal state. It's the right pattern for production and for high volume. The API reference covers the webhook payload and every option in detail.

From here, explore the full speech-to-text product and the deep dive on how accurate speech-to-text is in 2026.

View the full API reference → assemblyai.com/docs/api-reference/overview

Frequently asked questions

How do I authenticate with the AssemblyAI API?

Send your API key in the authorization header on every request — authorization: . There's no Bearer prefix; the value is your raw key. Store it in an environment variable like ASSEMBLYAI_API_KEY and get your key from the dashboard.

Do I poll for results or get them back immediately?

Transcription is asynchronous, so you poll. POST /v2/transcript returns an id and a status immediately, but not the transcript. Fetch the result with GET /v2/transcript/{id} every 3 seconds until status is completed or error. To avoid polling entirely, set webhook_url on submission and AssemblyAI pushes the result to you.

What do the transcript status values mean?

There are four: queued (waiting — usually only when rate-limited), processing (actively transcribing), completed (success, read text), and error (failed, read the error string). completed and error are the terminal states you poll toward.

How do I read confidence scores and word timestamps?

confidence is an overall float from 0.0 to 1.0. Each object in words[] has its own confidence plus start and end timestamps in milliseconds. Note that audio_duration is in seconds while word timestamps are in milliseconds — don't mix the units.

Why is my transcript status error?

The job was accepted but transcription failed. Read the error field for the reason. Common causes are an unsupported audio format, missing audio, or a URL the API couldn't reach. That's different from a 400, which means the request itself was malformed and shouldn't be blindly retried; server (5xx) errors, by contrast, can be resubmitted.

What can I do beyond a basic transcript?

Add request options to the POST body — speaker_labels, language_detection, punctuate, format_text, multichannel, redact_pii, filter_profanity, and webhook_url. The submit-poll-parse flow stays identical. See Submit a transcript and Get a transcript for full field references.

Fast ASR for Voice Agents: Bring Your Own Turn Detection

Mart Schweiger — Wed, 15 Jul 2026 15:48:04 +0000

There's a school of voice-agent development that treats turn detection as something you buy, not something you build. Pick a streaming STT provider, let its end-of-turn logic decide when the user is done, and move on. For a lot of teams that's the right move — and if you're weighing the options, our breakdown of turn detection vs forced endpoints is the place to start.

But some teams have already solved turn detection. They've tuned their own voice-activity detection over thousands of calls, they know their audio, and they trust their endpointing more than any default. For those teams, a streaming model's built-in turn logic isn't a feature — it's something to work around. What they want is narrower and faster: hand over a finished chunk of speech, get accurate text back, get out of the way.

That's the case for bringing your own turn detection and pairing it with fast ASR over HTTP.

Turn detection is an architectural decision, not a default

Here's the framing that matters. In a streaming setup, the STT model is a participant in the conversation — it's watching the audio and deciding, continuously, whether the user has finished. That's genuinely useful when you want the provider to own that judgment.

But it means the model is inserting its own decision between "user stopped talking" and "you get the transcript." If you already know the turn is over — because your VAD just fired — you don't want the model deliberating. You want it transcribing. Every millisecond the STT layer spends re-deciding a question you've already answered is latency you're adding for no benefit.

So the decision isn't "which provider has the best turn detection." For these teams it's "who owns the turn boundary?" If the answer is you, then the ideal STT layer is one that does exactly one thing: turn a finished clip into accurate text, fast.

Built-in vs. bring-your-own

Built-in (streaming). The model reads tonality, pacing, and rhythm to detect end-of-turn — with Universal-3.5 Pro Realtime, around 300ms — then commits the transcript. Great when you want conversation-aware endpointing handled for you, and when you value partial transcripts as the user speaks.

Bring-your-own (sync HTTP). Your VAD decides the turn is over and hands the completed clip to the Sync API, which returns the transcript in a single response at roughly 134ms p50, for clips up to 2 minutes (40 MB max), at $0.45/hr. No end-of-turn step on the model's side, because you already made that call. You trade mid-utterance streaming for a tighter, you-controlled loop.

The distinction isn't fast versus slow — both are fast. It's about who decides when the turn ends, and whether you want that decision inside or outside the STT layer.

The sub-500ms budget, with you owning the turn

Teams building responsive agents often target something under 500ms from end-of-speech to the start of the agent's reasoning. Here's roughly where the time goes when you own turn detection and use sync ASR:

Your VAD end-of-turn decision: however tight you've tuned it — often the biggest lever you control.
Sync transcription: ~134ms p50 for the clip.
Network overhead: tens of milliseconds, minimized by reusing a warm HTTP connection.

Because you're not waiting on the model to detect end-of-turn, the transcription step is close to pure processing time. That's the appeal: when you've already done the hard work of knowing when the user stopped, sync ASR adds very little on top before the LLM can start. The budget math gets a lot friendlier when the STT layer isn't also trying to be the turn detector.

One practical note that pays off here: keep the HTTP connection warm between turns. Reuse a single session so DNS, TCP, and TLS setup don't land on the critical path of every utterance. It's a small change that protects the latency you worked to win.

You don't trade accuracy for control

The reason this pattern works for serious teams is that owning turn detection doesn't cost you transcript quality. Sync returns the same accuracy as async — the same handling of names, numbers, and domain terms that you'd get from the batch path. In an agent, that's the number that matters most: if the transcript is wrong, the LLM answers the wrong question, and no amount of speed rescues the interaction.

So the tradeoff is clean. You give up the model's built-in end-of-turn detection and mid-utterance streaming. You keep full control of the conversation loop and full accuracy on the words. For a team that already trusts its VAD, that's a trade worth making.

When to hand turn detection back to us

Be honest about which camp you're in. Bring-your-own turn detection is the right pattern when you've genuinely tuned your VAD and you want the STT layer to stay out of the conversation logic. It is the wrong pattern if you're bringing your own turn detection mostly to avoid learning someone else's — in that case you're taking on the hardest part of voice UX to save yourself a config page.

If you'd rather not own endpointing, interruption handling, and the rest of the orchestration, hand it back. The Voice Agent API builds conversation-aware turn detection and barge-in directly into Universal-3.5 Pro Realtime, over one WebSocket, at a flat $4.50/hr. Bring your own turn detection when control is the goal; use the Voice Agent API when shipping fast is the goal. Both run on the same speech accuracy underneath — the difference is only how much of the loop you want to hold.

Frequently asked questions

What does "bring your own turn detection" mean for a voice agent?

Bringing your own turn detection means your application's own voice-activity detection decides when the user has finished speaking, rather than the speech-to-text model. Once your VAD fires, you hand the completed audio clip to a synchronous ASR endpoint and get the transcript back in one response. The STT layer does exactly one thing — turn a finished clip into accurate text — instead of also deliberating about when the turn ended.

When should I use my own VAD instead of a streaming model's built-in turn detection?

Use your own VAD when you've genuinely tuned it over real calls, you trust your endpointing more than any default, and you want the STT layer to stay out of the conversation logic. In that case a streaming model's built-in end-of-turn detection is redundant — it inserts a decision between "user stopped talking" and "you get the transcript" that you've already made. Stick with built-in turn detection if you value mid-utterance partial transcripts or you'd simply rather not own endpointing.

How do I build a low-latency voice agent under 500ms with sync ASR?

When you own turn detection, the sub-500ms budget breaks down into three pieces: your VAD's end-of-turn decision (the biggest lever you control), sync transcription at roughly 134ms p50, and network overhead of tens of milliseconds. Because you're not waiting on the model to detect end-of-turn, the transcription step is close to pure processing time. Keep the HTTP connection warm between turns — reuse one session so DNS, TCP, and TLS setup stay off the critical path.

Is synchronous HTTP ASR as accurate as streaming or batch transcription?

Yes. The Sync API returns the same accuracy as async transcription — the same handling of names, numbers, and domain terms you'd get from the batch path — because it runs on the same Universal-3.5 Pro model. Owning turn detection costs you the model's built-in endpointing and mid-utterance streaming, not transcript quality. In an agent that's the trade that matters, since a wrong transcript means the LLM answers the wrong question.

Should I bring my own turn detection or use AssemblyAI's Voice Agent API?

Bring your own turn detection when control is the goal — you've tuned your VAD and want to own the conversation loop, pairing it with fast sync ASR. Use the Voice Agent API when shipping fast is the goal: it builds conversation-aware turn detection and barge-in directly into Universal-3.5 Pro Realtime over one WebSocket at a flat $4.50/hr. Both run on the same speech accuracy underneath — the difference is only how much of the loop you want to hold.

What are the Sync API's limits and pricing?

The Sync API accepts clips from 80 ms up to 2 minutes (40 MB max) across 18 languages, runs on the flagship Universal-3.5 Pro model, and returns a finished transcript in a single HTTP response at roughly 134 ms p50. It's priced at $0.45/hr of audio, with keyterms prompting and conversation context included and no rate limits. For audio longer than two minutes, use the pre-recorded (async) API instead.

Sync HTTP for Voice Agents: Skip the WebSocket (2026)

Mart Schweiger — Wed, 15 Jul 2026 15:47:39 +0000

Most voice agent tutorials — including our own guide to building a chained STT-LLM-TTS architecture — assume WebSockets. You open a streaming connection to your speech-to-text provider, manage it for the life of the call, and handle a live event model. For a lot of agents, that's the right design.

But it's not the only one. Some of the most sophisticated voice-agent teams deliberately don't stream their transcription. They send audio to STT as plain HTTP requests — one request per turn — and they do it on purpose. This post is about that pattern: when a sync HTTP call beats a WebSocket, and how to build an agent around it.

Why anyone would skip streaming

Streaming feels like the obvious choice for live conversation, so the instinct is to treat "not streaming" as cutting a corner. It isn't. It's a different set of tradeoffs, and for the right team the HTTP path is the simpler, more scalable one.

Several large voice-agent teams asked us for exactly this — a sync HTTP API for real-time transcripts — for a specific reason: they already run their LLM and text-to-speech as HTTP services, and they want their whole stack to look the same. Stateless request/response services are easier to scale, easier to load-balance, and easier to reason about than long-lived stateful connections. They're willing to accept slightly higher per-turn latency to get an architecture where every component is a plain HTTP call.

The other reason is turn detection. If you already run your own voice-activity detection — deciding when the user has started and stopped talking — a streaming model's built-in end-of-turn logic is redundant. You don't want the STT layer guessing when the turn ends; you already know, because your VAD just told you. At that point a persistent socket is overhead. You have a discrete chunk of audio and you want text back. That's a request, not a stream.

How the sync HTTP pattern works

The shape is simple, and if you've built a cascaded agent it'll look familiar — just with the STT stage swapped from a socket to a call:

Your VAD detects the user has started speaking and buffers the audio.
Your VAD detects end-of-turn and closes the buffer — a clip, usually a few seconds.
You POST that clip to the Sync API and get the transcript back in the same response (~134ms p50, clips up to 2 minutes).
You hand the text to your LLM (an HTTP call), then the LLM's reply to your TTS (another HTTP call).
Repeat for the next turn.

Every stage is a stateless HTTP request. There's no connection to keep alive between turns, no streaming state to manage, no reconnection logic. The whole agent is a loop of request/response calls, which is exactly the property those teams were after. See the Sync API docs for the full request and response format.

# One turn, once your VAD has handed you a completed clip.
# Send audio as multipart/form-data with an "audio" part; model set via header.

import requests

session = requests.Session()  # reuse the connection across turns
SYNC_ENDPOINT = "https://sync.assemblyai.com/transcribe"
API_KEY = "YOUR_ASSEMBLYAI_API_KEY"

def transcribe_turn(wav_bytes: bytes) -> str:
    r = session.post(
        SYNC_ENDPOINT,
        headers={"Authorization": API_KEY, "X-AAI-Model": "u3-sync-pro"},
        files={"audio": ("clip.wav", wav_bytes, "audio/wav")},
        timeout=10,
    )
    r.raise_for_status()
    return r.json()["text"]

def handle_turn(wav_bytes):
    transcript = transcribe_turn(wav_bytes)   # STT — HTTP
    reply = run_llm(transcript)               # LLM — HTTP
    audio = synthesize(reply)                 # TTS — HTTP
    play(audio)

Reusing a single requests.Session across turns keeps the TCP/TLS connection warm, so you're not paying handshake cost on every utterance — a small detail that matters when you're chaining three HTTP calls per turn.

The tradeoffs, honestly

This pattern isn't free. Here's what you're trading.

You give up mid-utterance transcription. A streaming model can start feeding partial text to your LLM before the user finishes talking. The sync pattern waits for the complete clip, so the LLM starts a beat later. For many agents that's imperceptible; for latency-obsessed ones it's the reason to stream.

You own turn detection — which is the point, but also the work. The whole pattern assumes your VAD is good. If it isn't, you'll clip users off or leave dead air, and no fast transcription will save you. If you'd rather not own that logic, a streaming model with built-in end-of-turn detection is the better fit — and if you specifically want to keep owning it while getting fast ASR back, that's its own design worth reading up on separately.

You accept slightly higher per-turn latency for architectural simplicity. That's the deal the teams who asked for this made knowingly. If your product can absorb it, you get a stack that's dramatically easier to scale and debug.

What you keep is accuracy. The Sync API returns the same transcript quality as async, so choosing the HTTP path costs you nothing on the words themselves — which, in an agent, is the thing you least want to compromise. Get the input wrong and the LLM answers the wrong question.

When to use it — and when to reach for the Voice Agent API instead

Use the sync HTTP pattern when you already run your own orchestration and VAD, you want a uniform stateless stack, and your latency budget has room for per-turn request/response. It's a builder's pattern for teams who want maximum control over how the pieces fit together.

If you don't want to wire STT, LLM, TTS, turn detection, and interruption handling together yourself, that's a different job — and it's what the Voice Agent API is for. One WebSocket connection handles the full pipeline at a flat $4.50/hr, built on Universal-3.5 Pro Realtime, with turn detection and barge-in included. The sync HTTP pattern is for teams who want to own the orchestration; the Voice Agent API is for teams who want it handled. Both are valid — the right call depends on how much of the stack you want in your hands.

Frequently asked questions

What is the sync HTTP pattern for voice agents?

The sync HTTP pattern is a voice-agent design where each conversational turn is transcribed with a single stateless HTTP request instead of a persistent WebSocket stream. Your own voice-activity detection decides when a turn ends, you POST that completed audio clip to a synchronous speech-to-text endpoint, and the transcript comes back in the same response. Every stage of the agent — STT, LLM, TTS — becomes a plain request/response call, so there's no streaming connection to keep alive between turns.

When should a voice agent use a sync HTTP API instead of a streaming WebSocket?

Use the sync HTTP pattern when you already run your own turn detection and want your whole stack to be uniform, stateless HTTP services that are easy to scale, load-balance, and debug. It's the better fit when your latency budget can absorb waiting for a complete clip per turn in exchange for architectural simplicity. Choose streaming instead when you need mid-utterance partial transcripts or you'd rather the STT model handle end-of-turn detection for you.

Does the sync HTTP pattern add latency compared to streaming?

It adds a small amount. A streaming model can feed partial text to your LLM before the user stops talking, while the sync pattern waits for the complete clip and then returns a transcript in roughly 134 ms at p50. For many agents that difference is imperceptible; for latency-obsessed applications, streaming's mid-utterance head start is the reason to stream. The tradeoff buys you a dramatically simpler, stateless architecture.

Do I need my own voice-activity detection (VAD) to use the sync pattern?

Yes — the pattern assumes you own turn detection. Your VAD decides when the user has started and stopped talking, then hands a completed clip to the Sync API for transcription. That's the point of the pattern for teams who already run their own VAD, but it's also the work: if your turn detection is weak, you'll clip users off or leave dead air, and fast transcription won't fix that. If you'd rather not own that logic, a streaming model or the Voice Agent API with built-in turn detection is the better choice.

Should I use the sync HTTP pattern or AssemblyAI's Voice Agent API?

Use the sync HTTP pattern if you want to own the orchestration — wiring STT, LLM, TTS, turn detection, and interruption handling together yourself for maximum control. Use the Voice Agent API if you want that pipeline handled for you: one WebSocket connection at a flat $4.50/hr, built on Universal-3.5 Pro Realtime, with turn detection, barge-in, and tool calling included. Both are valid — the right call depends on how much of the stack you want in your hands.

How do I get started with the Sync API, and what are its limits?

Create a free AssemblyAI account, then POST a completed audio clip with your API key and read the transcript straight off the response — no WebSocket, no polling. The Sync API accepts clips from 80 ms up to 2 minutes (40 MB max) across 18 languages, runs on Universal-3.5 Pro, and is priced at $0.45/hr of audio with keyterms prompting and conversation context included. See the Sync API docs for the full request and response format.

Time to First Token: The Voice Agent Latency Metric

Mart Schweiger — Wed, 15 Jul 2026 15:47:32 +0000

Ask an STT vendor how fast they are and you'll get a word error rate and an average latency number. Both are real. Neither tells you whether a conversation with your voice agent will feel alive or feel broken.

Word error rate tells you how accurate the transcript is. Average latency tells you how fast the pipeline runs on average. But the thing a user actually experiences — the silence between "…and that's my question" and the agent's first sound back — isn't captured well by either. That gap is where voice agents win or lose, and the metric that governs it is time to first token.

If you're evaluating speech-to-text for a voice agent, this is the number to build your test around. Here's what it is, why the usual metrics miss it, and how to measure it on your own audio.

What time to first token actually means

Time to first token (TTFT) is the elapsed time from the moment the user stops speaking to the moment your system produces the first usable piece of output it can act on.

The phrase comes from the LLM world, where it means the delay before a model streams its first token. In a voice pipeline it's more useful to think of it end to end: the clock starts when the user finishes their turn, and stops when the first token the agent can use appears — the first chunk of finalized transcript that the LLM can start reasoning on.

That framing matters because a voice agent is a chain — speech-to-text, then an LLM, then text-to-speech — and each link has its own "first token" moment. But they're not independent. The STT stage gates everything downstream: the LLM can't produce a first token until it has text to read, and the TTS can't produce a first sound until the LLM has generated something to say. So the responsiveness the user feels is anchored on how quickly transcription commits to a usable result after the turn ends.

This is why TTFT is the metric that decides voice agents. It's the first domino.

Why WER and average latency don't tell you this

Two numbers dominate STT marketing, and both are genuinely useful for the questions they answer. They just don't answer this one.

Word error rate measures the wrong axis. WER tells you how many words the model got wrong across a whole transcript. It says nothing about when those words arrived. A model can post a beautiful WER and still feel sluggish in conversation because it waits for extra audio context before committing to a result. Accuracy and responsiveness are different axes, and a voice agent needs both. For the accuracy side of this story — and why WER itself is a slippery benchmark — see why your WER benchmark might be lying to you.

Average latency hides the moments that hurt. An average folds together the fast responses and the slow ones. But conversation is unforgiving about the slow ones — a single two-second stall in an otherwise snappy exchange is the thing the user remembers. If a vendor quotes you one latency number, ask which percentile it is. A p50 (median) and a p90 (the slow tail) can be worlds apart, and your users live in the tail as much as the middle.

End-of-turn detection is the hidden clock. Here's the subtle one. Before a streaming model can hand you a final transcript, it has to decide the user is done talking. Wait too long and you've added dead air before the LLM ever sees a word — latency the user blames on your agent. Decide too early and you cut them off. So turn detection isn't separate from latency; it's part of TTFT. A model that reads tonality, pacing, and rhythm to land an end-of-turn decision quickly is buying you responsiveness that no average-latency number will show. We go deep on this in our breakdown of turn detection vs forced endpoints.

The numbers, by transcription path

TTFT isn't a single value — it depends on how you're getting text back. The three transcription paths have very different profiles, and picking the right one for the interaction is half the battle.

Streaming (real-time). The model transcribes as audio arrives and commits final text shortly after the turn ends. With Universal-3.5 Pro Realtime, end-of-turn detection lands around 300ms by reading tonality, pacing, and rhythm rather than waiting on a fixed silence timer — so the first usable token reaches your LLM fast. This is the path built for live, back-and-forth conversation.
Sync (short-clip HTTP). For a short utterance sent as a single HTTP request, the Sync API returns the full transcript in one response at roughly 134ms p50. There's no turn-detection step to wait on because you decide when the clip ends. That makes TTFT and time-to-complete-transcript essentially the same moment — which is exactly why it's a strong fit for structured, turn-by-turn agents that own their own voice-activity detection.
Async (batch). Submit a recording, get it back in seconds to minutes. TTFT as a concept barely applies, because async isn't trying to be interactive — it optimizes for accuracy and full-file speech understanding, not first-token speed. If you're reaching for async in a live agent, you're using the wrong tool.

For the full tradeoff across these three paths — pricing, accuracy, and when to use each — see our guide to real-time vs batch transcription.

Zoom out to the whole agent and the STT number rolls up into an end-to-end budget. A well-tuned cascaded pipeline — STT, LLM, TTS — targets roughly one second from end of user speech to first agent audio, and AssemblyAI's Voice Agent API is built to that ~1s end-to-end target. Every one of those milliseconds starts with how fast transcription hands off the first token.

How to measure it yourself

Don't take anyone's quoted number — including ours — at face value. TTFT is easy to measure on your own audio, and your audio is the only benchmark that matters. A simple protocol:

Use realistic clips. Record utterances that sound like your actual users — the accents, the background noise, the domain terms, the short "yes"/"next" answers as well as the long ones. Clean read-aloud audio will flatter every vendor equally and tell you nothing.
Timestamp the two events. Mark the sample where the user stops speaking, and the timestamp when your system receives the first finalized token it can act on. The difference is your TTFT for that utterance.
Report percentiles, not an average. Collect a few hundred utterances and look at p50 and p90. The median tells you the typical experience; the tail tells you how often the agent feels stuck.
Separate the stages. Log first-token time for STT, LLM, and TTS independently. When something feels slow, you want to know which link to fix — and more often than not, the fix isn't where teams assume it is.

Do this once and the conversation with any vendor changes. You stop comparing marketing numbers and start comparing the only thing your users will ever feel.

Why this is the metric that matters

Accuracy is table stakes now — streaming models have closed most of the gap that used to make real-time feel like a compromise. What separates a voice agent that people happily talk to from one they hang up on is responsiveness, and responsiveness is governed by how fast the first token arrives after someone stops speaking.

So when you're evaluating speech-to-text for anything conversational, put TTFT at the center of the test. Ask for percentiles. Measure end-of-turn behavior, not just processing speed. And run it on audio that sounds like your users, because the felt experience of a conversation is decided in a window too small for an average to see.

Get the first token fast and everything downstream has room to work. Get it wrong and no amount of accuracy will make the agent feel alive.

Frequently asked questions

What is time to first token (TTFT) in a voice agent?

Time to first token is the elapsed time from the moment a user stops speaking to the moment the system produces the first usable piece of output it can act on. In a voice pipeline, that first token is the initial chunk of finalized transcript the LLM can start reasoning on — the domino that gates the LLM and text-to-speech stages downstream. It's the metric that most directly governs whether a conversation feels responsive.

Why isn't word error rate or average latency enough to evaluate voice-agent speed?

Word error rate measures how accurate a transcript is, not when the words arrived, so a model can post a great WER and still feel sluggish because it waits for extra context before committing. Average latency folds fast and slow responses together and hides the tail — and a single two-second stall is what a user remembers. To evaluate voice-agent responsiveness you need time to first token, reported as percentiles (p50 and p90), plus end-of-turn behavior.

What is a good time to first token, and what latency should a speech-to-text API hit for voice agents?

For live conversation, you want the first usable token within a few hundred milliseconds of the user finishing their turn. With streaming Universal-3.5 Pro Realtime, end-of-turn detection lands around 300 ms; a short clip sent to the Sync API returns a full transcript at roughly 134 ms p50. At the whole-agent level, a well-tuned STT-LLM-TTS pipeline targets about one second end-to-end from end of speech to first agent audio.

How does time to first token differ across streaming, sync, and async transcription?

Streaming transcribes as audio arrives and commits final text shortly after the turn ends (~300 ms end-of-turn with Universal-3.5 Pro Realtime), making it the path built for live conversation. Sync returns a full short-clip transcript in a single HTTP response at ~134 ms p50, with no turn-detection wait because you decide when the clip ends. Async optimizes for accuracy and full-file speech understanding over seconds to minutes, so TTFT barely applies — it's the wrong tool for a live agent.

How do I measure time to first token on my own audio?

Record clips that sound like your real users — accents, background noise, domain terms, and both short and long answers — then timestamp two events per utterance: when the user stops speaking and when your system receives the first finalized token. The difference is your TTFT. Collect a few hundred utterances, report p50 and p90 rather than an average, and log first-token time for STT, LLM, and TTS separately so you know which stage to fix.

How is turn detection related to time to first token?

Turn detection is part of TTFT, not separate from it. Before a streaming model can hand you a final transcript, it has to decide the user is done talking — wait too long and you add dead air the user blames on your agent; decide too early and you cut them off. A model that reads tonality, pacing, and rhythm to land an end-of-turn decision quickly (rather than waiting on a fixed silence timer) directly improves responsiveness that no average-latency number will reveal.

Build a Dictation App With the Sync API (Tutorial)

Mart Schweiger — Wed, 15 Jul 2026 15:47:06 +0000

Dictation feels like it should be a solved problem. You press a key, you talk, the words show up. But if you've ever tried to build it, you know the first version is almost always disappointing — you speak, then you wait, and the text lands a beat too late to feel like your own voice.

That lag is almost never the model. It's the plumbing around it.

In this tutorial we'll build a small dictation app — we're calling it Blurt — on top of AssemblyAI's Sync API. You hold a hotkey, speak a sentence or two, release, and the transcript drops straight into whatever text field you're in. Then we'll do the part most tutorials skip: cut the perceived latency roughly in half with a connection pre-warm trick, so "test, test, test" comes back in around 300ms instead of feeling like a round trip to the moon.

By the end you'll have a working app and a clear mental model for why the Sync API is the right tool for short, interactive speech — and where it isn't.

Why dictation doesn't want streaming

The instinct is to reach for streaming. Dictation is real-time-ish, streaming is real-time, case closed.

But look at what dictation actually is: short bursts of speech, one utterance at a time, with a hard start and a hard stop that the user controls by pressing and releasing a key. You don't need partial transcripts scrolling across the screen. You don't need the app guessing when you've finished talking — you told it, when you let go of the key. And you don't want to manage a persistent WebSocket, reconnection logic, and streaming state for what is, functionally, a series of two-second clips.

Streaming solves a problem dictation doesn't have (deciding when the speaker is done) and adds infrastructure dictation doesn't want (a stateful connection). What you actually want is dead simple: send a short clip over HTTP, get the text back in one response, right now.

That's the Sync API. One HTTP request, one clip up to 2 minutes, one response with the transcript — no polling, no webhooks, ~134ms p50 latency, and the same accuracy you already get from async transcription. It sits between async (submit a job, poll or wait for a webhook) and streaming (open a socket, manage a live session): the speed of real-time with the simplicity of a single request.

What we're building

Blurt is a desktop dictation utility. The flow is:

Hold a global hotkey to start recording.
Speak.
Release the key to stop.
The clip goes to the Sync API over HTTP.
The transcript is typed into the active application.

We'll build it in Python so the whole thing stays readable in one file. The same pattern ports cleanly to a browser extension, an Electron app, or a native menu-bar tool — the interesting part is the request pattern, not the UI shell.

Prerequisites: Python 3.9+, an AssemblyAI API key (grab one free), and a microphone. We'll use sounddevice to capture audio, pynput to listen for the hotkey and type the result, and requests for the API call:

pip install sounddevice pynput requests numpy

Step 1: Capture audio while a key is held

First, record audio for exactly as long as the hotkey is down. We buffer raw PCM frames and stop the moment the key comes up.

import sounddevice as sd
import numpy as np
import threading

SAMPLE_RATE = 16000  # 16 kHz mono is plenty for speech

class Recorder:
    def __init__(self):
        self._frames = []
        self._stream = None
        self._lock = threading.Lock()

    def _callback(self, indata, frames, time, status):
        with self._lock:
            self._frames.append(indata.copy())

    def start(self):
        self._frames = []
        self._stream = sd.InputStream(
            samplerate=SAMPLE_RATE,
            channels=1,
            dtype="int16",
            callback=self._callback,
        )
        self._stream.start()

    def stop(self) -> bytes:
        self._stream.stop()
        self._stream.close()
        with self._lock:
            audio = np.concatenate(self._frames, axis=0) if self._frames else
np.array([], dtype="int16")
        return audio.tobytes()

Nothing exotic here — we grab 16 kHz mono int16 audio, which is the sweet spot for speech recognition: small payloads, no meaningful accuracy loss versus higher sample rates.

Step 2: Send the clip to the Sync API

Here's the core of the app. We take the recorded bytes, wrap them as a WAV, and POST them in a single request. Because it's the Sync API, the transcript comes back in that same response — there's no transcript ID to poll. See the Sync API docs for the full request and response format.

import io
import wave
import requests

API_KEY = "YOUR_ASSEMBLYAI_API_KEY"

# Send audio as multipart/form-data with an "audio" part; model set via header.
SYNC_ENDPOINT = "https://sync.assemblyai.com/transcribe"

def pcm_to_wav(pcm_bytes: bytes, sample_rate: int = SAMPLE_RATE) -> bytes:
    buf = io.BytesIO()
    with wave.open(buf, "wb") as wf:
        wf.setnchannels(1)
        wf.setsampwidth(2)  # int16 = 2 bytes
        wf.setframerate(sample_rate)
        wf.writeframes(pcm_bytes)
    return buf.getvalue()

def transcribe_sync(pcm_bytes: bytes, session: requests.Session) -> str:
    wav_bytes = pcm_to_wav(pcm_bytes)
    response = session.post(
        SYNC_ENDPOINT,
        headers={"Authorization": API_KEY, "X-AAI-Model": "u3-sync-pro"},
        files={"audio": ("clip.wav", wav_bytes, "audio/wav")},
        timeout=10,
    )
    response.raise_for_status()
    return response.json()["text"]

One request in, one transcript out. No while transcript.status != "completed" loop, no webhook receiver to stand up. For clips up to 2 minutes, this is the whole integration.

Step 3: Wire up the hotkey and type the result

Now connect recording to the key and push the transcript into whatever app has focus:

from pynput import keyboard

recorder = Recorder()
kb_controller = keyboard.Controller()
session = requests.Session()  # reused — this matters in Step 4

HOTKEY = keyboard.Key.f9
is_recording = False

def on_press(key):
    global is_recording
    if key == HOTKEY and not is_recording:
        is_recording = True
        recorder.start()

def on_release(key):
    global is_recording
    if key == HOTKEY and is_recording:
        is_recording = False
        pcm = recorder.stop()
        text = transcribe_sync(pcm, session)
        kb_controller.type(text + " ")

with keyboard.Listener(on_press=on_press, on_release=on_release) as listener:
    print("Blurt is running. Hold F9 to dictate.")
    listener.join()

That's a working dictation app. Hold F9, say "the quick brown fox," release, and it types into your editor. But run it a few times and you'll feel it: there's a small hitch between releasing the key and seeing text. Let's kill it.

Step 4: The pre-warm trick that halves perceived latency

Here's the thing about a single HTTP request: the model isn't your only cost. Before the audio even reaches the transcription step, the client has to do DNS resolution, open a TCP connection, and complete the TLS handshake. On a cold connection that setup can rival the transcription itself — and it all happens after the user has finished speaking, which is exactly when they're waiting.

So move it earlier. The moment the user presses the hotkey — before they've said a word — fire a tiny warm-up request. That opens the connection and completes the handshake while the user is still talking. By the time they release the key and the real clip is ready, you're reusing a hot connection and paying only for transcription.

We already created a persistent requests.Session(), which reuses the underlying TCP/TLS connection across requests. We just need to warm it at key-down:

def prewarm(session: requests.Session):
    """Open the connection while the user is still speaking."""
    try:
        session.head(SYNC_ENDPOINT, timeout=2)
    except requests.RequestException:
        pass  # warm-up is best-effort; never block dictation on it

def on_press(key):
    global is_recording
    if key == HOTKEY and not is_recording:
        is_recording = True
        recorder.start()
        threading.Thread(target=prewarm, args=(session,), daemon=True).start()

In informal testing across a handful of runs, this roughly halved the end-to-end time for a short phrase — dictating "test, test, test" landed around 300ms instead of noticeably longer on a cold connection. These aren't formal benchmark numbers, but the effect is easy to feel: the transcript shows up while your finger is still lifting off the key.

The lesson generalizes. When you're optimizing an interactive speech feature, don't just look at model latency. Look at everything that happens between "user stops talking" and "text appears," and move as much of it as you can to before the user stops talking.

Step 5: Test it

Short clips first. Try one- and two-word phrases ("yes," "next slide") and a full sentence. Short utterances are where latency is most noticeable, so they're the real test.
Punctuation and casing. The Sync API returns formatted text, so "email me at sam at example dot com" should come back cleanly. Dictation lives or dies on this.
Cold vs. warm. Comment out the prewarm call and feel the difference. This is the whole point of the tutorial — make sure you can feel it too.
Clips over 2 minutes. Sync is built for short audio. If your users might hold the key for minutes, that's a signal you want async or streaming instead.

[VIDEO: embed Blurt demo screencast here]

Where to go from here

Blurt is intentionally small, but it's a real foundation. A few natural next steps:

Post-processing. Pipe the transcript through an LLM for light cleanup — fix "um"s, apply a formatting style, or translate on the fly. Keep it optional and off the hot path so latency-sensitive users aren't paying for a feature they didn't ask for.
Custom vocabulary. If your users dictate names, product terms, or medical language, prime the model so those come back right the first time.
Know when to switch tiers. Sync is the right call for short, interactive, user-triggered speech. If you find yourself wanting live partial transcripts as the user speaks, or handling clips longer than two minutes, that's your cue to look at streaming or async instead. We break down that decision in our guide to real-time vs batch transcription.

The reason dictation is worth getting right is that it's one of the few places where users feel your speech-to-text directly, with no interface in between. When the text lands the instant they stop talking, it feels like the app is reading their mind. When it lags, it feels broken. The model was never the hard part — the request pattern was, and now you have it.

Frequently asked questions

What is the best API for building a dictation app?

For dictation, a synchronous HTTP transcription API is usually the best fit because dictation is short, user-triggered speech with a clear start and stop. AssemblyAI's Sync API takes one short clip in a single HTTP request and returns a finished, formatted transcript in the same response at roughly 134 ms p50 — no WebSocket, no polling. Streaming and async APIs solve different problems (live sessions and long files, respectively), which is why they add plumbing a dictation app doesn't need.

Why use a sync HTTP API instead of streaming for dictation?

Dictation gives you a hard start and stop that the user controls with a key, so you don't need streaming's partial transcripts or its end-of-turn detection — the two things streaming is built for. A sync HTTP call avoids managing a persistent WebSocket, reconnection logic, and streaming state for what is functionally a series of two-second clips. You send the clip and get the text back in one request, which is both simpler to build and fast enough to feel instant.

How do I reduce latency in a dictation app?

The biggest hidden cost isn't the model — it's the DNS lookup, TCP connection, and TLS handshake that happen after the user stops speaking. Move that work earlier: the moment the user presses the hotkey, fire a small warm-up request over a reused HTTP session so the connection is already open by the time the clip is ready. In informal testing this roughly halved end-to-end time on short phrases, landing "test, test, test" around 300 ms.

How fast is the Sync API for short dictation clips?

The Sync API returns a finished transcript for a short clip in roughly 134 ms at the median (p50), in a single request/response with no polling or webhooks. Because you control when the clip ends, there's no turn-detection step to wait on — the transcript comes back essentially as fast as the model can produce it. Pairing that with a connection pre-warm makes the text appear the instant the user releases the key.

What are the Sync API's limits, and what does it cost?

The Sync API accepts clips from 80 ms up to 2 minutes (40 MB max) across 18 languages, runs on the flagship Universal-3.5 Pro model, and returns formatted text with punctuation and casing. It's priced at $0.45/hr of audio, with keyterms prompting and conversation context included and no rate limits. For anything longer than two minutes, use the pre-recorded (async) API instead.

When should I use streaming or async instead of sync for voice input?

Use streaming when you need live partial transcripts as the user speaks — an open-microphone experience or a real-time voice agent that reacts mid-sentence. Use async (batch) when you're transcribing long recordings where completeness and full-file speech understanding matter more than instant turnaround. Use sync for short, user-triggered clips like dictation, voice commands, and push-to-talk, where you want a finished transcript back immediately over one HTTP request.

Sync vs. Async Transcription: Which to Use (2026)

Mart Schweiger — Wed, 15 Jul 2026 15:46:58 +0000

You've got a recording and you want text back. For years that meant one thing at AssemblyAI: submit the file, wait for the job to finish, get a transcript. Async. It's reliable, it's cheap, and for a huge range of workloads it's exactly right.

But "wait for the job to finish" is doing a lot of work in that sentence. If your file is two minutes long and your user is staring at a spinner, waiting is the whole problem. That's the gap the Sync API fills — and it's why "which transcription path" is no longer a two-way question.

This post is about the two ways to transcribe a recording: async and sync. (If you're deciding between recorded and live audio in the first place — streaming versus the rest — start with our guide to real-time vs batch transcription, then come back here to choose between the two non-streaming paths.)

The one-sentence difference

Async transcription hands you a job: you submit audio, the work happens in the background, and you collect the result later by polling or via a webhook. Sync transcription hands you an answer: you POST a short clip and the transcript comes back in the same HTTP response — no job to track, no callback to wait for.

Everything else follows from that. Async is built for throughput and depth on files of any length. Sync is built for speed on short files, when a person or an agent is waiting on the other end.

How fast can each actually go?

This is the question that usually settles it, so let's be concrete.

Async processes the whole file and returns a single complete transcript, typically in seconds to a few minutes depending on file length and load. Crucially, it bills on audio duration ($0.21/hr on Universal-3.5 Pro), so a 30-minute file costs the same whether it comes back in 20 seconds or two minutes. You're optimizing for cost and completeness, not for the clock.

Sync is built to return a transcript for a short clip almost immediately — roughly 134ms p50 — in one request/response, with no polling and no webhooks. It's priced at $0.45/hr, the same rate as Universal-3.5 Pro Realtime, and handles clips up to 2 minutes (40 MB max). That cap is the point: it's a fast path for short audio, not a replacement for batch processing a podcast archive. Keyterms prompting and conversation context are included, and you can read the full request/response format in the Sync API docs.

Here's the part that matters for interactive apps: for a clip under two minutes, sync returns the full transcript faster than the clip took to record. That's "faster than real time" in the literal sense — you speak for eight seconds, you get the text back in a fraction of a second. Async gets there eventually; sync gets there before your user has looked up from the screen.

Here's how the two recorded-audio paths compare at a glance:

	Async (pre-recorded)	Sync
How you get the transcript	Submit a job, then poll or receive a webhook	One HTTP POST, transcript in the same response
Speed	Seconds to a few minutes, by file length and load	~134 ms p50 to a finished transcript
Audio length	Any length	Up to 2 minutes (max 40 MB)
Model	Universal-3.5 Pro	Universal-3.5 Pro
Speaker diarization & deep speech understanding	Yes	—
Price	$0.21/hr	$0.45/hr
Best for	Long files, batch volume, latency-tolerant workloads	Short clips a person or agent is waiting on

Same accuracy, different delivery

A fair worry: does the fast path cut corners on quality? It doesn't. Sync returns the same accuracy you already get from async — formatted text, punctuation, casing — because the tradeoff isn't accuracy, it's scope. Sync trades file length and depth of speech understanding for speed, not correctness on the words themselves.

What you give up with sync isn't accuracy — it's the heavy, full-file analysis that async is built for. Sync doesn't include speaker diarization, PII redaction, or the Speech Understanding models. Diarization across a long meeting, summarization, entity and topic detection over an hour of audio: those want the complete recording and the processing time to match. On a two-minute clip headed to a live UI, you don't need any of that. You need the words, now.

A decision you can make in one read

Reach for async when:

Files are long — meetings, podcasts, calls, media libraries.
You want the deepest speech understanding: full-file diarization, summaries, entity and topic detection.
Nobody's waiting on the result in real time — the transcript is consumed after the fact.
You're processing at volume and want to bill on audio duration.

Reach for sync when:

Clips are short (up to 2 minutes) — dictation, voice commands, one utterance at a time.
A person or an agent is waiting on the transcript right now.
You'd rather make one HTTP call than stand up a webhook receiver or poll a job.
You already control when the audio starts and stops, so you don't need a live stream.

A quick gut check: if you can't tolerate a spinner, and the clip is short, that's sync. If the file is long and the transcript is read later, that's async. And if you need live partial transcripts as someone speaks — a voice agent listening mid-sentence — that's neither; that's streaming, and the real-time vs batch guide covers where that line falls.

The bigger shift

For a long time the transcription decision was binary: fast-but-live (streaming) or accurate-but-delayed (async). Sync quietly breaks that binary. It says you can have a recorded-audio workflow — the simplicity of a request/response API, no streaming infrastructure — and get the answer back fast enough for a person to wait on it.

That's worth internalizing when you're architecting a product. The question isn't just "live or recorded" anymore. For recorded audio, it's "do I need this back while someone's waiting?" If yes and the clip is short, you no longer have to choose between the fast path and the simple one. Sync is both.

Frequently asked questions

What is the difference between sync and async transcription?

Sync transcription returns a finished transcript in the same HTTP response as your request, while async transcription submits a background job you collect later by polling or webhook. Sync is built for short clips (up to 2 minutes) where speed matters; async is built for files of any length where completeness and the deepest speech understanding matter. Both run on the same flagship model, so the words themselves are equally accurate — the difference is delivery, not quality.

How fast is a synchronous transcription API, and what response time can I expect?

AssemblyAI's Sync API returns a finished transcript in roughly 134 ms at the median (p50) for a short clip, in a single request/response with no polling or webhooks. For any clip under two minutes, that means the transcript comes back faster than the audio took to record — "faster than real time" in the literal sense. Async, by contrast, returns a complete transcript in seconds to a few minutes depending on file length and load.

When should I use sync instead of async batch transcription?

Use sync when the clip is short and a person or agent is waiting on the result right now — dictation, voice commands, IVR prompts, voicemail, or voice-agent turns where you handle turn detection yourself. Use async when files are long, nobody's waiting in real time, or you need full-file speaker diarization, summarization, and entity or topic detection. A quick gut check: if you can't tolerate a spinner and the clip is short, that's sync; if the file is long and the transcript is read later, that's async.

Is the Sync API the same as real-time streaming transcription?

No. Streaming transcribes live audio over a persistent WebSocket, returning partial words as someone speaks — the right tool for open-ended live sessions like captions and real-time voice agents. Sync is for recorded clips you already have in hand: you send one complete short clip over a single HTTP POST and get the full transcript back at once. If you need transcription mid-sentence during an open microphone, that's streaming, not sync.

Does synchronous transcription sacrifice accuracy for speed?

No. Sync returns the same transcription accuracy as async — the same formatted text, punctuation, and casing — because it runs on the same Universal-3.5 Pro model. The tradeoff is scope, not correctness: sync is capped at short clips and does not include speaker diarization, PII redaction, or the Speech Understanding models. For those heavier, full-file features, use the pre-recorded (async) or realtime APIs.

How do I get started with the Sync API, and what are its limits and pricing?

Create a free AssemblyAI account, then send a short audio clip in a single HTTP POST with your API key and read the transcript straight off the response — no job to track. The Sync API accepts clips from 80 ms up to 2 minutes (40 MB max) across 18 languages, and is priced at $0.45/hr of audio, the same rate as Universal-3.5 Pro Realtime, with keyterms prompting and conversation context included. See the Sync API docs for the full request and response format.

Voice Agent Architectures Explained

Mart Schweiger — Wed, 08 Jul 2026 16:14:28 +0000

Every voice agent does the same three things. It listens, it thinks, and it talks back. The interesting part isn't the what — it's the how. And the "how" splits into three very different architectures, each with its own trade-offs on latency, control, cost, and the one thing that quietly decides whether your agent is any good: accuracy.

So before you wire anything together, it's worth understanding the three patterns you can choose from — and why the step everyone treats as plumbing is actually the ceiling on how smart your agent can be.

The job of a voice agent: hear, think, speak

Strip away the framework names and a voice agent is a loop. A person says something. The agent turns that audio into text (speech-to-text, or STT). A language model reads the text and decides what to say (the LLM). That response gets turned back into audio (text-to-speech, or TTS). Then it waits for the next thing the person says and does it all again — ideally in about a second, because anything slower feels like a bad phone connection.

Here's the thing most architecture debates miss: the model that thinks is only ever reacting to what the first step heard. If the STT layer mishears "my account ends in 4-0-1-5" as "4-0-1-8," the LLM confidently helps with the wrong account. Get the input wrong and everything downstream is wrong too. Keep that in mind — it's the theme that runs through all three architectures below.

Architecture 1 — the chained STT→LLM→TTS pipeline

This is the classic approach, sometimes called the cascading or chained architecture. You pick a best-in-class model for each stage and connect them: a streaming speech-to-text API feeds transcripts to an LLM, and the LLM's response goes to a text-to-speech engine. Turn detection decides when the user is done talking so the LLM knows when to respond.

The upside is control. You choose the most accurate STT model, the LLM that best fits your use case, and the voice you want. You can tune each stage independently — swap the LLM without touching transcription, add domain vocabulary at the STT layer, change voices at the TTS layer. For teams that need a genuinely differentiated product, that granularity matters.

The cost is plumbing. You're now integrating three providers, reconciling three sets of latency, and debugging across three surfaces. Every millisecond of end-to-end latency is the sum of each hop plus the handoffs between them. It's the most flexible architecture and the most work.

Architecture 2 — a single speech-to-speech model

The newer approach collapses the pipeline into one multimodal model that takes audio in and emits audio out, reasoning over speech directly. It's elegant, and it's simple to start with — one API, one round trip.

But simplicity has a price, and it's control. When a single model does everything, you can't swap the "ears" for a more accurate transcription model, you can't inspect the transcript to see what it actually heard, and you can't tune turn-taking or entity recognition stage by stage. These models are also built primarily around conversational fluency rather than getting a spelled-out email address or a 16-digit card number exactly right. When the same black box handles listening and speaking, an input error is invisible until the agent says something wrong.

Speech-to-speech is a great fit for casual, low-stakes conversation. It's a harder sell when your agent has to capture names, numbers, and identifiers correctly on the first try.

Architecture 3 — a unified voice agent API

There's a third option that's easy to miss because it looks like the simple one from the outside and behaves like the flexible one on the inside: a unified voice agent API that runs the full STT→LLM→TTS pipeline for you behind a single connection.

AssemblyAI's Voice Agent API is one example. You connect to a single WebSocket, stream audio in, and get audio back. Under the hood it's still a pipeline — real speech-to-text, a real LLM, real text-to-speech — but you get one connection, one bill measured in hours instead of token math across three invoices, and one set of logs instead of three dashboards. It's a flat $4.50/hr for all three stages, with roughly one-second end-to-end latency, and it works natively with tools like Claude Code so you can go from docs to a working agent the same afternoon.

The framing that matters here is "invisible infrastructure." You're not adopting an opinionated agent platform where every agent ends up sounding the same. You keep control over conversation design, tool calling, and turn timing — you just skip the plumbing. It's built on Universal-3.5 Pro Streaming, so the listening step is handled by a model designed for exactly the hard tokens that trip agents up.

Choosing an architecture

There's no single right answer — there's a right answer for your product. Here's how the three patterns compare on the dimensions that usually decide it.

Dimension	Chained STT→LLM→TTS	Speech-to-speech	Unified voice agent API
Control over each stage	Highest	Lowest	High
Time to first working agent	Slowest (three integrations)	Fast	Fast
Transcript transparency	Full	None	Full
Billing & debugging surfaces	Three	One	One
Entity accuracy on hard tokens	Depends on chosen STT	Weaker	Strong (Universal-3.5 Pro Streaming)

Strong (Universal-3.5 Pro Streaming)

If you're a large team building a deeply custom product and you want to own every layer, the chained pipeline earns its complexity. If you're prototyping something casual, a speech-to-speech model gets you talking fast. And if you want production-grade accuracy and control without integrating three vendors, a unified API is the pragmatic middle path — which is why it's where most teams building their first serious voice product end up.

Why the listening step is the real ceiling

Notice what every row in that table has in common: none of them can fix a transcription mistake after the fact. The LLM can only reason over the words it was handed. So the accuracy of the speech-to-text layer sets the ceiling for the entire agent, no matter which architecture you pick.

That's why the input model matters more than its "plumbing" reputation suggests. Universal-3.5 Pro Streaming is built for the tokens agents get wrong most often — emails, names, phone numbers, account IDs — with sub-300-millisecond latency and native code-switching across 18 languages. It uses punctuation-based turn detection so the agent knows the difference between a pause and a finished thought. In a chained pipeline you'd choose it as your STT layer; in the Voice Agent API it's already there. Either way, it's doing the job that decides whether the rest of your stack has a chance.

For a closer look at where DIY stacks tend to strain under real traffic, our write-up on where voice agent stacks start showing their limits is a useful companion read.

The takeaway

Architecture is a series of trade-offs, but they all bottom out in the same place. Control versus simplicity, three bills versus one, transparency versus a black box — those are real decisions. What isn't up for debate is that the model doing the listening determines how good everything after it can be. Pick the architecture that fits your team, then give it the best possible ears. That's the part your customers will actually hear.

Want the hands-on version? Our guide to building with the Voice Agent API walks through a working agent, and what are AI voice agents covers the fundamentals if you're just getting oriented.

Frequently asked questions

What is a voice agent architecture?

A voice agent architecture is the way you connect the three core steps of a spoken conversation: speech-to-text (listening), a language model (thinking), and text-to-speech (speaking). The three common patterns are a chained STT→LLM→TTS pipeline, a single speech-to-speech model, and a unified voice agent API that runs the full pipeline behind one connection.

What is the difference between a cascading (STT→LLM→TTS) pipeline and a speech-to-speech model?

A cascading pipeline uses a separate, best-in-class model for each stage, giving you full control and a visible transcript but requiring you to integrate and debug three providers. A speech-to-speech model handles audio-in to audio-out in one model, which is simpler to start with but hides the transcript and gives you less control over per-stage accuracy and turn-taking.

Which voice agent architecture has the lowest latency?

All three can reach roughly one-second end-to-end latency when tuned well. A single speech-to-speech model removes handoffs between stages, while a well-built unified API like AssemblyAI's Voice Agent API reaches about one second end-to-end because the pipeline is optimized as a whole rather than assembled from three separate vendors.

Why does speech-to-text accuracy matter so much for voice agents?

The language model can only respond to the words the speech-to-text layer produces, so any transcription error propagates through the entire agent. This is why entity accuracy on emails, names, phone numbers, and account IDs is the ceiling on agent quality — and why models like Universal-3.5 Pro Streaming are built specifically for those hard tokens.

What is the best API for building a voice agent?

The best choice depends on how much control you need. Teams that want production accuracy without integrating three providers often choose a unified voice agent API; AssemblyAI's Voice Agent API runs STT, an LLM, and TTS through one WebSocket at a flat $4.50/hr, built on Universal-3.5 Pro Streaming for high entity accuracy.