APIs used to feel easy. I mean that literally, not as nostalgia. Send a request, get an answer, and the transaction closed. Even HATEOAS stayed inside that model: the API handed you the URI for the next step and the whole thing still behaved like furniture. The state was visible, the errors were HTTP status codes, and the connection was a detail the client library swallowed.
What streaming audio did to a calm mental model
The wobble came when I met streaming audio during a week at Deepgram. The connection did not end. It stayed open for minutes, audio arriving in chunks while results flowed back the other way at the same time. No 4xx code to catch. When something broke, the socket just dropped, or the stream went quiet. Years of webhooks at Vonage had never once shown me that shape.
That is a different kind of API, and it felt like the ground had moved.
The state you cannot see
What made it hard was not just the new lifecycle. It was state you can't see. A JSON payload is readable: field names, values, nesting, all right there. An audio file tells you nothing. Sample rate, bit depth, channel count, encoding all live in binary headers that stay invisible until you run a separate tool.
You can open two audio files in a file manager and they look identical. ffprobe disagrees. Most voice AI problems are audio problems, not API problems, and I didn't know that until I held real audio files. The format the phone saved was not the format the API wanted, and nothing in the first error message explained any of it.
The lesson: HTTP/2 bidirectional streaming
The destination, it turns out, is HTTP/2 true bidirectional streaming. Native to the protocol. One TCP connection carries many streams, and each side signals its own end independently, so you can keep receiving after you stop sending. Full-duplex at the stream level. Usually this rides on gRPC: one stream per RPC, length-prefixed framing, and a bidirectional streaming RPC is precisely send-up-while-receiving-down. gRPC metadata, key/value pairs riding the HTTP/2 headers and trailers, flows along mid-call.
The real-world proof is mundane in the good way. Cloud Speech-to-Text exposes streaming through gRPC only. Kaldi and Nuance style streaming ASR servers push binary speech up and partial hypotheses with word timestamps, confidence, and utterance info back down, all on the same open stream.
Two superpowers fall out of that, and they're the ones I keep thinking about. You can send feedback while you are still receiving audio. And you can receive metadata mid-stream, without closing anything. Not special JSON on a finished response. Live.
gRPC is the usual carrier but not the only one. WebSocket is browser-native and full-duplex at the message level, but it multiplexes nothing and has no streaming RPC semantics. SSE is strictly one-way, server to client. WebTransport over HTTP/3 is the emerging path for full duplex in the browser, because gRPC-web can't do client streaming or bidirectional streaming. So gRPC when you own both ends, WebSocket or WebTransport when the browser is the client.
Why this is the shape that pulls me
The specific capability that pulls me is a single gRPC stream carrying a discriminated union: audio bytes interleaved with control, metadata, and feedback messages, in both directions. That lets you deliver PCM mixed with metadata while you keep sending more data upstream. Two-way payloads at the same time. That never happened in request/response, and it is exactly the simultaneous conversation full-duplex AI products need.
Let me be honest about framing. I am not building this right now. It is a technology I keep coming back to, the transport I reach for in my head whenever I think about a full-duplex product. There is a difference between shipping it today and knowing the shape it has to take. The shape is HTTP/2 bidirectional streaming.
Where it lands
A voice agent is a pipeline: ASR into an LLM into TTS. Each stage streams into the next, and the invisible state is where the failures live. The Inside the Streaming Cascade Powering Voice AI is how those stages stay connected, and the agent pipeline below has its invisible layer highlighted because it is the same invisible state that broke my REST mental model, applied to a full-duplex conversation.
Years ago the extent of what I knew was APIs that ended. Now I look toward audio and data streaming over one full-duplex connection, where the model keeps talking and metadata keeps coming back mid-stream. I wrote out the architecture case in Full-Duplex Voice AI Needs a New Architecture, and this post is the journey side of the same coin. APIs stopped feeling easy. I don't regret that.
FAQ
Why is HTTP/2 bidirectional streaming the right transport for audio?
Because audio is continuous both ways and HTTP/2 streams handle that natively. One TCP connection carries many streams, and each side ends its own stream independently, so you keep receiving after you stop sending. That full-duplex, stream-level behaviour is what real-time audio and its metadata actually need, and it's the shape gRPC formalises.
Is gRPC the only way to do this?
No. gRPC is the usual carrier because it maps cleanly onto the stream, but WebSocket is full-duplex at the message level and WebTransport over HTTP/3 is the emerging browser path, since gRPC-web can't do client streaming or bidirectional streaming. Pick gRPC when you own both ends, WebSocket or WebTransport when the browser is the client.
Can you really send feedback while still receiving audio?
Yes. In a bidirectional streaming RPC the two directions are independent streams. You can send a control or feedback message upstream while audio or transcription is still flowing down to you, with nothing closed and no round-trip. That simultaneity is the whole point, and it's impossible in request/response.
How is this different from a plain WebSocket?
A WebSocket is full-duplex and browser-native, but it's message-level only. No multiplexing, no streaming RPC semantics, no per-stream ordering guarantees. HTTP/2 bidirectional streaming gives you length-prefixed message framing, many concurrent streams, and a defined request/response shape on a single connection, which is why gRPC builds on it.
Where does this fit for full-duplex AI products?
In a full-duplex voice product both parties speak at once, so the transport has to carry audio and metadata in both directions simultaneously. HTTP/2 bidirectional streaming is what that requires. It lets the model keep talking while it still receives input, which is exactly the simultaneous two-way payload full-duplex AI is built on.
Top comments (0)