Audio Input in the GPT-4o Realtime API

#ai #machinelearning #llm #programming

The Realtime API takes audio in and gives audio back over a persistent connection, with no transcription step in the middle. That single architectural fact — the model consumes audio tokens directly — is why its format constraints are strict and why its event model looks nothing like Chat Completions.

Not a transcription pipeline

The familiar way to build a voice application is three services in a row: speech-to-text, then a language model, then text-to-speech. Each hop adds latency and each hop throws information away — the transcription discards tone, pace, hesitation and emphasis, and the model never sees them.

The Realtime API collapses that. Audio is tokenised and fed to a speech-native model, which emits audio tokens back. Prosody survives the round trip in both directions. The cost of that design is that you are no longer sending a file to an endpoint: you hold a connection open, push audio into a server-side buffer as it is captured, and exchange events. That is a bidirectional WebSocket, not the one-way Server-Sent Events stream Chat Completions uses — so none of the accumulator patterns from the streaming chunk format transfer, and the instructions field on the session takes the place of the message role described in the developer role. OpenAI documents the whole event vocabulary in the Realtime API reference.

The Realtime API has been through a preview generation and a general availability generation, and several event names changed between them — audio delta events in particular were renamed. The names below are from the widely deployed preview vocabulary. Check the reference for the model and API version you are targeting and treat event names as version-specific.

The audio formats

Three formats are accepted, and the constraints on each are exact rather than advisory. Sending 44.1 kHz PCM labelled as pcm16 does not fail loudly; it produces a model that hears your user speaking too quickly.

pcm16 — 16-bit signed PCM, 24 kHz sample rate, mono, little-endian byte order. This is the format to use.
g711_ulaw — 8-bit µ-law, 8 kHz. For telephony.
g711_alaw — 8-bit A-law, 8 kHz. Also telephony, the non-North-American convention.

There is no MP3, no Opus, no AAC, no WAV container. Raw samples only — if you have a WAV file, strip the 44-byte header. Input and output formats are configured independently on the session, so a phone integration can take µ-law in and ask for µ-law out without resampling anywhere in your stack:

{
  "type": "session.update",
  "session": {
    "modalities": ["text", "audio"],
    "instructions": "You are a terse support agent. Never guess an order number.",
    "voice": "alloy",
    "input_audio_format": "pcm16",
    "output_audio_format": "pcm16",
    "input_audio_transcription": {"model": "whisper-1"},
    "turn_detection": {
      "type": "server_vad",
      "threshold": 0.5,
      "prefix_padding_ms": 300,
      "silence_duration_ms": 500
    }
  }
}

At 24 kHz, 16-bit, mono, audio is 48,000 bytes per second, so a 100-millisecond chunk is 4,800 bytes — about 6,400 characters once base64-encoded. That is the arithmetic to have in mind when choosing a chunk size: smaller chunks mean lower latency and more framing overhead.

The input buffer and its events

Audio does not arrive as a message. There is a server-side input buffer, you append to it, and at some point it becomes a conversation item. Three client events manage it:

input_audio_buffer.append — carries a base64-encoded chunk of raw audio in its audio field. Send these continuously as you capture. No response is expected per append.
input_audio_buffer.commit — closes the current buffer into a user message. Only needed when you are managing turns yourself.
input_audio_buffer.clear — discards what is buffered.

{
  "type": "input_audio_buffer.append",
  "audio": "UklGRiQAAABXQVZFZm10IBAAAAABAAEAgD4AAAB9AAACABAAZGF0YQ..."
}

The server side of the same buffer produces input_audio_buffer.speech_started, input_audio_buffer.speech_stopped and input_audio_buffer.committed. The first of those is the one that matters most for a good experience: it fires the moment the server detects the user has begun speaking, and it is your cue to stop playing whatever the assistant was saying. Barge-in is not automatic — the server stops generating, but the audio already in your playback buffer will keep playing unless you flush it.

Server VAD and manual turns

With turn_detection set to server_vad, the server decides when a turn ends and generates a response without being asked. Three parameters control it, and each maps to a real complaint:

threshold — how loud counts as speech, 0 to 1. Raise it in a noisy room; lower it for a quiet talker.
prefix_padding_ms — how much audio before the detected onset to include. This is what stops the first consonant being clipped.
silence_duration_ms — how long a pause must last to end the turn. Too low and the assistant interrupts people who are thinking; too high and every exchange feels sluggish. This is the parameter to tune first.

Set turn_detection to null and nothing happens automatically: you send input_audio_buffer.commit and then response.create yourself. That is the right mode for push-to-talk and for anything where a false turn end is worse than a slow one.

{"type": "input_audio_buffer.commit"}
{"type": "response.create",
 "response": {"modalities": ["audio", "text"]}}

The response then streams back as a sequence of delta events carrying base64 audio, terminated by a completion event and finally response.done, which carries the usage figures for the turn. The full event catalogue and its ordering rules are in the Realtime API event shape.

Getting the text as well

The model does not need a transcript to understand the user, but you almost certainly need one — for logs, for moderation, for a chat history, for support tooling. That is what input_audio_transcription in the session config is for: it runs a separate transcription model over the input audio in parallel and emits the result as its own event.

{
  "type": "conversation.item.input_audio_transcription.completed",
  "item_id": "item_A9f3",
  "content_index": 0,
  "transcript": "I need to change the delivery address on order 44182."
}

Two things follow. It is asynchronous relative to the response, so the transcript can arrive after the assistant has already started speaking — do not block on it. And it is a separate model looking at the same audio, so the transcript is a good record of what was said and not a record of what the speech-native model perceived. If the assistant misheard, the transcript may well be correct, which makes it evidence about the audio rather than evidence about the model.

Realtime is a WebSocket protocol with its own event vocabulary, not a request-response endpoint, so it sits outside the normalisation that gateways apply to chat traffic — including Multigrid’s. If you are building voice alongside text, expect the voice path to be a direct, separately instrumented integration rather than another model id behind the same call.