When people talk about AI voice agents, they usually talk about the model. They talk about prompts, tools, and how smart the reply sounds. That is only half the story.
The other half is much quieter, and much harder. It lives under the audio pipeline. It is called a jitter buffer.
If you get the jitter buffer wrong, your agent does not sound slow in a polite way. It sounds broken. It cuts people off mid-sentence. It answers into silence. It waits too long and kills the feeling of a real conversation.
This post is a deep dive into that piece. We will start from how voice travels on the internet, open an RTP packet and look at fields like sequence number and timestamp, then look at why fixed buffers fail for AI, and finally look at how adaptive jitter buffers try to solve the trade-off.
The internet does not deliver audio in a straight line
When you speak into a microphone, the sound starts as a continuous wave. Computers cannot ship that wave as one long stream across the public internet. So the system cuts the audio into tiny pieces called packets.
In voice systems, one packet often carries about 20 milliseconds of audio. That is a very small slice of speech. Your “hello” may travel as many packets, one after another.
Those packets usually travel over UDP. UDP is fast because it does not wait for a receipt. The sender does not sit there asking, “Did you get packet 17?” It just keeps sending.
That speed is useful for voice. The cost is reliability. Packets can:
- arrive late
- take different routes
- land out of order
- disappear completely
The variance in arrival time is called jitter.
If the receiving side plays each packet the moment it arrives, the audio does not sound like a human voice. It sounds like a skipping CD. You hear gaps, robotic jumps, and broken words.
So the problem is not only “send audio.” The problem is “rebuild smooth speech from messy packet arrival.”
Mic wave
-> cut into 20ms packets
-> wrap each slice in an RTP header
-> send over UDP
-> packets arrive late / early / out of order
-> without a buffer: glitchy audio
That is why voice systems do not send raw audio blobs with no labels. They send RTP packets. RTP is the envelope that tells the receiver how to put speech back together.
What is inside an RTP packet?
RTP means Real-time Transport Protocol. In WebRTC and most VoIP systems, your voice does not travel as “just UDP bytes.” It travels as RTP on top of UDP.
Think of each RTP packet as a small box with two parts:
- A header with labels the receiver needs
- A payload with the actual compressed audio
+--------------------------------------------------+
| RTP header |
| seq | timestamp | SSRC | payload type | ... |
+--------------------------------------------------+
| Payload: encoded audio (Opus, PCMU, etc.) |
+--------------------------------------------------+
The jitter buffer mostly cares about the header labels. Without those labels, late packets are just noise. With those labels, the receiver can reorder, detect loss, and decide when to play each slice of audio.
Here are the fields that matter most when you are building or debugging AI voice.
Sequence number
The sequence number is a simple counter. The sender increases it by one for every RTP packet it sends.
If the sender sends packets like this:
seq 101 -> seq 102 -> seq 103 -> seq 104
and the receiver gets:
seq 101 -> seq 103 -> seq 102 -> seq 104
the sequence number tells the buffer the true order. Packet 102 is not “wrong audio.” It is just late. The buffer can hold 103 for a moment, wait for 102, then play them in the right order.
Sequence numbers also help detect loss. If you see 101, then suddenly 104, you know 102 and 103 are missing for now. Maybe they will still arrive. Maybe they are gone. The buffer has to decide how long to wait before giving up.
In plain words:
- Sequence number answers: “Which packet comes next?”
- It is the main tool for ordering and loss detection
Timestamp
The timestamp is different from the sequence number, and this difference confuses many people at first.
The sequence number counts packets. The timestamp marks media time.
It tells the receiver when the first sample in this packet should sit on the audio timeline. It is not a wall-clock time like “3:41:02 PM on your laptop.” It is a media clock based on the audio sample rate.
A simple example helps.
Suppose you send 20ms of audio in each packet, and the clock runs at 8000 samples per second. Then each packet often advances the timestamp by:
8000 samples/sec * 0.020 sec = 160
So timestamps may look like:
packet A: timestamp 1600
packet B: timestamp 1760
packet C: timestamp 1920
Even if packet B arrives after packet C on the network, the timestamps still say B should play before C.
Why do we need both sequence number and timestamp?
Because one packet is not always one equal slice of time in every codec setup, and because silence, packetization, and frame packing can make timing more subtle. Sequence number keeps packet order. Timestamp keeps playout timing.
In plain words:
- Timestamp answers: “When should this audio be played on the media timeline?”
- It is the main tool for smooth playout and jitter measurement
Payload type
The payload type tells the receiver which codec is inside the packet.
Examples people see in real systems:
- PCMU / PCMA for classic telephony audio
- Opus for modern WebRTC voice
- other dynamic payload types negotiated during the call setup
If the receiver thinks the packet is Opus but the bytes are actually something else, decode fails. So this field is small, but it is important. It says, “Here is how you should unpack the payload.”
SSRC
SSRC means Synchronization Source. It is an ID for one media stream from one sender.
If your call has one microphone stream, that stream has one SSRC. If another person joins, or if a mixer creates a new stream, you get another SSRC.
The receiver uses SSRC to keep streams separate. Your jitter buffer for speaker A should not mix packets from speaker B just because both arrived on the same socket.
In plain words:
- SSRC answers: “Which talker or which stream does this packet belong to?”
Marker bit
The marker bit is a one-bit flag. In audio, it is often used to mark the start of a new talkspurt, meaning the moment someone starts speaking again after silence.
This matters for voice agents because silence and speech boundaries are exactly where VAD and turn-taking get sensitive. The marker bit is not the full VAD system by itself, but it is one more signal in the media layer about where a speech burst begins.
Payload
The payload is the actual compressed audio. This is the Opus frame, or PCMU samples, or whatever codec you negotiated.
Everything above the payload exists to protect this part. The header is the map. The payload is the voice.
A quick map of the important RTP fields
| Field | What it is | Why the jitter buffer cares |
|---|---|---|
| Sequence number | Packet counter | Reorder packets, detect gaps and loss |
| Timestamp | Media-clock time | Decide playout timing, measure jitter |
| Payload type | Codec ID | Decode the bytes correctly |
| SSRC | Stream ID | Keep one talker’s packets together |
| Marker bit | Talkspurt / frame hint | Help spot speech burst boundaries |
| Payload | Encoded audio | The actual sound the AI must hear |
What RTP does not do
RTP itself does not guarantee delivery. It does not retransmit lost packets by default. It does not fix jitter by itself.
RTP only labels the media carefully. The hard work still sits on the receiving side: hold packets, sort them, wait a little, and decide when silence is real silence versus a late packet.
That receiving-side logic is the jitter buffer.
There is also a partner protocol called RTCP. RTCP carries reports about quality: loss rate, jitter estimates, and timing stats between sender and receiver. Adaptive systems often use those reports, together with local arrival measurements, to decide whether the buffer should grow or shrink.
The jitter buffer is a waiting room for packets
A jitter buffer is the waiting room.
Instead of playing a packet the millisecond it arrives, the receiver holds packets for a short time. That short wait gives delayed packets a chance to show up. Then the system uses the sequence number to line them up, uses the timestamp to place them on the media timeline, and feeds clean audio forward at a steady rate.
Think of it like people entering a metro station during rush hour. If you open the gate for every person the second they arrive, the platform becomes chaotic. If you hold people for a few seconds and then let them board in order, the flow feels smooth.
In audio terms:
- RTP packets enter the buffer.
- The buffer reads sequence numbers and timestamps.
- The buffer waits a little for missing sequence numbers.
- Late packets get a chance to catch up.
- Packets leave the buffer in media order, at a constant pace.
- Speech-to-text (STT) hears cleaner audio.
A common starting point is something like 40 milliseconds of buffering. That number is not magic. It is a compromise. Enough wait to hide small network bumps. Not so much wait that the whole call feels delayed.
Arriving packets (messy timing)
|
v
[ jitter buffer ] <-- short waiting room
|
v
Smooth audio stream into STT / VAD / the AI pipeline
This idea is old in VoIP. Zoom-style calls have lived with it for years. The hard part starts when the listener on the other side is not a human. The listener is an AI agent with Voice Activity Detection, an LLM, and text-to-speech.
Why this is easy for Zoom and hard for AI voice agents
In a normal phone or video call, both sides are humans. Humans are flexible. If one packet is slightly late, your brain fills the gap. If the other person pauses for half a second, you wait. You do not instantly decide, “They finished talking, so I should start speaking now.”
An AI voice agent does not have that patience by default.
The agent pipeline usually looks like this:
- Capture caller audio
- Smooth it through a jitter buffer
- Run Voice Activity Detection (VAD)
- Transcribe speech to text
- Send text to an LLM
- Generate a reply
- Turn the reply into speech with TTS
- Stream that speech back to the caller
Every stage adds time. The jitter buffer is only one part of the total delay. But it sits at the front door, so bad choices here poison everything after it.
This creates a painful trade-off.
If the buffer is too large
Imagine a fixed buffer of 150 milliseconds.
Audio quality can look great. Delayed packets have more time to arrive. The STT engine hears cleaner speech. On paper, this feels safe.
In a live AI conversation, that “safe” wait is expensive.
You add 150ms before the AI even starts understanding you. Then you still pay for transcription time, model thinking time, and TTS time. By the time the agent answers, the caller already feels an awkward pause. A two-second gap destroys the illusion of natural talk. People stop trusting the product. They start speaking over the bot, or they hang up.
Large buffers protect quality. They also make the agent feel slow and unnatural.
If the buffer is too small
Now imagine a tiny buffer of 10 milliseconds.
Response time looks fast. The pipeline feels “real-time.” On a perfect network, this can work.
On a normal mobile network, it fails in a nasty way.
When a packet arrives late, a small buffer cannot wait for it. The packet gets dropped. In RTP terms, you may see a hole in the sequence numbers. The AI then hears a tiny hole of silence inside your sentence. Voice Activity Detection sees that silence and thinks you stopped talking. The agent decides it is its turn. It starts answering while you are still mid-thought.
That is the fatal AI flaw with small buffers.
The failure is not only bad audio. The failure is wrong turn-taking. The agent interrupts you because Wi-Fi had a hiccup, not because you finished your sentence.
Too large buffer:
better audio quality
higher latency
awkward pauses
conversation feels slow
Too small buffer:
lower latency
more packet drops
fake silence holes
VAD thinks you stopped
agent interrupts mid-sentence
This is why jitter buffering is one of the hardest parts of building AI voice agents. You are not only optimizing sound. You are protecting the social rhythm of a conversation.
Fixed buffers are too dumb for real networks
A fixed-size buffer assumes the network stays roughly the same. Real networks do not stay the same.
A caller can start on good office Wi-Fi, then walk into a lift. A mobile user can move from strong LTE into a crowded street. Packet delay can jump within seconds. A buffer that was perfect one minute ago can become too small or too large in the next minute.
That is why modern voice systems, including WebRTC stacks, move toward Adaptive Jitter Buffers (AJB).
An adaptive buffer behaves like a smart shock absorber on a car. On a smooth road, it stays tight and responsive. On a broken road, it softens and absorbs the bumps.
How an adaptive jitter buffer thinks
An adaptive jitter buffer keeps watching the network and changing its own size.
1. Network monitoring
It constantly measures variance using RTP metadata and arrival time.
A practical way to think about it:
- Compare arrival time on the wire with the packet’s RTP timestamp
- Watch gaps in the sequence number to estimate loss
- Track how often packets arrive out of order
- Use local stats, and sometimes RTCP reports, to see if the path is getting messy
This is not a one-time setup. It is continuous measurement during the call.
2. Dynamic resizing
If the connection looks clean, the buffer can shrink toward something like 10–20 milliseconds. That keeps the AI snappy. The agent can start thinking sooner, which helps the reply feel natural.
If the caller moves into weak cell coverage, the buffer can grow toward something like 60–80 milliseconds. That extra room catches delayed packets before they become silence holes. The goal is to stop VAD from hallucinating an end-of-turn.
So the buffer is not “set once and forget.” It breathes with the network.
3. Time stretching
Advanced systems go one step further. WebRTC’s NetEQ is a famous example in this family of ideas.
If the buffer is running empty, the system can slightly slow down playback for a few milliseconds. That buys time for the next packet to arrive. If the buffer is getting too full, it can slightly speed playback up.
The important part is that good algorithms try to do this without making your voice sound like a cartoon. Pitch stays natural. Timing flexes a little. The listener hears continuity instead of a hard glitch.
Good network:
shrink buffer -> lower delay -> faster AI turn start
Bad network:
grow buffer -> hide jitter -> protect VAD from fake silence
Buffer almost empty:
stretch time a little -> wait for next packet
Buffer too full:
compress time a little -> catch up without a hard jump
This tug-of-war never fully ends. Every live call is a negotiation between latency and stability.
Why VAD makes the stakes higher
In text chat, a late token is annoying. In voice, a late packet can change who is allowed to speak.
Voice Activity Detection is the component that guesses whether the human is still talking. Many agent systems use VAD to decide when to stop listening and start answering.
That decision is brittle when audio is broken.
A dropped packet can look like silence. A burst of jitter can look like the person paused. If your buffer is too aggressive, VAD fires early. The LLM starts generating. TTS starts speaking. Now the human and the bot are talking over each other.
Once that happens, recovery is hard. The caller feels interrupted. The transcript may be incomplete. The model answers a half-finished question. The product feels “dumb,” even if the model itself is strong.
So when people say “our voice agent interrupts users,” the root cause is not always the prompt. Often the root cause is audio timing at the edge of the pipeline.
The full latency budget, not just the buffer
It helps to look at the whole chain, because the jitter buffer never works alone.
A rough mental model:
| Stage | What it does | Why it hurts if it grows |
|---|---|---|
| Jitter buffer | Smooths packet arrival | Makes the agent start late |
| VAD | Detects end of speech | Wrong guess causes interruptions |
| STT | Turns audio into text | Slow STT adds dead air |
| LLM | Plans the reply | Long reasoning feels like silence |
| TTS | Turns text into voice | Slow first audio makes replies feel heavy |
| Playback | Streams speech back | Extra buffering on the way out also adds delay |
If each stage “only” adds a little delay, the total still becomes a conversation killer. That is why production voice teams obsess over the first audible response, not only the final answer quality.
A beautiful reply that arrives after two awkward seconds still feels broken.
What “good” looks like in production
A strong voice pipeline does not chase zero jitter buffer. Zero is a fantasy on the public internet.
Good systems chase a moving target:
- Keep buffer small when the network is stable.
- Grow only when measurement says growth is needed.
- Use sequence gaps and timestamp drift as early warning signs, not only “audio sounds bad.”
- Protect VAD from fake end-of-speech caused by drops.
- Start STT and model work as early as safely possible.
- Stream TTS output instead of waiting for the full sentence.
- Measure interruption rate, silence gaps, and time-to-first-audio as product metrics, not only as network metrics.
In other words, the buffer is not a pure networking detail. It is part of conversation design.
If you are building an AI calling product, these questions matter as much as model choice:
- How often does the agent cut the user off?
- How often does the user talk over the agent?
- How long is the quiet gap before the agent starts speaking?
- How does that gap change on mobile networks versus office Wi-Fi?
Those numbers tell you whether your jitter strategy is working.
A simple way to remember the whole idea
Here is the whole story in one short chain.
The internet ships voice as tiny UDP packets wrapped in RTP. Each RTP packet carries labels like sequence number, timestamp, payload type, and SSRC. Those packets do not arrive smoothly. Jitter is that uneven arrival. A jitter buffer uses sequence numbers to reorder packets and timestamps to place them on the media timeline. A buffer that is too large makes the AI slow. A buffer that is too small makes the AI interrupt people. Adaptive buffers watch the network and resize themselves. Advanced buffers also stretch or squeeze time a little to keep speech continuous. The hard part is not making audio “kind of work.” The hard part is protecting natural turn-taking while staying fast enough that the conversation still feels human.
That tug-of-war between fighting latency and preventing VAD misfires is the core engineering challenge of real-time conversational AI.
Closing
Models get the headlines. Jitter buffers do not.
But if you are building voice agents seriously, this is one of the places where product quality is won or lost. A clever prompt cannot fix a pipeline that keeps handing the model broken silence. A bigger model cannot hide a two-second pause that makes every reply feel late.
If you want AI voice to feel natural, you have to respect the physics of the network, the psychology of conversation, and the engineering trade-offs in between.
The waiting room has to be smart.
Top comments (0)