DEV Community

Cover image for Sub-Second Full-Duplex Voice AI: LiveKit WebRTC, Deepgram Nova-2 & POS State Machines
Shaheer Ul Hassan
Shaheer Ul Hassan

Posted on Originally published at shaheerulhassan.com

Sub-Second Full-Duplex Voice AI: LiveKit WebRTC, Deepgram Nova-2 & POS State Machines

#python #ai #webrtc #fastapi

Originally published on my systems engineering portfolio: shaheerulhassan.com/blog

Source Code on GitHub: github.com/samjanjua6/Haji-Cafe

Most voice AI tutorials teach a naive, turn-taking request-response loop: the user speaks, a timer waits for 2 seconds of silence, a .wav file is uploaded to Whisper, an LLM generates a complete response, a TTS engine synthesizes audio, and playback starts. In practice, this roundtrip takes 4.5 to 7 seconds.

Psychoacoustic research indicates that human conversational cadence collapses when turn-taking latency exceeds 800 milliseconds. Delays beyond this threshold feel awkward, provoke accidental interruptions, and destroy trust. In high-throughput operational environments—such as restaurant drive-thrus, café POS registers, or emergency dispatch—a voice agent that takes 5 seconds to reply is unusable.

Here is how we engineered an asynchronous, sub-500ms full-duplex conversational voice agent in Python integrating LiveKit WebRTC, Silero VAD, Deepgram Nova-2, Groq LPU inference, and transactional POS state machines.

1. Pipelined Interleaved Streaming & The Latency Budget

In naive architectures, each stage waits for the upstream stage to complete 100% of its workload. In our Pipelined Streaming Architecture, generation and synthesis overlap: the first sentence is sent to TTS while the LLM is still generating subsequent tokens.

Under this interleaved model, Time-to-First-Audio (TTFA) is bounded solely by the initial sentence chunk rather than the full response length:

T_TTFA = t_VAD + t_STT + t_LLM + t_TTS + t_net < 500 ms

Where:
• t_VAD = 120 ms (Adaptive trailing silence window)
• t_STT = 135 ms (Deepgram Nova-2 streaming WebSocket partial emit)
• t_LLM = 24 ms (Groq LPU Llama 3.3 70B generating first 12 tokens at >500 t/s)
• t_TTS = 160 ms (Streaming neural TTS time-to-first-byte)
• t_net = 25 ms (One-way WebRTC UDP packet delay)
-------------------------------------------------------------
Total Turnaround Time: ~464 ms (Sub-second target achieved!)
Enter fullscreen mode Exit fullscreen mode 
Sequential Model (Total: ~4,500ms):
[ User Speaks ] ──> [ Full Transcript ] ──> [ Full LLM Generation ] ──> [ Full TTS Audio ] ──> Play
0ms                1200ms                  2800ms                      4500ms

Pipelined Interleaved Streaming Model (Time-to-First-Audio: ~464ms):
[ User Speaks ]
──[ VAD Boundary (120ms) ]
    └──[ STT Interim (135ms) ]
         └──[ LLM Token 1-12 (24ms) ] ── (First Sentence Ready!)
              └──[ Streaming TTS Chunk (160ms) ]
                   └──[ WebRTC UDP Egress (25ms) ] ──► [ USER HEARS AUDIO: ~464ms ]
                        └──[ LLM Token 13-40 generates in background concurrently ]
Enter fullscreen mode Exit fullscreen mode

2. WebRTC UDP vs. TCP WebSockets: Eliminating Head-of-Line Blocking

Standard HTTP and WebSocket protocols rely on TCP, which enforces head-of-line blocking. A single dropped packet stalls the entire audio stream until retransmission succeeds. In spotty mobile networks or drive-thru lanes, this causes severe stutter.

Production voice AI must run on WebRTC UDP with Opus codecs. When packets arrive late, they are intentionally dropped, while the Opus codec applies Packet Loss Concealment (PLC) to synthesize missing phonemes with zero perceived latency.

💡 Key Takeaway: If your voice agent uses TCP WebSockets for raw audio streaming over public cellular networks, it will inevitably stutter under jitter. WebRTC over UDP is non-negotiable.

3. Solving Conversational Barge-In & Backchanneling

Full-duplex capability means both parties can speak simultaneously. However, naive barge-in detection suffers from two fatal failure modes:

A. Acoustic Echo Feedback

When the device plays the assistant's voice through external speakers, the microphone captures that audio and loops it back into the speech-to-text pipeline, causing the bot to interrupt itself mid-sentence. We enforce hardware-accelerated Acoustic Echo Cancellation (AEC) over WebRTC and maintain an in-memory egress playback register to subtract synthesized phonemes from incoming audio.

B. Backchannel Over-Sensitivity

Humans frequently emit affirmative conversational backchannels ("Mmhmm", "Right", "Yeah") while listening. If a bot halts every time a patron nods acoustically, conversation becomes impossible:

  • Energy Duration Debounce: Utterances under 180 ms matching affirmative lexicons are filtered out.
  • True Interruption: Sustained speech (> 180 ms across 3 consecutive frames) triggers an immediate cancellation token to abort egress audio in under 40 milliseconds.

Canceling a voice response requires three atomic operations:

  1. Aborting the active LLM async token generator.
  2. Sending a clear-buffer signal to the TTS stream.
  3. Commanding the LiveKit SFU to mute the egress WebRTC audio track.

4. The Markdown-to-Acoustic Regex AST Normalizer

Foundation language models are trained overwhelmingly on web text and markdown. When asked for menu prices or order summaries, models instinctively generate markdown tables, pipe characters (|), bold asterisks (**), bullet points (-), and currency symbols ($4.50).

If this raw text is passed to a neural text-to-speech engine, the voice reads out the formatting characters phonetically:

"Vertical bar caramel macchiato vertical bar dollar sign four point fifty vertical bar"

In our production engine, we built a dedicated regex AST acoustic parser that strips formatting tokens, converts numeric prices into natural spoken English ("four dollars and fifty cents"), and collapses multiline lists into spoken conversational clauses:

import re

def strip_markdown_for_speech(text: str) -> str:
    """Normalizes raw LLM markdown into clean, acoustic plain-text."""
    # Remove code blocks and inline code (using \x60 hex to avoid parser collision)
    text = re.sub(r"\x60{3}[\s\S]*?\x60{3}", "", text)  # strips code blocks
    text = re.sub(r"\x60[^\x60]+\x60", "", text)        # strips inline code
    text = re.sub(r"\|[^\n]+\|", "", text)
    text = re.sub(r"^#{1,6}\s+", "", text, flags=re.MULTILINE)
    text = re.sub(r"\*{1,3}([^*]+)\*{1,3}", r"\1", text)
    text = re.sub(r"\[([^\]]+)\]\([^\)]+\)", r"\1", text)
    text = re.sub(r"^\s*[-*+]\s+", "", text, flags=re.MULTILINE)
    # Convert currency ($4.50 -> 4 dollars and 50 cents)
    text = re.sub(r"\$(\d+)\.(\d{2})", r"\1 dollars and \2 cents", text)
    text = re.sub(r"\n{2,}", ". ", text)
    return re.sub(r"\s+", " ", text).strip()
Enter fullscreen mode Exit fullscreen mode

5. In-Flight Tool Execution & POS Price Immutability

Voice bots that only answer questions are toys. In an enterprise franchise operations platform, the voice agent must execute transactional mutations: checking real-time branch stock, validating staff authorization, applying discounts, and submitting orders to the kitchen.

Crucially, in multi-location food and beverage chains, menu prices fluctuate across branches and time. If an order record merely stores a foreign key to a menu item, subsequent menu price updates retroactively corrupt historical tax receipts and monthly franchise royalty calculations.

Our architecture enforces an immutable POS state machine:

  • When the voice agent places an order, it locks the price_at_purchase into the line item record.
  • The order lifecycle advances through strict state transitions: PENDING -> IN_PREPARATION -> COMPLETED.
  • The mutation automatically emits an asynchronous WebSocket event to the Kitchen Display System (KDS), updating the kitchen monitors in real time while the bot smoothly confirms the order with the patron.

6. Production Python Voice Pipeline Core

Below is the condensed asynchronous Python session coordinator with sentence frontier tokenization and cancellation token handling:

import re
import asyncio
from typing import AsyncGenerator
import httpx

class VoicePipelineSession:
    def __init__(self, groq_client, deepgram_key: str, elevenlabs_key: str):
        self.groq = groq_client
        self.deepgram_key = deepgram_key
        self.elevenlabs_key = elevenlabs_key
        self.is_speaking = False
        self.cancel_event = asyncio.Event()

    async def handle_barge_in(self):
        """Triggered when Silero VAD detects user speech exceeding 180ms debounce."""
        if self.is_speaking:
            self.cancel_event.set()
            self.is_speaking = False
            # Command WebRTC audio track to flush egress buffer
            await self.flush_audio_egress()

    async def stream_conversational_turn(self, user_transcript: str) -> AsyncGenerator[bytes, None]:
        self.cancel_event.clear()
        self.is_speaking = True
        sentence_buffer = ""

        # Stream tokens from Groq LPU (>500 t/s)
        response_stream = await self.groq.chat.completions.create(
            model="llama-3.3-70b-versatile",
            messages=[
                {"role": "system", "content": "You are Haji Cafe voice assistant. Be concise and natural."},
                {"role": "user", "content": user_transcript}
            ],
            stream=True
        )

        async for chunk in response_stream:
            if self.cancel_event.is_set():
                break
            delta = chunk.choices[0].delta.content or ""
            sentence_buffer += delta

            # Frontier chunking on sentence boundary
            if re.search(r"[.?!]\s*$", sentence_buffer):
                clean_sentence = strip_markdown_for_speech(sentence_buffer)
                if clean_sentence:
                    audio_bytes = await self.synthesize_speech_chunk(clean_sentence)
                    yield audio_bytes
                sentence_buffer = ""

        self.is_speaking = False
Enter fullscreen mode Exit fullscreen mode

Conclusion & Resources

Building enterprise voice AI is an engineering discipline of millisecond budgets, acoustic echo handling, and deterministic database state machines. When moving from toys to production, prioritize UDP over TCP, stream sentence frontiers, and decouple audio streaming from state mutations.

Top comments (0)