Building Voice Controlled Interfaces with LLM

Voice-controlled interfaces are moving from novelty to core infrastructure. Whether you are building a hands-free coding assistant, a voice-native CRM, or an embedded agent, the architecture is converging on a three-stage pipeline: speech-to-text, language model reasoning, and text-to-speech. This guide shows how to assemble that pipeline using modern APIs, with Oxlo.ai as the inference backbone.

The Voice Pipeline Architecture

A production voice interface typically executes three steps in sequence. First, an audio chunk is transcribed into text. Second, that text is processed by an LLM to generate a response. Third, the response is synthesized back into audio. Each step demands a different model family, but they can all be served through a single provider if the platform supports multimodal endpoints under one API schema.

Oxlo.ai offers exactly that: Whisper for transcription, dozens of LLMs for reasoning, and Kokoro for lightweight text-to-speech, all behind a fully OpenAI-compatible API that serves as a drop-in replacement for the standard SDK. Because Oxlo.ai uses request-based pricing, the cost of a single voice turn stays flat regardless of how long the user spoke or how much context the LLM receives. For agentic voice workflows that carry large system prompts or conversation history, this predictability is a significant advantage over token-based billing.

Transcribing Speech with Whisper

Start by capturing audio from the user. For web applications, the MediaRecorder API produces Blob objects. For Python services, PyAudio can stream raw PCM to a buffer. Once you have a file, send it to the transcriptions endpoint.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

with open("user_request.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3",
        file=audio_file
    )

user_text = transcript.text

Whisper Large v3 is available on Oxlo.ai with no cold starts, so the first request after idle time returns at the same speed as subsequent ones. If latency is critical, Whisper Turbo offers a faster alternative on the same endpoint.

Reasoning with an LLM

With the transcript in hand, pass it to a chat model. Voice agents often require fast turnaround, so a capable general-purpose model works well here. Llama 3.3 70B is a strong default, while Qwen 3 32B excels at multilingual input and tool use if your agent needs to call external APIs during the conversation.

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a concise voice assistant."},
        {"role": "user", "content": user_text}
    ],
    stream=True
)

reply_chunks = []
for chunk in response:
    if chunk.choices[0].delta.content:
        reply_chunks.append(chunk.choices[0].delta.content)

assistant_text = "".join(reply_chunks)

Streaming is essential for voice interfaces. Even though the full response must be collected before TTS can begin, streaming lets you measure time-to-first-token and abort early if the user interrupts. Oxlo.ai supports streaming across its LLM lineup, and the OpenAI SDK handles this without custom client code.

Synthesizing Voice with Kokoro

The final stage converts the assistant text into speech. Kokoro 82M is a compact, high-quality text-to-speech model available through Oxlo.ai's audio/speech endpoint. Its small footprint means synthesis is fast, which keeps end-to-end latency low.

speech_response = client.audio.speech.create(
    model="kokoro-82m",
    voice="default",
    input=assistant_text
)

with open("assistant_response.wav", "wb") as f:
    f.write(speech_response.content)

Because the endpoint is OpenAI-compatible, you can swap voices or formats by changing the request parameters, with no vendor-specific SDK to install.

Wiring It All Together

A minimal Python service that ties the three stages together looks like this:

import openai

class VoiceAgent:
    def __init__(self, api_key: str):
        self.client = openai.OpenAI(
            base_url="https://api.oxlo.ai/v1",
            api_key=api_key
        )

    def process_turn(self, audio_path: str) -> bytes:
        # Step 1: Transcribe
        with open(audio_path, "rb") as f:
            transcript = self.client.audio.transcriptions.create(
                model="whisper-large-v3",
                file=f
            )

        # Step 2: Reason
        chat = self.client.chat.completions.create(
            model="llama-3.3-70b",
            messages=[{"role": "user", "content": transcript.text}],
            stream=True
        )

        text = "".join(
            chunk.choices[0].delta.content 
            for chunk in chat 
            if chunk.choices[0].delta.content
        )

        # Step 3: Speak
        speech = self.client.audio.speech.create(
            model="kokoro-82m",
            voice="default",
            input=text
        )

        return speech.content

This class encapsulates a single voice turn. In production, you will want to add interruption handling, barge-in detection, and a websocket layer to stream audio chunks rather than writing files to disk.

Cost and Latency Considerations

Voice interfaces generate unpredictable input lengths. A user might ask a one-word question or dictate five minutes of context. Under token-based pricing, the LLM portion of the bill scales with every syllable. Under Oxlo.ai's request-based model, each API call incurs one flat fee, so the cost of a voice turn is bounded to three requests regardless of transcript length or context window size. For long-context agents that append full conversation history to every prompt, Oxlo.ai can be 10-100x cheaper than token-based providers.

Latency is the other critical variable. The pipeline is only as fast as its slowest stage. Whisper Large v3 on Oxlo.ai starts immediately with no cold-start penalty, and Kokoro 82M synthesizes in milliseconds. The dominant factor is usually the LLM's time-to-first-token, which streaming helps mask. For latency-sensitive deployments, you can also cache system prompts at the application layer to minimize per-request overhead.

Conclusion

Building a voice-controlled interface is fundamentally an exercise in pipeline orchestration. You need reliable transcription, capable reasoning, and fast synthesis, all reachable through a consistent API. Oxlo.ai provides all three under one roof, with OpenAI SDK compatibility that lets you prototype in minutes. The request-based pricing model removes the uncertainty of token counts from variable-length speech, giving voice agents a cost structure that matches their interaction model. Developers can even prototype on the free tier, which includes 60 requests per day and a 7-day full-access trial. If you are evaluating infrastructure for your next voice project, start at the Oxlo.ai pricing page and run the three-stage pipeline above against your own audio samples.