Integrating Oxlo with LLM for Voice Assistants

#aiinfrastructure #oxlo #ai

Voice assistants are only as good as the loop that connects hearing, reasoning, and speaking. A production pipeline typically stitches together three separate inference calls: speech-to-text transcription, LLM reasoning with tool access, and text-to-speech synthesis. When that loop runs on token-based billing, every filler word, system prompt, and turn of conversation inflates the bill. Oxlo.ai removes that uncertainty with flat per-request pricing and a fully OpenAI-compatible stack that includes audio, reasoning, and vision models.

The Voice Assistant Stack

A production voice assistant runs a tight loop. Audio capture feeds a speech-to-text model. The resulting text hits an LLM alongside conversation history and tool definitions. The LLM response drives a text-to-speech model and optional client actions. Latency and cost at every stage determine whether the product feels alive or abandoned.

Why Request-Based Pricing Fits Voice

Voice interactions are rarely single-turn. Users pause, correct themselves, or invoke tools. Under token-based billing, every filler word, system prompt, and history entry adds cost. Because Oxlo.ai charges one flat rate per API request, your cost per conversation stays predictable even as context windows grow. For agentic voice assistants that maintain long sessions or loop through tool calls, this can reduce inference spend significantly compared to token-based providers. See exact rates on the Oxlo.ai pricing page.

Setting Up the Pipeline

Oxlo.ai exposes fully OpenAI-compatible endpoints. You can point an existing Python voice stack at https://api.oxlo.ai/v1 without rewriting client logic.

Speech-to-text uses Whisper Large v3, Whisper Turbo, or Whisper Medium. Text generation pulls from Llama 3.3 70B for general dialogue, Qwen 3 32B for multilingual workflows, or DeepSeek R1 671B MoE when the assistant must reason through complex user requests. Text-to-speech uses Kokoro 82M for fast, lightweight synthesis.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

# 1. Transcribe user audio
with open("user_query.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3",
        file=audio_file
    )

# 2. Generate response with tool use enabled
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a helpful voice assistant."},
        {"role": "user", "content": transcript.text}
    ],
    tools=[{
        "type": "function",
        "function": {
            "name": "check_calendar",
            "description": "Check the user's calendar",
            "parameters": {
                "type": "object",
                "properties": {
                    "date": {"type": "string"}
                }
            }
        }
    }],
    stream=True
)

# Collect the streamed text
reply_text = ""
for chunk in response:
    if chunk.choices[0].delta.content:
        reply_text += chunk.choices[0].delta.content

# 3. Synthesize speech
speech = client.audio.speech.create(
    model="kokoro-82m",
    voice="af",
    input=reply_text
)

with open("response.mp3", "wb") as f:
    f.write(speech.content)

Choosing Models for Each Stage

STT: Whisper Large v3 offers the highest accuracy for noisy environments. Whisper Turbo trades a small amount of accuracy for lower latency, which matters when you are streaming partial results to the LLM.

Reasoning: For general assistance, Llama 3.3 70B provides strong instruction following and low latency. If your assistant operates across languages, Qwen 3 32B handles multilingual reasoning and agent workflows. When users ask the assistant to write or debug code via voice, DeepSeek R1 671B MoE or DeepSeek V4 Flash deliver deep reasoning without the per-token penalty of running a large parameter model on metered billing.

TTS: Kokoro 82M is a compact text-to-speech model that starts instantly. Because Oxlo.ai has no cold starts on popular models, the first request after silence returns audio in milliseconds, not seconds.

Reducing Latency with Streaming and Tool Use

Perceived latency in voice is a product of three variables: time-to-first-transcript, time-to-first-token from the LLM, and time-to-first-audio byte. Oxlo.ai supports streaming responses for chat completions, so you can begin buffering text for the TTS engine before the LLM finishes thinking.

Function calling lets the assistant act rather than talk. A voice assistant that needs to check the weather or modify a calendar can fire a tool call, receive the result, and generate a concise confirmation. Because Oxlo.ai charges per request, a multi-step tool loop costs the same whether it involves one tool call or three, provided each step is a single API request.

Getting Started

You can prototype this pipeline on the Oxlo.ai free tier, which includes 60 requests per day and access to more than 16 models, including a 7-day full-access trial. When you move to production, the Pro and Premium plans offer fixed daily request allotments that make voice assistant margins predictable. For teams running dedicated workloads, the Enterprise plan provides custom unlimited access with dedicated GPUs.

Point your OpenAI SDK client to https://api.oxlo.ai/v1 and swap in Whisper, Llama 3.3 70B, and Kokoro 82M to see how flat per-request pricing affects your voice stack economics. Full details are available at the Oxlo.ai pricing page.

DEV Community