Integrating LLM with Speech Recognition and Synthesis Models

Voice agents and conversational AI systems require a coordinated pipeline of speech recognition, language reasoning, and speech synthesis. Orchestrating these stages across separate providers introduces unnecessary latency, authentication overhead, and pricing fragmentation. Oxlo.ai unifies all three capabilities, from Whisper Large v3 to Kokoro 82M text-to-speech and flagship LLMs such as Kimi K2.6 and Llama 3.3 70B, behind a single OpenAI-compatible API with flat per-request pricing.

Pipeline Architecture

A production voice agent typically follows a three-stage loop:

• Audio input is transcribed by an ASR model.
• The transcript, plus conversation history and system instructions, is processed by an LLM.
• The generated text is streamed to a TTS model for audio output.

Running these on disparate platforms means managing multiple API keys, inconsistent latency, and token-cost accumulation across services. Oxlo.ai collapses this stack into one endpoint base URL, https://api.oxlo.ai/v1, with no cold starts on popular models.

Speech-to-Text with Whisper

Oxlo.ai offers Whisper Large v3, Turbo, and Medium through the audio/transcriptions endpoint. Because the platform is fully OpenAI SDK compatible, you can use the same Python client you already know.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

with open("input.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3-turbo",
        file=audio_file,
        response_format="text"
    )

print(transcript)

Whisper Large v3 Turbo is ideal for real-time applications where latency matters. The transcription returns plain text or JSON, which feeds directly into the chat completion stage.

Language Model Reasoning

Once you have the transcript, you route it to an LLM. For general-purpose dialogue, Llama 3.3 70B works well. For advanced reasoning, agentic coding, or vision-enabled turns, Kimi K2.6 offers a 131K context window. Oxlo.ai supports function calling, JSON mode, multi-turn conversations, and streaming responses, so your voice agent can query APIs, return structured data, or begin synthesizing audio before the full response completes.

response = client.chat.completions.create(
    model="kimi-k2-6",
    messages=[
        {"role": "system", "content": "You are a concise voice assistant. Keep responses under two sentences."},
        {"role": "user", "content": transcript}
    ],
    stream=False
)

reply_text = response.choices[0].message.content

Because Oxlo.ai charges per request, not per token, stuffing the context window with long system prompts or extensive conversation history does not increase the cost of this API call. This is a structural advantage over token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale.

Text-to-Speech with Kokoro

For the final stage, Oxlo.ai hosts Kokoro 82M text-to-speech through the audio/speech endpoint. It is lightweight and fast, making it suitable for low-latency delivery to users.

speech_response = client.audio.speech.create(
    model="kokoro-82m",
    voice="voice_id",  # replace with your preferred Kokoro voice
    input=reply_text,
    response_format="mp3"
)

with open("output.mp3", "wb") as f:
    f.write(speech_response.content)

End-to-End Example

Here is a minimal but complete Python script that ties the three stages together. It assumes you have the OpenAI SDK installed and an Oxlo.ai API key.

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=os.getenv("OXLO_API_KEY"))

def voice_agent(audio_path: str, output_path: str = "response.mp3"):
    # 1. Transcribe
    with open(audio_path, "rb") as f:
        transcription = client.audio.transcriptions.create(
            model="whisper-large-v3-turbo",
            file=f,
            response_format="text"
        )

    # 2. Reason
    chat = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": "You are a helpful voice assistant."},
            {"role": "user", "content": transcription}
        ]
    )
    text = chat.choices[0].message.content

    # 3. Synthesize
    speech = client.audio.speech.create(
        model="kokoro-82m",
        voice="voice_id",  # replace with your preferred Kokoro voice
        input=text,
        response_format="mp3"
    )

    with open(output_path, "wb") as f:
        f.write(speech.content)

    return text

if __name__ == "__main__":
    reply = voice_agent("input.wav")
    print("Assistant said:", reply)

This single-client approach eliminates the need to juggle separate SDKs for speech and language models.

Cost and Scaling Considerations

Voice agents are inherently stateful. A production system may send thousands of tokens of conversation history and tool definitions on every turn. On token-based