Using LLM for Speech Recognition and Voice Control

#aiinfrastructure #oxlo #ai

Speech recognition and voice control systems are moving beyond traditional command-and-response patterns. By routing audio through a large language model, you can capture intent from noisy transcripts, maintain multi-turn conversational context, and trigger precise function calls. The result is a voice interface that understands nuance rather than matching rigid keywords. Oxlo.ai provides the necessary inference stack for this pipeline, combining Whisper transcription, open-source LLM reasoning, and text-to-speech synthesis behind a single OpenAI-compatible API with flat per-request pricing.

The Speech-to-Intent Pipeline

A production voice assistant typically runs three stages. First, an automatic speech recognition model converts user audio to text. Second, an LLM parses the transcript for intent, extracts entities, and decides whether to invoke external tools. Third, a text-to-speech model delivers the response back to the user. Oxlo.ai covers all three stages through its audio and chat endpoints, so you can keep the entire loop on one platform.

Voice transcripts, especially from meetings or continuous dictation, can grow long quickly. Because Oxlo.ai uses request-based pricing rather than token-based metering, the cost of sending a lengthy transcript or a detailed system prompt does not scale with input length. For teams building persistent voice agents, this flat cost per API request removes the pricing penalty that token-based providers apply to long context. For long-context transcription follow-ups or agentic voice workflows, this can be 10-100x cheaper than token-based alternatives.

Step 1: Transcription with Whisper

Oxlo.ai hosts Whisper Large v3, Whisper Turbo, and Whisper Medium through the audio/transcriptions endpoint. These models handle noisy audio, multiple speakers, and varied accents. Because the endpoint is fully OpenAI SDK compatible, you can switch to Oxlo.ai by changing the base URL.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_API_KEY"
)

with open("command.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3",
        file=audio_file
    )

print(transcript.text)

Step 2: Reasoning and Tool Use

Once you have the transcript, route it to an LLM for reasoning. Oxlo.ai offers models such as Llama 3.3 70B for general-purpose parsing and Qwen 3 32B for multilingual agent workflows. You can attach function schemas so the model emits structured tool calls instead of free text, which makes voice control deterministic and safe.

Because Oxlo.ai does not charge by the token, you can include large tool definitions, extended system prompts, or long conversation histories without inflating cost. This is a sharp contrast to token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, where every additional character in the prompt increases the bill.

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a home automation assistant. Parse user commands and call the appropriate tool."},
        {"role": "user", "content": transcript.text}
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "set_temperature",
                "description": "Set the thermostat temperature",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "room": {"type": "string", "enum": ["living room", "bedroom"]},
                        "temp": {"type": "number"}
                    },
                    "required": ["room", "temp"]
                }
            }
        }
    ],
    tool_choice="auto"
)

if response.choices[0].message.tool_calls:
    tool_call = response.choices[0].message.tool_calls[0]
    print("Tool called:", tool_call.function.name)
    print("Arguments:", tool_call.function.arguments)

Step 3: Voice Synthesis

After the LLM decides on a response or confirms an action, you can stream the reply through a text-to-speech model. Oxlo.ai hosts Kokoro 82M, a lightweight but natural sounding TTS model, via the audio/speech endpoint.

speech = client.audio.speech.create(
    model="kokoro-82m",
    voice="af",
    input="Temperature set to 22 degrees in the living room."
)

with open("response.mp3", "wb") as f:
    f.write(speech.content)

Why Oxlo.ai for Voice Workloads

Voice agents create unique inference patterns: long audio transcripts, multi-turn session history, and frequent tool loops. Oxlo.ai is built to handle these patterns efficiently.

Flat per-request pricing: Unlike token-based competitors such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, Oxlo.ai charges one flat cost per API request regardless of prompt length. For long-context transcription follow-ups or agentic voice workflows, this can be 10-100x cheaper.
No cold starts on popular models: Voice interfaces demand consistent latency. Oxlo.ai keeps frequently used models warm, so you do not pay a startup penalty on the first request after idle time.
OpenAI SDK drop-in replacement: The base URL https://api.oxlo.ai/v1 and full OpenAI API compatibility mean you can reuse existing transcription, chat, and speech client code without rewriting your transport layer.
Unified model catalog: With 45+ models across 7 categories, you can optimize each stage independently. Use Whisper for audio, a fast code model for structured command parsing, and Kokoro for speech, all under one account.

Pricing details are available at https://oxlo.ai/pricing.

<h2 id

DEV Community