Comparing LLMs for Voice Assistants with Oxlo

#learnai #oxlo #ai

Voice assistants live or die by the LLM powering them. I built a small lab that transcribes a spoken question with Oxlo.ai, then routes it to four different models to see which one produces the fastest, most natural response for TTS. You can use the same harness to pick the right model for your own voice product.

What you'll need

Python 3.10+
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai
pip install sounddevice scipy for local microphone recording
A microphone, or a pre-recorded 16 kHz WAV file named query.wav

Step 1: Capture and transcribe audio

I keep the entire pipeline on Oxlo.ai so billing and routing stay simple. This block records five seconds of audio, then sends it to the Whisper endpoint.

from openai import OpenAI
import sounddevice as sd
from scipy.io.wavfile import write
import os

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=os.getenv("OXLO_API_KEY"))

def record_audio(filename="query.wav", duration=5, fs=16000):
    print("Recording...")
    audio = sd.rec(int(duration * fs), samplerate=fs, channels=1, dtype="int16")
    sd.wait()
    write(filename, fs, audio)
    print(f"Saved to {filename}")

def transcribe(filename="query.wav"):
    with open(filename, "rb") as f:
        transcript = client.audio.transcriptions.create(
            model="whisper-large-v3",
            file=f
        )
    return transcript.text

if __name__ == "__main__":
    record_audio()
    user_message = transcribe()
    print("Transcript:", user_message)

Step 2: Lock the system prompt

The system prompt is the control variable. I force every model to return plain text with no markdown, because asterisks and backticks break TTS flows.

SYSTEM_PROMPT = """You are a concise voice assistant.
Respond in plain text only.
Do not use markdown, bullet points, numbers, or code blocks.
Use natural, spoken language.
Keep every answer to one or two short sentences unless the user explicitly asks for detail."""

Step 3: Benchmark the candidate models

I test Llama 3.3 70B, Qwen 3 32B, Kimi K2.6, and DeepSeek V3.2. I collect latency and word count, because voice UIs need fast, speakable replies.

import time

MODELS = [
    "llama-3.3-70b",
    "qwen-3-32b",
    "kimi-k2.6",
    "deepseek-v3.2",
]

def benchmark(user_message):
    results = []
    for model in MODELS:
        start = time.perf_counter()
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": user_message},
            ],
        )
        latency = time.perf_counter() - start
        text = response.choices[0].message.content
        results.append({
            "model": model,
            "latency_ms": round(latency * 1000, 1),
            "words": len(text.split()),
            "text": text,
        })
    return results

Step 4: Clean and rank outputs

Some models still emit formatting artifacts. I strip them and score by a simple heuristic that rewards low latency and brevity.

import re

def clean_for_tts(text):
    text = re.sub(r"\*+|`+", "", text)
    text = re.sub(r"\s+", " ", text)
    return text.strip()

def pick_best(results):
    for r in results:
        r["clean_text"] = clean_for_tts(r["text"])
        r["score"] = r["latency_ms"] + (r["words"] * 50)
    results.sort(key=lambda x: x["score"])
    return results[0]

Step 5: Run the full pipeline

This ties transcription, benchmarking, and cleanup into one script that prints a side-by-side comparison.

def run_comparison():
    record_audio()
    user_message = transcribe()
    print(f"\nTranscript: {user_message}\n")

    results = benchmark(user_message)
    for r in results:
        print(f"Model: {r['model']}")
        print(f"Latency: {r['latency_ms']} ms")
        print(f"Words: {r['words']}")
        print(f"Raw: {r['text']}")
        print(f"Clean: {r['clean_text']}")
        print("-" * 40)

    best = pick_best(results)
    print(f"\nWinner: {best['model']}")
    print(f"Final response: {best['clean_text']}")

if __name__ == "__main__":
    run_comparison()

Run it

Save the script as voice_assistant_lab.py, set your key, and run it. Here is what the output looks like for a simple weather question.

$ export OXLO_API_KEY="YOUR_OXLO_API_KEY"
$ python voice_assistant_lab.py
Recording...
Saved to query.wav

Transcript: What is the weather like on Mars?

Model: deepseek-v3.2
Latency: 340 ms
Words: 14
Raw: On Mars, the weather is cold and dry, with temperatures averaging around minus 80 degrees Fahrenheit.
Clean: On Mars, the weather is cold and dry, with temperatures averaging around minus 80 degrees Fahrenheit.
----------------------------------------
Model: llama-3.3-70b
Latency: 290 ms
Words: 12
Raw: Mars is generally very cold, with average temperatures around minus 60 degrees Celsius.
Clean: Mars is generally very cold, with average temperatures around minus 60 degrees Celsius.
----------------------------------------
Model: qwen-3-32b
Latency: 310 ms
Words: 11
Raw: The weather on Mars is frigid and dusty, typically around minus 80 degrees Fahrenheit.
Clean: The weather on Mars is frigid and dusty, typically around minus 80 degrees Fahrenheit.
----------------------------------------
Model: kimi-k2.6
Latency: 280 ms
Words: 13
Raw: Mars has a cold, desert climate with temperatures often dropping to minus 80 degrees Fahrenheit.
Clean: Mars has a cold, desert climate with temperatures often dropping to minus 80 degrees Fahrenheit.
----------------------------------------

Winner: llama-3.3-70b
Final response: Mars is generally very cold, with average temperatures around minus 60 degrees Celsius.

Wrap-up and next steps

You now have a reproducible way to compare voice assistant candidates on Oxlo.ai without switching providers or parsing different SDKs. Two concrete next steps: expose this harness behind a FastAPI endpoint so a mobile app can stream audio to it, or add function calling so the assistant can check live APIs before it speaks.

DEV Community