Building Chatbots with LLM, NLU, and Speech Recognition

#learnai #oxlo #ai

We will build a voice-first support triage bot that ingests customer audio, transcribes it with Whisper, extracts intent via structured LLM inference, and drafts a contextual reply. It is useful for SaaS teams that want to automate first-line support without gluing together separate providers for speech, NLU, and dialogue. Oxlo.ai hosts Whisper, Qwen, and Llama behind one OpenAI-compatible endpoint, so the entire pipeline runs against a single base URL.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai
A sample audio file named support_request.wav for testing

Step 1: Configure the client

Instantiate the SDK once and point it at Oxlo.ai. I pull the key from the environment so it never sits in source control.

import os
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY"),
)

Step 2: Transcribe speech

Send the audio to Oxlo.ai's transcription endpoint. I use whisper-large-v3 because it handles noisy microphone audio and accents well.

def transcribe(audio_path: str) -> str:
    with open(audio_path, "rb") as audio_file:
        result = client.audio.transcriptions.create(
            model="whisper-large-v3",
            file=audio_file,
        )
    return result.text

user_input = transcribe("support_request.wav")
print("Transcript:", user_input)

Step 3: Extract intent and entities

Feed the raw transcript to qwen-3-32b with a strict system prompt that forces JSON. I enable JSON mode so the output stays machine readable without regex hacks.

INTENT_PROMPT = """You are an NLU engine.
Read the support message and return a JSON object with exactly these keys:
- intent: one of [billing, technical, sales, unknown]
- product: the product mentioned, or null
- urgency: one of [low, medium, high]
- summary: a 10-word summary of the issue
Respond with valid JSON only. No markdown, no explanation."""

def extract_entities(transcript: str) -> dict:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": INTENT_PROMPT},
            {"role": "user", "content": transcript},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

entities = extract_entities(user_input)
print("Entities:", entities)

Step 4: Draft the agent response

Pass the original transcript and parsed entities to llama-3.3-70b. The system prompt below adapts tone based on urgency and intent, keeping replies short so they work well in voice or chat windows.

SYSTEM_PROMPT = """You are a concise technical support agent for a SaaS platform.
Use the provided intent and entity JSON to tailor your answer.
If urgency is high, acknowledge it and offer an escalation path.
If the intent is billing, keep the tone calm and direct.
If technical, ask one clarifying question maximum.
Always respond in two sentences or fewer."""

def draft_response(transcript: str, entities: dict) -> str:
    context = f"User said: {transcript}\nExtracted entities: {json.dumps(entities)}"
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": context},
        ],
    )
    return response.choices[0].message.content

reply = draft_response(user_input, entities)
print("Agent:", reply)

Run it

Tie the stages together and execute the full pipeline against your sample audio.

if __name__ == "__main__":
    audio_file = "support_request.wav"

    transcript = transcribe(audio_file)
    entities = extract_entities(transcript)
    reply = draft_response(transcript, entities)

    print("\n--- Final Output ---")
    print(f"Intent:  {entities['intent']}")
    print(f"Urgency: {entities['urgency']}")
    print(f"Reply:   {reply}")

Expected output:

Transcript: My dashboard won't load since this morning and I have a demo in an hour.
Entities: {'intent': 'technical', 'product': 'dashboard', 'urgency': 'high', 'summary': 'Dashboard not loading before demo'}
Agent: I see the dashboard is not loading and your demo is soon. Have you tried a hard refresh or clearing your browser cache?

--- Final Output ---
Intent:  technical
Urgency: high
Reply:   I see the dashboard is not loading and your demo is soon. Have you tried a hard refresh or clearing your browser cache?

Next steps

Wrap the pipeline in a FastAPI endpoint so your frontend can stream user voice uploads directly to the bot. Alternatively, push the final reply through Oxlo.ai's Kokoro TTS endpoint to return spoken audio instead of text, giving you a true voice-in, voice-out agent.

DEV Community