Integrating LLM with Speech Recognition Models

We are building a voice memo pipeline that transcribes audio with Whisper and extracts structured notes using an LLM. This is useful for anyone who needs to turn long meeting recordings into actionable summaries without manual typing. Oxlo.ai makes this cheap and predictable, because a flat per-request price means a two-hour recording costs the same as a thirty-second clip, and the follow-up LLM call does not scale with transcript length like it does on token-based providers. See https://oxlo.ai/pricing for plan details.

What you'll need

• An Oxlo.ai API key from https://portal.oxlo.ai
• Python 3.10 or newer
• The OpenAI SDK: pip install openai
• A sample audio file named memo.mp3 (WAV works too)

Oxlo.ai hosts both Whisper and open-source LLMs on the same API, so we only need one key and one client.

Step 1: Configure the client

Oxlo.ai is fully OpenAI SDK compatible. We point the client at the Oxlo.ai base URL and set the API key.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Step 2: Transcribe audio with Whisper

We send the audio file to Oxlo.ai's audio/transcriptions endpoint using whisper-large-v3. Because Oxlo.ai uses request-based pricing, this single request covers the entire file no matter how long the recording is.

with open("memo.mp3", "rb") as audio_file:
    transcription = client.audio.transcriptions.create(
        model="whisper-large-v3",
        file=audio_file,
    )

raw_text = transcription.text
print("Transcript preview:", raw_text[:200])

Step 3: Define the extraction prompt

The system prompt tells the LLM to act as a meeting assistant and return strict JSON. Keeping the instructions explicit reduces parsing errors later.

SYSTEM_PROMPT = """You are a meeting assistant. Read the transcript and produce a JSON object with exactly these keys:
- summary: a concise paragraph of the discussion
- action_items: a list of strings, each a specific task with an owner if mentioned
- key_decisions: a list of strings describing decisions made
- sentiment: one of positive, neutral, or negative

Do not include markdown fences or commentary. Output only valid JSON."""

Step 4: Generate structured notes

We pass the transcript to llama-3.3-70b and enable JSON mode. On Oxlo.ai, this LLM call is one flat request, so a fifty-thousand-character transcript costs the same as a short paragraph. There are no cold starts, so the response comes back immediately.

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": raw_text},
    ],
    response_format={"type": "json_object"},
    temperature=0.2,
)

Step 5: Assemble the full script

Here is the complete agent in one file. It transcribes the memo and prints the structured JSON result.

from openai import OpenAI
import json

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

SYSTEM_PROMPT = """You are a meeting assistant. Read the transcript and produce a JSON object with exactly these keys:
- summary: a concise paragraph of the discussion
- action_items: a list of strings, each a specific task with an owner if mentioned
- key_decisions: a list of strings describing decisions made
- sentiment: one of positive, neutral, or negative

Do not include markdown fences or commentary. Output only valid JSON."""

def process_memo(audio_path: str):
    with open(audio_path, "rb") as audio_file:
        transcription = client.audio.transcriptions.create(
            model="whisper-large-v3",
            file=audio_file,
        )
    raw_text = transcription.text

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": raw_text},
        ],
        response_format={"type": "json_object"},
        temperature=0.2,
    )

    return json.loads(response.choices[0].message.content)

if __name__ == "__main__":
    result = process_memo("memo.mp3")
    print(json.dumps(result, indent=2))

Run it

Save the script as memo_agent.py, replace YOUR_OXLO_API_KEY, and run it with your audio file in the same directory.

python memo_agent.py

Example output:

{
  "summary": "The team discussed Q3 roadmap priorities and agreed to migrate the transcription pipeline to Oxlo.ai for flat-rate pricing.",
  "action_items": [
    "Alice to finalize the API schema by Friday",
    "Bob to provision the staging environment"
  ],
  "key_decisions": [
    "Migrate the transcription pipeline to Oxlo.ai",
    "Deprecate the legacy summarization service"
  ],
  "sentiment": "positive"
}

Next steps

Add speaker labels by chunking the audio or prompting the LLM to identify distinct voices in the transcript. You could also wire this into a Slack bot using Oxlo.ai's function calling support, so the agent posts action items directly instead of printing to stdout.