The Role of LLM in Audio Analysis: Opportunities and Challenges

#learnai #oxlo #ai

Most teams sit on hundreds of hours of recorded calls that never get reviewed because listening is slow. In this tutorial, I will show you how to build a lightweight pipeline that transcribes a meeting recording and extracts action items, key decisions, and sentiment using an LLM. The entire stack runs on Oxlo.ai, from the Whisper transcription endpoint to the chat model that structures the output.

What you'll need

Python 3.10 or higher
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai
A sample audio file named meeting.mp3 in your working directory

Step 1: Transcribe audio with Whisper

Oxlo.ai hosts Whisper Large v3 with no cold starts, so the first request is as fast as the hundredth. We point the OpenAI client at Oxlo.ai's audio endpoint and read a local file.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

def transcribe_audio(path: str) -> str:
    with open(path, "rb") as audio_file:
        transcript = client.audio.transcriptions.create(
            model="whisper-large-v3",
            file=audio_file,
            response_format="text"
        )
    return transcript

raw_text = transcribe_audio("meeting.mp3")
print(f"Transcript length: {len(raw_text)} characters")

Step 2: Define the analysis system prompt

We want structured JSON back, so we give the model a strict system prompt and let it reason through the transcript. I keep the prompt versioned in code so I can tune it later without touching the business logic.

SYSTEM_PROMPT = """You are an audio analysis assistant. Your job is to read a meeting transcript and produce a structured analysis.

Output strictly valid JSON with these keys:
- summary: a concise 2-sentence summary of the discussion
- sentiment: an object with label (positive, neutral, or negative) and justification (1 sentence)
- decisions: an array of concrete decisions made
- action_items: an array of objects, each with owner (name or Unknown), task (string), and deadline (date or Unknown)
- key_topics: an array of main topics discussed

Rules:
- If the transcript is empty or unreadable, return empty arrays and set sentiment label to "neutral".
- Do not add markdown code fences around the JSON.
- Use double quotes for all strings.
"""

Step 3: Analyze the transcript with Llama 3.3 70B

Now we feed the transcript into Llama 3.3 70B, Oxlo.ai's general-purpose flagship. Because Oxlo.ai uses flat per-request pricing, a long transcript does not inflate the cost. See https://oxlo.ai/pricing for details.

import json

def analyze_transcript(transcript: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Analyze this transcript:\n\n{transcript}"}
        ],
        temperature=0.2
    )
    
    content = response.choices[0].message.content
    return json.loads(content)

analysis = analyze_transcript(raw_text)
print(json.dumps(analysis, indent=2))

Step 4: Wrap the pipeline into a single function

I like to expose one clean entrypoint that handles the file path and returns the final analysis. This makes it easy to drop into a FastAPI route or a CLI later.

def process_meeting_audio(path: str) -> dict:
    print("Transcribing audio...")
    transcript = transcribe_audio(path)
    
    print("Running LLM analysis...")
    result = analyze_transcript(transcript)
    
    result["metadata"] = {
        "audio_file": path,
        "transcript_length": len(transcript)
    }
    return result

if __name__ == "__main__":
    output = process_meeting_audio("meeting.mp3")
    print(json.dumps(output, indent=2))

Run it

Assuming your environment variable OXLO_API_KEY is set and meeting.mp3 is present, run python analyze_meeting.py. You should see output similar to this.

$ python analyze_meeting.py
Transcribing audio...
Running LLM analysis...
{
  "summary": "The team aligned on Q3 roadmap priorities and agreed to delay the mobile redesign by two weeks.",
  "sentiment": {
    "label": "neutral",
    "justification": "The discussion was pragmatic with no strong positive or negative language."
  },
  "decisions": [
    "Delay mobile redesign to August 15",
    "Allocate two additional engineers to the API gateway project"
  ],
  "action_items": [
    {"owner": "Sarah", "task": "Update Jira timeline", "deadline": "2024-07-20"},
    {"owner": "Unknown", "task": "Schedule load-testing session", "deadline": "Unknown"}
  ],
  "key_topics": [
    "Q3 roadmap",
    "mobile redesign",
    "API gateway",
    "resource allocation"
  ],
  "metadata": {
    "audio_file": "meeting.mp3",
    "transcript_length": 3420
  }
}

Wrap-up

You now have a working audio analysis pipeline that turns raw meeting recordings into structured data. If you want to take this further, add speaker diarization with pyannote.audio before transcription so the LLM can attribute action items to specific people. You could also wrap the script in a FastAPI handler and use Oxlo.ai's JSON mode to lock the output schema even tighter for downstream tools.

DEV Community