Integrating LLM with Speech Recognition: Best Practices and Examples

Building applications that combine speech recognition with large language models means orchestrating at least two distinct inference steps: audio transcription and text generation. The way you connect these steps determines your end-to-end latency, your total cost, and how easily you can maintain the pipeline in production. A typical pattern is to send audio to a Whisper model, feed the resulting transcript into an LLM for summarization or structured extraction, then optionally synthesize a response with text-to-speech. Oxlo.ai provides all three capabilities, transcription, chat, and speech synthesis, behind a single OpenAI-compatible API with request-based pricing that stays predictable even when your transcripts grow long.

Architecture Overview

A typical voice-to-insight pipeline has three stages. First, an audio transcription model converts raw speech to text. Second, an LLM processes that text for tasks like summarization, entity extraction, or question answering. Third, an optional text-to-speech model returns a spoken response.

Running this on separate services means managing multiple SDKs, authentication schemes, and pricing models. Oxlo.ai consolidates these into one endpoint structure. You can call /v1/audio/transcriptions with Whisper Large v3, then pass the text to /v1/chat/completions with Llama 3.3 70B or DeepSeek V3.2, using the same API key and base URL. This reduces integration surface area and lets you optimize the entire flow as a single unit.

Choosing Your Models

For transcription accuracy, Oxlo.ai offers Whisper Large v3, Whisper Turbo, and Whisper Medium. Large v3 is the right choice when you need precise diarization or noisy audio handling. Turbo trades a small amount of accuracy for significantly faster turnaround on clean audio.

For the LLM stage, select a model that matches your output complexity. Llama 3.3 70B works well for general summarization and conversational responses. If you need reasoning over long transcripts, DeepSeek R1 671B MoE or Kimi K2.6 with 131K context windows handle extended input without losing track of earlier segments. For coding or agentic workflows triggered by voice commands, Qwen 3 32B and Minimax M2.5 are solid options.

Because Oxlo.ai uses flat per-request pricing, the cost of sending a long transcript to a large-context model does not scale with token count. This makes it practical to feed entire meeting recordings or lengthy dictations into an LLM in a single shot rather than chunking the text to save money. See https://oxlo.ai/pricing for current plan details.

Code Example: Transcription to Structured Output

The following Python example uses the OpenAI SDK configured for Oxlo.ai. It transcribes an audio file, then asks an LLM to extract action items as JSON.

import openai
import json

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

# Step 1: Transcribe audio
with open("meeting.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3",
        file=audio_file,
        response_format="text"
    )

# Step 2: Extract structured data with an LLM
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {
            "role": "system",
            "content": "You are an assistant that extracts action items from meeting transcripts. Respond only with valid JSON."
        },
        {
            "role": "user",
            "content": f"Extract action items from this transcript:\n\n{transcript}"
        }
    ],
    response_format={"type": "json_object"}
)

action_items = json.loads(response.choices[0].message.content)
print(action_items)

Handling Long-Form Audio

Whisper models have a fixed context window, typically around 30 seconds of audio per internal frame, but the API can accept files much longer than that. The service handles chunking internally, but you should still verify that the returned transcript maintains speaker consistency across boundaries.

If the resulting transcript exceeds the LLM's context window, you have two options. You can chunk the text and process each section independently, or you can switch to a model with a larger context. On Oxlo.ai, Kimi K2.6 supports 131K tokens, and DeepSeek V4 Flash handles up to 1M tokens. With request-based pricing, choosing the larger context model does not inflate your cost proportional to input length, so you can often skip complex chunking logic and send the full transcript directly.

Latency and Cost Optimization

Speech-to-text pipelines often suffer from double billing: you pay for transcription tokens, then pay again for LLM input tokens, with both costs rising as the audio duration increases. Token-based providers charge more for longer transcripts and longer prompts. Oxlo.ai charges one flat rate per API request. For agentic or long-context workloads, this can reduce costs significantly because a 50,000-token transcript costs the same to process as a 500-token transcript.

To minimize latency, run transcription and LLM inference sequentially but keep both on the same provider to avoid cross-region network hops. If you need real-time responses, use streaming for the chat completion stage. Oxlo.ai supports streaming responses on all chat models, so you can start processing the LLM output as soon as the first tokens arrive.

Best Practices

Prompt with context. When you send a transcript to the LLM, include metadata such as speaker names, timestamps, or domain hints in the system prompt. This improves extraction accuracy without requiring fine-tuning.

Use JSON mode for downstream automation. Setting response_format to json_object forces the model to return parseable output. This is useful when the transcript feeds into a CRM, ticketing system, or database.

Validate audio before sending. Ensure your files are in a supported format and sample rate. Re-encoding audio on the client side prevents unnecessary API failures.

Separate concerns. Do not ask the LLM to perform transcription cleanup and reasoning in the same call if the tasks require different model strengths. Run Whisper first, then route the clean text to a reasoning model like DeepSeek R1 671B MoE or GLM 5.

Monitor end-to-end latency. Log the duration from audio upload to final LLM output. If transcription is the bottleneck, switch to Whisper Turbo. If reasoning is slow, enable streaming or choose a lighter model like Qwen 3