Speech recognition has moved beyond simple transcription. Modern pipelines pair automatic speech recognition (ASR) with large language models to correct errors, add speaker labels, format output, and extract structured data. These hybrid systems raise new engineering questions around chunking, prompting, latency, and cost, especially when transcripts grow long. This guide covers practical patterns for integrating LLMs with speech recognition tasks, with concrete code and infrastructure considerations.
Why Combine LLMs with ASR
Traditional ASR outputs raw text with limited context awareness. LLMs can fix homophone errors, expand abbreviations, apply domain-specific terminology, and structure output. For example, a medical dictation pipeline might use Whisper to generate a draft, then prompt Llama 3.3 70B or Qwen 3 32B to normalize terms and format clinical notes. The result is higher accuracy without retraining the acoustic model.
Architecture Patterns
Two dominant patterns exist today.
Cascaded ASR then LLM. Audio goes to an ASR model, then the transcript plus a system prompt go to an LLM. This is flexible, debuggable, and lets you upgrade each component independently. You can swap Whisper Turbo for speed on short clips, or Whisper Large v3 for accuracy on noisy audio, without changing the downstream LLM logic.
End-to-end speech LLM. A single multimodal model processes audio directly. These are emerging but less available in open weights and often require more compute for comparable accuracy on general domains.
For production today, the cascaded approach is safer and more cost-predictable.
Pre-processing and Audio Handling
Before transcription, normalize audio to 16kHz mono. For long files, chunk by silence or fixed intervals with overlap to avoid boundary errors. Track timestamps so the LLM can reference them during post-processing.
When using Oxlo.ai, you can send audio directly to the audio/transcriptions endpoint using Whisper Large v3 or Whisper Turbo for speed. The endpoint returns segments with timestamps, which you can feed into downstream LLM prompts.
Prompt Engineering for Transcription Accuracy
ASR models are sensitive to prompt context. With Whisper, the initial prompt can bias vocabulary toward domain terms. Keep prompts under 224 tokens for Whisper. For post-processing LLMs, structure prompts into sections: role, input transcript, formatting rules, and output schema.
An example system prompt for an LLM post-processor:
You are a transcript editor. Fix spelling errors, remove filler words, and format as Markdown dialogue. Preserve timestamps. Do not summarize.Post-processing with LLMs
After transcription, route the text to an LLM for several refinement tasks:
- Diarization assistance: prompt the model to infer speaker changes based on context and paragraph breaks.
- Formatting: convert raw text to JSON, SOAP notes, or meeting minutes with strict schemas.
- Redaction: remove PII using guided instructions or direct LLM editing.
Because transcripts from meetings, podcasts, or legal depositions can run to tens of thousands of tokens, token-based billing inflates costs quickly. Oxlo.ai uses flat per-request pricing, so a long transcript costs the same as a short one. This makes it significantly cheaper for long-context post-processing compared to token-based providers. See https://oxlo.ai/pricing for plan details.
Code Example: Building a Pipeline
The following Python example uses the OpenAI SDK with Oxlo.ai to transcribe audio and then post-process the result. Changing providers requires only updating the base_url.
import openai
import os
client = openai.OpenAI(
api_key=os.environ.get("OXLO_API_KEY"),
base_url="https://api.oxlo.ai/v1"
)
Step 1: Transcribe with Whisper
with open("meeting.wav", "rb") as audio_file:
transcript = client.audio.transcriptions.create(
model="whisper-large-v3",
file=audio_file,
response_format="verbose_json",
timestamp_granularities=["segment"]
)
raw_text = " ".join([seg.text for seg in transcript.segments])
Step
Top comments (0)