Building Language Translation Apps with LLM, Speech Recognition, and Computer Vision

Building a production-grade language translation application requires more than swapping text between languages. Modern use cases demand real-time speech recognition, visual context understanding, and low-latency inference across heterogeneous models. This article walks through a practical architecture that combines speech-to-text, vision, large language models, and text-to-speech into a single pipeline, with concrete code examples you can run today.

Architecture Overview

A robust translation app typically follows a four-stage pipeline. First, audio input is transcribed into text. Second, optional image frames provide visual context. Third, an LLM translates the source text into the target language while preserving tone and context. Fourth, the translated text is synthesized back into speech. Each stage can hit a different endpoint, so your infrastructure must support audio, vision, and chat models without cold starts or token-based billing surprises.

Oxlo.ai hosts over 45 open-source and proprietary models across seven categories, including audio, vision, chat, and speech. You get Whisper for transcription, vision models like Gemma 3 27B and Kimi VL A3B for image understanding, multilingual LLMs such as Qwen 3 32B and Llama 3.3 70B for translation, and Kokoro 82M for text-to-speech. All are accessible through the standard chat/completions, audio/transcriptions, and audio/speech endpoints, with full OpenAI SDK compatibility. Because Oxlo.ai uses request-based pricing, a long audio transcript or a detailed visual prompt costs the same flat rate per request regardless of input length. That predictability matters when you are processing multi-minute audio or high-resolution image frames.

Speech-to-Text with Whisper

Start by capturing audio and sending it to a transcription endpoint. Oxlo.ai hosts Whisper Large v3, Turbo, and Medium, so you can balance accuracy against speed.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

with open("conversation.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3",
        file=audio_file,
        response_format="text"
    )

source_text = transcript

The returned text becomes the input for your translation stage. If your users speak in bursts, Whisper Turbo keeps latency low without sacrificing multilingual coverage.

Adding Visual Context with Vision Models

In scenarios like tourist translation or industrial maintenance, the camera feed carries information that pure text misses. A sign, diagram, or warning label changes the meaning of a phrase. You can pass the image directly to a vision-capable model alongside the transcript.

Oxlo.ai offers Gemma 3 27B and Kimi VL A3B for exactly this. Both accept image inputs through the chat completions endpoint, which is fully OpenAI SDK compatible.

import base64

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("scene.jpg")

response = client.chat.completions.create(
    model="gemma-3-27b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": f"Describe the visual context for this text: '{source_text}'"},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}}
            ]
        }
    ]
)

visual_context = response.choices[0].message.content

This context is then fed into the translation prompt so the LLM knows whether "exit" refers to a door or a financial strategy.

The Translation Core

The heart of the app is the LLM that performs the actual translation. You need a model with strong multilingual reasoning, support for system instructions, and tool use if you plan to chain additional steps. Oxlo.ai carries several options. Qwen 3 32B excels at multilingual reasoning and agent workflows, making it ideal for translation pipelines that later trigger bookings or lookups. Llama 3.3 70B serves as a general-purpose flagship with broad language coverage. For deep reasoning or complex technical documentation, DeepSeek R1 671B MoE or Kimi K2.6 provide advanced chain-of-thought capabilities.

Because Oxlo.ai is fully OpenAI SDK compatible, switching models is a single parameter change. There are no cold starts on popular models, so your translation API responds immediately even after idle periods.

translation_prompt = f"""
You are a professional translator. Translate the following text into Japanese.
Preserve tone and formality. Use the visual context provided to resolve ambiguities.

Source text: {source_text}
Visual context: {visual_context}
"""

response = client.chat.completions.create(
    model="qwen3-32b",
    messages=[
        {"role": "system", "content": "You translate accurately and concisely."},
        {"role": "user", "content": translation_prompt}
    ],
    temperature=0.3
)

translated_text = response.choices[0].message.content

Text-to-Speech Output

Once you have the translated text, synthesize natural speech for the user. Oxlo.ai offers Kokoro 82M, a lightweight text-to-speech model that runs fast enough for real-time apps.

speech = client.audio.speech.create(
    model="kokoro-82m",
    voice="af",
    input=translated_text,
    response_format="mp3"
)

with open("output.mp3", "wb") as f:
    f.write(speech.content)

With streaming responses supported on chat endpoints, you can even begin synthesizing speech while the final sentence is still being generated.

Putting the Pipeline Together

Here is a minimal but complete script that wires all four stages. It assumes you have an Oxlo.ai API key and the openai Python package installed.

import openai

import base64

client = openai.OpenAI(

    base_url="https://api.oxlo.ai/v1",

    api_key="YOUR_OXLO_API_KEY"

)

def translate_media(audio_path, image_path, target_lang="Japanese"):

    # 1. Transcribe

    with open(audio_path, "rb") as f:

        transcript = client.audio.transcriptions.create(

            model="whisper-large-v3", file=f, response_format="text"

        )

# 2. Visual context
with open(image_path, "rb") as f:
    b64 = base64.b64encode(f.read()).decode("utf-8")

vision = client.chat.completions.create(
    model="gemma-3-27b",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": f"Context for: '{transcript}'"},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
        ]
    }]
)

# 3. Translate
translation = client.chat.completions.create(
    model="qwen3-32b",
    messages=[
        {"role": "system", "content": f"Translate to {target_lang}. Preserve tone."},
        {"role": "user", "content": f"Text: {transcript}\nVisual context: {vision.choices[0].message.content}"}
    ],
    temperature=0.3
)

# 4. Speak
speech = client.audio.speech.create(
    model="kokoro-82m",
    voice="af",
    input=translation.choices[0].message.content,
    response_format="mp3"
)

with open("translation.mp3", "wb") as f:
    f.write(speech.content)