Building Language Translation Models with LLMs

#aiinfrastructure #oxlo #ai

Machine translation has moved beyond phrase-based and neural encoder-decoder systems. Modern large language models capture nuance, idioms, and domain terminology with far less task-specific training, but turning that capability into a reliable pipeline requires deliberate architecture choices around prompting, context management, and cost control.

Why LLMs for Translation

Traditional neural machine translation (NMT) models excel at high-resource language pairs and short sentences, but they struggle with context windows that span paragraphs, inconsistent terminology across documents, and low-resource dialects. LLMs address these gaps through in-context learning and massive multilingual pretraining. A single model can translate, summarize, and adapt tone without retraining or cascading pipeline complexity.

The tradeoff is inference cost and latency. Long documents inflate token counts, and agentic correction loops add further overhead. This makes pricing structure and context length limits central engineering constraints, not just footnotes.

Architecture Patterns

Production translation systems rarely send raw text to a chat endpoint and return the first response. Three patterns dominate:

Chunked direct translation: Split documents at sentence or paragraph boundaries, translate each chunk with a persistent system prompt, and reassemble. Simple, but risks inconsistent terminology.
Reflection and refinement: Use an initial translation pass followed by a critique pass, either from the same model or a smaller verifier. This improves fluency but doubles (or triples) request volume.
Retrieval-augmented translation: Inject glossary entries, style guides, and previous translations into the context window. This keeps terminology consistent across chunks without fine-tuning.

For low-latency workflows, streaming responses let you begin displaying text before the final token arrives. For quality-critical workflows, JSON mode lets you enforce structured output such as segmented alignments or confidence scores alongside translated strings.

Prompt Engineering for Quality

Translation quality depends heavily on prompt design. A bare instruction like "Translate to French" loses domain context and tone. Effective prompts specify:

Target locale and register, e.g., Canadian French, formal medical tone.
Domain constraints, e.g., preserve Markdown formatting, do not translate code blocks.
Glossary overrides, e.g., "translate 'server' as 'serveur', not 'serveuse'".
Output schema, especially when using JSON mode to receive parallel arrays of source and target segments.

Few-shot examples in the target domain reduce hallucination and stabilize style. When translating long documents, prepend a persistent system prompt that includes the glossary, then stream chunks to maintain context.

Scaling Translation with Oxlo.ai

Long-context and agentic workloads expose the weaknesses of token-based billing. Every glossary entry, every paragraph of source text, and every reflection pass adds input tokens. Providers that bill per token can make large-document translation prohibitively expensive, especially when you need to maintain context across 100K+ tokens for technical manuals or legal contracts.

Oxlo.ai uses flat per-request pricing. One API call costs the same regardless of prompt length, which makes it predictable and significantly cheaper for long-context translation and multi-pass agentic refinement. You can send entire chapters, full glossaries, and chain-of-thought instructions in a single request without watching token meters spin.

The platform runs 45+ models, including options that are particularly strong for multilingual work:

Qwen 3 32B: Strong multilingual reasoning and agent workflow support.
Kimi K2.6: 131K context window, advanced reasoning, and agentic coding capabilities that transfer well to structured translation tasks.
Llama 3.3 70B: A solid general-purpose workhorse for high-throughput translation.
DeepSeek V4 Flash: 1M context window and efficient MoE architecture for near-state-of-the-art open-source reasoning on entire documents.

All endpoints are fully OpenAI SDK compatible, so switching an existing translation pipeline requires only a base URL change to https://api.oxlo.ai/v1. There are no cold starts on popular models, which keeps latency consistent even when traffic spikes.

You can explore the exact cost structure on the Oxlo.ai pricing page.

Code Example: Document Translation

The following Python script translates a Markdown file using the OpenAI SDK pointed at Oxlo.ai. It uses a system prompt that enforces terminology, streams the response, and preserves formatting.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

SYSTEM_PROMPT = """You are a professional technical translator.
Translate the user's Markdown document into Canadian French.
Rules:
- Preserve all Markdown syntax exactly.
- Translate 'server' as 'serveur'.
- Use formal register.
- Output only the translated text, no explanations."""

def translate_chunk(text: str) -> str:
    response = client.chat.completions.create(
        model="qwen3-32b",  # or llama-3.3-70b, kimi-k2.6
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text}
        ],
        stream=True,
        temperature=0.3,
    )
    output = ""
    for chunk in response:
        delta = chunk.choices[0].delta.content or ""
        output += delta
    return output

# Example: translate a paragraph
source = "## Installation Guide\nFirst, start the server with `launch.sh`."
print(translate_chunk(source))

Because Oxlo.ai bills per request, sending a 4,000-token prompt with a 2,000-token completion costs the same as a 500-token exchange. This flat rate makes budget planning straightforward when you batch large files or run reflection loops where a second model critiques the first.

Evaluation and Iteration

Automated metrics like BLEU and chrF++ correlate poorly with human judgments of LLM translations. A better approach is LLM-as-judge: use a separate, high-capability model to score fluency, accuracy, and terminology adherence against a reference rubric. Back-translation, where you translate to the target language and then back to the source, surfaces semantic drift and hallucinations.

Set up a small golden dataset of challenging segments, domain-specific idioms, and edge cases. Run them through your pipeline on each prompt or model change. Track regression with deterministic sampling so you can compare outputs side by side.

Conclusion

Building a translation system on LLMs is now a matter of API orchestration and prompt design rather than training custom seq2seq models. The remaining engineering challenges are cost control at scale, context fidelity across long documents, and consistent terminology.

Oxlo.ai addresses the cost and context problems directly through flat per-request pricing and models with extensive context windows. If your translation workload involves long inputs, multi-pass refinement, or unpredictable document lengths, moving to a request-based provider removes the token-count anxiety that complicates pipeline design.