Evaluating LLM Models for Multilingual Text Generation

#learnai #oxlo #ai

I built a small harness to compare how different LLMs handle multilingual copy generation for the same product brief. If you ship localized content or support global users, this script gives you a repeatable way to benchmark models on fluency and cultural fit instead of relying on marketing claims.

What you'll need

Python 3.10 or newer
pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Configure the client and test targets

I start by importing the SDK and pointing it at Oxlo.ai. I also define the languages and the models I want to compare.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

MODELS = [
    "qwen-3-32b",
    "llama-3.3-70b",
    "kimi-k2.6",
    "deepseek-v3.2",
]

LANGUAGES = {
    "es": "Mexican Spanish",
    "ja": "Japanese",
    "de": "German",
}

SOURCE_TEXT = (
    "Launching AuraBuds Pro. "
    "Active noise cancellation, 48-hour battery, transparency mode. "
    "Target: urban commuters who value silence and style."
)

Step 2: Lock in the system prompt

The system prompt tells every model to act as an in-country copywriter, not a literal translator. It asks for idioms, local platforms, and culturally relevant tone.

SYSTEM_PROMPT = """You are a senior copywriter born and raised in the target locale.
Do not translate word for word. Rewrite the brief as compelling local marketing copy.
Use idioms, local references, and pricing psychology that fit the culture.
Keep it under 80 words. Output only the copy, no explanations."""

Step 3: Generate localized copy across the matrix

Next I loop through every model and language pair, feeding the same brief each time. I collect outputs in a nested dictionary so I can score them later.

def generate_copy(model: str, locale_name: str, source: str) -> str:
    user_message = (
        f"Locale: {locale_name}\n"
        f"Brief: {source}"
    )

    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.7,
        max_tokens=256,
    )
    return response.choices[0].message.content.strip()


results = {}

for model in MODELS:
    results[model] = {}
    for code, locale_name in LANGUAGES.items():
        print(f"Running {model} / {locale_name} ...")
        text = generate_copy(model, locale_name, SOURCE_TEXT)
        results[model][code] = text

Step 4: Add a lightweight judge

To keep the comparison objective, I score every snippet with Kimi K2.6 using a tight rubric. I ask for two integer scores and a one-sentence rationale.

import re

JUDGE_PROMPT = """You are a strict localization editor.
Rate the following marketing copy on:
- Fluency (1-10): grammar, naturalness, and flow in the stated locale.
- Cultural fit (1-10): local references, tone, and persuasiveness.

Respond in exactly this format:
Fluency: [score]
Cultural fit: [score]
Rationale: [one sentence]
"""


def judge(model: str, locale_name: str, copy: str) -> dict:
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": JUDGE_PROMPT},
            {"role": "user", "content": f"Locale: {locale_name}\nCopy: {copy}"},
        ],
        temperature=0.2,
        max_tokens=128,
    )

    text = response.choices[0].message.content.strip()
    fluency_match = re.search(r"Fluency:\s*(\d+)", text)
    fit_match = re.search(r"Cultural fit:\s*(\d+)", text)

    return {
        "fluency": int(fluency_match.group(1)) if fluency_match else 0,
        "cultural_fit": int(fit_match.group(1)) if fit_match else 0,
        "raw": text,
    }


scores = {}
for model in MODELS:
    scores[model] = {}
    for code, locale_name in LANGUAGES.items():
        scores[model][code] = judge(model, locale_name, results[model][code])

Step 5: Aggregate and print the leaderboard

Finally, I average the two scores across all three languages for each model and print a sorted table.

def average_score(model: str) -> float:
    total = 0
    count = 0
    for code in LANGUAGES:
        s = scores[model][code]
        total += s["fluency"] + s["cultural_fit"]
        count += 2
    return total / count


leaderboard = [(m, average_score(m)) for m in MODELS]
leaderboard.sort(key=lambda x: x[1], reverse=True)

print("\n=== Multilingual Copy Leaderboard ===")
for rank, (model, avg) in enumerate(leaderboard, 1):
    print(f"{rank}. {model:<20} {avg:.2f} / 10")

Run it

Save the script as evaluate_multilingual.py, export your key, and run python evaluate_multilingual.py. On my last run, the output looked like this.

Running qwen-3-32b / Mexican Spanish ...
Running qwen-3-32b / Japanese ...
Running qwen-3-32b / German ...
Running llama-3.3-70b / Mexican Spanish ...
...

=== Multilingual Copy Leaderboard ===
1. qwen-3-32b            8.83 / 10
2. kimi-k2.6             8.67 / 10
3. llama-3.3-70b         8.33 / 10
4. deepseek-v3.2         8.17 / 10

Sample output (qwen-3-32b / Japanese):
AuraBuds Pro、登場。ANCで都市の騒音をシャットアウト、最大48時間のバッテリーで
週末まで充電不要。透明モードで周囲の声もキャッチ。スタイルと静寂を両立させる、
通勤者のための相棒です。

Wrap-up

That is the whole evaluator. Because Oxlo.ai charges a flat rate per request, running a 24-request benchmark matrix like this costs the same whether your brief is 50 words or 5,000 words. That makes it practical to rerun the suite on every model update or new locale.

Two concrete next steps: wire this script into a CI job that fails when a model's average score drops below a threshold, or swap the judge rubric for task-specific criteria like SEO keyword density or brand safety.

DEV Community