Building Language Models for Low-Resource Languages with LLM

#engineering #oxlo #ai

We are building a synthetic data generator that produces validated Swahili instruction-response pairs for fine-tuning small language models. This pipeline solves the cold-start problem for low-resource languages where human-curated datasets are scarce or expensive to produce. We will run everything through Oxlo.ai because its flat per-request pricing keeps costs predictable even when we batch long prompts for translation and back-translation.

What you'll need

Before starting, grab an Oxlo.ai API key from https://portal.oxlo.ai. You also need Python 3.10 or newer and the OpenAI SDK installed with pip install openai. Because Oxlo.ai charges a flat rate per request, you can pack long system prompts and few-shot examples without worrying about token costs. See https://oxlo.ai/pricing for the latest plan details.

Step 1: Bootstrap the Oxlo.ai client

I always start by verifying the endpoint and model. This snippet initializes the OpenAI-compatible client and asks Qwen 3 32B for a Swahili greeting.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Translate 'Hello, how can I help you today?' into Swahili."},
    ],
)
print(response.choices[0].message.content)

Step 2: Generate English seed pairs

We need raw material. I prompt the model for five diverse instruction-response pairs in a strict line format so we can parse them without extra dependencies.

import json

PAIR_GENERATION_PROMPT = """Generate 5 diverse instruction-response pairs for a helpful assistant.
Each pair must follow this exact format:
INSTRUCTION: 
RESPONSE: 
Separate pairs with one blank line."""

response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=[
        {"role": "system", "content": "You follow formatting instructions exactly."},
        {"role": "user", "content": PAIR_GENERATION_PROMPT},
    ],
)
raw = response.choices[0].message.content

pairs = []
for block in raw.strip().split("\n\n"):
    inst = ""
    resp = ""
    for line in block.splitlines():
        if line.startswith("INSTRUCTION:"):
            inst = line.replace("INSTRUCTION:", "").strip()
        elif line.startswith("RESPONSE:"):
            resp = line.replace("RESPONSE:", "").strip()
    if inst and resp:
        pairs.append({"instruction": inst, "response": resp})

print(f"Generated {len(pairs)} pairs.")

Step 3: Translate into Swahili

This is the core localization agent. I lock the model to translation-only output and instruct it to adapt cultural references for East African contexts.

TRANSLATION_SYSTEM_PROMPT = """You are a professional English-to-Swahili translator.
Translate the user's text into natural Swahili. Adapt examples, currencies, and cultural references to fit East African contexts.
Output only the translation. Do not add explanations."""

def translate(text: str) -> str:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": TRANSLATION_SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
    )
    return response.choices[0].message.content.strip()

for pair in pairs:
    pair["swahili_instruction"] = translate(pair["instruction"])
    pair["swahili_response"] = translate(pair["response"])
    print(pair["swahili_instruction"][:60])

Step 4: Back-translate and score

To validate quality without a human reviewer, I back-translate the Swahili response into English and score semantic fidelity against the original. Any pair scoring below 8 is flagged for removal.

import re

BACK_TRANSLATION_PROMPT = """Back-translate the following Swahili text into English.
Then compare it to the original English meaning and rate semantic fidelity on a scale of 1 to 10.
Return exactly:
BACK: 
SCORE: 

Swahili text:
{text}

Original English meaning:
{original}"""

def evaluate(pair: dict) -> dict:
    prompt = BACK_TRANSLATION_PROMPT.format(
        text=pair["swahili_response"],
        original=pair["response"]
    )
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": "You are a strict translation evaluator."},
            {"role": "user", "content": prompt},
        ],
    )
    text = response.choices[0].message.content
    back_match = re.search(r"BACK:\s*(.+)", text)
    score_match = re.search(r"SCORE:\s*(\d+)", text)
    pair["back_translation"] = back_match.group(1).strip() if back_match else ""
    pair["score"] = int(score_match.group(1)) if score_match else 0
    return pair

for pair in pairs:
    evaluate(pair)
    print(f"Score: {pair['score']} | {pair['instruction'][:50]}...")

Step 5: Filter and export

Finally, I keep only high-fidelity pairs and write them to JSONL. This format drops straight into training frameworks like Hugging Face TRL or Unsloth.

filtered = [p for p in pairs if p["score"] >= 8]

with open("swahili_data.jsonl", "w", encoding="utf-8") as f:
    for p in filtered:
        record = {
            "messages": [
                {"role": "user", "content": p["swahili_instruction"]},
                {"role": "assistant", "content": p["swahili_response"]}
            ]
        }
        f.write(json.dumps(record, ensure_ascii=False) + "\n")

print(f"Exported {len(filtered)} high-quality pairs to swahili_data.jsonl.")

Run it

Executing the full script produces a mix of scores. When I ran this pipeline yesterday, one creative writing task scored a 6 because the model localized an idiom too aggressively, so it was correctly dropped.

$ python build_swahili_dataset.py
Generated 5 pairs.
Je, mji mkuu wa Tanzania ni upi?
Andika hadithi fupi kuhusu siku ya mvua...
Score: 9 | What is the capital of Tanzania?...
Score: 6 | Write a short story about a rainy market day...
Exported 4 high-quality pairs.

You can inspect swahili_data.jsonl to see the final messages ready for supervised fine-tuning.

Wrap-up

You now have a working pipeline that generates training data for low-resource languages using Oxlo.ai. A concrete next step is to wrap the translation and scoring calls in asyncio so you can process hundreds of seeds in parallel without blocking on each request. After that, run a small LoRA fine-tune on a 1B-parameter model to verify that the synthetic Swahili actually improves downstream task performance.

DEV Community