Route machine translation by risk, not by a single score

#ai #llm #translation #testing

"Will AI replace translators?" is the wrong question for anyone building a localization pipeline. The useful question is which segments can ship without a person, which need a person to check them, and which need a person to write them. Answer that per piece of content, before anything gets translated.

Olga Beregovaya, VP of AI at Smartling, gave the clearest version of the division of labor I've heard on the show:

AI does the heavy lifting, introduces predictability, does a lion chunk of the work, takes us much closer to human parity than ever before. And the human translator or fact checker or validator really compensates for the delta that AI is not able to cover.

Olga Beregovaya, Smartling

Your routing logic decides where that delta lives.

Why can't a fluent translation be trusted on its own?

Because the expensive failures read well. Beregovaya describes an Italian researcher receiving smooth Italian text in which the phenomena described were "100% American culture centric." In her words, "you do lose on factual accuracy and anthropological and phenomenological local phenomena." A readability check passes that text.

The risk grows with language coverage. "The less covered and less represented the languages in the model, the higher the odds are that biases will be introduced," she says. And she does not assume a large language model is the right engine for every pair: "Quite often, neural machine translation would do much better than a large language model would do."

What do automatic scores tell you?

BLEU compares an output with one or more human reference translations using modified n-gram precision and a brevity penalty. Its correlation with human judgment is high when scores are averaged over a test corpus, while scores on individual sentences often diverge from human judgment. METEOR, proposed by Banerjee and Lavie at Carnegie Mellon in 2005, aligns words by exact match, stem and synonym and weights recall more heavily, which catches some of what BLEU misses.

Both still compare strings with a reference. Neither knows whether a sentence is culturally right for its reader. So I use them the way their own evidence supports: to compare engines and spot regressions over a batch, and never to approve a single segment for a high-stakes page.

Which signals should move a segment between lanes?

Four signals, checked before and after machine output:

Content risk. Beregovaya scopes fully unattended translation to content with a shorter shelf life, higher risk tolerance and instant turnaround. Opinion portals and forums are her examples. I would not put product pages, legal text or brand campaigns in that lane.
Language coverage. "If you feed Swahili and all you have at your disposal is an English tokenizer, it's not gonna get you far." Her phrase for shipping into a language the model cannot reliably vectorize is "upload and pray." A poorly covered language moves up a lane no matter what the overlap score says.
Reference material. Style guides that "can be a 100 pages," translation memories, dictionaries and do-not-translate lists belong in retrieval, not pasted into every prompt. Pulling only the relevant entries is how her team grounds output, and stuffing everything in is, as she put it, a "waste of fortune, make no money."
Latency. Her team's platform work went into queuing, caching and multithreading to cut inference time. A perfect prompt that takes five minutes to run is, in her words, "obviously not gonna fly" for instant content.

What does the routing look like in code?

A sketch, with lanes and thresholds you would set from your own content and reviewers. It is not Smartling's pipeline.

# Sketch only: decide the lane before translating.
LANES = ("machine_only", "machine_plus_review", "human_with_ai_assist")

def route(segment, well_covered_langs, batch_score=None):
    if segment.brand_critical or segment.regulated:
        return "human_with_ai_assist"
    if segment.target_lang not in well_covered_langs:
        return "machine_plus_review"
    short_lived = segment.shelf_life_days <= 7 and segment.risk_tolerance == "high"
    # Batch scores flag a regressed engine; they never approve a single segment.
    engine_ok = batch_score is None or batch_score >= segment.min_batch_score
    if short_lived and engine_ok:
        return "machine_only"
    return "machine_plus_review"

The default lane is machine output plus review. Content earns its way down to machine-only; it does not start there.

For more conversations on deploying AI inside large organizations, see the best episodes for enterprise AI leaders.

Checklist

Decide the lane per segment before translation, from content risk and language coverage.
Use BLEU or METEOR to compare engines over a batch, not to approve a sentence.
Move poorly covered languages to a reviewed lane regardless of score.
Put style guides and translation memories behind retrieval instead of in every prompt.
Keep a human on brand-critical and regulated content.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

DEV Community