Detecting the Dominant Language in a Mixed-Script Document

#ai #machinelearning #tutorial #llm

A scanned contract with an Arabic body and an English annex, a Russian research paper with English references, a Japanese manual with a romaji index — running one detector over the concatenated text gives you one label and throws away the structure that made the document useful. The partition is available before any model runs, for free, from Unicode itself.

Why one guess per document is wrong

The whole-document call fails in two directions and both are expensive. If the document is 80% Arabic and 20% English, the label is ar, and every downstream step — the tokenizer choice, the embedding model, the chunk size, the OCR engine, the translation target — is now applied to the English annex too. If the split is nearer 50/50, the label is a coin flip between two answers that are each wrong for half the file.

There is also a quieter failure. Concatenating text across a script boundary produces n-grams that span the boundary and exist in no language at all. Those spurious features are noise added to a decision that was already hard.

The correct output for a mixed-script document is not a label. It is a list of ranges with a label each, plus a dominant language derived from those ranges by whatever weighting your application needs — which is usually characters, sometimes blocks, and occasionally “whichever language the first heading is in”.

Partition by script first

Every Unicode code point carries a Script property, defined in Unicode Standard Annex #24. It is a lookup, not a guess: U+0645 is Arabic, U+0416 is Cyrillic, U+3042 is Hiragana. Partitioning on it is deterministic and costs one table lookup per character.

Two special values do the real work. Common covers characters shared by all scripts — spaces, digits, most punctuation — and Inherited covers combining marks that take the script of the base character they attach to. Neither should start or break a run: a full stop between two Arabic sentences must not split them, and a combining mark must never be separated from its base.

import regex   # the third-party regex module, which supports \p{Script=...}

SCRIPTS = ["Arabic", "Cyrillic", "Greek", "Hebrew", "Devanagari",
           "Han", "Hiragana", "Katakana", "Hangul", "Thai", "Latin"]

def script_of(ch: str) -> str:
    for name in SCRIPTS:
        if regex.match(r"\p{Script=" + name + r"}", ch):
            return name
    return "Other"

def script_runs(text: str):
    """Yield (script, start, end), absorbing Common/Inherited into the run."""
    runs, current, start = [], None, 0
    for i, ch in enumerate(text):
        if ch.isspace() or regex.match(r"[\p{Common}\p{Inherited}]", ch):
            continue                      # never starts or breaks a run
        s = script_of(ch)
        if current is None:
            current, start = s, i
        elif s != current:
            runs.append((current, start, i))
            current, start = s, i
    if current is not None:
        runs.append((current, start, len(text)))
    return runs

Detect per block, aggregate after

Script runs are not the unit you want to detect on — a run can be one word. Use the document’s own structure as the block boundary (paragraph, table cell, PDF text block, OCR region), compute the script histogram of each block, and act on that:

A block that is over ~90% one non-Latin script has its language narrowed to the languages written in that script. Cyrillic narrows to about a dozen candidates; Hangul narrows to one; Thai narrows to one. Run the statistical detector only to pick within the family, and discard any label whose script does not match.
A predominantly Latin block gets no help from script and must go to the detector with the full length gate and confidence threshold applied.
A genuinely mixed block — two scripts each over 20% — is the code-switching case, and should be labelled as mixed and carried forward as such rather than forced to one language.

Aggregate to a dominant language at the end by summing characters per label, not by counting blocks. A document with forty short English headings and four long Arabic sections is Arabic; block counting says English.

Where the Han script breaks the partition

The Script property answers “which writing system”, not “which language”, and for the CJK block those come apart. Chinese, Japanese and Korean text can all contain characters with Script=Han, so a Han run narrows the candidate set to three languages rather than one.

The resolution is positive evidence from the scripts that are unambiguous:

Any Hiragana or Katakana in the block means Japanese. Ordinary Japanese prose cannot avoid kana for long, so even a short paragraph almost always contains some.
Any Hangul means Korean. Mixed hangul-hanja text is Korean with Han characters in it, not Chinese.
Han with no kana and no hangul is Chinese by default — but a Japanese heading, a name, or a compound noun of four kanji is Han with no kana, so short blocks should inherit the surrounding document’s answer rather than getting their own.

A parallel trap: Latin characters inside a Japanese document are frequently not English. Product names, acronyms and romaji inside otherwise Japanese text will be detected as some European language by a Latin-only detector. Require a minimum run length before you let a Latin block claim its own label inside a non-Latin document.

The procedure end to end

Normalise the text first, to NFC. Otherwise a decomposed character may split into a base and a combining mark that your run logic handles inconsistently. See the difference between the normalization forms for which form to pick.
Split into structural blocks — paragraphs, cells, OCR regions. Never a fixed character window; a window that cuts mid-sentence produces exactly the boundary-spanning n-grams you are trying to avoid.
Compute a script histogram per block using the run logic above, ignoring Common and Inherited.
Resolve the script to a candidate language set, applying the Han rules. If the set has exactly one member, you are done for that block and no classifier runs.
Run the statistical detector only on the remaining blocks, and mask its output to the candidate set — a fr label on a Cyrillic block is an artefact and should be dropped, not returned.
Emit ranges, then derive the dominant language by character mass. Keep the ranges: whatever consumes this — translation, indexing, chunking — needs them more than it needs the single label.

The usual reason to detect per block is that different blocks then go to different models — a strong Arabic model for the body, a cheaper one for the English annex. That means two providers, two request shapes and two streaming formats behind one document pipeline; a gateway that presents one API and one key across providers keeps the per-block routing decision as a string in your code rather than as a second SDK integration.