A language detector given “Ik moet die deploy pipeline nog even fixen before the release” will answer nl with a comfortable-looking score, and will say nothing at all about the four English words. That is not the detector failing to notice. It is the detector doing exactly what it was built to do, on a kind of input it was never given a way to describe.
What a detector actually computes
Every widely deployed language identifier — fastText’s lid.176, Google’s CLD3, langid.py, the classifiers behind most cloud “detect language” endpoints — is a document classifier over character n-grams. It slices the input into overlapping runs of two to five characters, looks each one up in a learned embedding or weight table, sums or averages those vectors across the entire input, and pushes the single resulting vector through a softmax over its label set.
The averaging step is the whole story. Once the n-grams of a sentence have been pooled into one vector, the classifier has no representation of where in the string any piece of evidence came from. A sentence is, to the model, a bag of character fragments with no ordering and no boundaries. Two languages that each contributed fragments have been stirred into one soup before the decision layer ever runs. Fastext’s own language identification documentation describes the model as a supervised classifier over a document, and the training corpora it names — Wikipedia, Tatoeba, SETimes — are collections of monolingual sentences.
A labelled sentence through the arithmetic
Take the sentence above and label every token by hand, which is the step nobody does and the step that makes the failure obvious:
Ik(nl) moet(nl) die(nl) deploy(en) pipeline(en) nog(nl)
even(nl) fixen(nl/en-derived) before(en) the(en) release(en)
Dutch characters : 34
English characters : 39
Ambiguous ("fixen"): 5 Dutch morphology, English stem
Now consider what the pooled n-gram vector contains. The trigram ik and the bigram ij-adjacent patterns push toward Dutch. The trigrams the, rel, eas and lin push toward English. But pipeline, deploy and release are also extremely common in Dutch technical writing, so their n-grams carry weight for Dutch in a model trained on any corpus that includes Dutch technical prose. The result is a distribution that leans Dutch, and the label that comes back is one word: nl.
Flip the ratio — keep the same construction but let English supply forty more characters — and the label flips to en with a similar score. The label is a function of the character mass each language contributed. That is majority-language bias stated precisely: not “detectors struggle with mixed text”, but “the pooling operation discards everything except the mixture proportions, and the argmax reports the larger share”.
There is no label for “mixed”
The second half of the problem is that even a detector that somehow knew the input was mixed has nowhere to put that fact. The output space is a fixed list of language codes. There is no mixed class, no und class in most implementations, and crucially no mechanism to return two labels with weights that mean “60% of this string is Dutch”.
People read the second-place softmax score as if it meant that, and it does not. The softmax is normalised over mutually exclusive hypotheses: the numbers answer “if this document is in exactly one language, which one?”. A 0.62/0.31 split does not mean the text is 62% Dutch. It means the model is uncertain between two single-language hypotheses, which is also what you get from a short monolingual string, from a proper noun, and from a pair of closely related languages. Three completely different situations produce the same shaped output, so you cannot recover which one you are in from the numbers alone.
Shared script removes the last cheap signal
Code-switching between languages that use different scripts is the easy case, because the Unicode Script property of each character is a hard, non-statistical signal: a run of Devanagari is not English, no model required. That is the strategy behind detecting the dominant language in a mixed-script document.
The painful cases are the common ones, and they are all single-script. Hinglish written in Latin, Spanglish, Dutch or German or Scandinavian technical speech with English nouns dropped in, Arabizi, Taglish — every character in the string has Script=Latin, so the script property partitions nothing. The detector is left with n-gram statistics as its only evidence, on input where the n-grams of both languages are drawn from the same character inventory and frequently from the same subwords.
What to do instead of a document label
The fix is to stop asking for one label per document, because that question has no correct answer for this input.
- Ask per span, not per document. Word-level and span-level language identification is a different task with its own research literature and its own labelled data — the LinCE benchmark and the CALCS workshop series exist precisely because document-level LID does not transfer. If you need to know that four words were English, you need a tagger, not a classifier.
- Detect on a window, then aggregate. A cheap approximation: run the document detector over a sliding window of five to ten tokens and keep the sequence of labels. You will see the switch points as a label change, and you get a proportion instead of an argmax. It is noisy at the boundaries, and it is still far more informative than one code.
- Use the label as a prior, never as a gate. Routing a support ticket to a Dutch-only queue on a
nllabel, when the ticket is half English, is how code-switched messages get misrouted. Treat the label as one input to a decision that has a fallback, not as ground truth. - Let a language model read it. An instruction-tuned model asked to reply in the language of the user’s message handles code-switched input far better than a bag-of-n-grams classifier, because it is conditioned on the whole sequence in order. It is slower and it costs a call, so use it where the statistical detector reports low confidence rather than everywhere.
The general rule: a detector’s output is a compression of the input down to one symbol. If the property you care about was destroyed by that compression, no amount of threshold tuning will bring it back.
Top comments (0)