Document-level language detection returns one string. No threshold, no ensemble and no larger model changes that, because the return type is the limitation. Finding the word where a sentence changes language is a sequence-labelling problem and has to be built as one.
This is a different task, not a better detector
The usual first attempt is to run a document-level detector over a sliding window and look for the point where the label flips. It fails for a reason worth understanding, because the reason recurs: detectors like lid.176 and CLD3 are trained on documents and their accuracy collapses on short inputs. Below roughly a clause, the character n-gram statistics they rely on are too sparse to separate related languages, so a five-word window produces a label sequence that flickers. You cannot distinguish a real switch from window noise, because both look like a flip.
The task that has an answer is: given a token sequence, assign a language label to each token. That is the same shape as named entity recognition or part-of-speech tagging, and it inherits their tooling, their architectures and their evaluation practice.
The label set
Do not invent one. The inventory from the First Shared Task on Language Identification in Code-Switched Data (Solorio et al., EMNLP 2014), reused by the LinCE benchmark (LREC 2020), covers the cases you will actually hit:
lang1 first language of the pair
lang2 second language of the pair
ne named entity — belongs to neither, e.g. "Flipkart", "Netflix"
mixed morphologically mixed token, e.g. "crasheó", "textear"
ambiguous valid in both languages, e.g. "no", "me", "a", "un"
fw foreign word from a third language
other punctuation, emoji, numbers, URLs
unk unidentifiable
The three categories that a naive design omits are the three that carry the work. ne exists because brand names have no language and forcing them into one is the largest single source of label noise. ambiguous exists because no is a word in both English and Spanish, me is a word in both, and a labeller forced to choose produces training data that teaches the model a coin flip. mixed exists because crasheó is genuinely both, as the Spanglish pipeline works through.
Once tokens are labelled, a switch point is derived rather than predicted: it is any boundary between adjacent tokens whose labels are lang1 and lang2 in either order, ignoring ne, other and ambiguous tokens in between. That ignoring rule is not a detail — a brand name sitting between a Hindi and an English token is one switch, not two.
A baseline that works without training
Before fine-tuning anything, build the lexicon baseline. It is embarrassingly effective for pairs that do not share vocabulary, and it gives you a floor to beat:
from collections import Counter
# Frequency lists from any monolingual corpus for each language.
EN = load_freq("en.txt") # token -> count
ES = load_freq("es.txt")
def label_token(tok: str) -> str:
t = tok.lower().strip(".,!?¿¡")
if not t or not t.isalpha():
return "other"
en, es = EN.get(t, 0), ES.get(t, 0)
if en == 0 and es == 0:
return "unk"
if en and es:
# both lists know it: only call it if one is far more likely
ratio = (en + 1) / (es + 1)
if 0.2 < ratio < 5:
return "ambiguous"
return "lang2" if ratio >= 5 else "lang1"
return "lang2" if en else "lang1"
def switch_points(tokens):
labels = [label_token(t) for t in tokens]
real = [(i, l) for i, l in enumerate(labels) if l in ("lang1", "lang2")]
return [real[k + 1][0] for k in range(len(real) - 1)
if real[k][1] != real[k + 1][1]]
The frequency-ratio test is the part that earns its keep. A hard membership check labels no as Spanish or English depending on which list you consulted first; the ratio test declines to guess and returns ambiguous, and the switch-point derivation skips it. The baseline’s real weakness is not ambiguity but out-of-vocabulary tokens: romanised Hindi and Spanglish coinages are unk across the board, so the baseline is much stronger on Spanish-English than on Hindi-English.
The sequence-labelling version
- Start from a multilingual encoder. XLM-R or a similar model that saw both languages in pretraining. A monolingual encoder cannot represent one side of the pair and no amount of fine-tuning fixes that.
- Add a token classification head over the eight labels. This is the standard token-classification setup — the same code path as named entity recognition, with a different label list.
- Align labels to subwords carefully. The tokenizer splits words into subwords; your labels are per word. Label the first subword of each word and mask the rest with
-100so they are excluded from the loss. Getting this wrong is the most common silent bug in the whole procedure, and it shows up as suspiciously high accuracy on frequent short words. - Train on LinCE or GLUECoS if your pair is covered — Hindi-English, Spanish-English, Nepali-English, Modern Standard Arabic with Egyptian Arabic — and on a few thousand hand-labelled sentences of your own traffic if it is not. Token labelling is unusually cheap to annotate: a bilingual annotator labels several hundred sentences an hour because each decision is one word.
- Derive switch points from the label sequence with the skip rule above, and store them as character offsets on the record. Downstream components need the offsets, not the labels.
One post-processing step earns its place. Switches are rare relative to token count — most adjacent token pairs are same-language — so an unsmoothed per-token classifier produces isolated single-token flips that are almost always errors rather than real switches. Adding a transition constraint, either a CRF layer over the tag sequence or a rule that discards a language run shorter than two tokens unless it is a known insertion, removes a large share of false switch points at little cost to recall. The exception to guard is the genuine single-word insertion, which is the most common switch type of all, so tune the minimum run length against your own data rather than setting it to two and forgetting it.
Decide too what happens when the labels come back unusable — a sentence of mostly unk, or a near-even split with no matrix language. Downstream consumers need a defined behaviour for that case rather than a best guess, which is the general argument made in language detection fallback strategies.
Evaluating it without fooling yourself
Per-token accuracy is the wrong headline metric and it will make a bad model look excellent. Most tokens in a code-switched corpus belong to the matrix language, so a model that predicts the majority label for every token scores high while finding no switch points at all. Report instead:
- Switch-point F1. Precision and recall over the derived boundaries, not over tokens. This is the number the task is actually about.
- Per-label F1, including the rare labels.
mixedandneare a small fraction of tokens and a large fraction of the errors that matter downstream. - Sentence-level exact match. The proportion of sentences labelled entirely correctly. It is a harsh metric and it correlates with what a downstream consumer of the labels experiences.
- Accuracy on monolingual sentences. Your corpus contains plenty of them. A model that hallucinates switches in monolingual text is worse than no model, because it converts a working path into a broken one.
Top comments (0)