The bug report says the filter let an obvious slur through. There is no error, no log line and no low-confidence score to look at, because nothing ran. A word-list filter with no list for a language does not perform badly in it — it performs no operation at all, and returns clean.
The symptom: nothing happens
The libraries most products reach for are English-first by design. The widely used Node package bad-words ships an English list. Python profanity libraries ship English lists. The most common multilingual resource, the “List of Dirty, Naughty, Obscene and Otherwise Bad Words”, covers dozens of languages but with wildly uneven depth: the English file has hundreds of entries and several language files have a few dozen, assembled once and rarely revised.
Because a missing list produces a clean verdict rather than an error, nothing in your telemetry distinguishes “we checked and it was fine” from “we had nothing to check with”. That is the actual defect, and it is a code-structure defect rather than a linguistic one.
Zero is not weak
It is worth being blunt about the arithmetic. Recall is the fraction of true positives found. With no entries for a language, that fraction is zero — not low, exactly zero — for every input in it. Your English test suite reports the same green as before, because the filter’s English behaviour did not change.
The distinction matters because the two conditions have different fixes. Weak coverage is improved by tuning: thresholds, better matching, more entries. Zero coverage is not improved by any amount of tuning; only the presence of data changes it. Teams routinely spend a quarter on the first when they have the second, because the dashboard looks the same. Make the pipeline distinguish them before anything else, exactly as moderation gaps in low-resource languages argues for the general case.
Five reasons the list does not transfer
Adding entries is necessary and it is not sufficient, because the matching machinery built for English encodes English assumptions.
Morphology
English profanity has few surface forms. In an inflected or agglutinative language, one lemma is dozens or hundreds of forms: Russian nouns decline across six cases and two numbers, Turkish and Finnish agglutinate suffixes without bound. An exact-match list holds the citation form and misses every inflected use, which is most of them. The general shape of this problem is in agglutinative languages and NLP assumptions.
Normalisation and homoglyphs
The same word can be spelled with precomposed or decomposed accents, with diacritics stripped, or with characters swapped for visually-identical ones from another script — Cyrillic а at U+0430 for Latin a at U+0061 is the classic. Add zero-width joiners, repeated letters, and digit substitutions, and one list entry needs to match an unbounded set of strings, and the precomposed and decomposed spellings are not the same bytes at all. See NFC and NFKC normalisation for which normal form to fold to and why the choice is not free.
No word boundaries
Matching in English relies on \b. Thai, Khmer, Japanese and Chinese do not put spaces between words, so that anchor never fires and the fallback is substring matching. Substring matching in CJK produces far more false positives than the English Scunthorpe problem, because short character sequences recur constantly inside unrelated words. Segmenting first with a proper word breaker is the only reliable fix, and it makes the filter dependent on a segmenter whose accuracy in that language you now also have to know.
Register, target and variety
Offensiveness depends on who says a word to whom. Reclaimed terms invert by speaker. A word that is a hard slur in one national variety is mild or neutral in another, so a single list for “Spanish” or “Arabic” is wrong in both directions simultaneously — over-blocking one region’s ordinary speech while missing another’s abuse.
Code-switching and transliteration
The offensive token frequently sits inside a message that is otherwise in another language, often romanised. Language detection labels the message by its majority language and routes it to that list, so the minority-language token is never checked against the list that contains it. Arabizi substitutes digits for letters as a matter of convention, not evasion, so the strings do not resemble the native-script entries at all. See language detection on code-switched text for why the routing step is the thing that fails here.
Normalising before matching
Every list-based approach should share one normalisation function, applied identically to the input and to the list entries at load time. Applying it to only one side is a common and silent bug.
// Applied to input AND to every list entry, at load time.
export function foldForMatching(s: string): string {
return s
.normalize("NFKC") // compatibility forms, full-width -> ASCII
.toLowerCase()
.normalize("NFD")
.replace(/\p{Mn}/gu, "") // strip combining marks
.replace(/[\u200B-\u200D\uFEFF]/g, "") // zero-width joiners and spaces
.replace(/(.)\1{2,}/gu, "$1$1") // collapse runs of 3+ to 2
.normalize("NFC");
}
Two cautions on that function. Stripping combining marks is right for matching and wrong for storage — never write the folded form back. And it must not be applied blindly to every language: in Vietnamese the diacritics are the letters, so stripping them merges distinct words, as Vietnamese diacritics and search matching sets out. Fold per language, using the rules for that language, not one global fold.
Beyond folding: map confusable characters using Unicode’s security-mechanisms data on confusables rather than a hand-written homoglyph table, match on stems or lemmas where you have a stemmer for the language, and for spaceless scripts segment with an ICU break iterator before matching rather than falling back to substrings.
A coverage audit that fails the build
- Enumerate the languages actually present in your traffic, by running detection over a sample of real messages. Not the languages your UI is translated into — those are different sets and the difference is usually large.
- For each language above a traffic threshold, assert at load time that a list exists and has a minimum number of entries. Make a missing list a startup failure, not a warning. This one change converts the entire class of bug from silent to loud.
- Emit a per-check event recording which lists were consulted. A verdict of “clean” with an empty list set is a distinct outcome and should be queryable.
- Run every applicable list on every message rather than routing by detected language. Lists are small and the cost is negligible next to the cost of routing a code-switched message wrongly.
- Build a small labelled evaluation set per language — a few hundred items, native speakers, including hard negatives such as reclaimed usage and homographs — and report recall and precision per language. Never aggregate: an aggregate figure improves when English traffic grows.
- For languages where morphology or code-switching makes a list hopeless, escalate to a multilingual classifier for that subset only, and accept the per-item cost on a fraction of traffic rather than all of it.
A list is a lower bound on what a language needs and never the whole control. The realistic goal is that the filter’s coverage is known and reported per language, so that the gap is a documented risk rather than an assumption nobody has tested.
Top comments (0)