Most text processing assumes tokens are separated by spaces. Japanese and Korean quietly violate that assumption in different ways, and pipelines built on English defaults produce subtly wrong output rather than obvious errors.
Japanese: no spaces at all
私は本を読んでいます
No delimiters. Segmentation requires a morphological analyser with a dictionary — MeCab, Sudachi, Janome. And they legitimately disagree about compound nouns: 東京都 can be one token or 東京 + 都 depending on the dictionary, and both are defensible.
Consequences that catch people out:
- Character count is not word count. Any length heuristic tuned on English is meaningless here.
-
Naive
split()yields one token. Anything downstream — TF-IDF, keyword extraction, readability — silently produces garbage rather than failing. - Three scripts interleave. Hiragana, katakana and kanji mix within a single word. Script-based splitting seems clever and is wrong.
Korean: spaces exist but agglutinate
Korean has spacing, which lulls you into thinking split() works. It does not, because particles attach to stems:
학교에서 school + at
학교를 school + (object marker)
학교에서 and 학교를 are the same noun with different particles. Without morphological analysis they are two unrelated vocabulary items, which wrecks frequency counts and matching.
Hangul also composes: 한 is a single syllable block built from three jamo. Depending on Unicode normalisation (NFC vs NFD) the same visible text has different code point counts — a classic source of off-by-N bugs in length limits.
What this means for detection and rewriting
Anything scoring "how machine-like is this text" on English features — burstiness, sentence-length variance, function word ratios — needs those features recomputed on morphemes, not characters or space-delimited chunks. Ported directly, the scores are noise.
Rewriting is harder still. Preserving meaning while altering phrasing requires respecting:
- Keigo levels (Japanese politeness registers) — casually rewriting 丁寧語 into plain form changes social meaning entirely
- Korean speech levels — same problem, different system
- Particle correctness after any reordering
Practical stack
- Normalise Unicode first (NFC), before anything else touches the text
- Morphological analysis with an explicit dictionary choice — pin the version, results change between them
- Compute features over morphemes
- Validate on native text, never on machine-translated English
That last one matters most: translated text has different statistical properties from natively written text, so validating on it produces a system tuned for the wrong distribution.
This is the problem I work on at Suikou AI — multilingual humanizing and detection with the morphological layer done properly for Japanese and Korean.
Caveat
Analyser choice is a real trade-off. Sudachi has better modern coverage, MeCab has more ecosystem tooling. Neither is strictly correct — pick one and pin it, because switching mid-project invalidates every threshold you tuned.
Top comments (0)