What Breaks When Your App Has to Reason About Chinese Characters

Most internationalization advice assumes text is a bag of words separated by spaces. Chinese breaks that assumption at every layer of the stack, and if your product's core value depends on reasoning about individual characters — not just displaying them — you end up rewriting infrastructure you assumed was solved.

I spend most of my time on a system that analyzes Chinese characters for naming: stroke counts, five-element (五行) classification, tone patterns, and sourcing candidate characters from classical poetry corpora. Here's what actually bit us.

A "Character" Is Not a Character

The first thing that breaks is len().

name = "𠮷田"          # note: the rare variant of 吉, U+20BB7
print(len(name))       # Python 3: 2 -- correct

Python 3 gets this right because strings are sequences of code points. JavaScript does not:

const name = "𠮷田";
name.length            // 3  -- UTF-16 surrogate pair counted as two
[...name].length       // 2  -- correct

Any CJK-aware code path in JS must use Array.from() / spread / for...of, never .length or charAt. This matters immediately for us because a name's stroke-count math is per-character. Counting one character as two silently produces a wrong numerological result — and it never throws, it just returns a plausible wrong answer. Those are the worst bugs.

The second thing that breaks is your database column. Extension B/C/D characters (U+20000 and up) are 4-byte UTF-8. MySQL's utf8 charset is a 3-byte subset and will truncate or reject them:

ALTER TABLE characters
  CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;

Note utf8mb4_bin, not a general collation. Unicode collations for CJK do fuzzy things you never want in a lookup table — you want exact code point identity.

Normalization Will Quietly Destroy Your Data

Everyone reaches for NFKC because it "cleans up" input. For CJK it is destructive.

Unicode has a compatibility block of CJK ideographs (U+F900–U+FAFF) that exists purely for round-tripping legacy encodings. NFC and NFKC both map most of them onto the unified codepoint:

import unicodedata
s = "例"                        # CJK COMPATIBILITY IDEOGRAPH-F9B5
unicodedata.normalize("NFC", s)     # -> the unified ideograph, different codepoint

That's usually fine. What is not fine is NFKC on mixed content: it converts fullwidth forms to ASCII (Ａ → A), collapses fullwidth punctuation, and rewrites halfwidth kana. In a naming corpus, fullwidth vs halfwidth is real signal about the source document.

Our rule ended up being: NFC for storage, never NFKC, and normalize once at the ingest boundary. Anything that normalizes on read will eventually disagree with an index built before the rule changed.

Then there's variation selectors. The Ideographic Variation Sequences at U+E0100+ attach to a base ideograph to select a specific glyph, and they survive NFC. Dedupe candidates with a plain hash set and 葛 and 葛 + U+E0100 are two entries. We strip variation selectors for the index key but preserve them on the display string — two fields, not one.

Traditional ↔ Simplified Is Not a Lookup Table

This is where most implementations get sloppy. The mapping is not a bijection.

• One simplified character maps to several traditional ones: 发 is both 發 (to emit) and 髮 (hair). 后 covers 後 (after) and 后 (empress).
• Some conversions are only correct in one direction.
• Context decides, and character-level tables have no context.

For anything user-facing, use OpenCC rather than a homemade table — it does phrase-level conversion, which resolves a good chunk of the ambiguity:

from opencc import OpenCC
cc = OpenCC('s2twp')   # simplified -> traditional, Taiwan idiom
cc.convert("头发")      # 頭髮 -- phrase context picks 髮, not 發

But for a naming system, phrase context often doesn't exist — you're evaluating a single character in isolation. Our approach: store the character in both scripts as separate rows with an explicit variant_group_id, and never auto-convert at query time. Let the user pick the script; don't guess.

Stroke count has the same shape of problem. It's not one number per character — it depends on the script (简 vs 繁), and traditional counts differ between the Kangxi dictionary convention and modern standards for radicals like 艹 (3 strokes in modern counting, 6 under Kangxi where it is treated as 艸). Naming numerology systems typically demand Kangxi counts specifically. If you pull stroke data from a general Unihan field like kTotalStrokes, you get the modern count, and your results will be off for a large fraction of characters. Unihan is the right source, but you have to read the right property. Getting that right — Kangxi strokes, five-element classification, and tone-pattern checking against 诗经/楚辞 source lines — is the core of what we ship at BabyNameAI, and the stroke-count semantics were more work than the entire model layer sitting on top.

Indexing Classical Chinese

Classical Chinese has no word delimiters and no reliable segmenter — tools like jieba are trained on modern Mandarin and will happily mis-segment a Tang line. So don't segment.

Index character n-grams instead:

-- Postgres: trigram index sidesteps a CJK-hostile default text parser
CREATE EXTENSION pg_trgm;
CREATE INDEX ON poem_lines USING gin (line gin_trgm_ops);

For Elasticsearch, the built-in cjk analyzer (which produces bigrams) beats a standard analyzer, and beats ik_smart for classical text since IK's dictionary is modern vocabulary. Bigrams over a few hundred thousand lines of 诗经/楚辞/唐诗 is a small index and it recalls correctly on the queries that matter: "find every line containing this character in second position of a two-character compound."

One last trap: tone data. Pinyin tone marks are combining diacritics under NFD (zhōng decomposes to o + U+0304) but precomposed under NFC, so comparing differently-normalized pinyin silently fails. And classical tone categories (平上去入) do not map onto modern Mandarin tones — the entering tone 入声 disappeared in Mandarin but survives in Cantonese and Min. Checking a name against classical prosody rules needs a 中古音 rime table, not modern pinyin.

None of this is exotic once you've hit it. Every "just use Unicode" tutorial simply stops right before the interesting part.