What Breaks When Your App Has to Reason About Chinese Characters

Yu'an Zhou — Sat, 25 Jul 2026 08:10:47 +0000

Most internationalization advice assumes text is a bag of words separated by spaces. Chinese breaks that assumption at every layer of the stack, and if your product's core value depends on reasoning about individual characters — not just displaying them — you end up rewriting infrastructure you assumed was solved.

I spend most of my time on a system that analyzes Chinese characters for naming: stroke counts, five-element (五行) classification, tone patterns, and sourcing candidate characters from classical poetry corpora. Here's what actually bit us.

A "Character" Is Not a Character

The first thing that breaks is len().

name = "𠮷田"          # note: the rare variant of 吉, U+20BB7
print(len(name))       # Python 3: 2 -- correct

Python 3 gets this right because strings are sequences of code points. JavaScript does not:

const name = "𠮷田";
name.length            // 3  -- UTF-16 surrogate pair counted as two
[...name].length       // 2  -- correct

Any CJK-aware code path in JS must use Array.from() / spread / for...of, never .length or charAt. This matters immediately for us because a name's stroke-count math is per-character. Counting one character as two silently produces a wrong numerological result — and it never throws, it just returns a plausible wrong answer. Those are the worst bugs.

The second thing that breaks is your database column. Extension B/C/D characters (U+20000 and up) are 4-byte UTF-8. MySQL's utf8 charset is a 3-byte subset and will truncate or reject them:

ALTER TABLE characters
  CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;

Note utf8mb4_bin, not a general collation. Unicode collations for CJK do fuzzy things you never want in a lookup table — you want exact code point identity.

Normalization Will Quietly Destroy Your Data

Everyone reaches for NFKC because it "cleans up" input. For CJK it is destructive.

Unicode has a compatibility block of CJK ideographs (U+F900–U+FAFF) that exists purely for round-tripping legacy encodings. NFC and NFKC both map most of them onto the unified codepoint:

import unicodedata
s = "例"                        # CJK COMPATIBILITY IDEOGRAPH-F9B5
unicodedata.normalize("NFC", s)     # -> the unified ideograph, different codepoint

That's usually fine. What is not fine is NFKC on mixed content: it converts fullwidth forms to ASCII (Ａ → A), collapses fullwidth punctuation, and rewrites halfwidth kana. In a naming corpus, fullwidth vs halfwidth is real signal about the source document.

Our rule ended up being: NFC for storage, never NFKC, and normalize once at the ingest boundary. Anything that normalizes on read will eventually disagree with an index built before the rule changed.

Then there's variation selectors. The Ideographic Variation Sequences at U+E0100+ attach to a base ideograph to select a specific glyph, and they survive NFC. Dedupe candidates with a plain hash set and 葛 and 葛 + U+E0100 are two entries. We strip variation selectors for the index key but preserve them on the display string — two fields, not one.

Traditional ↔ Simplified Is Not a Lookup Table

This is where most implementations get sloppy. The mapping is not a bijection.

One simplified character maps to several traditional ones: 发 is both 發 (to emit) and 髮 (hair). 后 covers 後 (after) and 后 (empress).
Some conversions are only correct in one direction.
Context decides, and character-level tables have no context.

For anything user-facing, use OpenCC rather than a homemade table — it does phrase-level conversion, which resolves a good chunk of the ambiguity:

from opencc import OpenCC
cc = OpenCC('s2twp')   # simplified -> traditional, Taiwan idiom
cc.convert("头发")      # 頭髮 -- phrase context picks 髮, not 發

But for a naming system, phrase context often doesn't exist — you're evaluating a single character in isolation. Our approach: store the character in both scripts as separate rows with an explicit variant_group_id, and never auto-convert at query time. Let the user pick the script; don't guess.

Stroke count has the same shape of problem. It's not one number per character — it depends on the script (简 vs 繁), and traditional counts differ between the Kangxi dictionary convention and modern standards for radicals like 艹 (3 strokes in modern counting, 6 under Kangxi where it is treated as 艸). Naming numerology systems typically demand Kangxi counts specifically. If you pull stroke data from a general Unihan field like kTotalStrokes, you get the modern count, and your results will be off for a large fraction of characters. Unihan is the right source, but you have to read the right property. Getting that right — Kangxi strokes, five-element classification, and tone-pattern checking against 诗经/楚辞 source lines — is the core of what we ship at BabyNameAI, and the stroke-count semantics were more work than the entire model layer sitting on top.

Indexing Classical Chinese

Classical Chinese has no word delimiters and no reliable segmenter — tools like jieba are trained on modern Mandarin and will happily mis-segment a Tang line. So don't segment.

Index character n-grams instead:

-- Postgres: trigram index sidesteps a CJK-hostile default text parser
CREATE EXTENSION pg_trgm;
CREATE INDEX ON poem_lines USING gin (line gin_trgm_ops);

For Elasticsearch, the built-in cjk analyzer (which produces bigrams) beats a standard analyzer, and beats ik_smart for classical text since IK's dictionary is modern vocabulary. Bigrams over a few hundred thousand lines of 诗经/楚辞/唐诗 is a small index and it recalls correctly on the queries that matter: "find every line containing this character in second position of a two-character compound."

One last trap: tone data. Pinyin tone marks are combining diacritics under NFD (zhōng decomposes to o + U+0304) but precomposed under NFC, so comparing differently-normalized pinyin silently fails. And classical tone categories (平上去入) do not map onto modern Mandarin tones — the entering tone 入声 disappeared in Mandarin but survives in Cantonese and Min. Checking a name against classical prosody rules needs a 中古音 rime table, not modern pinyin.

None of this is exotic once you've hit it. Every "just use Unicode" tutorial simply stops right before the interesting part.

Shipping a Chrome Extension That Works Across Sites You Do Not Control

Yu'an Zhou — Fri, 24 Jul 2026 19:21:11 +0000

Building a Chrome extension that saves items from arbitrary third-party sites means every site is a hostile, unstable environment. Here is what actually broke, and what held.

Manifest V3 changes the shape of everything

The persistent background page is gone. Service workers terminate — aggressively, after roughly 30 seconds of inactivity — and any in-memory state dies with them.

Anything that must survive goes into chrome.storage:

// ❌ dies with the worker
let cache = {}

// ✅ survives
await chrome.storage.local.set({ cache })

This is the single most common source of "works when I test it, fails in the wild" bugs. Your worker is alive during active testing and dead during real usage patterns.

Content script isolation cuts both ways

Content scripts run in an isolated world: you see the DOM, not the page's JavaScript. You cannot read the page's variables, and the page cannot see yours.

Good for safety, awkward when the data you want lives in a framework's state rather than the DOM. Options are injecting into the main world (more risk, more capability) or scraping rendered DOM (fragile but isolated). I default to DOM scraping and accept the fragility.

Selectors rot, so degrade instead of breaking

Third-party markup changes without warning. A selector working today breaks next week with no notice and no error you will see.

What helps:

Layered fallbacks — semantic markup first (article, [itemprop], OpenGraph tags), site-specific selectors only as a last resort
Fail visibly, not silently — if extraction fails, tell the user rather than saving an empty record
Never assume shape — el?.textContent?.trim() ?? '' everywhere; a null deref in a content script can break the host page, which is much worse than your feature not working

Permissions: ask for less

<all_urls> triggers heavier review and scares users at install time. activeTab plus optional_host_permissions requested at first use is a better trade — fewer install-time objections, and review goes faster.

Store review realities

Justify every permission in the listing, specifically. Vague justifications get rejected.
Privacy policy required if you touch user data at all.
First review is slow; updates are usually much faster.
A permission added later triggers full re-review — worth batching permission changes rather than shipping them one at a time.

I shipped one of these for BabyNameAi — cross-site saving, a daily classical-poetry card, and offline lookup.

Caveat

Extension APIs are still shifting under MV3. Anything you read from before 2023 — including plenty of still-top-ranked results — may describe APIs that no longer exist.

DEV Community: Yu'an Zhou