Two Chinese phrases that any reader would call the same thing come back with a cosine similarity you would expect from unrelated text. The reason is upstream of the model: the tokenizer decided where the words were, and it decided by counting, not by understanding.
There are no word boundaries to read
Written Chinese has no spaces. A sentence is a run of characters, and the reader segments it into words as part of comprehension. Every system that processes it has to make the same decision, and unlike a reader it has to make it before it has understood anything.
Modern embedding models do not run a word segmenter. They run a byte-level BPE tokenizer over the UTF-8 bytes, and a common Chinese character occupies three bytes. The merge table was learned by counting which byte sequences co-occur in the training corpus, so a high-frequency two-character word becomes a single token, while a valid but rarer word stays as separate character tokens — or, worse, gets split across a byte boundary and shares no unit with anything.
This is also why the usual lexical fallback is unavailable. In an English system, when the vector search misses you still have BM25 on whitespace-delimited terms. In Chinese there are no terms until something produces them, so a hybrid index needs either a segmenter such as jieba or an ICU break iterator, or a character-bigram field. Skipping that is how a Chinese pipeline ends up with vector search as its only recall mechanism.
A worked ambiguous string
Take 研究生命起源. There are two readings:
研究生 / 命起源 → "graduate student" + a fragment
研究 / 生命 / 起源 → "research" + "life" + "origin"
The second is the sensible one — research into the origin of life — but 研究生 (graduate student) is an extremely common three-character word, so a merge table built by frequency has a strong incentive to represent it as one unit. If it does, the sentence is tokenized around a word that is not in it, and every downstream representation is built on that mistake.
The classic example is 南京市长江大桥, which reads either as Nanjing City / Yangtze River Bridge or as Nanjing / Mayor / Jiang Daqiao, a person’s name. Both segmentations are grammatical. Nothing in the character sequence decides it; only the surrounding context does, and the tokenizer does not look at context.
A transformer can partly recover from a bad segmentation, because attention lets later layers recombine whatever the tokenizer produced. Partly is the operative word. The recovery is better when the pieces are frequent characters the model has seen in many contexts and worse when they are rare, which means it is worst exactly where you need it most: technical vocabulary, product names and transliterated foreign words.
Frequency decides the unit, not meaning
Here is the part that explains the synonym behaviour. Suppose 电脑 (computer, the everyday word) is frequent enough in the training corpus to be one token, and 计算机 (computer, the formal or technical word) is not, so it becomes three character tokens.
The model’s representation of the single frequent token is learned directly from tens of thousands of contexts. Its representation of the three-token sequence is composed at inference time from three units, each of which appears in an enormous range of unrelated words — 机 alone appears in machine, aircraft, opportunity and organic. The composition can be good, but it is a different kind of object: assembled rather than memorised. The two representations are not built the same way, so there is no reason for them to be near-identical, and cosine similarity between them reflects that.
Frequency also varies by variety. Mainland, Taiwanese, Hong Kong and Singaporean written Chinese differ in vocabulary, so which member of a synonym set is the frequent one depends on which corpus the tokenizer was trained on. A model trained mostly on mainland text will treat mainland vocabulary as units and treat the Taiwanese equivalent as fragments.
The synonym pairs this breaks
- Register pairs.
电脑against计算机;土豆against马铃薕for potato. Colloquial and formal forms of the same referent, with very different corpus frequencies. - Regional pairs.
出租车,的士and计程车are all taxi;软件and软体are both software. A user in one market types one and your documentation uses another. - Script variants. Simplified and traditional forms of the same word are different code points entirely, so
头发and頭髮share nothing. Conversion is not a clean round trip either: traditional後(after) and后(empress) both simplify to后, so converting back is a guess. See how models handle the two scripts.
What actually helps
In rough order of effort against payoff:
- Normalize script variant at ingest and at query time, in the same direction, and keep the original for display. Do this before you measure anything else, or your measurements include a bug.
- Add a lexical field. A character-bigram index needs no segmenter and no dictionary: index every adjacent character pair, do the same to the query, and run BM25 over that. It recovers exact-substring matches that vector search drops, and it is immune to segmentation entirely.
- Expand queries with a synonym table for the terms your users actually type. This is unfashionable and it works, because the failure is a vocabulary mismatch and a vocabulary mismatch has a vocabulary fix. A few hundred entries covers most of a product domain.
- Fuse the two result lists by rank rather than by score. Vector scores and BM25 scores are not on the same scale and normalizing them is guesswork; reciprocal rank fusion needs no calibration.
- Only then evaluate a different embedding model, on a test set of your own synonym pairs. Chunk sizes matter here too, since a token budget buys far fewer characters in Chinese than the same budget in English — size CJK chunks in tokens, not characters.
The diagnostic that tells you which problem you have is cheap: embed both members of ten synonym pairs and ten unrelated pairs, and compare the two distributions. If synonym pairs are not clearly separated from unrelated pairs, the model is the problem. If they are separated but your live queries still miss, the problem is segmentation or normalization in the pipeline around it.
Top comments (0)