DEV Community: jack

Unicode normalization is not a deduplication key

jack — Tue, 22 Sep 2026 17:29:47 +0000

I had 84,000 kaomoji to deduplicate and a key that looked obviously correct:

key = "".join(unicodedata.normalize("NFKC", text).split())

Strip the whitespace, normalize the Unicode, group by the result. Every
deduplication tutorial I have ever read says to normalize first. It took me one
afternoon of reviewing the output to realize that this key had quietly decided
(╥_╥) and (╥﹏╥) were the same record.

They are two different faces. The wavy mouth is a harder cry than the flat one.

NFKC folds four characters that matter here

NFKC is compatibility normalization. The K stands for compatibility, and the
point of it is to fold characters that exist in Unicode only for round-tripping
with older encodings: the fullwidth Latin letters, the halfwidth katakana, the
circled numbers, the ligatures. It maps them onto their "normal" equivalents so
that ｈｅｌｌｏ and hello compare equal.

That is correct when the text is prose and you want Ｅ－ｍａｉｌ to match
E-mail. It stops being correct once the characters are the content themselves.
Here is what it does to four characters that carry meaning in a kaomoji:

Codepoint	Name	NFKC result
U+FE4F	WAVY LOW LINE `﹏`	`_` (U+005F)
U+FF3C	FULLWIDTH REVERSE SOLIDUS `＼`	`\` (U+005C)
U+FF64	HALFWIDTH IDEOGRAPHIC COMMA `､`	`、` (U+3001)
U+3000	IDEOGRAPHIC SPACE	(U+0020)

The first one is the crying face. ﹏ is a wavy mouth and _ is a flat one, and
NFKC folds them together because one of them was originally a typesetting variant
of the other. The second one is a pair of arms. ＼(^o^)／ is a wide celebratory
shrug, \(^o^)/ is a narrow one, and after NFKC they are the same string.

The third one is a case where NFKC is right. A fullwidth and a halfwidth
ideographic comma really are the same punctuation mark, and (≖、≖╬) and
(≖､≖╬) really are one entry. That is what makes this hard. The normalization
is not simply wrong. It is right about maybe a third of what it touches, and you
can't tell which third without looking.

233 proposed merges, or 6,944

This is easy to measure on a published dataset, so you do not have to take my word
for it. The file below is the merged corpus, 82,109 entries, already deduplicated
on exact string match. Every collision found here is a proposed merge between
two entries that a human previously decided were different.

import gzip, json, unicodedata as ud, collections

def groups(keyfn):
    buckets = collections.defaultdict(set)
    for line in gzip.open("data/kaomoji.jsonl.gz", "rt", encoding="utf-8"):
        t = json.loads(line)["text"]
        buckets[keyfn(t)].add(t)
    g = [v for v in buckets.values() if len(v) > 1]
    return len(g), sum(len(v) for v in g)

strip_only = lambda t: "".join(t.split())
with_nfkc  = lambda t: "".join(ud.normalize("NFKC", t).split())

print(groups(strip_only))   # (233, 500)
print(groups(with_nfkc))    # (6944, 15038)

Stripping whitespace alone proposes 233 merges. Adding NFKC proposes 6,944,
covering 15,038 of the 82,109 entries. Auto-merging on that key deletes about
8,000 records, and you never find out which ones, because a merge leaves no
trace.

Some of what it finds is genuinely useful. These are real groups from that run:

(╥_╥)              (╥﹏╥)
(・∀・)ノ            (･∀･)ノ
(╯°□°)╯︵ ┻━┻       (╯°□°）╯︵ ┻━┻
(T_T)              (T＿T)          (Ｔ＿Ｔ)

The second row is a real duplicate. Halfwidth versus fullwidth katakana middle
dot is not something anyone chose, it is an artifact of which IME the author was
typing in. The third row is a table flip where one of the two parentheses came
out fullwidth, which is also an accident. The first row is two different faces.
The fourth row is three faces if you care about stroke weight and one face if you
don't.

There is no rule that separates those four rows. I tried several: character
class, width consistency within a single entry, whether the fold lands in the
mouth region. All of them got the first row wrong in one direction or the other.

Whitespace carries pose information

Stripping whitespace has the same problem on a smaller scale, because whitespace
is load-bearing in this data:

ʕ·ᴥ·ʔ          bear, facing you
ʕ·ᴥ·　ʔ         bear, looking left
ʕ　·ᴥ·ʔ         bear, looking right

( •_•)>⌐■-■    before the sunglasses
(•_•)>⌐■-■     also before the sunglasses, typed by someone else

The first three are three entries. The last two are one. Whether a given space is
a pose or a typo is not recoverable from the string, and I don't think it is
recoverable at all without knowing what the thing depicts.

Three keys, one of which is allowed to merge

key_exact = text                       # byte-identical; safe to auto-merge
key_loose = "".join(text.split())      # whitespace only; PROPOSES merges
key_nfkc  = "".join(ud.normalize("NFKC", text).split())   # output field only

key_exact collisions merge automatically with no review. key_loose collisions
go to a human queue, and 233 groups is an afternoon. key_nfkc is computed,
stored as a field in the dataset, and never consulted by the merge pipeline.
Consumers who want aggressive folding for search or for their own matching can
have it. It just doesn't get to delete anything.

That last key is the part I got wrong twice. My first instinct after it burned me
was to rip it out entirely, which would have been the opposite mistake, because
the folding is genuinely useful for recall. If somebody searches for (╥﹏╥) you
want the flat-mouthed one in the results too. So I kept computing it and took
away its ability to destroy a record.

Keep the queue small enough that you will actually read it

Normalization tells you which records might be the same. Whether they are the
same is a different question, and I am
not sure that one has an algorithmic answer.

So normalize to find candidates, keep the raw string as the only identity, and
put a human in front of anything that would destroy a record. If the human queue
is too big to review, the candidate key is too aggressive and needs tightening.
Nobody reviews 6,944 groups. They trust the key and move on, which is exactly how
the records disappear without anyone noticing.

The corpus is on GitHub with the raw JSONL and the verification scripts, and on
npm and PyPI as kaomoji-dataset if you just want the data:

npm i kaomoji-dataset
pip install kaomoji-dataset

https://github.com/Funovate/fontvibe-kaomoji

If you want to look at the entries rather than the code, the browsable version is
at fontvibe.ai/tools/kaomoji. Data is CC BY
4.0.

I merged the seven public kaomoji collections. They barely overlap.

jack — Tue, 22 Sep 2026 16:02:59 +0000

I wanted a complete list of kaomoji, the ( ͡° ͜ʖ ͡°) and ¯\_(ツ)_/¯ family. There isn't one.
So I merged the public collections instead, and the merge turned up something I
wasn't expecting: they hardly overlap at all.

Seven collections, 21 pairs, 0.3% median overlap

The ones I found:

fdw/rofimoji, the biggest English-side set
Allaman/emoji.nvim
aoguai/rime_kaomoji_dict
mtripg6666tdr/Kaomoji_proj
the Japanese IME dictionaries
kaosute
an older list of my own

Across the 21 pairwise combinations the median Jaccard overlap is 0.3%. Four
pairs share not a single entry. One pair clears 5%, rofimoji and emoji.nvim,
which are both English-side Linux pickers, so that one is no surprise.

The IME dictionaries are where most kaomoji actually live, and they share almost
nothing with any of the English sets.

Each entry in the merged file keeps the sources it came from, so you can check
this yourself:

import gzip, json, itertools, collections, statistics

by = collections.defaultdict(set)
for line in gzip.open("data/kaomoji.jsonl.gz", "rt", encoding="utf-8"):
    r = json.loads(line)
    for src in r["sources"]:
        by[src].add(r["text"])

o = [len(by[a] & by[b]) / len(by[a] | by[b]) for a, b in itertools.combinations(by, 2)]
print(statistics.median(o), max(o), sum(1 for x in o if x == 0))
# 0.003  0.079  4

None of the collections is close to complete, and after seeing those numbers I
understand why.

Coverage checked at run time

Saying "this has everything" is not worth much on its own, so
scripts/verify_coverage.py downloads each upstream file when you run it and
diffs it against the merged set. Nothing is precomputed.

Upstream	Entries	Covered	Missing
rofimoji	1,562	100.0%	0
emoji.nvim	2,016	99.9%	3
rime_kaomoji_dict	959	99.4%	6
Kaomoji_proj	2,166	97.8%	48

It prints all 57 misses, so you can look at them and disagree with me about what
counts as a kaomoji. Three rounds of that took rofimoji from 93.0% to 100.0%.

Labels, and where they fall down

Deduplicated it comes to 82,109 entries. Around 70k carry emotion, intent and
subject labels from a controlled vocabulary in en, ja, zh, es, pt and de, so
嬉しい, 开心, feliz and happy return the same entries instead of whichever English
word the original author happened to use.

Three things worth knowing before you use it:

The labels are coarse. 69,679 labelled entries share only 605 distinct label combinations, because they were applied in bundles rather than judged one at a time. Fine for pulling up a mood, not for finding one specific face.
About 15% is left unlabelled rather than guessed at.
Most of the corpus is 顔文字＋セリフ, a face plus a line of Japanese dialogue, which is not much use if you cannot read it.

That last one is why the English-language picker at
fontvibe.ai/tools/kaomoji shows 11,020, the
subset that renders safely outside Japanese contexts, while the full 82,109 sits
in the dataset and on the Japanese page.

Getting it

npm i kaomoji-dataset
pip install kaomoji-dataset

Both read a gzip off disk, so they are Node and Python only for now. They will
not bundle for a browser. The repo has the raw JSONL and a CSV as well:
github.com/Funovate/fontvibe-kaomoji

106 of the entries are not collected from anywhere. I drew them, for feelings
that had no kaomoji yet: 躺平, 社死, saudade, mamihlapinatapai. Those are CC0.

If you know of a collection I missed, say so and I will run the coverage script
against it.