Unicode normalization is not a deduplication key

I had 84,000 kaomoji to deduplicate and a key that looked obviously correct:

key = "".join(unicodedata.normalize("NFKC", text).split())

Strip the whitespace, normalize the Unicode, group by the result. Every
deduplication tutorial I have ever read says to normalize first. It took me one
afternoon of reviewing the output to realize that this key had quietly decided
(╥_╥) and (╥﹏╥) were the same record.

They are two different faces. The wavy mouth is a harder cry than the flat one.

NFKC folds four characters that matter here

NFKC is compatibility normalization. The K stands for compatibility, and the
point of it is to fold characters that exist in Unicode only for round-tripping
with older encodings: the fullwidth Latin letters, the halfwidth katakana, the
circled numbers, the ligatures. It maps them onto their "normal" equivalents so
that ｈｅｌｌｏ and hello compare equal.

That is correct when the text is prose and you want Ｅ－ｍａｉｌ to match
E-mail. It stops being correct once the characters are the content themselves.
Here is what it does to four characters that carry meaning in a kaomoji:

Codepoint	Name	NFKC result
U+FE4F	WAVY LOW LINE ﹏	_ (U+005F)
U+FF3C	FULLWIDTH REVERSE SOLIDUS ＼	\ (U+005C)
U+FF64	HALFWIDTH IDEOGRAPHIC COMMA ､	、 (U+3001)
U+3000	IDEOGRAPHIC SPACE	(U+0020)

The first one is the crying face. ﹏ is a wavy mouth and _ is a flat one, and
NFKC folds them together because one of them was originally a typesetting variant
of the other. The second one is a pair of arms. ＼(^o^)／ is a wide celebratory
shrug, \(^o^)/ is a narrow one, and after NFKC they are the same string.

The third one is a case where NFKC is right. A fullwidth and a halfwidth
ideographic comma really are the same punctuation mark, and (≖、≖╬) and
(≖､≖╬) really are one entry. That is what makes this hard. The normalization
is not simply wrong. It is right about maybe a third of what it touches, and you
can't tell which third without looking.

233 proposed merges, or 6,944

This is easy to measure on a published dataset, so you do not have to take my word
for it. The file below is the merged corpus, 82,109 entries, already deduplicated
on exact string match. Every collision found here is a proposed merge between
two entries that a human previously decided were different.

import gzip, json, unicodedata as ud, collections

def groups(keyfn):
    buckets = collections.defaultdict(set)
    for line in gzip.open("data/kaomoji.jsonl.gz", "rt", encoding="utf-8"):
        t = json.loads(line)["text"]
        buckets[keyfn(t)].add(t)
    g = [v for v in buckets.values() if len(v) > 1]
    return len(g), sum(len(v) for v in g)

strip_only = lambda t: "".join(t.split())
with_nfkc  = lambda t: "".join(ud.normalize("NFKC", t).split())

print(groups(strip_only))   # (233, 500)
print(groups(with_nfkc))    # (6944, 15038)

Stripping whitespace alone proposes 233 merges. Adding NFKC proposes 6,944,
covering 15,038 of the 82,109 entries. Auto-merging on that key deletes about
8,000 records, and you never find out which ones, because a merge leaves no
trace.

Some of what it finds is genuinely useful. These are real groups from that run:

(╥_╥)              (╥﹏╥)
(・∀・)ノ            (･∀･)ノ
(╯°□°)╯︵ ┻━┻       (╯°□°）╯︵ ┻━┻
(T_T)              (T＿T)          (Ｔ＿Ｔ)

The second row is a real duplicate. Halfwidth versus fullwidth katakana middle
dot is not something anyone chose, it is an artifact of which IME the author was
typing in. The third row is a table flip where one of the two parentheses came
out fullwidth, which is also an accident. The first row is two different faces.
The fourth row is three faces if you care about stroke weight and one face if you
don't.

There is no rule that separates those four rows. I tried several: character
class, width consistency within a single entry, whether the fold lands in the
mouth region. All of them got the first row wrong in one direction or the other.

Whitespace carries pose information

Stripping whitespace has the same problem on a smaller scale, because whitespace
is load-bearing in this data:

ʕ·ᴥ·ʔ          bear, facing you
ʕ·ᴥ·　ʔ         bear, looking left
ʕ　·ᴥ·ʔ         bear, looking right

( •_•)>⌐■-■    before the sunglasses
(•_•)>⌐■-■     also before the sunglasses, typed by someone else

The first three are three entries. The last two are one. Whether a given space is
a pose or a typo is not recoverable from the string, and I don't think it is
recoverable at all without knowing what the thing depicts.

Three keys, one of which is allowed to merge

key_exact = text                       # byte-identical; safe to auto-merge
key_loose = "".join(text.split())      # whitespace only; PROPOSES merges
key_nfkc  = "".join(ud.normalize("NFKC", text).split())   # output field only

key_exact collisions merge automatically with no review. key_loose collisions
go to a human queue, and 233 groups is an afternoon. key_nfkc is computed,
stored as a field in the dataset, and never consulted by the merge pipeline.
Consumers who want aggressive folding for search or for their own matching can
have it. It just doesn't get to delete anything.

That last key is the part I got wrong twice. My first instinct after it burned me
was to rip it out entirely, which would have been the opposite mistake, because
the folding is genuinely useful for recall. If somebody searches for (╥﹏╥) you
want the flat-mouthed one in the results too. So I kept computing it and took
away its ability to destroy a record.

Keep the queue small enough that you will actually read it

Normalization tells you which records might be the same. Whether they are the
same is a different question, and I am
not sure that one has an algorithmic answer.

So normalize to find candidates, keep the raw string as the only identity, and
put a human in front of anything that would destroy a record. If the human queue
is too big to review, the candidate key is too aggressive and needs tightening.
Nobody reviews 6,944 groups. They trust the key and move on, which is exactly how
the records disappear without anyone noticing.

---

The corpus is on GitHub with the raw JSONL and the verification scripts, and on
npm and PyPI as kaomoji-dataset if you just want the data:

npm i kaomoji-dataset
pip install kaomoji-dataset

https://github.com/Funovate/fontvibe-kaomoji

If you want to look at the entries rather than the code, the browsable version is
at fontvibe.ai/tools/kaomoji. Data is CC BY
4.0.