DEV Community

jack
jack

Posted on AI-assisted

I merged the seven public kaomoji collections. They barely overlap.

I wanted a complete list of kaomoji, the ( ͡° ͜ʖ ͡°) and ¯\_(ツ)_/¯ family. There isn't one.
So I merged the public collections instead, and the merge turned up something I
wasn't expecting: they hardly overlap at all.

Seven collections, 21 pairs, 0.3% median overlap

The ones I found:

Across the 21 pairwise combinations the median Jaccard overlap is 0.3%. Four
pairs share not a single entry. One pair clears 5%, rofimoji and emoji.nvim,
which are both English-side Linux pickers, so that one is no surprise.

The IME dictionaries are where most kaomoji actually live, and they share almost
nothing with any of the English sets.

Each entry in the merged file keeps the sources it came from, so you can check
this yourself:

import gzip, json, itertools, collections, statistics

by = collections.defaultdict(set)
for line in gzip.open("data/kaomoji.jsonl.gz", "rt", encoding="utf-8"):
    r = json.loads(line)
    for src in r["sources"]:
        by[src].add(r["text"])

o = [len(by[a] & by[b]) / len(by[a] | by[b]) for a, b in itertools.combinations(by, 2)]
print(statistics.median(o), max(o), sum(1 for x in o if x == 0))
# 0.003  0.079  4
Enter fullscreen mode Exit fullscreen mode

None of the collections is close to complete, and after seeing those numbers I
understand why.

Coverage checked at run time

Saying "this has everything" is not worth much on its own, so
scripts/verify_coverage.py downloads each upstream file when you run it and
diffs it against the merged set. Nothing is precomputed.

Upstream Entries Covered Missing
rofimoji 1,562 100.0% 0
emoji.nvim 2,016 99.9% 3
rime_kaomoji_dict 959 99.4% 6
Kaomoji_proj 2,166 97.8% 48

It prints all 57 misses, so you can look at them and disagree with me about what
counts as a kaomoji. Three rounds of that took rofimoji from 93.0% to 100.0%.

Labels, and where they fall down

Deduplicated it comes to 82,109 entries. Around 70k carry emotion, intent and
subject labels from a controlled vocabulary in en, ja, zh, es, pt and de, so
嬉しい, 开心, feliz and happy return the same entries instead of whichever English
word the original author happened to use.

Three things worth knowing before you use it:

  • The labels are coarse. 69,679 labelled entries share only 605 distinct label combinations, because they were applied in bundles rather than judged one at a time. Fine for pulling up a mood, not for finding one specific face.
  • About 15% is left unlabelled rather than guessed at.
  • Most of the corpus is 顔文字+セリフ, a face plus a line of Japanese dialogue, which is not much use if you cannot read it.

That last one is why the English-language picker at
fontvibe.ai/tools/kaomoji shows 11,020, the
subset that renders safely outside Japanese contexts, while the full 82,109 sits
in the dataset and on the Japanese page.

Getting it

npm i kaomoji-dataset
pip install kaomoji-dataset
Enter fullscreen mode Exit fullscreen mode

Both read a gzip off disk, so they are Node and Python only for now. They will
not bundle for a browser. The repo has the raw JSONL and a CSV as well:
github.com/Funovate/fontvibe-kaomoji

106 of the entries are not collected from anywhere. I drew them, for feelings
that had no kaomoji yet: 躺平, 社死, saudade, mamihlapinatapai. Those are CC0.

If you know of a collection I missed, say so and I will run the coverage script
against it.

Top comments (0)