DEV Community

Cover image for One invisible character broke skin tones for 12 emoji, and I shipped it
Peter
Peter

Posted on

One invisible character broke skin tones for 12 emoji, and I shipped it

#javascript #unicode #emoji #debugging

👍 supports five skin tones. My emoji API said it supported none.

The data was right. Both upstream sources were right. The bug was one line of lookup code and a character you cannot see.

The symptom

I had just published an emoji API and was checking endpoints by hand:

curl https://eeemoji.com/api/v1/emojis/thumbs-up
Enter fullscreen mode Exit fullscreen mode 
{
  "emoji": "👍",
  "name": "thumbs up",
  "skinToneSupport": false,
  "skinToneVariants": null
}
Enter fullscreen mode Exit fullscreen mode

That is plainly wrong. Thumbs up has had skin tone variants since Emoji 1.0.

The tempting move here is to patch the response. Hard-code a list of hand emoji, set the flag, move on. I nearly did. Instead I went looking for why, and the why turned out to affect 152 records, not the ten or so I had spotted.

Two sources, both correct

My data bundle is generated from two packages:

First thought: they disagree, and I picked the wrong one. So I checked:

thumbs up      emojibase skins=5    unicode-emoji-json skin_tone_support=True
thumbs down    emojibase skins=5    unicode-emoji-json skin_tone_support=True
raised hand    emojibase skins=5    unicode-emoji-json skin_tone_support=True
raised fist    emojibase skins=5    unicode-emoji-json skin_tone_support=True
Enter fullscreen mode Exit fullscreen mode

Both sources say yes. Neither is wrong. My generator was producing false from two inputs that both said true, which is the sort of result that means you are looking at the wrong layer.

The actual bug

Here is the line:

const skinSupport = unicodeEmojiData[eb.emoji]?.skin_tone_support ?? false;
Enter fullscreen mode Exit fullscreen mode

eb.emoji is emojibase's emoji character. unicodeEmojiData is keyed by emoji character. This looks like it cannot fail.

It fails for 152 of 1,949 emoji, because the two packages disagree about something invisible:

// emojibase's `emoji` field for thumbs up
[...eb.emoji].map(c => c.codePointAt(0).toString(16))
// => ['1f44d', 'fe0f']     <-- includes U+FE0F

// unicode-emoji-json's key for the same emoji
// => '1f44d'               <-- no U+FE0F
Enter fullscreen mode Exit fullscreen mode

U+FE0F is the variation selector. It is a zero-width character that tells the renderer "draw this as colour emoji, not monochrome text". It has no visual form of its own. Both strings print as 👍 and look identical in every editor, diff and terminal I use.

So the lookup missed. And then:

?? false
Enter fullscreen mode Exit fullscreen mode

swallowed the miss and returned a confident, wrong answer. No exception, no warning, no undefined creeping into the output. Just false, in 152 records, shipped to production and served over a public API.

Why the fallback is the real villain

The FE0F mismatch is a fact of life. Emoji data packages disagree about variation selectors constantly, and both are defensible: emojibase gives you the fully-qualified sequence, unicode-emoji-json gives you the base.

The thing that turned a mismatch into a bug is ?? false.

?? on a lookup you expect to succeed converts "I could not find this" into "the answer is no". Those are completely different statements, and the second one is unfalsifiable from the output. If the code had thrown, or produced undefined, or logged once, I would have caught it in minutes.

// what I wrote
const skinSupport = lookup[key]?.skin_tone_support ?? false;

// what would have caught it
const entry = lookup[key] ?? lookup[stripVariationSelectors(key)];
if (!entry) throw new Error(`No entry for ${key} (${toCodePoints(key)})`);
const skinSupport = entry.skin_tone_support;
Enter fullscreen mode Exit fullscreen mode

The rule I took from this: a ?? default on a lookup that should always hit is a silent-failure generator. Use it for genuinely optional data. For a required join, fail loudly.

Fixing it against the actual authority

Rather than patch the generator and re-run it, I reconciled the whole bundle against Unicode's own emoji-test.txt, which lists every fully-qualified sequence including toned ones. If the file contains 1F44D 1F3FB, thumbs up supports skin tones. That is the authority, not a package's interpretation of it.

The script derives both the flag and the variant list, so it can be re-run when Unicode ships a new version:

curl -o emoji-test.txt https://unicode.org/Public/emoji/latest/emoji-test.txt
EMOJI_TEST_FILE=emoji-test.txt node scripts/fix-skin-tone-data.js --check
Enter fullscreen mode Exit fullscreen mode

Result: 26 records corrected, 311 to 330 with skin tone support. Thumbs up and down, raised hand, raised fist, the four pointing hands, ear, snowboarder, surfer, swimmer, wrestlers, bunny ears, ballet dancer.

Three bugs in the fix itself

I wrote --check mode before --write mode, which is the only reason this section exists rather than a story about corrupting 1,900 records.

1. Stripping every FE0F. My first attempt removed all variation selectors before looking up the toned sequence. But the toned form of 👱‍♀️ is:

1F471 1F3FB 200D 2640 FE0F
      ^tone            ^this one survives
Enter fullscreen mode Exit fullscreen mode

Only the selector attached to the base is replaced by the modifier. Later ones belong to other parts of the sequence. Stripping all of them made 128 emoji look like they had lost skin tone support. Caught because 👱‍♀️ obviously has tones, so the rule had to be wrong.

2. Multi-person emoji tone every figure. 🧑‍🤝‍🧑 is not person + tone + handshake + person. It is:

1F9D1 tone 200D 1F91D 200D 1F9D1 tone
Enter fullscreen mode Exit fullscreen mode

Both people. My bundle had stored only the first figure toned, which renders as a toned person beside an untoned one. The flag was right and the variants were wrong, so the fix now checks those independently.

3. The fix for #2 matched everything. My "tone every person code point" function returns the sequence unchanged when there are no person code points. An unchanged sequence is of course present in the file, so every emoji "passed" and the script proposed 1,376 gains including 😀 grinning face. Fixed by requiring that a modifier was actually inserted.

Each of those passed typecheck and lint. None would have been caught by anything except running the thing and reading the output.

What I would tell past me

  1. When output contradicts every input, the bug is in the join, not the data. I spent time diffing two sources that agreed with each other.
  2. ?? false on a required lookup is a lie generator. It answers a question you did not ask.
  3. Invisible characters need a code point view. console.log(emoji) shows you nothing. [...s].map(c => c.codePointAt(0).toString(16)) shows you the bug.
  4. Write --check before --write. My data-fix script had three bugs. Dry run caught all three.

The API this came out of is at eeemoji.com/api. Free, no key, and the skin tone flags are now reconciled against Unicode 17.0. If you are pulling emoji data from more than one package, it is worth checking how many of your joins are quietly falling through to a default.

Top comments (0)