Emoji look simple: one little picture. For developers, they are one of the most common sources of subtle bugs, such as truncated messages, broken character counters, mangled database rows and cursor positions that jump. This post explains how emoji are encoded and how to handle them correctly.
Emoji are Unicode characters, sometimes several
Each basic emoji has a Unicode code point. ๐ is U+1F600 ("grinning face"), and ๐ is U+1F480 ("skull"). These code points live outside the Basic Multilingual Plane, which already causes trouble in languages that use UTF-16 internally.
"๐".length; // 2 (two UTF-16 code units, a surrogate pair)
[..."๐"].length; // 1 (one code point)
Modifiers and sequences
Many emoji that look like a single character are actually sequences:
-
Skin tone modifiers: ๐๐ฝ = ๐
U+1F44D+ ๐ฝU+1F3FD -
Variation selectors: โค๏ธ = โค
U+2764+U+FE0F(emoji presentation) -
Zero-width joiner (ZWJ) sequences: ๐ฉโ๐ป = ๐ฉ +
U+200D+ ๐ป - Flags: ๐จ๐ณ = regional indicator C + regional indicator N
-
Keycaps: 1๏ธโฃ =
1+U+FE0F+U+20E3
The family emoji ๐จโ๐ฉโ๐งโ๐ฆ is seven code points and eleven UTF-16 code units.
const family = "๐จโ๐ฉโ๐งโ๐ฆ";
family.length; // 11
[...family].length; // 7
Grapheme clusters: what users actually see
Users think in grapheme clusters, meaning what looks like one character on screen. To count those, use Intl.Segmenter:
const seg = new Intl.Segmenter("en", { granularity: "grapheme" });
[...seg.segment("Hi ๐จโ๐ฉโ๐งโ๐ฆ!")].length; // 5
In Python, the standard library counts code points. Use the regex module's \X or the grapheme package for clusters:
import regex
len(regex.findall(r"\X", "Hi ๐จโ๐ฉโ๐งโ๐ฆ!")) # 5
Common bugs and fixes
1. Truncation breaks emoji. str.slice(0, 140) can cut a surrogate pair or ZWJ sequence in half and leave a "๏ฟฝ". Truncate by grapheme cluster instead.
2. Character counters disagree. Twitter-style limits, SMS segments and your UI may each count differently. Decide which unit your limit uses and document it.
3. MySQL drops emoji. The legacy utf8 charset in MySQL stores only up to 3 bytes per character. Use utf8mb4 for columns, tables and the connection.
ALTER TABLE messages CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
4. Reversing strings. str.split("").reverse() scrambles emoji. Reverse grapheme clusters, not code units.
5. Regex character classes. /^.$/ will not match a single emoji in JavaScript without the u flag, and even then it will not match a ZWJ sequence. Consider \p{Extended_Pictographic} with the u or v flag for detection.
Names and metadata: CLDR
Official emoji names and keywords come from Unicode's CLDR (Common Locale Data Repository). CLDR gives each emoji a short name, such as "loudly crying face", and search keywords, in many languages. This is the data behind emoji pickers on phones and keyboards.
When debugging user content, it helps to see exactly which code points are in a string and what they are called. Emoji Translator is handy for this. Paste a string, and it lists each emoji with its official name, keywords and code points, so you can spot a stray variation selector or a ZWJ sequence quickly.
Rendering differences
The same code point is drawn by different vendor fonts on Apple, Google, Microsoft and Samsung. Newer emoji may appear as a blank box ("tofu") or as separate components on older systems. A ZWJ sequence unsupported by the font falls back to its parts, so ๐ฉโ๐ป becomes ๐ฉ๐ป.
Checklist for emoji-safe applications
- [ ] Store text as UTF-8, with
utf8mb4in MySQL - [ ] Count and truncate by grapheme cluster
- [ ] Use Unicode-aware regex
- [ ] Test with skin tones, flags, ZWJ families and keycaps
- [ ] Do not assume one emoji = one character = one byte
Emoji are a great stress test for your text handling. If your app handles ๐จโ๐ฉโ๐งโ๐ฆ correctly, it probably handles most of the world's scripts correctly too.
Top comments (0)