DEV Community

Tim Nguyen
Tim Nguyen

Posted on

Unicode, UTF-8, UTF-16, surrogate pairs, and why JavaScript `length` lies

#javascript #node #webdev #unicode

Introduction

If you've ever written this in JavaScript:

console.log("🐣".length); // 2 πŸ€”
Enter fullscreen mode Exit fullscreen mode

or

console.log("πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦".length); // 11 😱
Enter fullscreen mode Exit fullscreen mode

you've probably wondered:

"How can one visible emoji have a length of 11?"

This article explains everything you need to knowβ€”from Unicode to UTF encodings, surrogate pairs, grapheme clusters, and the correct way to count characters in JavaScript.

Unicode: The Universal Character Set

Unicode is not an encoding.

Unicode is simply a standard that assigns every character a unique number called a code point.

Examples:

Character Unicode Code Point
A U+0041
δ½  U+4F60
πŸ˜€ U+1F600
🐣 U+1F423

Think of Unicode as a giant dictionary.

A  -> U+0041
B  -> U+0042
δ½  -> U+4F60
🐣 -> U+1F423
Enter fullscreen mode Exit fullscreen mode

Notice Unicode only defines the number.

It does not define how those numbers are stored in memory.

UTF-8, UTF-16 and UTF-32

UTF means Unicode Transformation Format.

Each UTF is simply a different way of encoding Unicode code points.

Character Unicode UTF-8 UTF-16 UTF-32
A U+0041 1 byte 2 bytes 4 bytes
δ½  U+4F60 3 bytes 2 bytes 4 bytes
🐣 U+1F423 4 bytes 4 bytes (2 code units) 4 bytes

UTF-8

UTF-8 uses between 1 and 4 bytes.

Advantages:

  • Most efficient for English text
  • Backward compatible with ASCII
  • Standard for the web
  • Used by JSON, HTML, Linux, HTTP...

Today, UTF-8 dominates the internet.

UTF-16

UTF-16 stores data in 16-bit code units.

Characters inside the Basic Multilingual Plane (BMP) use one code unit.

Characters outside the BMP use two code units.

JavaScript strings are UTF-16.

That's why:

"ABC".length
// 3
Enter fullscreen mode Exit fullscreen mode

and

"δ½ ".length
// 1
Enter fullscreen mode Exit fullscreen mode

work as expected.

UTF-32

UTF-32 stores every Unicode code point using exactly 4 bytes.

Advantages:

  • Very simple indexing
  • Every code point occupies one slot

Disadvantages:

  • Huge memory usage
  • 4Γ— larger than UTF-8 for English text

Because of this, UTF-32 is rarely used for general-purpose applications.

Why Emoji Break length

Consider:

"🐣"
Enter fullscreen mode Exit fullscreen mode

Unicode:

U+1F423
Enter fullscreen mode Exit fullscreen mode

This value is larger than 0xFFFF.

A single UTF-16 code unit can only represent:

0x0000
...
0xFFFF
Enter fullscreen mode Exit fullscreen mode

So UTF-16 needs two code units.

Surrogate Pairs

Unicode reserved two special ranges:

High Surrogates

D800 - DBFF

Low Surrogates

DC00 - DFFF
Enter fullscreen mode Exit fullscreen mode

An emoji becomes:

🐣

↓

D83D DC23
Enter fullscreen mode Exit fullscreen mode

Therefore JavaScript stores:

[D83D][DC23]
Enter fullscreen mode Exit fullscreen mode

So:

"🐣".length
// 2
Enter fullscreen mode Exit fullscreen mode

because JavaScript counts UTF-16 code units.

Why for...of Works Better

Strings are iterable.

for (const ch of "🐣") {
    console.log(ch);
}
Enter fullscreen mode Exit fullscreen mode

prints

🐣
Enter fullscreen mode Exit fullscreen mode

not

D83D
DC23
Enter fullscreen mode Exit fullscreen mode

The iterator automatically detects surrogate pairs.

Therefore

[..."🐣"].length
// 1
Enter fullscreen mode Exit fullscreen mode

works.

But for...of Still Isn't Correct

Now look at this emoji:

πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦
Enter fullscreen mode Exit fullscreen mode

Visually:

πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦
Enter fullscreen mode Exit fullscreen mode

Internally:

πŸ‘¨
ZWJ
πŸ‘©
ZWJ
πŸ‘§
ZWJ
πŸ‘¦
Enter fullscreen mode Exit fullscreen mode

There are 7 Unicode code points.

So

[..."πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦"].length
// 7
Enter fullscreen mode Exit fullscreen mode

Still wrong.

What is ZWJ?

ZWJ stands for Zero Width Joiner.

Unicode:

U+200D
Enter fullscreen mode Exit fullscreen mode

It is an invisible character.

Its job is:

Join the surrounding characters into one displayed glyph.

Example:

Without ZWJ:

πŸ‘© πŸ’»
Enter fullscreen mode Exit fullscreen mode

With ZWJ:

πŸ‘©β€πŸ’»
Enter fullscreen mode Exit fullscreen mode

Internally:

πŸ‘©
+
ZWJ
+
πŸ’»
Enter fullscreen mode Exit fullscreen mode

The rendering engine displays:

πŸ‘©β€πŸ’»
Enter fullscreen mode Exit fullscreen mode

as a single emoji.

Grapheme Clusters

Humans don't think in Unicode code points.

We think in visible characters.

Unicode calls those grapheme clusters.

Examples:

Text Code Points Visible Characters
A 1 1
δ½  1 1
🐣 1 1
πŸ‡»πŸ‡³ 2 1
πŸ‘πŸ½ 2 1
é 2 1
πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦ 7 1

This is what users expect when they see:

Maximum 100 characters

How Does JavaScript Know?

Modern JavaScript provides

Intl.Segmenter
Enter fullscreen mode Exit fullscreen mode

It implements the Unicode Text Segmentation algorithm (UAX #29).

Instead of asking:

"How many UTF-16 code units?"

or

"How many Unicode code points?"

it asks:

"Where are the grapheme boundaries?"

Unicode maintains a database of character properties.

Examples:

πŸ‘¨

Extended_Pictographic

ZWJ

Zero Width Joiner

🏽

Extend

πŸ‡»

Regional Indicator
Enter fullscreen mode Exit fullscreen mode

The algorithm walks through the string and decides whether each boundary should be broken.

Very roughly:

Read next code point

↓

Look up Unicode properties

↓

Apply Unicode rules

↓

Break?

or

Continue current grapheme?
Enter fullscreen mode Exit fullscreen mode

No machine learning.

No AI.

Just deterministic Unicode rules.

The Right Way to Count Characters 

const segmenter = new Intl.Segmenter(undefined, {
    granularity: "grapheme"
});

function characterCount(str) {
    let count = 0;

    for (const _ of segmenter.segment(str)) {
        count++;
    }

    return count;
}
Enter fullscreen mode Exit fullscreen mode

Examples:

characterCount("Hello")
// 5

characterCount("δ½ ε₯½")
// 2

characterCount("🐣")
// 1

characterCount("πŸ‡»πŸ‡³")
// 1

characterCount("πŸ‘πŸ½")
// 1

characterCount("πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦")
// 1

characterCount("é")
// 1
Enter fullscreen mode Exit fullscreen mode

This works correctly across languages, emoji, combining marks, flags, and ZWJ sequences.

Comparison

Method Counts Correct for Emoji Correct for Flags Correct for ZWJ Correct for Combining Marks
str.length UTF-16 code units ❌ ❌ ❌ ❌
for...of / Array.from() Unicode code points βœ… ❌ ❌ ❌
Intl.Segmenter Grapheme clusters βœ… βœ… βœ… βœ…

Key Takeaways

  • Unicode defines characters.
  • UTF-8, UTF-16, and UTF-32 are different encodings of Unicode.
  • JavaScript strings are UTF-16.
  • String.length counts UTF-16 code units, not characters.
  • for...of iterates Unicode code points, which is better but still not enough.
  • User-visible characters are called grapheme clusters.
  • Intl.Segmenter is the correct solution for counting characters in modern JavaScript.

If you're building user-facing features like character limits, text editors, or social apps, Intl.Segmenter is the API you should be using.

Top comments (1)

Collapse
 
frank_signorini profile image
Frank •

The length property lying about Unicode characters is a common gotcha, but what's the best way to accurately count characters in JavaScript, especially with surrogate pairs? I'd love to swap ideas on this.