DEV Community

Easy Formatter
Easy Formatter

Posted on

Your word counter returns 0 for half the planet

Here's a one-line test. Run it in your console right now:

"مرحبا بالعالم".match(/\b\S+\b/g)?.length ?? 0
Enter fullscreen mode Exit fullscreen mode

It returns 0. Not "2". Zero. That string is "hello world" in Arabic.

Now try the same pattern on Chinese, Japanese, Russian, Greek, Hebrew, Thai and Hindi. Same answer every time:

input /\b\S+\b/g
the quick brown fox 4
مرحبا بالعالم 0
你好世界 0
こんにちは世界 0
привет мир 0
γεια σου κόσμε 0
שלום עולם 0
สวัสดีชาวโลก 0
नमस्ते दुनिया 0

I found this in my own code, in a diff tool that had been reporting "Removed: 0, Added: 0" over two visibly different panes. The panes were highlighted. The counter said nothing had changed. Both were right, according to the regex.

Why \b does this

\b is a word boundary, meaning the position between a \w and a non-\w. And \w in JavaScript is exactly this:

[A-Za-z0-9_]
Enter fullscreen mode Exit fullscreen mode

That's it. ASCII letters, ASCII digits, underscore. Not "letters". Not "word characters" in any linguistic sense. Sixty-three specific code points.

So in مرحبا, there is no \w anywhere, which means there is no boundary anywhere, which means \b\S+\b never matches. The regex isn't failing. It is correctly reporting that a string of Arabic contains no ASCII word boundaries.

This is not a JavaScript quirk you can flag your way out of. The u flag doesn't change \w. Neither does v. \w is ASCII by specification and will stay that way.

Fix one: Unicode property escapes

\p{L} matches any Unicode letter, \p{N} any number. With the u flag:

const WORD_RE = /[\p{L}\p{N}][\p{L}\p{N}_'’-]*/gu;

"مرحبا بالعالم".match(WORD_RE).length;  // 2
"привет мир".match(WORD_RE).length;     // 2
"the quick brown fox".match(WORD_RE).length;  // 4
Enter fullscreen mode Exit fullscreen mode

The leading class is deliberate: a word has to start with a letter or digit, so a bare - or ... is not a word. The continuation class allows internal apostrophes and hyphens, so don't, it’s and well-known each count once. It agrees with \b\S+\b on every ASCII case I threw at it, and counts the rest of the world too.

This is the fix I shipped. It is also not the whole truth, and the next section is the part most posts leave out.

Where the regex still lies

Run it against Hindi and Chinese:

"नमस्ते दुनिया".match(WORD_RE).length;  // 5, should be 2
"你好世界".match(WORD_RE).length;         // 1, arguably 2
Enter fullscreen mode Exit fullscreen mode

Two different failures.

Devanagari over-splits. नमस्ते is written with combining marks. The virama and the vowel sign are Unicode category Mn (nonspacing mark), which is neither \p{L} nor \p{N}. So the character class treats them as separators and shatters one word into pieces. The same applies across Indic scripts.

CJK under-splits. 你好世界 is two words, 你好 (hello) and 世界 (world), and the continuation class swallows the entire run as one match. There is no space to split on, and no amount of character-class tuning finds the boundary, because the boundary is a fact about Chinese, not about characters.

If you only serve Latin, Cyrillic, Greek, Arabic and Hebrew, the regex is fine and it's what I'd use. If Indic or CJK matters to you, it isn't.

Fix two: Intl.Segmenter

Intl.Segmenter does real locale-aware segmentation, and it's been in every major browser since 2022 (Firefox was last, in 125):

const seg = new Intl.Segmenter(undefined, { granularity: "word" });
const countWords = (s) =>
  [...seg.segment(s)].filter((x) => x.isWordLike).length;

countWords("नमस्ते दुनिया");  // 2
countWords("你好世界");         // 2
countWords("สวัสดีชาวโลก");    // 3
Enter fullscreen mode Exit fullscreen mode

isWordLike is what filters out the whitespace and punctuation segments. Without it you're counting separators too.

Side by side:

input \b\S+\b \p{L} regex Intl.Segmenter
the quick brown fox 4 4 4
مرحبا بالعالم 0 2 2
привет мир 0 2 2
สวัสดีชาวโลก 0 3 3
你好世界 0 1 2
नमस्ते दुनिया 0 5 2
... --- !!! 0 0 0

The cost is that it allocates a segment object per token, so it is meaningfully slower than a regex scan on large inputs. Worth measuring if you're counting on every keystroke. And it takes a locale, which you may not know.

The sibling bug: character counts

Same class of mistake, one line away:

"😀".length;  // 2
Enter fullscreen mode Exit fullscreen mode

String.prototype.length counts UTF-16 code units, not characters. Anything outside the Basic Multilingual Plane (emoji, most CJK extensions, historic scripts) is a surrogate pair and counts twice. Type three emoji into a counter built on .length and it tells you six.

Spreading the string iterates code points instead:

[..."😀"].length;  // 1
Enter fullscreen mode Exit fullscreen mode

That's the cheap fix and it's usually enough. But code points aren't characters either:

string .length code points graphemes
😀 2 1 1
👨‍👩‍👧 8 5 1
नमस्ते 6 6 3

A ZWJ family emoji is one thing on screen and five code points. For what a user would call "characters", you need Intl.Segmenter again:

const graphemes = new Intl.Segmenter(undefined, { granularity: "grapheme" });
[...graphemes.segment("👨‍👩‍👧")].length;  // 1
Enter fullscreen mode Exit fullscreen mode

There's also a normalization trap hiding here: café can be four code points (precomposed é) or five (e + combining acute). They render identically. "café".normalize("NFC") before counting if the source is untrusted.

What I'd actually do

  • Counting for a UI badge, Latin-ish audience: use the \p{L} regex. It's one line, it's fast, and it's a massive improvement over \b.
  • Counting where CJK or Indic users exist: use Intl.Segmenter. The regex will be confidently wrong, which is worse than slow.
  • Never \b\S+\b. It has no audience. It's wrong for most of the world and the alternative costs one line.
  • Never bare .length for anything a user is told is a character count.

Pick one definition and put it in exactly one module. I had two, a diff view counting with the regex and a stats panel counting with text.trim().split(/\s+/), and they disagreed the moment a token held no letter or digit. ... is one word to split and zero to the regex. Two places in the same app gave two answers for the same paragraph, and the bug report that finally surfaced it was "the word count is wrong", which is the least debuggable sentence in the English language.


This came out of building Easy Formatter, a set of browser-only text and data tools. The diff view is here if you want to see the counting in situ. Everything runs client-side, which is why I couldn't quietly fix any of this on a server and had to actually understand it.

Every number in this post is reproducible. Paste this into a file and run it with Node 18+:

const OLD = /\b\S+\b/g;
const NEW = /[\p{L}\p{N}][\p{L}\p{N}_'’-]*/gu;
const seg = new Intl.Segmenter(undefined, { granularity: "word" });
const segCount = (s) => [...seg.segment(s)].filter((x) => x.isWordLike).length;

const cases = {
  English: "the quick brown fox", Arabic: "مرحبا بالعالم",
  Chinese: "你好世界", Japanese: "こんにちは世界", Thai: "สวัสดีชาวโลก",
  Hindi: "नमस्ते दुनिया", Russian: "привет мир", punctuation: "... --- !!!",
};

for (const [k, v] of Object.entries(cases)) {
  console.log(
    k.padEnd(12),
    "\\b:", String((v.match(OLD) || []).length).padStart(2),
    " regex:", String((v.match(NEW) || []).length).padStart(2),
    " Segmenter:", String(segCount(v)).padStart(2),
  );
}
Enter fullscreen mode Exit fullscreen mode

Top comments (0)