DEV Community

Cover image for Styled Unicode Breaks Character Counters
Eugen
Eugen

Posted on

Styled Unicode Breaks Character Counters

#javascript #unicode #webdev

A 76-character bio, styled once, no longer has one useful length.

const s = "𝐚𝐞𝐬𝐭𝐑𝐞𝐭𝐒𝐜";
[...s].length                          // 9   code points (and 9 graphemes here)
s.length                               // 18  UTF-16 units β€” what String.length counts
new TextEncoder().encode(s).length     // 36  UTF-8 bytes  β€” what storage counts
Enter fullscreen mode Exit fullscreen mode

Three defensible answers, factor of four on one word. I found this while auditing a counter beside field limits I do not own. The counter reported code points; the destination's unit was unknown. That is already enough to make a green "safe" verdict dishonest.

Styled letters from the Mathematical Alphanumeric Symbols block live in the Supplementary Plane: 1 code point = 2 UTF-16 units = 4 UTF-8 bytes. If you display one number next to a limit documented only as "characters", you are making a claim about a unit the destination never published.

What a real bio line does

Same 76-code-point line, four styles. Measured, not recalled:

style code points UTF-16 UTF-8 bytes
plain 76 76 76
Bold 𝐀 76 134 250
Script π’œ 76 117 225
Small Caps α΄€ 76 76 155

A code-point counter calls Bold "76". The same text occupies 134 UTF-16 units or 250 UTF-8 bytes. If the destination budgets one of those units, a verdict based on another is meaningless. Documentation that says only "characters" does not resolve it.

Honest UI: three states, not a green number

Measure all three. If the destination explicitly publishes a byte limit, enforce it. If it publishes a character limit without naming the unit, keep bytes visible but do not compare them to the same number: byte and character budgets are not interchangeable.

const measure = (s) => ({
  codePoints: [...s].length,
  utf16: s.length,
  bytes: new TextEncoder().encode(s).length,
});

// ok    β€” fits by code points and UTF-16; bytes are reported separately
// risk  β€” fits by code points, not by UTF-16: the field decides, and it did not say
// over  β€” too long even by the most generous count
const state = (m, limit) =>
  m.codePoints > limit ? "over" : m.utf16 > limit ? "risk" : "ok";
Enter fullscreen mode Exit fullscreen mode

risk is the honest answer when the platform has not told you the unit. The only way to name the unit is to paste a known-length styled string into the live field and see where it cuts. Until then, do not draw a green bar.

The next failure is truncation, not the count

.slice(), .substring(), and [0..n] on a JavaScript string cut at UTF-16 boundaries. An odd index inside a Supplementary character keeps half a surrogate pair. Many encoders and renderers replace that lone surrogate with U+FFFD:

"𝐚𝐞𝐬𝐭𝐑𝐞𝐭𝐒𝐜".slice(0, 7)   // "𝐚𝐞𝐬\uD835"  β†’ renders 𝐚𝐞𝐬�
Enter fullscreen mode Exit fullscreen mode

Truncate on code points β€” and on grapheme clusters if you allow combining marks (aΜΆ is two code points):

const cutCodePoints = (s, n) => [...s].slice(0, n).join("");
const cutGraphemes = (s, n) =>
  [...new Intl.Segmenter(undefined, { granularity: "grapheme" }).segment(s)]
    .slice(0, n)
    .map((g) => g.segment)
    .join("");
Enter fullscreen mode Exit fullscreen mode

Where the rest of the measurements live

I did not want a second guessed number in the UI, so I ran this against a shipping 22-style catalog rather than a handful of examples. Length units, the NFKC rule that flattens 17 of those 22 styles, coverage holes per family, and how /\d/ in JavaScript disagrees with Python on πŸπŸŽπŸπŸ” are written up with the reproducing code in Unicode text in fields you do not own (CC0).

Checklist I actually use:

[ ] Count in all three units; never show one number as if it were the limit
[ ] Truncate on code points or graphemes, never on UTF-16 indices
[ ] Normalise before validating; know whether your storage path normalises
[ ] Test rendering on a device you did not compose the text on
[ ] Disclose coverage holes instead of silently substituting another letter
Enter fullscreen mode Exit fullscreen mode

The catalog those numbers were measured on is the type-style-copy tool at fontius.app. The skill is the document; the tool is just where the 22 families live.

Research disclosure: The catalog measurements come from focused local tests against the 22 shipped styles. I did not establish which unit any named third-party platform uses; that requires testing the field itself.

AI-assistance disclosure: AI tools assisted with research navigation, code verification, and drafting. I reviewed the evidence and take responsibility for the claims and conclusions.

Top comments (1)

Collapse
 
dang-phong profile image
Dang Phong •

Really appreciated the care you put into separating code points, UTF-16 units, and UTF-8 bytes instead of treating a platform's unspecified β€œcharacter limit” as something developers can safely guess.

The distinction between counting and truncation is especially valuable. Your grapheme-safe truncation example addresses an issue that many text tools overlook.

One small extension that might be interesting for the validation pipeline is to test the relationship between compatibility normalization and styled Unicode input.

For example, 𝐀 (U+1D400) occupies two UTF-16 code units, but NFKC maps it to the ordinary letter A (U+0041).

That means the original and normalized representations can have different lengths, in addition to different character identities.

For a field that performs normalization before indexing or validation, it may be useful to measure both representations while retaining the original input for display.

Two references that discuss these distinctions:

Disclosure: I am affiliated with Levenshtein.net.

Thanks for sharing the measurements and code. A small before-and-after normalization test would complement your existing character-counting examples nicely.