DEV Community

Cover image for Styled Unicode Breaks Character Counters
Eugen
Eugen

Posted on

Styled Unicode Breaks Character Counters

#javascript #unicode #webdev

A 76-character bio, styled once, no longer has one useful length.

const s = "𝐚𝐞𝐬𝐭𝐑𝐞𝐭𝐒𝐜";
[...s].length                          // 9   code points (and 9 graphemes here)
s.length                               // 18  UTF-16 units β€” what String.length counts
new TextEncoder().encode(s).length     // 36  UTF-8 bytes  β€” what storage counts
Enter fullscreen mode Exit fullscreen mode

Three defensible answers, factor of four on one word. I found this while auditing a counter beside field limits I do not own. The counter reported code points; the destination's unit was unknown. That is already enough to make a green "safe" verdict dishonest.

Styled letters from the Mathematical Alphanumeric Symbols block live in the Supplementary Plane: 1 code point = 2 UTF-16 units = 4 UTF-8 bytes. If you display one number next to a limit documented only as "characters", you are making a claim about a unit the destination never published.

What a real bio line does

Same 76-code-point line, four styles. Measured, not recalled:

style code points UTF-16 UTF-8 bytes
plain 76 76 76
Bold 𝐀 76 134 250
Script π’œ 76 117 225
Small Caps α΄€ 76 76 155

A code-point counter calls Bold "76". The same text occupies 134 UTF-16 units or 250 UTF-8 bytes. If the destination budgets one of those units, a verdict based on another is meaningless. Documentation that says only "characters" does not resolve it.

Honest UI: three states, not a green number

Measure all three. If the destination explicitly publishes a byte limit, enforce it. If it publishes a character limit without naming the unit, keep bytes visible but do not compare them to the same number: byte and character budgets are not interchangeable.

const measure = (s) => ({
  codePoints: [...s].length,
  utf16: s.length,
  bytes: new TextEncoder().encode(s).length,
});

// ok    β€” fits by code points and UTF-16; bytes are reported separately
// risk  β€” fits by code points, not by UTF-16: the field decides, and it did not say
// over  β€” too long even by the most generous count
const state = (m, limit) =>
  m.codePoints > limit ? "over" : m.utf16 > limit ? "risk" : "ok";
Enter fullscreen mode Exit fullscreen mode

risk is the honest answer when the platform has not told you the unit. The only way to name the unit is to paste a known-length styled string into the live field and see where it cuts. Until then, do not draw a green bar.

The next failure is truncation, not the count

.slice(), .substring(), and [0..n] on a JavaScript string cut at UTF-16 boundaries. An odd index inside a Supplementary character keeps half a surrogate pair. Many encoders and renderers replace that lone surrogate with U+FFFD:

"𝐚𝐞𝐬𝐭𝐑𝐞𝐭𝐒𝐜".slice(0, 7)   // "𝐚𝐞𝐬\uD835"  β†’ renders 𝐚𝐞𝐬�
Enter fullscreen mode Exit fullscreen mode

Truncate on code points β€” and on grapheme clusters if you allow combining marks (aΜΆ is two code points):

const cutCodePoints = (s, n) => [...s].slice(0, n).join("");
const cutGraphemes = (s, n) =>
  [...new Intl.Segmenter(undefined, { granularity: "grapheme" }).segment(s)]
    .slice(0, n)
    .map((g) => g.segment)
    .join("");
Enter fullscreen mode Exit fullscreen mode

Where the rest of the measurements live

I did not want a second guessed number in the UI, so I ran this against a shipping 22-style catalog rather than a handful of examples. Length units, the NFKC rule that flattens 17 of those 22 styles, coverage holes per family, and how /\d/ in JavaScript disagrees with Python on πŸπŸŽπŸπŸ” are written up with the reproducing code in Unicode text in fields you do not own (CC0).

Checklist I actually use:

[ ] Count in all three units; never show one number as if it were the limit
[ ] Truncate on code points or graphemes, never on UTF-16 indices
[ ] Normalise before validating; know whether your storage path normalises
[ ] Test rendering on a device you did not compose the text on
[ ] Disclose coverage holes instead of silently substituting another letter
Enter fullscreen mode Exit fullscreen mode

The catalog those numbers were measured on is the type-style-copy tool at fontius.app. The skill is the document; the tool is just where the 22 families live.

Research disclosure: The catalog measurements come from focused local tests against the 22 shipped styles. I did not establish which unit any named third-party platform uses; that requires testing the field itself.

AI-assistance disclosure: AI tools assisted with research navigation, code verification, and drafting. I reviewed the evidence and take responsibility for the claims and conclusions.

Top comments (0)