DEV Community

nrdxn
nrdxn

Posted on

Stripping invisible Unicode usually corrupts your text. Here is how to not do that.

#security #unicode #javascript #webdev

You have met this bug. You copy a command from a documentation page, paste it
into a terminal, and it fails. The command looks exactly right. You retype it
by hand and it works.

curl -fsSL https://example.com/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

The version that failed contained U+00A0 NO-BREAK SPACE instead of U+0020. Your
shell sees a different token. Your eyes see nothing.

Same family of bug, different disguises:

  • Two lines in a diff are visibly identical and the diff insists they differ — one ends with U+200B ZERO WIDTH SPACE.
  • A search for a word in your own document finds nothing — U+00AD SOFT HYPHEN sits in the middle of it, courtesy of a word processor.
  • A username passes your uniqueness check twice — one of them uses Cyrillic U+0430 where you expected Latin a.
  • A code review reads one way and compiles another — U+202E RIGHT-TO-LEFT OVERRIDE, the Trojan Source class of attack.

So you reach for a cleaner. And this is where it usually goes wrong.

The naive fix quietly destroys valid text

The obvious implementation is a blacklist:

// Please do not ship this.
const clean = s => s.replace(/[\u200B-\u200D\u2060\uFEFF\u00AD]/g, "");
Enter fullscreen mode Exit fullscreen mode

Run it on this string and watch what happens:

clean("\u{1F468}\u200D\u{1F469}\u200D\u{1F466}");
// input:  one family emoji  = U+1F468 U+200D U+1F469 U+200D U+1F466
// output: three separate people, both joiners gone
Enter fullscreen mode Exit fullscreen mode

U+200D ZERO WIDTH JOINER is not junk there. It is the entire mechanism by which
emoji sequences exist. Strip it and you have silently rewritten the content.

It gets worse with human languages:

const persian = "\u0645\u06CC\u200C\u062E\u0648\u0627\u0646\u0645";  // Persian for "I read"
clean(persian);  // drops the U+200C between yeh and khah -> a different spelling
// U+200C ZERO WIDTH NON-JOINER removed -> میخوانم, a different spelling
Enter fullscreen mode Exit fullscreen mode

U+200C and U+200D do real orthographic work in Persian, Arabic, and Indic
scripts. In Devanagari, the sequence U+0915 U+094D U+200D U+0937 renders as a different
conjunct than the same string with the joiner removed. A cleaner that removes them is not cleaning, it
is corrupting — and because the characters are invisible, nobody notices until
much later.

Judge joiners by their neighbours

The fix is to stop treating a code point as intrinsically good or bad and look
at what surrounds it. A joiner between two pictographs is doing its job. A
joiner between two Latin letters is not:

const RE_PICTO   = /\p{Extended_Pictographic}/u;
const RE_JOINING = /[\p{Script=Arabic}\p{Script=Devanagari}\p{Script=Hebrew}\p{Script=Thaana}]/u;

const isEmojiish  = cp => cp !== undefined &&
  (RE_PICTO.test(String.fromCodePoint(cp)) || (cp >= 0x1F3FB && cp <= 0x1F3FF));
const needsJoiner = cp => cp !== undefined && RE_JOINING.test(String.fromCodePoint(cp));

function isSuspiciousJoiner(cp, prev, next){
  if (cp === 0x200D && isEmojiish(prev) && isEmojiish(next)) return false;  // emoji sequence
  if ((cp === 0x200C || cp === 0x200D) &&
      (needsJoiner(prev) || needsJoiner(next))) return false;             // Persian / Indic shaping
  return cp === 0x200C || cp === 0x200D;                                  // "a" U+200D "b"
}
Enter fullscreen mode Exit fullscreen mode

\p{Script=...} and \p{Extended_Pictographic} are standard in JS regex with
the u flag, so this costs you no dependencies.

The same mistake, one level up: homoglyphs

paypal.com with a Cyrillic U+0443 in place of y is a real attack. The naive
response is to flag every Cyrillic letter that resembles a Latin one — which
means every Russian, Ukrainian, Serbian or Bulgarian text you touch lights up
like a Christmas tree and, if you "fix" it, gets mangled into gibberish.

The signal is not "this letter is Cyrillic". The signal is script mixing
inside a single word:

function mixedScriptWords(text){
  const flagged = new Set();
  for (const m of text.matchAll(/[\p{L}\p{M}\p{N}_]+/gu)){
    let latin = false, other = false;
    for (const ch of m[0]){
      const c = ch.codePointAt(0);
      if ((c >= 0x41 && c <= 0x5A) || (c >= 0x61 && c <= 0x7A)) latin = true;
      else if ((c >= 0x0400 && c <= 0x052F) || (c >= 0x0370 && c <= 0x03FF)) other = true;
    }
    if (latin && other) flagged.add(m[0]);   // "pa"+U+0443+"pal" yes, "Привет" no
  }
  return flagged;
}
Enter fullscreen mode Exit fullscreen mode

Привет, как дела? produces zero findings. paуpal.com produces one, with a
concrete repair. That is the difference between a tool you can run on real input
and a tool you run once and never trust again.

The full cast of characters

Worth knowing what you are actually looking for:

Class Examples Why it matters
Zero-width U+200B, U+2060, U+FEFF, U+00AD changes length, hashes, diffs, search
Bidi controls U+202E, U+202D, U+2066, U+2069 Trojan Source: text reads ≠ text runs
Tag characters U+E0000–U+E007F invisible ASCII channel; hidden text and watermarks
Exotic spaces U+00A0, U+202F, U+3000 the broken-curl classic
Homoglyphs U+0430, U+0443, U+FF41 spoofed domains, duplicate identities
Private use / U+FFFD U+E000–U+F8FF fingerprint of a broken encoding round-trip

Tag characters deserve special mention: U+E0041 maps to A, U+E0042 to B, and
so on for the whole printable ASCII range. A paragraph can carry an entire
hidden message — or an instruction aimed at an LLM reading the page — with no
visual trace whatsoever.

What I built

Invisibles — one HTML file that
implements all of the above. Paste text, every hidden character becomes a
labelled chip, each class is a separate toggle, and the findings table lists
every code point with a severity and its replacement.

No dependencies, no build step, and no network calls at all: the page makes zero
requests, so you can save it and use it offline on text you would never paste
into someone else's server. selftest.mjs lifts the pure logic out of the HTML
and runs 61 assertions in Node — including the emoji, Persian and Devanagari
carve-outs, and HTML escaping of the rendered output.

Source: https://github.com/nrdxn/invisibles (MIT)
Live, preloaded with a sample containing one of everything:
https://nrdxn.github.io/invisibles/?demo

If you maintain a cleaner of your own, the three test cases worth stealing are a
ZWJ emoji family, a Persian word with U+200C in it, and an ordinary Russian
sentence. If all three survive untouched, you are ahead of most of the field.

Top comments (0)