DEV Community

Invisible Character
Invisible Character

Posted on

Invisible Unicode characters are silently breaking your strings

#unicode #javascript #python #webdev

Invisible Unicode characters are silently breaking your strings

You copy text from a chat, a doc, or a webpage. It looks empty. But str.length says 3. Your === comparison fails. Your form validation rejects input the user swears they typed.

Nothing is wrong with your code. The string contains characters you cannot see.

The usual suspects

A small set of Unicode code points renders as completely blank in most fonts, yet counts as a real character:

  • U+200B Zero Width Space
  • U+200C Zero Width Non-Joiner
  • U+200D Zero Width Joiner
  • U+FEFF Zero Width No-Break Space (BOM)
  • U+3164 Hangul Filler
  • U+2800 Braille Pattern Blank

They sneak in through copy-paste from messaging apps, rich text editors, and web pages. U+3164 is also the character people use on purpose for blank WhatsApp messages and empty Instagram names. Same characters, two very different problems.

Finding them in JavaScript 

function findInvisible(str) {
  const targets = new Set([0x200B, 0x200C, 0x200D, 0xFEFF, 0x3164, 0x2800]);
  const found = [];
  for (const ch of str) {
    const cp = ch.codePointAt(0);
    if (targets.has(cp)) {
      found.push('U+' + cp.toString(16).toUpperCase().padStart(4, '0'));
    }
  }
  return found;
}
Enter fullscreen mode Exit fullscreen mode

Finding them in Python 

import unicodedata

def find_invisible(s):
    targets = {0x200B, 0x200C, 0x200D, 0xFEFF, 0x3164, 0x2800}
    return [
        (c, f"U+{ord(c):04X}", unicodedata.name(c, "UNKNOWN"))
        for c in s if ord(c) in targets
    ]
Enter fullscreen mode Exit fullscreen mode

Stripping them

Once you know what you are dealing with, removal is one line. JavaScript:

const clean = str.replace(/[\u200B-\u200D\uFEFF\u3164\u2800]/g, '');
Enter fullscreen mode Exit fullscreen mode

(That character class contains U+200B through U+200D, U+FEFF, U+3164, and U+2800.)

Python:

import re
clean = re.sub(r'[\u200B-\u200D\uFEFF\u3164\u2800]', '', s)
Enter fullscreen mode Exit fullscreen mode

Be careful with U+200D: it is load-bearing inside emoji sequences (family emojis, skin-tone modifiers). Stripping it blindly from user content will break emoji. Only strip it when you know the field should be plain text.

When you do not want to run code

Not every case deserves a script. If you just need to check a suspicious string, or you want to copy a specific invisible character for a blank message or username, there are free web tools for it. I built one at invisiblechar.com: paste any text into the detector and it highlights each invisible code point by name, and the generator gives you one-click copy for U+3164, U+200B, and the rest, plus a full Unicode table of 107 blank characters.

Invisible characters are a feature of Unicode, not a bug. They break things only when you do not know they are there.

Top comments (0)