DEV Community

Text Repeaterr
Text Repeaterr

Posted on Edited on

Invisible characters that break your code (and how to find them)

You copy a value from a web page, a PDF or a chat message. It looks correct. But the password fails, JSON.parse throws, or two strings that look identical are not equal.

Often the cause is a character you cannot see. This post covers the usual suspects, why normal cleanup misses them, and how to find them in JavaScript, Python and the terminal.

The usual suspects

Character Code Where it tends to come from
Zero Width Space U+200B Web pages, chat apps, rich text editors
Byte Order Mark (Zero Width No-Break Space) U+FEFF Files saved as "UTF-8 with BOM", some CSV exports
No-Break Space U+00A0 Word processors, HTML  , PDFs
Narrow No-Break Space U+202F Formatted numbers and dates in some locales
Soft Hyphen U+00AD Text copied from typeset documents
Word Joiner U+2060 Rich text and some keyboards
Direction marks and overrides U+200E, U+200F, U+202A–U+202E, U+2066–U+2069 Mixed left-to-right and right-to-left text
Zero Width Joiner / Non-Joiner U+200D / U+200C Emoji sequences, Persian and Indic scripts

Most of these are in the Unicode "Cf" (format) category. They have no width, so editors and browsers show nothing.

Why trim() and \s don't catch them

This surprises people, so here it is in code:

/\s/.test('​');   // false: a zero width space is not whitespace
/\s/.test('');   // true
/\s/.test(' ');   // true

JSON.stringify('  a​'.trim());
// "a​"  <- the zero width space is still there
Enter fullscreen mode Exit fullscreen mode

Python behaves similarly:

'​'.isspace()   # False
''.isspace()   # False
' '.isspace()   # True

'  a​ '.strip()
# 'a​ '  <- both survive
Enter fullscreen mode Exit fullscreen mode

So a value can pass your "trim and compare" step and still not match.

Three bugs they cause

1. String comparison fails

'abc' === 'a​bc';   // false, though both print as abc
Enter fullscreen mode Exit fullscreen mode

This is the classic "the password is correct but login fails" bug, and the reason a spreadsheet lookup won't match a value that looks identical.

2. JSON won't parse

A BOM at the start of a file breaks JSON.parse:

JSON.parse('{"a":1}');
// SyntaxError: Unexpected token '', "{"a":1}" is not valid JSON
Enter fullscreen mode Exit fullscreen mode

The error message even contains the character, which is why it looks like there's nothing wrong. In Python, open such files with encoding='utf-8-sig' and the BOM is dropped for you.

3. Code that "looks fine" doesn't run

A zero width space pasted into an identifier or a config key gives you an undefined variable or a missing setting, on a line that looks perfect.

Finding them in JavaScript

List each hidden character with its code and position:

const hidden = /[­​-‏‪-‮⁠-⁤⁦-⁩]/g;

function findHidden(text) {
  return [...text.matchAll(hidden)].map(m =>
    'U+' + m[0].codePointAt(0).toString(16).toUpperCase().padStart(4, '0') +
    ' at ' + m.index
  );
}

findHidden('pass​word and end');
// [ 'U+200B at 4', 'U+FEFF at 14' ]
Enter fullscreen mode Exit fullscreen mode

No-break spaces are not in that pattern on purpose. They are visible as a gap, so it's usually better to turn them into normal spaces than to delete them:

const cleaned = text
  .replace(/[­​‎‏‪-‮⁠-⁤⁦-⁩]/g, '')
  .replace(/[  ]/g, ' ');
Enter fullscreen mode Exit fullscreen mode

Finding them in Python

The unicodedata module names every character, which makes the output readable:

import unicodedata

def find_hidden(text):
    for i, ch in enumerate(text):
        if unicodedata.category(ch) == 'Cf' or ch in '  ':
            print(i, f'U+{ord(ch):04X}', unicodedata.name(ch, '?'))

find_hidden('pass​word and end')
# 4 U+200B ZERO WIDTH SPACE
# 9 U+00A0 NO-BREAK SPACE
# 14 U+FEFF ZERO WIDTH NO-BREAK SPACE
Enter fullscreen mode Exit fullscreen mode

Finding them in files

With GNU grep and a UTF-8 locale:

grep -nP '[\x{200B}-\x{200F}\x{2060}\x{FEFF}]' file.txt
Enter fullscreen mode Exit fullscreen mode

cat -A file.txt also shows them as byte sequences. A zero width space appears as M-bM-^@M-^K.

Recent versions of VS Code highlight many invisible and confusable characters by default, so it's worth checking that the Unicode highlighting settings are on in your editor.

Don't delete ZWJ and ZWNJ blindly

The zero width joiner (U+200D) is what turns three separate emoji into one family emoji. Strip it and 👨‍👩‍👧 falls apart into 👨👩👧. The zero width non-joiner (U+200C) is a normal part of written Persian and several Indian scripts.

That's why the cleanup pattern above leaves U+200C and U+200D alone. Remove them only when you're cleaning code, passwords, IDs or spreadsheet keys, where they never belong.

A quick way to check by hand

If you just want to paste a piece of text and see what's hiding in it, I built a free hidden character checker. It marks each character with its name and code and can remove them, with an option to keep emoji joiners. It runs in the browser, so the text isn't uploaded.

Have you been bitten by one of these? I'd like to hear which character it was and where it came from.

Top comments (0)