You copy a value from a web page, a PDF or a chat message. It looks correct. But the password fails, JSON.parse throws, or two strings that look identical are not equal.
Often the cause is a character you cannot see. This post covers the usual suspects, why normal cleanup misses them, and how to find them in JavaScript, Python and the terminal.
The usual suspects
| Character | Code | Where it tends to come from |
|---|---|---|
| Zero Width Space | U+200B | Web pages, chat apps, rich text editors |
| Byte Order Mark (Zero Width No-Break Space) | U+FEFF | Files saved as "UTF-8 with BOM", some CSV exports |
| No-Break Space | U+00A0 | Word processors, HTML , PDFs |
| Narrow No-Break Space | U+202F | Formatted numbers and dates in some locales |
| Soft Hyphen | U+00AD | Text copied from typeset documents |
| Word Joiner | U+2060 | Rich text and some keyboards |
| Direction marks and overrides | U+200E, U+200F, U+202A–U+202E, U+2066–U+2069 | Mixed left-to-right and right-to-left text |
| Zero Width Joiner / Non-Joiner | U+200D / U+200C | Emoji sequences, Persian and Indic scripts |
Most of these are in the Unicode "Cf" (format) category. They have no width, so editors and browsers show nothing.
Why trim() and \s don't catch them
This surprises people, so here it is in code:
/\s/.test(''); // false: a zero width space is not whitespace
/\s/.test(''); // true
/\s/.test(' '); // true
JSON.stringify(' a'.trim());
// "a" <- the zero width space is still there
Python behaves similarly:
''.isspace() # False
''.isspace() # False
' '.isspace() # True
' a '.strip()
# 'a ' <- both survive
So a value can pass your "trim and compare" step and still not match.
Three bugs they cause
1. String comparison fails
'abc' === 'abc'; // false, though both print as abc
This is the classic "the password is correct but login fails" bug, and the reason a spreadsheet lookup won't match a value that looks identical.
2. JSON won't parse
A BOM at the start of a file breaks JSON.parse:
JSON.parse('{"a":1}');
// SyntaxError: Unexpected token '', "{"a":1}" is not valid JSON
The error message even contains the character, which is why it looks like there's nothing wrong. In Python, open such files with encoding='utf-8-sig' and the BOM is dropped for you.
3. Code that "looks fine" doesn't run
A zero width space pasted into an identifier or a config key gives you an undefined variable or a missing setting, on a line that looks perfect.
Finding them in JavaScript
List each hidden character with its code and position:
const hidden = /[----]/g;
function findHidden(text) {
return [...text.matchAll(hidden)].map(m =>
'U+' + m[0].codePointAt(0).toString(16).toUpperCase().padStart(4, '0') +
' at ' + m.index
);
}
findHidden('password and end');
// [ 'U+200B at 4', 'U+FEFF at 14' ]
No-break spaces are not in that pattern on purpose. They are visible as a gap, so it's usually better to turn them into normal spaces than to delete them:
const cleaned = text
.replace(/[---]/g, '')
.replace(/[ ]/g, ' ');
Finding them in Python
The unicodedata module names every character, which makes the output readable:
import unicodedata
def find_hidden(text):
for i, ch in enumerate(text):
if unicodedata.category(ch) == 'Cf' or ch in ' ':
print(i, f'U+{ord(ch):04X}', unicodedata.name(ch, '?'))
find_hidden('password and end')
# 4 U+200B ZERO WIDTH SPACE
# 9 U+00A0 NO-BREAK SPACE
# 14 U+FEFF ZERO WIDTH NO-BREAK SPACE
Finding them in files
With GNU grep and a UTF-8 locale:
grep -nP '[\x{200B}-\x{200F}\x{2060}\x{FEFF}]' file.txt
cat -A file.txt also shows them as byte sequences. A zero width space appears as M-bM-^@M-^K.
Recent versions of VS Code highlight many invisible and confusable characters by default, so it's worth checking that the Unicode highlighting settings are on in your editor.
Don't delete ZWJ and ZWNJ blindly
The zero width joiner (U+200D) is what turns three separate emoji into one family emoji. Strip it and 👨👩👧 falls apart into 👨👩👧. The zero width non-joiner (U+200C) is a normal part of written Persian and several Indian scripts.
That's why the cleanup pattern above leaves U+200C and U+200D alone. Remove them only when you're cleaning code, passwords, IDs or spreadsheet keys, where they never belong.
A quick way to check by hand
If you just want to paste a piece of text and see what's hiding in it, I built a free hidden character checker. It marks each character with its name and code and can remove them, with an option to keep emoji joiners. It runs in the browser, so the text isn't uploaded.
Have you been bitten by one of these? I'd like to hear which character it was and where it came from.
Top comments (0)