Why does copied text look identical but break your code?

#webdev #unicode #debugging #python

This is the clean version of the example:

total = 10
count = 2
print(total / count)  # 👩‍💻

Inserting a zero-width space after count produces an error:

  File "demo.py", line 2
    count = 2
         ^
SyntaxError: invalid non-printable character U+200B

The inserted character is U+200B. It has no visible width, so the two versions look the same. Python rejects it on that line.

The first Python block deliberately contains no U+200B. To create the failing version, generate the file with explicit escapes:

python3 -c "open('demo.py','w').write('total = 10\ncount\u200b = 2\nprint(total / count)  # \U0001F469\u200d\U0001F4BB\n')"
python3 demo.py

That gives the SyntaxError above (Python 3.13).

It doesn't always crash. In a JSON key, the lookup just quietly misses:

import json
d = json.loads('{"user\u200bId": 42}')
print(d.get("userId"))  # None
print(list(d))          # ['user\u200bId']

Seeing it

Print the repr instead of the text:

$ python3 -c "print(repr(open('demo.py').read()))"
'total = 10\ncount\u200b = 2\nprint(total / count)  # 👩\u200d💻\n'

Two hidden characters show up. \u200b after count is the bug. \u200d is a zero-width joiner. It glues 👩 and 💻 into one "woman technologist" emoji.

To get line numbers, use GNU grep built with PCRE support. The -P option is not supported by the grep bundled with macOS:

$ grep -nP '\x{200B}' demo.py
2:count = 2

Widen the class when needed, e.g. '[\x{200B}\x{00A0}\x{FEFF}]' for zero-width space, no-break space and BOM.

Don't strip everything

A regex that removes every invisible character will fix this file and break other text. Some of these characters are doing real work:

U+200D (zero-width joiner): emoji sequences like 👩‍💻, and shaping in some Indic scripts.
U+200C (zero-width non-joiner): used in normal Persian writing.
U+FE0E / U+FE0F (variation selectors): text vs emoji presentation.
U+00A0 (no-break space): sometimes intended in prose. In code it's almost always a bug.

So remove only what broke the code. Here that's the U+200B:

$ python3 -c "
s = open('demo.py').read().replace('\u200b', '')
open('fixed.py', 'w').write(s)
print(repr(s))"
'total = 10\ncount = 2\nprint(total / count)  # 👩\u200d💻\n'

$ python3 fixed.py
5.0

It runs, and the joiner in the emoji is still there.

In the browser

I'm dark, and I built InvisiChar for this. It's free, and the text is processed locally in your browser. Everything above works without it. It's handy when the text isn't in a file and you want to go through findings one at a time.

For the browser example, use the contents of the generated demo.py, which includes U+200B. The Inspect screenshot below shows 2 findings: the zero-width space is marked "Suggested removal", and the zero-width joiner is "Protected by default". Nothing changes until you confirm.

In Clean you pick a preset (Prose, Code & data, International text) and tick the occurrences to remove. The review below selects only U+200B and leaves the joiner unselected.