DEV Community

frank
frank

Posted on

Invisible Unicode Characters Can Break Markdown Without Looking Different

#security #markdown #webdev #unicode

Two Markdown lines can look identical and still produce different HTML. Before blaming the parser, inspect the code points.

# heading
#​heading
Enter fullscreen mode Exit fullscreen mode

The second line contains U+200B ZERO WIDTH SPACE after #. CommonMark requires the ATX marker sequence to be followed by a space, tab, or line ending, so this is a paragraph rather than a heading.

Make the invisible input observable 

const codePoints = value => [...value].map(char =>
  `U+${char.codePointAt(0).toString(16).toUpperCase().padStart(4, '0')}`
)

console.log(codePoints('#\u200Bheading'))
// U+0023 U+200B U+0068 ...
Enter fullscreen mode Exit fullscreen mode

When the source came from an unknown editor, first verify that it is actually a text file and inspect its encoding. This guide to opening Markdown files is a useful checklist; the important step is inspection, not conversion.

What three fixed parser versions produced

I tested Node.js 25.3.0 with Marked 18.0.7, markdown-it 14.1.0, and commonmark.js 0.31.2. These are observed outputs under default options, not claims about every Markdown platform.

Input Marked markdown-it commonmark.js
# heading heading heading heading
leading U+FEFF before # paragraph, FEFF retained paragraph, FEFF omitted in output paragraph, FEFF omitted in output
U+200B after # paragraph paragraph paragraph
U+00A0 after # heading paragraph paragraph
fullwidth ＃ or ＊ plain text plain text plain text

The NBSP result is the useful warning: a character that looks like a space creates a real cross-parser difference. “Whitespace” is not one universal Markdown category.

A BOM is positional

UTF-8 uses EF BB BF as its optional signature. After decoding, that is U+FEFF. Unicode says a recognized initial BOM should be removed before text processing. The same code point in the middle of text is not automatically a BOM and must not be globally deleted.

In the experiment, a leading U+FEFF prevented # from being the first character seen by the Markdown grammar. An internal alpha\uFEFFbeta remained present in all three outputs. Fix BOM handling at the decoding boundary, not with replaceAll('\uFEFF', '').

Zero width does not mean zero semantics

U+200B is a format control used to indicate a word or line-break opportunity. Its effect depends on position:

  • after #, it prevents ATX-heading recognition;
  • inside **bo\u200Bld**, all three parsers still create <strong>, while preserving U+200B in the text node.

So a blanket “remove invisible characters” rule is both too broad and too weak.

Look-alike punctuation is different punctuation

＃ is U+FF03, not U+0023. ＊ is U+FF0A, not U+002A. All three parsers treated the fullwidth examples as ordinary text.

NFKC normalization can map some compatibility characters to ASCII, but silently normalizing user source changes data. A safer editor warns, previews the proposed change, and lets the author approve it.

Bidirectional controls require a different review model

UAX #9 distinguishes logical order from display order. Directional controls can change what reviewers see while parsers continue reading logical code-point order.

With U+202E inside a Markdown URL, none of the three parsers simply discarded it. Marked and commonmark.js percent-encoded it; markdown-it serialized the host differently through IDNA processing. The safe review target is therefore not just the visible source. Inspect the code points, parsed href, and final resolved URL.

A practical ingestion policy

  1. Handle an initial BOM during byte decoding.
  2. Preserve raw bytes in regression fixtures.
  3. Flag Cf characters and non-ASCII look-alike punctuation near Markdown delimiters.
  4. Show code point, line, column, and Unicode name in diagnostics.
  5. Preview repairs instead of silently rewriting source.
  6. Validate links after parsing and URL resolution.

Invisible-character bugs are boundary bugs: the decoder, editor, Markdown grammar, HTML serializer, and browser may each make a different decision. Good tooling makes those decisions observable.

Would you prefer an editor that automatically cleans invisible characters, or one that always preserves the source and only warns?

Top comments (0)