DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Writing an HTML entity encoder from scratch: which chars really need escaping, and why you iterate by code point

#javascript #html #webdev #programming

HTML has exactly five characters that mean something to the parser: & starts a reference, < and > delimit tags, and "/' delimit attribute values. If any of those appear literally in text you didn't intend as markup, the page breaks — or someone injects a <script>. The fix is a character reference (an "entity"). Let me build a lossless, two-way encoder/decoder in under 100 lines, no library, and clear up what actually needs escaping.

An entity is a character reference

It always starts with & and ends with ;. In between is either a name or a number (a Unicode code point in decimal or hex). Three spellings of the same character:

"&copy;"    // named
"&#169;"    // numeric, decimal
"&#xA9;"    // numeric, hexadecimal
Enter fullscreen mode Exit fullscreen mode

Only five characters must be escaped — and context decides which

Everything else — letters, digits, é, ™, even emoji — is legal as-is in a UTF-8 document. Which of the five matters depends on where the text lands:

Character In text In "double-quoted" attr In 'single-quoted' attr
& must must must
< must recommended recommended
> recommended safe safe
" safe must safe
' safe safe must

Escaping all five is always safe, which is why general-purpose encoders just handle the lot. Keep them in a set so the encoder can ask, in O(1), "is this dangerous?"

const CORE = { "&":1, "<":1, ">":1, '"':1, "'":1 };
const isUnsafe = ch => CORE[ch] === 1;
Enter fullscreen mode Exit fullscreen mode

Encode: iterate by code point

for (const ch of text) yields whole code points — an emoji comes out as one ch, not two broken surrogate halves. For each character, escape it if it's unsafe (or, in "encode everything" mode, if it's non-ASCII), otherwise copy it straight through.

function encode(text, opts){
  let out = "";
  for (const ch of text){                     // code-point iteration
    const cp = ch.codePointAt(0);
    const escape = isUnsafe(ch) || (opts.scope === "all" && cp > 0x7F);
    out += escape ? toEntity(ch, cp, opts) : ch;
  }
  return out;
}

function toEntity(ch, cp, opts){
  if (opts.mode === "named" && TO_NAME[ch] !== undefined)
    return "&" + TO_NAME[ch] + ";";                 // &copy;
  return opts.radix === "hex"
    ? "&#x" + cp.toString(16) + ";"                 // &#xA9;
    : "&#"  + cp + ";";                             // &#169;
}
Enter fullscreen mode Exit fullscreen mode

This also quietly dodges the classic bug. If you chain .replace(/</g,...).replace(/&/g,...), a real & you just wrote gets re-escaped into &amp;amp;. Reading each source character exactly once and writing its finished entity means the ampersand of an entity you just produced is never re-examined. No ordering, no double-encoding.

Decode: one regex, three shapes

A reference is always &, a body, then ;. The body is # + digits (decimal), #x + hex digits, or letters (a name). One alternation captures all three; a replace callback dispatches.

const ENT = /&(#[0-9]+|#[xX][0-9a-fA-F]+|[a-zA-Z][a-zA-Z0-9]*);/g;

function fromEntity(m, body){
  if (body[0] === "#"){
    const hex = body[1] === "x" || body[1] === "X";
    const cp  = parseInt(body.slice(hex ? 2 : 1), hex ? 16 : 10);
    if (!(cp >= 0 && cp <= 0x10FFFF)) return m;      // out of range -> leave alone
    try { return String.fromCodePoint(cp); } catch (e){ return m; }
  }
  return NAMED[body] !== undefined ? NAMED[body] : m;   // unknown name -> verbatim
}
Enter fullscreen mode Exit fullscreen mode

Two things make decoding safe. Numeric references are Unicode code points, so String.fromCodePoint correctly rebuilds astral characters like 🚀 as a surrogate pair. And a decoder must be conservative: an unknown name like &widget; or an out-of-range number is returned unchanged, never dropped or turned into a "?" box.

The round-trip guarantee — and what it is not

A correct codec satisfies decode(encode(x)) === x for every string, because every entity the encoder emits is one the decoder recognises.

const x = '<b>café & "quotes"</b>';
decode(encode(x, { mode:"named", scope:"all", radix:"hex" })) === x;  // true
Enter fullscreen mode Exit fullscreen mode

One caveat worth internalising: entities are not sanitization. They neutralise markup in an HTML text/attribute context, but do nothing for a javascript: URL, an unquoted attribute, or a value interpolated into inline JS/CSS — those need their own escaping or an allow-list.

Try the live two-way tool — toggle named vs numeric, decimal vs hex, and watch each character map to its code point: https://dev48v.infy.uk/solve/day53-html-entity-encoder.html

Top comments (0)