My Unicode escaper turns one emoji into two \\u escapes. JSON is fine with that, a Python string literal is not

#unicode #javascript #python

Background

I run a site of small browser-based tools, and one of them escapes text into \uXXXX sequences and back.

https://hashitosystem.com/en/tools/unicodeescape/

I mostly use it to paste non-ASCII text into JSON or JavaScript source safely.
Today I fed it an emoji (U+1F600, the grinning face) and got back two escapes instead of one.
I wanted to know whether that output is actually safe to paste everywhere I paste it, so I checked JavaScript, JSON and Python.

What the tool outputs

The escape function walks the string one UTF-16 code unit at a time.
charCodeAt(i) returns the code unit at position i, and anything above 127 is written as \u plus four hex digits.
This is the real function from the page, unchanged.

function hex4(n){return('000'+n.toString(16)).slice(-4);}
function escapeUnicode(text,all){
  var out='';
  for(var i=0;i<text.length;i++){
    var c=text.charCodeAt(i);
    if(c>127||all){out+='\\u'+hex4(c);}
    else{out+=text[i];}
  }
  return out;
}

I called it in Node.js 24 with the letter A, the emoji built with String.fromCodePoint(0x1F600), and the hiragana character U+3042.
The output was this:

A\ud83d\ude00\u3042

The hiragana became one escape, but the emoji became two: \ud83d and \ude00.

Why one character becomes two escapes

A \uXXXX escape has room for four hex digits, which is 16 bits, so it can only name a code point up to U+FFFF.
That range is called the Basic Multilingual Plane (BMP).
U+1F600 is above it.
JavaScript strings are stored as UTF-16, an encoding that represents characters above U+FFFF as two 16-bit units called a surrogate pair: a high surrogate in D800-DBFF followed by a low surrogate in DC00-DFFF.
Because the tool walks code units rather than code points, it writes each half of the pair as its own escape.

JavaScript and JSON join the pair back

In JavaScript this is fine, because the string is UTF-16 anyway.
Here I parsed the tool's output as a JSON string, where e is the escaped text from above:

const s = JSON.parse('"' + e + '"');
console.log(s.codePointAt(1).toString(16));
console.log(s === 'A' + String.fromCodePoint(0x1F600) + '\u3042');

It printed 1f600 and true.
JSON's grammar (RFC 8259, section 7) says a character outside the BMP is escaped exactly this way, as a 12-character pair, so any conforming JSON parser should join it.

https://www.rfc-editor.org/rfc/rfc8259#section-7

Python string literals do not

Python 3 strings are sequences of code points, not UTF-16 units.
So a \ud83d\ude00 escape in a Python string literal produces two separate code points, each a lone surrogate.
To test it without my editor rewriting the escape into a real emoji, I built the literal from chr(92) (the backslash) and evaluated it with ast.literal_eval, which parses a Python literal exactly as source code would.

import json, ast
bs = chr(92)
s = ast.literal_eval("'" + bs + "ud83d" + bs + "ude00'")
print('literal len', len(s), [hex(ord(c)) for c in s])

t = json.loads('"' + bs + "ud83d" + bs + 'ude00"')
print('json len', len(t), hex(ord(t)))

s8 = ast.literal_eval("'" + bs + "U0001F600'")
print('U8 len', len(s8), hex(ord(s8)))

try:
    s.encode('utf-8')
except UnicodeEncodeError as e:
    print('encode error:', e)

fixed = s.encode('utf-16', 'surrogatepass').decode('utf-16')
print('fixed len', len(fixed), hex(ord(fixed)))

With Python 3.11.9 the output was:

literal len 2 ['0xd83d', '0xde00']
json len 1 0x1f600
U8 len 1 0x1f600
encode error: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
fixed len 1 0x1f600

So the same escaped text behaves differently depending on who reads it.
json.loads joins the pair into one character, as the JSON spec asks.
A Python string literal keeps two broken halves, and the error only shows up later, when something tries to write the string to a file or a socket as UTF-8.
The Python spelling for this character is \U0001F600, with a capital U and eight hex digits.
If you already have a string with a pair in it, the surrogatepass round trip on the last line joins it back.

Why I am keeping the output as it is

I thought about switching the tool to code points and printing \u{1f600} instead.
That form is valid in JavaScript source since ES2015, and the tool's unescape side already accepts it.
But it is not valid JSON, and JSON is the main place I paste this output, so the surrogate pair is the safer default.
The trade-off is that the output is specifically for JSON and JavaScript, not a universal escape format.

If you paste \uXXXX output from a JavaScript-based tool into Python source, check for anything in the D800-DFFF range first, and write those characters as \U escapes instead.

This article is about my own side project. It was written with AI assistance.

DEV Community