Unicode Escapes in JSON and JavaScript: Syntax, Surrogates and Interchange

Escapes are source notation, not a separate encoding

A Unicode escape describes a character or code unit using ASCII characters inside a programming language or data format. The literal six-character sequence backslash-u-0-0-E-9 is not the same thing as the decoded Unicode character é; an escape parser converts the former into the latter. Confusing source text, decoded runtime strings and UTF-8 bytes leads to double-escaping errors and malformed interchange.

JSON and JavaScript share the \uXXXX idea but have different string-literal grammars and capabilities. JSON specifies a four-hex-digit Unicode escape syntax. ECMAScript additionally supports code-point escapes of the form \u{1F600} in contexts that allow them; this braced form is not valid JSON escape syntax. Both ecosystems may also represent supplementary scalars using a UTF-16 surrogate pair of \uXXXX escapes.

Escape mechanisms by environment

Hexadecimal digits are case-insensitive in the four-digit escapes. JSON also defines special escapes such as \b, \f, \r, \t, a quotation mark and a backslash. Unlike JavaScript, JSON does not permit comments or arbitrary single-quoted string literals; using a JavaScript expression as JSON can therefore fail even when each intended character is legitimate Unicode.

Representing a supplementary code point

U+1F600 GRINNING FACE is above U+FFFF, so it cannot fit into a single four-hex-digit escape. Subtract 0x10000 to get V = 0xF600; compute the high surrogate 0xD800 + (V shifted right ten bits) = D83D, and low surrogate 0xDC00 + (V AND 0x3FF) = DE00. Writing \uD83D\uDE00 creates the valid pair, which represents a single Unicode scalar on interpretation.

The same scheme applies to mathematical styled letters. For U+1D400 MATHEMATICAL BOLD CAPITAL A, the surrogate units are D835 and DC00. A JSON serializer might emit the actual UTF-8 bytes for 𝐀 instead of ASCII escapes, or emit \uD835\uDC00. Both are valid ways to represent the same scalar value when decoded correctly.

const emojiA = "\uD83D\uDE00";
const emojiB = "\u{1F600}";
const styled = "\uD835\uDC00";
console.log(emojiA === emojiB); // true
console.log(emojiA.length, [...emojiA].length); // 2, 1
console.log(styled.codePointAt(0).toString(16)); // 1d400

RFC 8259 allows a dangerous syntax edge case

RFC 8259 recognizes a subtle interoperability problem: its JSON string grammar can admit an escape such as \uDEAD even though that is a lone low surrogate, not a Unicode scalar. Such a member name or string value may be syntactically accepted by some JSON parsers while different implementations disagree on how to represent or encode it. Valid JSON syntax alone is not sufficient proof that all decoded strings contain well-formed Unicode scalar sequences.

Cross-runtime services should validate reconstructed strings if their protocol requires valid Unicode scalar text. In JavaScript, JSON.parse() can produce a JavaScript string containing an isolated surrogate. An application might choose to reject it, convert it to U+FFFD under an explicit loss policy, or preserve the code unit in an internal format that understands its limitation. The choice should be documented and tested rather than silently delegated to the next UTF-8 encoder.

const value = JSON.parse('"\\uDEAD"');
console.log(value.length);            // 1
console.log(value.isWellFormed());    // false
console.log(value.charCodeAt(0).toString(16)); // dead
// Do not assume another language can encode this as strict UTF-8.

JSON.stringify, backslashes and double-escaping

JavaScript's JSON.stringify() produces JSON text, escaping quotes, backslashes and control characters as required. It need not escape every non-ASCII code point; a literal é or emoji encoded as UTF-8 can be perfectly valid JSON. Modern well-formed stringification also escapes isolated surrogate code units rather than emitting an ill-formed Unicode sequence in the JSON text itself.

The common double-escaping defect occurs when one component serializes an already serialized JSON string again. A consumer that expects a JSON object but receives a JSON string containing another JSON document must parse twice. Conversely, blindly replacing \u substrings before parsing can corrupt legitimately escaped backslashes or produce values that were never intended as Unicode escapes.

const payload = { label: "café 😀", plain: "\\u0041" };
const json = JSON.stringify(payload);
const back = JSON.parse(json);
console.log(back.label === payload.label); // true
console.log(back.plain === "\\u0041");     // true, literal backslash-u
console.log(JSON.parse('"\\u0041"') === "A"); // true
console.log(JSON.stringify("😀").includes("😀")); // true in common engines

Code point iteration is not UTF-16 indexing

JavaScript strings are indexed in 16-bit code units, which makes the length of a supplementary character equal to two. By contrast, the for…of iterator advances across a valid surrogate pair as a single code point. A parsing or validation routine that slices a string at a raw code-unit boundary may accidentally introduce the very unpaired surrogate that later fails JSON interchange.

Use String.fromCodePoint() to construct a scalar deliberately; String.fromCharCode() constructs UTF-16 units and must be given both surrogate units for supplementary characters. Before producing wire-format JSON from arbitrary user input, validate whether unpaired surrogate units can occur in the internal representation. Make a decision to reject or replace them before crossing a protocol boundary.

Python JSON interoperability fixtures

Python's json module can emit literal non-ASCII text when configured with ensure_ascii=False, or ASCII escape forms when ensure_ascii=True. Both can represent valid strings; the visible JSON byte count differs. Python may parse a lone-surrogate JSON escape into a Python string holding that code point value, but strict UTF-8 encoding of it fails.

import json
original = {"letter": "é", "face": "😀"}
safe_ascii = json.dumps(original, ensure_ascii=True)
assert "\\ud83d\\ude00" in safe_ascii.lower()
assert json.loads(safe_ascii) == original
raw_utf8 = json.dumps(original, ensure_ascii=False).encode("utf-8")
assert json.loads(raw_utf8.decode("utf-8")) == original
bad = json.loads('"\\uDEAD"')
try:
    bad.encode("utf-8", "strict")
    raise AssertionError("accepted lone surrogate")
except UnicodeEncodeError:
    pass

UTF-8 on the wire, BOM handling and transport concerns

RFC 8259 requires JSON exchanged between systems outside a closed ecosystem to be encoded in UTF-8. JSON generators must not prepend a BOM to network-transmitted JSON; parsers may tolerate one for interoperability. JavaScript source-file escapes and HTTP payload bytes are separate concerns: a JSON document consisting of ASCII escape notation still has a byte encoding, normally UTF-8.

For exact comparison of identifiers and keys, remember that JSON parsing does not automatically normalize Unicode. The precomposed é and the canonically decomposed sequence e\u0301 can result in different scalar strings even though they are canonically equivalent. A protocol may establish its own normalization rule; simply parsing JSON does not enforce one.

Acceptance tests for a JSON/JavaScript bridge

Standards and technical references

1. RFC 8259 — JSON strings, Unicode and encoding
2. ECMAScript — String literal escape grammar
3. MDN — JSON.stringify behavior
4. MDN — String.fromCodePoint
5. Unicode Standard Chapter 3 — Surrogates and encoding forms

Independent technical research. Byte representation, scalar identity and presentation are separate interoperability dimensions.