HTML Character References vs. Unicode Characters: Parsing, Serialization and Escaping

An HTML entity is source syntax; a Unicode character is data. When a browser parses ©, it creates the character © (U+00A9) in a text node. That distinction matters when an editor exports markup, compares strings or double-decodes content from the clipboard.

Character references are syntax in HTML source

HTML supports named character references such as & and numeric references such as © or ©. The parser resolves them according to the HTML tokenizer's current state and produces Unicode text nodes or attribute values.

The source spelling is not part of a normal DOM Text node once parsed. A human may type actual © directly into UTF-8 HTML instead; the parsed character is then the same U+00A9 scalar. Encoding and HTML parsing solve different problems.

Escape metacharacters for the output context

An HTML serializer must encode ampersands and less-than signs where they could be interpreted as markup or character references. Attribute values have additional quoting rules. Not every non-ASCII character needs to be serialized as a numeric reference.

Converting every Unicode scalar to &#x... may be valid as a representation but harms readability and is not needed for ordinary UTF-8 output. Escape by context: text nodes, double-quoted attributes, URL components and JavaScript strings have different parsers.

Named references and semicolon rules have edge cases

HTML's named character reference table contains many aliases and names, and some historical references can be accepted without a final semicolon in certain contexts. However, acceptance differs between text and attribute contexts and can be ambiguous with following characters.

A custom regular expression that replaces substrings resembling &name is not a substitute for a standards-conforming tokenizer. Use the browser's HTML parser or a compliant library; do not invent simplified decoding rules for security-critical input.

Numeric references can be invalid or adjusted

Not every numeric value denotes a Unicode scalar. U+D800 is a surrogate code point and cannot represent a Unicode scalar value; values above U+10FFFF are invalid. HTML's tokenizer has defined error recovery, including replacement behavior for many invalid references and legacy control mappings.

Do not promise numeric character references preserve arbitrary 32-bit integers. Check precise HTML tokenization rules instead of assuming the result is a literal character corresponding to any input number.

Parsing an ampersand twice can corrupt user text

If the user writes the literal text & into an editor, rendering it as HTML requires encoding its ampersand. After one HTML parse, the text may be the five characters & as intended, or a single ampersand depending on the original source and escaping layer.

The application should never repeatedly call an entity decoder on already parsed DOM text. A second decode might turn a user's literal source demonstration into a different character. Record whether the inspected artifact is HTML source, DOM text or a serialized clipboard representation.

Unicode normalization does not decode HTML entities

NFC and NFD apply Unicode canonical-equivalence rules to sequences of characters; NFKC and NFKD apply compatibility mappings. None of these functions is an HTML character-reference parser. Running normalize('NFC') on the literal six-character string © does not turn it into ©.

Conversely, parsing an HTML entity does not make two canonically equivalent Unicode sequences identical. If the document needs NFC for search or storage, make that a separate, documented transformation.

Script, attribute and URL contexts need distinct treatment

Inside <script> raw-text handling differs from ordinary text nodes. HTML attribute parsing, embedded JavaScript string literals and URL query components each have different syntax and escaping. A function named escapeHTML cannot safely cover them all.

Untrusted strings placed into HTML should go through correct contextual output encoding or safe DOM APIs such as textContent. Inserting user text with innerHTML is a different, dangerous operation if that text can contain tags or event handlers.

Emoji, mathematical alphabets and supplementary characters

A Unicode character above U+FFFF can be represented in HTML using a single numeric character reference for its scalar value, for example 😀 for U+1F600 GRINNING FACE. JavaScript may represent that scalar using a UTF-16 surrogate pair in memory.

The HTML numeric reference uses the scalar value, not two surrogate references. This illustrates the distinction among HTML input syntax, Unicode code points, UTF-16 code units and visible glyphs.

Design round-trip fixtures for source and DOM

Test &, <, 😀, literal non-ASCII text, combining marks and a literal string that looks like an entity. Parse once, inspect textContent, serialize, then parse again and compare expected scalars.

A successful round trip preserves the intended text content under a defined policy; it need not recreate the exact same HTML source spelling. Distinguish lexical round trips of source markup from semantic round trips of DOM text.

Worked example: concrete source and observable output

This compact experiment isolates a defined data transformation. Use a standards-compliant parser or browser in an authorized test environment; a demonstration of raw HTML input is not an instruction to inject untrusted content into a live DOM. Treat displayed characters, HTML source spelling and serialized bytes as distinct measurements.

const box = document.createElement('div');
box.innerHTML = '© 😀'; // Example source spelling escaped once
console.log(box.textContent); // The literal text "© 😀"
box.innerHTML = '© 😀';
console.log(box.textContent); // "© 😀"
console.log([...box.textContent].map(c => c.codePointAt(0).toString(16)));
// Use textContent for untrusted literal text instead of innerHTML.

For a reproducible issue report, include the initial sequence of Unicode scalar values, whether Unicode normalization is expected, and the exact API boundary under test. If a result is different only in font appearance, record that separately from a change of text content.

Interoperability matrix and expected invariants

Interpreting this matrix requires a declared contract: exact scalar identity is stronger than canonical equivalence, and canonical equivalence is different from visual similarity. An implementation may preserve text while intentionally transforming formatting or serialization syntax.

Validation workflow and failure classification

1. HTML source. Record the input specimen ©, define expected output, and compare the original Unicode code points to the observed representation. the parser decodes a named reference. If the display differs, inspect the parser, DOM, selection or rendering layer before rewriting the source text.
2. Numeric source. Record the input specimen 😀, define expected output, and compare the original Unicode code points to the observed representation. one scalar value U+1F600. If the display differs, inspect the parser, DOM, selection or rendering layer before rewriting the source text.
3. DOM text. Record the input specimen ©, define expected output, and compare the original Unicode code points to the observed representation. actual Unicode character, not entity markup. If the display differs, inspect the parser, DOM, selection or rendering layer before rewriting the source text.
4. Normalization. Record the input specimen NFC / NFKC, define expected output, and compare the original Unicode code points to the observed representation. does not decode HTML references. If the display differs, inspect the parser, DOM, selection or rendering layer before rewriting the source text.
5. UTF-16 memory. Record the input specimen 😀, define expected output, and compare the original Unicode code points to the observed representation. one scalar stored in two code units. If the display differs, inspect the parser, DOM, selection or rendering layer before rewriting the source text.

Once the failure layer has been identified, create a regression fixture with the actual source, expected data and browser or parser version. Test ordinary and adversarial strings, including UTF-16 surrogate pairs, decomposed combining accents, nonbreaking spaces and zero-width characters when relevant.

Common misdiagnoses

• Character references are syntax in HTML source: A common false conclusion is that all text representations share the same semantics. Preserve the original source, document the transformation rule, and compare the exact intermediate stage discussed above before choosing a fix.
• Escape metacharacters for the output context: A common false conclusion is that all text representations share the same semantics. Preserve the original source, document the transformation rule, and compare the exact intermediate stage discussed above before choosing a fix.
• Named references and semicolon rules have edge cases: A common false conclusion is that all text representations share the same semantics. Preserve the original source, document the transformation rule, and compare the exact intermediate stage discussed above before choosing a fix.
• Numeric references can be invalid or adjusted: A common false conclusion is that all text representations share the same semantics. Preserve the original source, document the transformation rule, and compare the exact intermediate stage discussed above before choosing a fix.
• Parsing an ampersand twice can corrupt user text: A common false conclusion is that all text representations share the same semantics. Preserve the original source, document the transformation rule, and compare the exact intermediate stage discussed above before choosing a fix.

Respect the distinction between valid Unicode data and hostile markup. A sanitizer makes decisions about executable HTML; a Unicode-aware editor makes decisions about text identity and caret boundaries. One does not replace the other.

Primary specifications and implementation references

These are the source specifications and API documents for the behaviors described. Version-dependent browser behavior should be verified against the actual implementation. The distinction between normative algorithms and recommended application design is intentional.

• WHATWG HTML: Named character references. Reference for standards terminology or the corresponding browser and Unicode API; confirm the edition and feature support for the target environment.
• WHATWG HTML: Tokenization. Reference for standards terminology or the corresponding browser and Unicode API; confirm the edition and feature support for the target environment.
• MDN Character references. Reference for standards terminology or the corresponding browser and Unicode API; confirm the edition and feature support for the target environment.
• Unicode Core Specification. Reference for standards terminology or the corresponding browser and Unicode API; confirm the edition and feature support for the target environment.

Related research

Within the Clipboard, Editors & Markup virtual silo, this reference is part of the larger Unicode interoperability library. Additional published studies address neighboring data or editing layers:

• unicode escapes json javascript
• rich text plain text conversion
• unicode scalar values surrogates