UTF-16 Surrogate Pairs: Calculation, Validation and String Boundaries

What a UTF-16 code unit represents

UTF-16 represents a Unicode scalar value with either one 16-bit code unit or a pair of 16-bit surrogate code units. Scalars U+0000..U+FFFF outside the surrogate interval U+D800..U+DFFF use one unit. Supplementary scalars U+10000..U+10FFFF use exactly two code units. UTF-16 code units are not the same thing as grapheme clusters, displayed glyphs, or even individual scalar values in the supplementary range.

The range U+D800..U+DBFF is reserved for high surrogates and U+DC00..U+DFFF for low surrogates. These values are legal UTF-16 code units only when they appear as a correctly ordered pair. They are not Unicode scalar values and must not be independently encoded into UTF-8 or UTF-32. A supplementary emoji or mathematical letter may occupy two UTF-16 units even if rendered as a single symbol.

Deriving the surrogate pair formula

For scalar U between U+10000 and U+10FFFF, subtract 0x10000 to obtain a 20-bit number V. Split V into ten upper bits and ten lower bits. The high code unit is 0xD800 + (V >> 10); the low code unit is 0xDC00 + (V & 0x3FF). Reassembly reverses the process: 0x10000 + ((high - 0xD800) << 10) + (low - 0xDC00).

For U+1F600 GRINNING FACE, V = 0xF600. The high surrogate is 0xD83D and the low surrogate is 0xDE00. For U+1D400 MATHEMATICAL BOLD CAPITAL A, the pair is D835 DC00. Those hexadecimal words describe UTF-16 units; serializing them into bytes requires a separate endian choice.

What makes a sequence well-formed?

A conforming UTF-16 sequence cannot contain a high surrogate unless the next code unit is a low surrogate, or a low surrogate unless immediately preceded by its matching high surrogate. A high surrogate followed by another high surrogate, a BMP letter or end of input is ill-formed. Similarly, a low surrogate that appears at the beginning of a string is ill-formed.

Some string APIs permit ill-formed internal UTF-16 sequences, even though standard UTF encodings for interchange require well-formed input. This is an API representation issue, not permission to call an isolated surrogate a Unicode character. A decoder's error policy might reject, replace or expose problematic units; callers need to know which behavior is used before using the data as a filename, identifier or database key.

JavaScript: length, codePointAt and iteration

JavaScript exposes its indexed strings as sequences of 16-bit code units. Indexing with text[i] or querying length counts units, while for…of and [...text] iterate string code points, combining valid surrogate pairs. codePointAt(index) reads a supplementary scalar when called at the high-surrogate position, but passing an index at the low surrogate yields that low surrogate's numeric value.

const smile = "😀";
console.log(smile.length);                  // 2 UTF-16 units
console.log(smile.charCodeAt(0).toString(16)); // d83d
console.log(smile.charCodeAt(1).toString(16)); // de00
console.log(smile.codePointAt(0).toString(16)); // 1f600
console.log([...smile].length);             // 1 code point
console.log(String.fromCodePoint(0x1F600) === smile); // true
console.log(smile.slice(0, 1).length);      // 1: lone high surrogate

Never assume a code-point iterator automatically identifies user-perceived characters. A flag emoji made from two regional indicators, or an emoji ZWJ sequence, contains multiple scalar values. Extended grapheme segmentation (UAX #29) better approximates cursor and deletion boundaries. Even grapheme clusters are not interchangeable with shaped glyph counts.

Lone surrogates in modern JavaScript

Recent JavaScript exposes String.prototype.isWellFormed() and toWellFormed(). The first reports whether the UTF-16 sequence contains unpaired surrogates; the second returns a copy with those unpaired units replaced by U+FFFD. This replacement is lossy by definition. The methods are helpful before converting input to a scalar-based interchange format but should not be used to silently conceal a validation failure when exact preservation is required.

const broken = "\uD83D";
console.log(broken.isWellFormed()); // false
console.log(broken.toWellFormed()); // "�"
console.log(broken.toWellFormed().codePointAt(0).toString(16));
// fffd
const ok = "\uD83D\uDE00";
console.log(ok.isWellFormed()); // true

The UTF-8 TextEncoder also maps unpaired surrogate input to the replacement scalar as part of its scalar-based encoding. That is very different from a custom encoder incorrectly emitting UTF-8 bytes that encode surrogate code points directly, which must be rejected as malformed UTF-8.

Python round trips and strict decoding

Ordinary Python strings expose Unicode code point integers to ord(). UTF-16 byte serialization is explicit: choose UTF-16LE or UTF-16BE where the protocol defines the order, or UTF-16 with a signature according to Python's codec behavior. Round-trip tests should compare scalar sequences and not only the display.

value = "A😀𝐀"
raw = value.encode("utf-16-le")
assert raw.hex(" ") == "41 00 3d d8 00 de 35 d8 00 dc"
assert raw.decode("utf-16-le", errors="strict") == value
try:
    bytes.fromhex("3d d8").decode("utf-16-le", errors="strict")
    raise AssertionError("accepted lone high surrogate")
except UnicodeDecodeError:
    pass

Notice how the UTF-16LE bytes for the surrogate pair D83D DE00 are 3D D8 00 DE, not D8 3D DE 00. The high and low surrogates remain in logical order; only the byte order within each 16-bit unit changes. Confusing logical code-unit order with byte order is a common source of corrupt data.

The unsafe midpoint problem

Truncating a JavaScript string at a raw code-unit length can split a pair. An editor that removes just the first unit of U+1F600 leaves a lone low surrogate, and downstream text encoding may replace it with U+FFFD. For limits measured in bytes, scalar values, or grapheme clusters, use the corresponding boundary-aware algorithm instead of a generic slice by length.

An Intl.Segmenter created with granularity: "grapheme" yields grapheme-like text segments for user-visible editing. It does not normalize Unicode, fix malformed sequences, enforce protocol-specific byte limits or guarantee identical behavior across all Unicode data versions. Keep tests tied to the runtime's segmentation version where consistency is critical.

UTF-16 encoding form versus byte encoding scheme

UTF-16 is often discussed without a byte order because it is a sequence of 16-bit code units. To transmit bytes, UTF-16BE stores each 16-bit unit's most significant byte first; UTF-16LE stores its least significant byte first. A byte order mark (U+FEFF) at the start of some streams can signal the endian form, but the precise policy depends on the surrounding file format and encoding label.

A UTF-16 string is not inherently smaller than UTF-8 text. ASCII-heavy English typically needs two bytes per BMP scalar in UTF-16 versus one in UTF-8; supplementary scalars use four bytes in either scheme. JavaScript string internal memory accounting may include runtime optimization and is separate from serialized byte counts.

Verification cases and expected failures

Run round trips for all BMP boundaries, supplementary boundaries and malformed pairs. Assert bytes, code units and decoded scalar values separately. For display-facing features, add a grapheme and font-fallback test rather than assuming that a correct UTF-16 pair implies a single rendered glyph.

Standards and implementation references

1. Unicode Standard Chapter 3 — UTF-16 Encoding Form
2. Unicode FAQ — UTF-16, surrogate pairs and byte order
3. UAX #29 — Unicode Text Segmentation
4. MDN — String.prototype.isWellFormed
5. MDN — String.prototype.toWellFormed

Independent Unicode interoperability reference. Always validate encoded data separately from font rendering and user-perceived characters.