Unicode Scalar Values, Surrogates, and the Exact UTF-16 Encoding Model

Codespace, code points, and scalar values

Unicode defines a codespace from U+0000 to U+10FFFF, divided into 17 planes numbered 0 to 16. A code point is a position in that codespace, including positions that have not been assigned to characters. A Unicode scalar value is a code point excluding the surrogate range U+D800 through U+DFFF. This distinction is fundamental for interpreting UTF-8, UTF-16, and UTF-32 data.

There are 1,112,064 possible scalar values (1,114,112 code-point positions minus 2,048 surrogates). This does not mean Unicode assigns characters to every scalar value: unassigned positions, private-use values, and noncharacters have different statuses. Conversely, “outside the BMP” does not mean invalid—supplementary scalar values are an ordinary part of Unicode text.

Do not use the number of assigned characters in a particular Unicode release as the range limit of an encoder. Assigned-character counts change between versions; the codespace and the scalar-value exclusion rule do not.

UTF-16 encoding: BMP and supplementary characters

UTF-16 is an encoding form built from 16-bit code units. A BMP scalar value outside the surrogate range is encoded as one code unit whose numeric value equals the code point. A supplementary scalar value (U+10000..U+10FFFF) is encoded as two code units: first a high surrogate in D800..DBFF, then a low surrogate in DC00..DFFF.

The supplementary interval contains exactly 0x100000 possible scalar values, or 1,048,576 positions. The product of 1,024 possible high surrogates and 1,024 possible low surrogates equals that capacity. An isolated surrogate code unit does not designate a character and is not itself a Unicode scalar value.

Surrogate pair formulas and a worked example

For a supplementary scalar value U, first calculate V = U − 0x10000, where 0 ≤ V ≤ 0xFFFFF. Split V into a high ten-bit field and a low ten-bit field:

high = 0xD800 + (V >> 10)
low  = 0xDC00 + (V & 0x3FF)

U = 0x10000
  + ((high - 0xD800) << 10)
  +  (low  - 0xDC00)

For U+1F600 GRINNING FACE, V = 0xF600. The high ten bits are 0x3D and the low ten bits are 0x200; its UTF-16 sequence is D83D DE00. Serialized in UTF-16BE, those units become bytes D8 3D DE 00. In UTF-16LE, each unit’s bytes are reversed: 3D D8 00 DE.

The formula has two validation requirements often skipped in homegrown codecs: U must be between 0x10000 and 0x10FFFF for pair encoding, and both input code units must belong to the appropriate ranges when decoding. Do not silently pair two highs, two lows, or a trailing high surrogate with the next arbitrary BMP code unit.

Ill-formed sequences, lone surrogates, and error handling

A lone high surrogate, a lone low surrogate, or an incorrectly ordered pair is an ill-formed UTF-16 code-unit sequence. In a well-formed Unicode interchange encoding, a surrogate code point must not be emitted as a stand-alone encoded character. Different APIs adopt different failure policies: strict rejection, replacement with U+FFFD REPLACEMENT CHARACTER, or preservation in an intermediate language-specific string type.

JavaScript strings expose UTF-16 code units and can contain lone surrogate code units. That does not make such strings well-formed Unicode text. TextEncoder converts a lone surrogate to the UTF-8 bytes of U+FFFD, whereas encodeURIComponent() throws URIError for an isolated surrogate. Thus, two APIs processing the same JavaScript string can handle malformed input differently.

const valid = "\uD83D\uDE00";
console.log(valid.length);                    // 2 UTF-16 units
console.log(valid.codePointAt(0).toString(16)); // "1f600"
console.log([...valid].length);               // 1 code point

const lone = "\uD800";                        // unpaired high surrogate
console.log(lone.length);                     // 1 code unit
console.log(new TextEncoder().encode(lone));  // EF BF BD (U+FFFD)
console.log(lone.isWellFormed());             // false (modern JS)

When a runtime lacks String.prototype.isWellFormed(), validate the code-unit sequence explicitly. For security-sensitive pipelines, decide whether malformed sequences are rejected or replaced and apply that policy at the ingress boundary, not unpredictably in each downstream renderer.

The notion “Unicode string” varies between programming languages: Python 3 strings are sequences of code points at the language level and can still, through certain APIs, contain surrogate values that fail strict UTF-8 encoding. Java and JavaScript represent ordinary string indices as UTF-16 code units. Rust str requires valid UTF-8 and therefore cannot contain isolated surrogate scalar values. Always document the language runtime and error mode in interoperability tests.

Noncharacters, private use, and unassigned code points

A common validator mistake is to label all code points without assigned graphic characters “invalid.” They are not the same category:

• Surrogate code points: D800..DFFF are excluded from Unicode scalar values and must not be encoded as scalar characters in UTF-8 or UTF-32.
• Noncharacters: Such as U+FDD0..U+FDEF and the last two code points of each plane (U+FFFE, U+FFFF, U+1FFFE, etc.) are valid scalar values. They are permanently reserved for internal use and never assigned abstract characters. Unicode encoding forms can represent them, and their appearance alone is not proof of ill-formed UTF data.
• Private-use code points: BMP U+E000..U+F8FF and supplementary private-use areas represent valid scalars whose meaning is assigned by private agreement. Their semantics are not standardized across applications.
• Unassigned/reserved code points: Valid scalar positions may be assigned in future Unicode versions. An application can choose to limit its accepted repertoire, but that is an application policy, not an encoding constraint.

Whether to allow noncharacters or private-use scalars in an identifier, protocol field, or document is a separate profile decision. Standards such as UTS #39 define security-related identifier processing; valid UTF-8 by itself is not an authorization or safety guarantee.

Implementation patterns in JavaScript and Python

For code-point iteration in JavaScript, use for…of or spread iteration, not naive index-based iteration. codePointAt(index) consumes the pair only when called at a high-surrogate position; at the following low-surrogate index, it returns that low surrogate’s numeric value. String.fromCodePoint() validates its numeric input, while String.fromCharCode() accepts individual 16-bit code units.

function scalarValue(cp) {
  return Number.isInteger(cp) &&
    cp >= 0 && cp <= 0x10FFFF &&
    !(cp >= 0xD800 && cp <= 0xDFFF);
}

function utf16UnitsForScalar(cp) {
  if (!scalarValue(cp)) throw new RangeError("Not a scalar");
  if (cp <= 0xFFFF) return [cp];
  const v = cp - 0x10000;
  return [0xD800 + (v >> 10),
          0xDC00 + (v & 0x3FF)];
}
console.log(utf16UnitsForScalar(0x1F600).map(
  x => x.toString(16).toUpperCase().padStart(4, "0")
));
// ["D83D", "DE00"]

In Python, ord() returns the code-point number of a one-element string, and chr() produces a string for a code-point position. When emitting UTF-8, the default strict encoder rejects isolated surrogates:

face = "\U0001F600"
assert len(face) == 1
assert ord(face) == 0x1F600
assert face.encode("utf-8") == bytes.fromhex("F0 9F 98 80")

try:
    "\ud800".encode("utf-8")
except UnicodeEncodeError:
    print("Lone surrogate rejected")

Validation boundaries and test vectors

Test both decoding correctness and error-policy correctness. Passing only valid BMP samples will not catch off-by-one errors, bad endianness, overlong UTF-8, unpaired UTF-16 units, unexpected replacement policy, or truncated supplementary sequences.

Common integration failures

• Assuming string.length means “number of Unicode characters” in JavaScript or Java.
• Breaking a surrogate pair when truncating text to a UI length limit, producing replacement glyphs or URI encoding exceptions.
• Converting UTF-16 units individually to UTF-8 rather than decoding into scalar values before encoding.
• Treating the high/low pair as two independent characters in regular expressions, indexes, or substring algorithms.
• Rejecting noncharacters as “malformed UTF” without an actual protocol requirement.
• Assuming every valid scalar is assigned a visible graphic character in the latest Unicode release.

Primary specifications and further reading

1. The Unicode Standard 18.0.0, encoding forms and definitions.
2. Unicode FAQ — UTF-8, UTF-16, UTF-32, and BOM, surrogate model and malformed units.
3. Unicode FAQ — Private-use characters and noncharacters, their allocation status and validity.
4. UAX #44 — Unicode Character Database, code point categories and property definitions.
5. MDN — String.prototype.isWellFormed(), detecting unpaired UTF-16 surrogates.

Independent technical commentary on public standards. Unicode property and library behavior must be checked against the version shipped with each runtime.