Malformed Unicode Encoding Sequences: Detection, Recovery and Security

Well-formed Unicode streams versus damaged code units

A Unicode scalar value ranges from U+0000 through U+10FFFF, excluding U+D800..U+DFFF. UTF-8, UTF-16 and UTF-32 are encoding forms for scalar sequences. A malformed encoding sequence is a group of code units or bytes that cannot be interpreted as such a sequence under the relevant form. It is not the same as an unusual assigned character, an invisible control or an unsupported font glyph.

Examples include an illegal UTF-8 leading byte, a truncated UTF-8 multibyte prefix at end of stream, a lone UTF-16 high surrogate, a UTF-16 low surrogate without its preceding high surrogate, and a UTF-32 word with a surrogate or value greater than U+10FFFF. The application must define what happens when these arrive from untrusted input.

Malformed UTF-8: lead, continuation and shortest-form rules

Modern UTF-8 uses one to four bytes per scalar. Continuation bytes have the form 10xxxxxx, hexadecimal 80..BF. Legal leading bytes are ASCII 00..7F, C2..DF, E0..EF and F0..F4, with additional range constraints on the second byte in certain sequences. An isolated 80..BF byte cannot begin a character, and C0..C1 or F5..FF are forbidden as modern UTF-8 leading bytes.

Overlong forms are especially important for security. If one component treats an invalid byte sequence as an ASCII slash and another rejects it, path validation and authorization can disagree. Validating before policy checks, then using the same decoded representation through the stack, avoids canonicalization gaps.

Malformed UTF-16 and the internal string trap

UTF-16 stores supplementary scalars as a high surrogate D800..DBFF immediately followed by a low surrogate DC00..DFFF. A lone high surrogate at end of buffer, a low surrogate at the beginning, or a high surrogate followed by a BMP letter is ill-formed as serialized UTF-16. It is incorrect to encode an isolated surrogate independently into UTF-8 or UTF-32.

Some runtimes, notably JavaScript, represent strings as sequences of UTF-16 code units and can hold unpaired surrogates internally. A JavaScript string containing \uD800 does not thereby become valid Unicode scalar data. String.prototype.isWellFormed() can detect such units, while toWellFormed() substitutes U+FFFD for each unpaired surrogate, losing exact information.

const broken = "\uD800";
console.log(broken.isWellFormed());  // false
console.log(broken.toWellFormed());  // U+FFFD replacement character
console.log([..."\uD83D\uDE00"].length); // 1 valid scalar
console.log("\uD83D\uDE00".length);       // 2 code units

Invalid UTF-32 words and noncharacter confusion

UTF-32 uses one 32-bit code unit per Unicode scalar. Values U+D800..U+DFFF and values above U+10FFFF are malformed as scalars, even though they fit inside a 32-bit integer. A stream with a byte count that is not a multiple of four is truncated for an explicitly endian UTF-32 codec.

Noncharacters such as U+FDD0, U+FFFE and U+10FFFF are reserved for internal-use conventions, but they remain valid Unicode scalar values and can be well-formed in UTF encodings. A higher-level interchange profile might reject them; do not conflate profile restrictions with encoding conformance. U+FEFF used as a BOM is likewise distinct from the malformed-byte question.

Fatal errors, replacement, and information loss

A strict decoder rejects malformed input. A replacement decoder emits U+FFFD for invalid subsequences according to its specified algorithm, allowing processing to continue. The replacement character is a valid scalar, but its appearance means the original bytes are no longer reconstructible from the decoded string alone. An ignore policy is even more lossy because it removes offending data without evidence.

Replacement granularity matters: robust UTF-8 decoding follows the Unicode recommendation to replace maximal subparts of an ill-formed subsequence rather than silently skipping arbitrary portions or decoding overlong forms. Different nonconforming decoders can yield different counts of replacement characters. Test the decoder implementation and Unicode version rather than assuming that every implementation replaces identical byte groups.

const malformed = Uint8Array.of(0xED, 0xA0, 0x80);
try {
  new TextDecoder("utf-8", {fatal: true}).decode(malformed);
  throw new Error("Should reject malformed UTF-8");
} catch (error) {
  if (!(error instanceof TypeError)) throw error;
}
console.log(new TextDecoder("utf-8").decode(malformed));
// Replacement characters, not a valid surrogate scalar.

Streaming chunks must not be validated in isolation

Consider a valid four-byte encoding F0 9F 98 80 for U+1F600. A network chunk containing F0 9F is incomplete, not necessarily malformed. A decoder maintaining state across chunks can retain that prefix and consume the remaining 98 80 when it arrives. If the transport definitively ends after F0 9F, the same prefix is malformed at end of stream.

const bytes = Uint8Array.of(0xF0,0x9F,0x98,0x80);
const dec = new TextDecoder("utf-8",{fatal:true});
const first = dec.decode(bytes.slice(0,2),{stream:true});
const rest = dec.decode(bytes.slice(2),{stream:true});
const finish = dec.decode(); 
console.log(first + rest + finish === "😀"); // true

Boundary-aware validation is essential for WebSocket frames, file streams, multipart uploads and database chunks. Splitting a UTF-8 stream into arbitrary chunks and decoding each independently can inject replacement characters into otherwise well-formed data.

Python strict-mode fixtures

Python's strict codecs make malformed sequences observable as UnicodeDecodeError. The following test contains both valid and rejected streams. A permissive mode may be desirable for logs shown to humans, but security-sensitive identifiers and protocol payloads should have a documented rejection policy.

ok = bytes.fromhex("f0 9f 98 80").decode("utf-8", "strict")
assert ok == "😀"
invalid = ["c0 af", "ed a0 80", "f4 90 80 80", "e2 82"]
for case in invalid:
    try:
        bytes.fromhex(case).decode("utf-8", "strict")
        raise AssertionError(f"accepted {case}")
    except UnicodeDecodeError:
        pass
assert bytes.fromhex("ed a0 80").decode("utf-8", "replace") != "😀"

Security boundaries: validate bytes before interpreting paths

Malformed data should not be allowed to travel across layers that interpret it differently. Overlong encoding attacks, mixed strict and replacement decoders, and CESU-8 surrogate encodings can create ambiguous identities. CESU-8 and modified UTF-8 may encode surrogate code units for compatibility purposes; such sequences are not valid UTF-8 and must not be silently accepted as conforming UTF-8.

Different application rules apply after decoding: Unicode normalization can fold canonically equivalent strings, case-folding may alter search identities, and confusable detection may flag lookalike characters. These are independent tasks. A validator that accepts well-formed UTF-8 cannot on that basis claim that a username or file path is secure, human-readable or visually unambiguous.

A decoder conformance checklist for implementers

Standards and technical references

1. Unicode Standard Chapter 3 — Conformance and ill-formed sequences
2. UTR #36 — Unicode Security Considerations
3. RFC 3629 — UTF-8 restrictions
4. WHATWG Encoding Standard — UTF-8 decoding error algorithm
5. MDN — TextDecoder.fatal
6. Python codec error handling

Independent technical research. Use strict byte-level evidence to diagnose encoding defects before transforming data.