UTF-8 Byte Sequences: Encoding Rules, Boundaries and Validation

UTF-8 encodes scalar values into octets

UTF-8 is a variable-length Unicode encoding form. It serializes a Unicode scalar value—a code point excluding the surrogate range U+D800..U+DFFF—into one, two, three or four 8-bit code units (octets). One octet is identical to ASCII for U+0000..U+007F. This preservation of ASCII byte values is one of the reasons UTF-8 fits protocols, source files and modern web content.

Encoding and decoding are not glyph operations. The glyph rendered for U+1D400 may be bold, missing or substituted by a fallback font; its UTF-8 bytes are still defined exactly. Likewise, é encoded as U+00E9 uses different bytes from the canonically equivalent sequence U+0065 U+0301. Normalization and encoding solve different problems.

The one-to-four-byte bit templates

The initial octet determines the expected sequence length; continuation octets have their two most significant bits set to 10. A conforming decoder checks both the pattern and whether the decoded integer is a valid scalar represented with the shortest possible sequence.

Legal continuation octets range from 80 to BF in hexadecimal. An octet from C0 or C1 cannot begin a valid shortest-form UTF-8 sequence, nor can F5..FF begin any valid sequence under RFC 3629. Literal octets FE and FF are never part of well-formed UTF-8 text. The original older proposals with five- and six-octet sequences do not define valid modern UTF-8.

Encoding U+1F600 step by step

The supplementary scalar U+1F600 has a 21-bit binary representation placed into the four-byte template 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx. Grouping its bits into the available payload positions yields F0 9F 98 80. This form is independent of CPU endianness: UTF-8 has no byte-order variation.

Each code point maps to a unique shortest well-formed UTF-8 sequence. However, an arbitrary string of multiple Unicode scalars can have multiple encodings of visually equivalent text because scalar sequences themselves may differ canonically, and because glyph layout is not an encoding function. A robust text pipeline can choose NFC if its application requires canonical normalization, but should never confuse NFC with UTF-8 byte validation.

Boundary octets a validator must reject

Decoders must reject overlong representations, UTF-8 encodings of surrogate values, values above U+10FFFF and truncated or malformed continuation sequences. The lead byte alone does not capture every constraint; particular second bytes have narrower legal ranges.

The remaining continuation octets in each sequence must also be within 80..BF. RFC 3629 and Unicode conformance disallow a decoder from silently treating overlong forms as a different way of spelling the same scalar. These cases have security implications: different decoders accepting and rejecting the same bytes can create bypasses in validation or path processing.

Working encode/decode examples

In web JavaScript, TextEncoder produces UTF-8 bytes from a JavaScript string. TextDecoder can decode UTF-8 with replacement behavior by default or reject malformed input with the fatal option. JavaScript strings are UTF-16 code-unit sequences, so a lone surrogate supplied to TextEncoder is replaced with U+FFFD as part of converting input to valid Unicode scalar text.

const data = "Aé€😀𝐀";
const bytes = new TextEncoder().encode(data);
console.log(Array.from(bytes, b =>
  b.toString(16).padStart(2, "0")).join(" "));
// 41 c3 a9 e2 82 ac f0 9f 98 80 f0 9d 90 80
const roundTrip = new TextDecoder("utf-8", {fatal: true}).decode(bytes);
console.log(roundTrip === data); // true

try {
  new TextDecoder("utf-8", {fatal: true})
    .decode(Uint8Array.of(0xC0, 0xAF)); // overlong '/'
} catch (e) {
  console.log(e.name); // TypeError
}

Python exposes strict conversion explicitly through .encode("utf-8") and .decode("utf-8", errors="strict"). An exception from strict decoding is important information; using errors="ignore" silently discards data, while replacing invalid input deliberately loses byte-level fidelity.

text = "Aé€😀𝐀"
raw = text.encode("utf-8")
print(raw.hex(" "))
assert raw.decode("utf-8", errors="strict") == text
for raw_bad in (bytes.fromhex("c0 af"),
                bytes.fromhex("ed a0 80"),
                bytes.fromhex("f4 90 80 80")):
    try:
        raw_bad.decode("utf-8", errors="strict")
        raise AssertionError("accepted malformed bytes")
    except UnicodeDecodeError:
        pass

Streaming and chunk boundaries

A transport may deliver only part of a multi-octet sequence. A valid UTF-8 stream divided between chunks can therefore contain an incomplete chunk without containing malformed text. A streaming decoder retains incomplete lead/continuation state across input boundaries; it cannot simply decode each chunk independently with separate replacement behavior.

const enc = new TextEncoder().encode("😀"); // f0 9f 98 80
const dec = new TextDecoder("utf-8", { fatal: true });
const first = dec.decode(enc.slice(0, 2), { stream: true });
const second = dec.decode(enc.slice(2), { stream: true });
const finish = dec.decode();
console.log(first + second + finish === "😀"); // true

At end of stream a trailing incomplete prefix must be handled using the selected error policy. Converting partial bytes into U+FFFD prematurely can corrupt otherwise valid network text. For byte-oriented protocols, specify whether malformed input triggers rejection, replacement or an error report; never allow ambiguous processing of untrusted byte strings.

Bytes versus user-visible character counts

The UTF-8 byte size of A, é, € and 😀 is respectively 1, 2, 3 and 4 in their precomposed/scalar forms. Four bytes does not mean four characters. The mathematical styled capital 𝐀 also occupies four UTF-8 bytes; compared with ASCII A, this affects byte-limited fields and storage budgets without changing the number of visible letters.

For an application offering copyable styled Unicode text, a useful check is the difference between new TextEncoder().encode(text).length, the count of scalar values and the number of user-perceived grapheme clusters. Each answers a different question. The US Fancy Text tool illustrates why visually similar output can impose different byte limits on downstream services.

File formats, signatures and diagnostic procedure

UTF-8 does not need a BOM for endianness. An optional initial EF BB BF can act as an encoding signature, but some protocols and file formats treat it specially or reject it. A BOM is not part of the encoded payload of a normal ASCII-first UTF-8 text file. Determine transport or format requirements rather than inserting it universally.

When a file appears corrupted, first capture the original bytes before any lossy decode. Next verify the declared encoding and use a strict UTF-8 decoder. Compare the resulting scalar sequence with the intended content. Distinguish the wrong-encoding symptom often called mojibake (e.g. UTF-8 bytes interpreted as a different legacy encoding) from invalid UTF-8, normalization changes or missing fonts.

Conformance-oriented byte test vectors

Test the chosen error policy on malformed input and document it. A conforming UTF-8 validator must not report malformed data as a valid scalar sequence; a replacement decoder may produce U+FFFD as a recovery strategy. Both behaviors are useful when chosen deliberately for the surrounding protocol.

Standards and implementation references

1. RFC 3629 — UTF-8, a transformation format of ISO 10646
2. Unicode Standard Chapter 3 — Conformance, Encoding Forms
3. Unicode FAQ — UTF-8 and byte-order marks
4. WHATWG Encoding Standard
5. MDN — TextDecoder
6. UAX #15 — Unicode Normalization Forms

Independent Unicode interoperability research. Examples describe encoded data, not a guarantee about how every font, editor, or platform will render it.