Byte Order Marks and Endianness in UTF-8, UTF-16 and UTF-32

A BOM is a signature at the start of a stream

The Unicode character U+FEFF BYTE ORDER MARK can serve as an initial signature in encoded text. The byte sequence representing U+FEFF varies with the UTF encoding scheme and, for UTF-16 or UTF-32, its endianness. When interpreted at the beginning of a stream under a BOM-aware protocol, those bytes can identify the byte order or text encoding. U+FEFF is not a universal delimiter and does not belong indiscriminately in every text document.

Endianness describes the order of octets within multi-byte code units. Big-endian stores the most significant octet first; little-endian stores the least significant octet first. This affects UTF-16 (16-bit units) and UTF-32 (32-bit units). UTF-8 consists of octets arranged in a fixed sequence and does not have endian variants, so a UTF-8 BOM cannot indicate byte order.

Exact BOM byte patterns

These are signatures for encoded text streams, not a list of arbitrary zero-width formatting characters. Although U+FEFF historically also appeared as a zero-width no-break space, modern text should normally use U+2060 WORD JOINER for the word-joining use case. U+FEFF in the middle of text is not a second BOM; its handling is distinct and may reveal concatenation or editing mistakes.

Why decoding order must check long signatures first

UTF-32LE starts with FF FE 00 00, which contains the complete UTF-16LE signature FF FE as a prefix. A naive detector that checks for UTF-16LE first will misclassify a UTF-32LE BOM. When supporting both protocols and when no authoritative higher-level encoding declaration overrides sniffing, test the four-byte patterns before two-byte patterns.

Byte sniffing alone is not a universal way to detect every file's encoding. An unmarked UTF-16LE document beginning with ASCII may include zero bytes that suggest UTF-16, but that is heuristic rather than a guarantee. A rigorous interface carries an explicit encoding label, especially for files without BOMs; any fallback heuristic must be documented as such.

Web encoding precedence is protocol-specific

HTML and the WHATWG Encoding Standard define how a browser extracts character encoding information. An initial BOM is a strong signal in the web encoding sniffing algorithm, but the rules depend on the consumer and the format. It is misleading to assert a single precedence rule for every file format, programming language or transport.

The web platform strongly prefers UTF-8 for HTML. An initial UTF-8 BOM is usually unnecessary when the server declares UTF-8 and the document carries appropriate metadata. When files are concatenated, a BOM from the second fragment may become U+FEFF in the middle of the result and cause surprising output. A tool that prepends a BOM to every generated string can therefore introduce invisible content instead of improving compatibility.

Reading signed files in Python

Python distinguishes the generic signature-aware codecs from explicitly endian codecs. For example, utf-8-sig can consume an initial UTF-8 BOM during decoding, whereas ordinary utf-8 decoding preserves an initial U+FEFF character in the resulting string. The utf-16 and utf-32 generic codecs use BOM-based conventions for decoding and produce BOMs on encoding.

sample = "A"
assert sample.encode("utf-8") == bytes.fromhex("41")
assert sample.encode("utf-8-sig") == bytes.fromhex("ef bb bf 41")
assert bytes.fromhex("ef bb bf 41").decode("utf-8-sig") == "A"
assert bytes.fromhex("ef bb bf 41").decode("utf-8") == "\ufeffA"

assert sample.encode("utf-16-le") == bytes.fromhex("41 00")
assert sample.encode("utf-16-be") == bytes.fromhex("00 41")
assert sample.encode("utf-32-le") == bytes.fromhex("41 00 00 00")

If a protocol mandates a particular byte order, use utf-16-le, utf-16-be, utf-32-le or utf-32-be directly rather than trusting platform defaults. BOM handling is part of the file format contract, not an arbitrary choice by a user interface.

BOM behavior in browser text APIs

The Encoding Standard defines TextDecoder options including ignoreBOM, whose name can be counterintuitive. In web TextDecoder, the default ignoreBOM: false causes a leading BOM to be recognized and omitted from the decoded output for supported Unicode encodings. With ignoreBOM: true, the decoder ignores its BOM-skipping behavior and can expose the U+FEFF character instead. Always test the actual API rather than deducing the behavior from the option name.

const bytes = Uint8Array.of(0xEF, 0xBB, 0xBF, 0x41);
const stripped = new TextDecoder("utf-8").decode(bytes);
const retained = new TextDecoder("utf-8", {
  ignoreBOM: true
}).decode(bytes);
console.log(stripped === "A");            // true
console.log(retained.codePointAt(0).toString(16)); // feff

Web TextEncoder emits UTF-8 and does not insert a BOM by default. Decoding plain text from a byte buffer and displaying an already-decoded DOM string are separate operations; a BOM at the beginning of a byte stream is consumed at the decoding layer when the applicable rules specify it.

Byte-level examples: the letter A

Notice that the logical scalar U+0041 never changes. What changes is the byte representation and possibly the presence of an initial encoding signature. If an app displays garbled characters because a UTF-16LE file was treated as UTF-8, Unicode's character identity is not at fault; the byte-to-text decoding step used the wrong scheme.

Tricky boundaries and common integration failures

A shell script starting with UTF-8 BOM bytes may no longer begin with the exact ASCII bytes 23 21 (#!) expected by a shebang interpreter. A BOM unexpectedly inserted before a JSON-like data prefix or CSV header can become part of a key or header name when the consumer fails to strip it. Such problems often appear as a mysterious U+FEFF prefix rather than a visible error.

A file having no BOM is not necessarily non-Unicode; UTF-8 without a signature is standard practice. Conversely, the first bytes FF FE do not always suffice to classify a file without considering the four-byte UTF-32LE possibility. Some protocols explicitly forbid a BOM; others use one to decide byte order. Choose based on the protocol instead of a universal “always add” or “always remove” rule.

Never strip every occurrence of U+FEFF in a document to remove a BOM. Only an encoding-aware leading-signature rule can justify consuming the initial marker. A U+FEFF later in a string may indicate concatenation, accidental insertion or legacy word-joining use; handle it as data or flag it for review according to the application.

A reliable forensic workflow

When text appears corrupted, capture the first 8–16 raw bytes and the content type or encoding declaration before attempting repair. Check any four-byte signature, then applicable shorter signatures; apply the protocol-specific precedence rule; choose a strict decoder; and verify that the decoded scalar values match expectations. Distinguish byte-order errors from unrelated normalization, glyph fallback or text-shaping problems.

Record the result as concrete evidence: raw hex bytes, declared encoding, decoder used, scalar sequence, and any lossy substitutions. UTF-16 and UTF-32 decoders must also validate surrogate rules and maximum scalar range rather than assuming that a valid signature makes the remainder of the stream valid. The BOM establishes at most the claimed format; it does not validate all payload units.

Reproducible acceptance and rejection cases

Repeat tests across file upload, editor save, HTTP response processing and CSV export/import where the product uses these routes. A robust result is correct bytes and scalar values, not just a plausible screenshot; byte signatures and font rendering belong to different layers.

Standards and implementation references

1. Unicode FAQ — Byte Order Mark and UTF encoding forms
2. W3C Internationalization — The BOM in HTML
3. WHATWG Encoding Standard — BOM handling and encoding sniffing
4. MDN — TextDecoder.ignoreBOM
5. Python codecs — Signature-aware UTF codecs

Independent Unicode interoperability reference. Always validate encoded data separately from font rendering and user-perceived characters.