Unicode String Length: UTF-8 Bytes, UTF-16 Units, Code Points and Graphemes

“Length” is incomplete without a unit

A string length may refer to serialized UTF-8 bytes, UTF-16 code units, Unicode code points or grapheme clusters perceived as characters. These metrics are not interchangeable, especially for supplementary mathematical symbols, combining sequences and emoji. The same string can have radically different lengths under each definition even when it occupies a single perceived text element on screen.

Unicode encodes abstract characters as code points and uses encoding forms for serialized representation. UTF-8 expresses scalar values using one to four bytes each; UTF-16 uses one or two 16-bit code units for each scalar; UTF-32 uses one 32-bit unit per scalar. Extended grapheme clusters are segmentation units defined by UAX #29. Pixel width and glyph count are additional measures that no one Unicode length function supplies.

Exact measurements of common test strings

The numbers assume valid Unicode scalar sequences and the default extended grapheme rules. For the woman-technologist emoji, four UTF-8 bytes represent U+1F469, three represent U+200D and four represent U+1F4BB, totaling eleven bytes. UTF-16 needs two, one and two code units respectively. Do not estimate network payload size by counting screen glyphs.

JavaScript length counts UTF-16 units

JavaScript's String.prototype.length reports UTF-16 code units; it does not report Unicode scalar count. [...s] advances through a valid surrogate pair as one code point, but combining sequences and emoji ZWJ sequences still contribute multiple array elements. new TextEncoder().encode(s).length provides UTF-8 byte length for the string after the encoder's handling of internal string data. Intl.Segmenter with granularity:"grapheme" supplies a user-facing cluster count.

const segmenter = new Intl.Segmenter("en",{granularity:"grapheme"});
function measures(text) {
  if (!text.isWellFormed()) throw new TypeError("Unpaired UTF-16 surrogate");
  return {
    utf8Bytes: new TextEncoder().encode(text).length,
    utf16Units: text.length,
    codePoints: [...text].length,
    graphemes: [...segmenter.segment(text)].length
  };
}
console.log(measures("👩‍💻"));
// { utf8Bytes: 11, utf16Units: 5, codePoints: 3, graphemes: 1 }
console.log(measures("e\u0301"));
// { utf8Bytes: 3, utf16Units: 2, codePoints: 2, graphemes: 1 }

By checking isWellFormed() before measurement, the function avoids claiming that the internal JavaScript string contains a valid Unicode scalar sequence when it holds an unpaired surrogate. An ordinary TextEncoder conversion will replace ill-formed surrogate units under the web platform's string conversion rules, so byte counts of the encoded replacement are not the same as bytes for a hypothetical invalid UTF-8 encoding.

Python code-point counts are different from JavaScript length

For normal Python 3 strings, len(s) counts code points, not UTF-16 code units or graphemes. Encoding explicitly with s.encode("utf-8", "strict") computes bytes and reports invalid surrogate values as errors. To count UTF-16 units for a valid string, encode using an explicitly endian UTF-16 codec that does not prepend a BOM and divide the resulting byte count by two.

s = "👩‍💻"
assert len(s) == 3
assert len(s.encode("utf-8")) == 11
assert len(s.encode("utf-16-le")) // 2 == 5
assert len("e\u0301") == 2
assert len("e\u0301".encode("utf-8")) == 3
# A conforming UAX #29 implementation is needed for graphemes;
# neither Python len() nor slicing supplies that segmentation.

Using utf-16 instead of utf-16-le introduces a BOM by default in many Python encoders, adding two bytes to the serialized result. Always identify encoding, byte order and BOM policy when measuring or comparing serialized sizes.

Canonical normalization can change code point and byte counts

U+00E9 and e followed by U+0301 are canonically equivalent, but the former takes two UTF-8 bytes, one scalar and one UTF-16 unit, while the latter takes three UTF-8 bytes, two scalars and two UTF-16 units. NFC typically composes the latter to U+00E9; NFD decomposes the former. Thus code-point count and byte length are not invariant under canonical normalization, even when grapheme count stays one in this example.

Compatibility normalization NFKC can be more destructive. It may transform mathematical bold A into ordinary A, reducing serialized bytes from four to one while discarding the encoded mathematical identity. A text system that promises exact copy-paste fidelity should not silently apply such a transform to the only stored copy.

Choose different length policies for different constraints

An HTTP or database storage limit often applies to bytes, while a JavaScript UI's naive maxlength or custom count may apply code units depending on the element and platform. A message composer can reasonably display a grapheme-cluster limit because people care about perceived characters. A backend might also need independent UTF-8 byte limits to protect database capacity, and scalar/normalization validation for protocol safety.

No single metric simultaneously guarantees a memory budget, visual width and perceived character count. CJK ideographs can occupy more display columns than a Latin letter in terminal environments; emoji ZWJ clusters can be visually wider than both. Shaped ligatures may combine multiple letters into one glyph. Pixel width requires measuring shaped glyph runs using actual fonts and layout settings, not multiplying grapheme count by an average character width.

Glyph count, terminal columns and screen width are separate

Unicode East Asian Width properties help some terminal applications reason about character columns, but are not a universal pixel-width metric. Font fallback, script shaping, line-break rules and emoji presentation affect visual advances. A zero-width joining character may contribute no own visible glyph while changing its neighboring sequence, so character counts alone do not predict layout.

For the US Fancy Text tool, one mathematical bold scalar may take four UTF-8 bytes and two UTF-16 units yet count as one grapheme. A destination's missing-font box does not mean the character's scalar count changed. Inspect the data separately from rendering when diagnosing cross-platform copy and paste.

Safe truncation needs a clearly chosen boundary

Slicing a UTF-8 buffer at an arbitrary byte limit may cut through a multibyte sequence and produce malformed UTF-8. Slicing JavaScript by code unit can split a surrogate pair, and slicing by code point can separate combining marks or emoji components. If truncation is specified in user-perceived characters, enumerate extended grapheme clusters and retain complete clusters; if it is also limited by bytes, measure the encoded result after adding each complete cluster.

function capByGraphemesAndBytes(text,maxClusters,maxBytes) {
  const parts = new Intl.Segmenter("en",{granularity:"grapheme"}).segment(text);
  const encoder=new TextEncoder();
  let out="",count=0;
  for(const {segment} of parts){
    if(count >= maxClusters) break;
    if(encoder.encode(out + segment).length > maxBytes) break;
    out+=segment; count++;
  }
  return out;
}
console.log(capByGraphemesAndBytes("A😀B",2,5)); // A😀
console.log(capByGraphemesAndBytes("e\u0301X",1,3)); // é

Counting streamed text with boundaries

UTF-8 byte counts add across chunks because they measure octets, provided the original byte stream is observed directly. UTF-16 code units can be counted in chunks if the encoding representation is clear, but a chunk ending with a high surrogate may not represent a complete scalar. Code point counts and grapheme counts cannot always be computed by independently decoding and segmenting arbitrary chunks, because bytes or grapheme clusters may be split across chunks.

For streaming grapheme counting, the state machine needs the appropriate preceding context for combining marks, regional indicator parity, emoji ZWJ sequences and other UAX #29 rules. Use tested incremental segmentation or apply the algorithm to the complete logical string.

Exact length regression vectors

Use the Unicode conformance corpus GraphemeBreakTest.txt for segmentation and RFC 3629 for UTF-8 validation. Track measurement units in API field names and database schemas; an unqualified length field creates unnecessary interoperability defects.

Standards and implementation references

1. UAX #29 — Unicode Text Segmentation
2. RFC 3629 — UTF-8
3. UAX #15 — Unicode Normalization
4. MDN — TextEncoder
5. MDN — Intl.Segmenter
6. Python string types and length

Independent technical reference. Editing behavior may be tailored; preserve source text and test against version-matched Unicode data.