Unicode-Safe String Truncation: Grapheme Boundaries, Byte Budgets and Normalization

Truncation is not one operation: a database may enforce a byte limit, an API may specify UTF-16 units, and a caption preview may need a fixed number of extended grapheme clusters. A substring cut at an arbitrary offset can leave a surrogate half, remove a combining mark from its base or break an emoji ZWJ sequence. Choose the metric first, then cut at a boundary valid for that metric.

State the constraint before writing code

A limit of 100 can mean UTF-8 bytes, Unicode code points, UTF-16 code units, grapheme clusters or rendered width. These metrics diverge dramatically for emoji and combining sequences. There is no single correct function named truncateUnicode without further arguments.

A user-facing character limit usually benefits from grapheme-aware behavior, while transport limits must be enforced in actual encoded bytes. Rendered pixel width additionally depends on font, size, shaping and layout and cannot be predicted from scalar count alone.

Do not split a UTF-8 multibyte sequence

The UTF-8 representation of U+1F600 contains four bytes. Cutting a byte array after the first three yields an invalid sequence. A decoder operating in strict mode should report an error, while replacement mode can emit U+FFFD.

For byte-budget output, iterate complete Unicode scalars or grapheme clusters, encode each candidate and stop before adding a unit that would exceed the budget. Decide whether to append an ellipsis and reserve its encoded size before choosing the cut point.

UTF-16 substring can bisect a surrogate pair

In JavaScript, '😀'.length is 2. slice(0,1) yields an isolated high surrogate code unit, not a valid scalar value. This can cause replacement characters or serialization inconsistencies at a later boundary.

When APIs require UTF-16 positions (such as DOM Ranges), round cut points to valid scalar boundaries at minimum. For visible text, grapheme boundaries are usually better than code-point boundaries.

Grapheme-safe truncation protects complex sequences

The decomposed accented letter é and the joined profession emoji 👩‍💻 each commonly form one extended grapheme cluster. Cutting by code-point count can detach the combining accent or leave an incomplete joined sequence.

Intl.Segmenter can enumerate grapheme segments in JavaScript; ICU BreakIterator performs related segmentation in other environments. Treat default boundaries as a useful editing convention, not a promise that all fonts render one glyph per segment.

Code-point truncation is appropriate only for a code-point contract

Iterating JavaScript strings with for...of yields code points rather than raw UTF-16 halves, making it safer than arbitrary code-unit slicing for valid source strings. But it can still truncate a combining mark separately from its base.

In a scientific API that genuinely limits scalars, code-point truncation is legitimate and should be named as such. Do not expose its output as 'first N visible characters' when ZWJ, variation selectors and combining sequences are possible.

Ellipses consume budget and have their own semantics

An appended U+2026 HORIZONTAL ELLIPSIS is one scalar and commonly one grapheme, but three ASCII dots are three scalars. In UTF-8 the ellipsis takes three bytes. The choice affects exact byte budgets and preview comparisons.

Use a deliberate output contract: whether ellipsis counts toward the limit, whether it is appended only on actual truncation and how readers can recover the full content. Never modify stored identifiers simply to fit a narrow visual card.

Normalization changes lengths in different units

NFC composes some decomposed sequences without changing their typical grapheme count, while NFKC can change styled mathematical letters to ordinary code points. A limit applied before normalization might produce a different accepted set than a limit applied after it.

Define an explicit pipeline order: parse input, validate well-formedness, apply a declared normalization if required, segment under a chosen version, then enforce the output budget. Log transformations if exact round-trip fidelity matters.

Width-based truncation is a layout problem

In a responsive web UI, CSS text-overflow:ellipsis can visually clip a string without changing its DOM source value. This is different from generating a shorter stored Unicode string and can be preferable for account labels or dynamic translations.

Visual clipping can still hide security-critical suffixes or important qualifiers. Provide an accessible full-text reveal mechanism, especially for origins, paths and names, and verify keyboard focus and assistive-technology behavior.

Test against supplementary and multi-scalar fixtures

Cover ASCII, é, é, thumbs-up with modifier, regional indicator flags, emoji ZWJ, RTL text and strings containing U+FE0F. For each, assert all metrics and expected retained substring.

Run the same helper over strict byte limits, zero-length cases, an ellipsis longer than the budget and strings with malformed surrogates. A correct helper must never produce invalid UTF-8 or overstate the number of preserved grapheme clusters.

Reproducible technical example

This controlled specimen demonstrates a single subsystem, not a comprehensive conformance claim. Use exact Unicode scalar values, record language/runtime versions and distinguish byte decoding, normalized equality, layout rendering and application validation. A visual match does not guarantee a reversible text interchange.

const segmenter=new Intl.Segmenter('en',{granularity:'grapheme'});
function truncateGraphemes(text,max){
 if(!Number.isInteger(max)||max<0) throw new RangeError('max');
 const iterator=segmenter.segment(text)[Symbol.iterator]();
 let out='';
 for(let i=0;i<max;i++){const next=iterator.next();if(next.done) return out;out+=next.value.segment;}
 return out;
}
console.log(truncateGraphemes('e\u0301👩\u200D💻A',2)); // e + accent, joined emoji
// For UTF-8 byte budgets, measure TextEncoder bytes separately.

For reproducible defect reports, provide the original encoded data, the first observable transformation, the expected result and the tested API versions. The same logical text can produce different font glyphs, and an accidental scalar replacement can be invisible when those glyphs look alike.

Interoperability and evidence matrix

Each row covers one independent contract. Use code-point equality when assessing text integrity, supported locale algorithms for boundary or sorting work and assistive technology feedback for reading comprehension. Avoid substituting one test for another.

Verification notes and regression acceptance

1. UTF-16 slice. Use slice(0,n) as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: may split surrogate pairs. Record false positives and separate presentation differences from changed data.
2. Scalar iterator. Use for...of as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: won't split valid surrogate pairs, can split clusters. Record false positives and separate presentation differences from changed data.
3. Grapheme iterator. Use Intl.Segmenter as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: preferred default for visible character count. Record false positives and separate presentation differences from changed data.
4. UTF-8 byte budget. Use TextEncoder as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: must include ellipsis bytes. Record false positives and separate presentation differences from changed data.
5. CSS ellipsis. Use text-overflow as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: visual clipping does not change stored string. Record false positives and separate presentation differences from changed data.

Include canonical equivalence, supplementary emoji, combining sequences, directional controls and locale-sensitive cases where applicable. Keep all expected outputs versioned rather than changing them automatically during upgrades. For byte-to-text errors use explicit strict or replacement policies; for identity checks use the actual standard named by the product.

Handling platform differences

An updated Unicode data version can introduce new assignments and properties, while font and OS support may lag. Test the actual deployed environments and record differences as supported variation, expected tailoring or defects. Preserving original input is critical to distinguishing these outcomes.

Primary technical specifications and implementation references

The sources below provide authoritative normative algorithms and API documentation. Conformance to one specification does not imply that all aspects of encoding, rendering, identity, accessibility and security are correct without end-to-end tests.

• UAX #29 Unicode Text Segmentation — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• ECMA-402 Intl.Segmenter — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• WHATWG Encoding Standard — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• CSS Text Level 3 — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• Unicode Normalization UAX #15 — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.

Related published research

Continue within the Text APIs & Implementation Patterns virtual silo or read closely related published Unicode technical references below.

• javascript intl segmenter
• javascript strings code points
• text selection grapheme clusters
• string length bytes units codepoints graphemes

This research reference is designed to support practical diagnostics and verifiable implementation decisions rather than optimizing separate article keywords.