ICU Unicode Text APIs: BreakIterator, Normalizer2, Collator and Locale Data

International Components for Unicode (ICU) brings together implementations of Unicode algorithms and locale-sensitive services. Its APIs serve different purposes: Normalizer2 handles normalization, BreakIterator handles text boundaries and Collator orders strings according to collation data. Applications should not treat these components as interchangeable 'Unicode cleaners'.

Separate normalization, segmentation and collation

Normalization determines canonical or compatibility representations of text. Segmentation identifies character, word, sentence and line boundaries. Collation provides culturally and contextually appropriate sort orders and comparison keys. Each has a different contract.

A valid NFC conversion does not tell an editor where to place the cursor; a grapheme iterator does not produce a locale-aware alphabetical sort order; and a collation key is generally not suitable as a display string. Choose one API per problem.

Normalizer2 provides multiple normalization forms

ICU Normalizer2 supports normalization schemes including NFC/NFD and NFKC/NFKD; some implementations provide specialized comparison-oriented variants such as NFKC_Casefold. Operations can normalize strings and check if an input is normalized.

Use form-specific code and version-aware tests rather than applying an arbitrary normalization everywhere. Compatibility folding can change styled mathematical characters, which may be inappropriate for a text generator intended to preserve exact Unicode output.

BreakIterator implements boundary analysis

ICU's BreakIterator can segment grapheme-like character clusters, words, sentences and line break opportunities, with locale-sensitive tailoring. It does not generally return one item per Unicode scalar when in character mode.

Word and line break results depend on locale and dictionary availability for scripts without explicit spaces, including Thai. A browser's Intl.Segmenter may use ICU internally but can expose a different abstraction and different ICU data version.

Collator applies locale-aware comparison rules

Collator compares and sorts strings using ICU/Unicode collation data, with configurable strength and other attributes. This enables language-sensitive ordering that differs from byte order or JavaScript ordinal comparison.

Do not assume one sort key is stable across all Unicode or CLDR updates. Store original strings and reindex derived collation keys after changing locale data or comparator settings; otherwise uniqueness and pagination may change unexpectedly.

UnicodeSet and property access support inspection

ICU includes APIs for Unicode character properties and sets. These can classify Script, Script_Extensions, general category and many derived properties using the bundled UCD version.

A Script property identifies a writing system, not a user's language or trustworthiness. A security profile requires additional standards such as UTS #39, field-specific restrictions and collision policies.

Encoding conversion belongs at a different layer

ICU provides conversion facilities for character encodings. Conversion maps bytes into Unicode text or between encodings under declared error policies; it is not transliteration of language between scripts.

Avoid treating a conversion failure as a font fallback defect. With strict error handling, invalid byte sequences should be reported; replacement mode may insert U+FFFD. Record original bytes before attempting another conversion.

Locale identifiers and BCP 47 require careful mapping

ICU locale IDs can express language, script, region and keywords; modern APIs support relationships with BCP 47 language tags. Locale services use CLDR datasets and may provide fallback when a particular resource is unavailable.

A successful locale constructor does not mean every option is implemented exactly. Query resolved locale or API attributes and use appropriate language/script metadata; the locale does not alter the Unicode identity of stored strings.

C and C++ boundary offsets need explicit units

ICU's UTF-16-oriented APIs may expose offsets in 16-bit code units, while UTF-8-oriented APIs measure bytes. The C/C++ layer distinguishes UChar, UChar32 and encoded data buffers.

Never mix an offset from a UTF-16 BreakIterator with a UTF-8 byte-buffer index. Document conversions carefully, especially around supplementary emoji or decomposed combining marks; wrong offset units cause corruption even if segmentation itself is correct.

ICU/CLDR upgrades require regression plans

ICU bundles Unicode/CLDR data and library implementation versions; changes can alter word boundaries, case mappings, collation order and supported emoji properties. Cross-runtime tests should record the ICU and Unicode/CLDR versions.

Before upgrade, run a corpus of accented strings, mixed scripts, numeric collation, emoji ZWJ, sentence punctuation, Thai words and malformed input. Classify changes as normative data updates, tailoring changes or unexpected regressions.

Reproducible technical example

This controlled specimen demonstrates a single subsystem, not a comprehensive conformance claim. Use exact Unicode scalar values, record language/runtime versions and distinguish byte decoding, normalized equality, layout rendering and application validation. A visual match does not guarantee a reversible text interchange.

// Illustrative ICU4C API relationships (not a full buildable program):
// Normalizer2::getNFCInstance(status)       -> canonical normalization
// BreakIterator::createCharacterInstance(locale,status) -> grapheme boundaries
// BreakIterator::createWordInstance(locale,status)      -> word boundaries
// Collator::createInstance(locale,status)  -> locale-aware comparison
// Check if returned offsets are UTF-16 code units or UTF-8 bytes
// in the specific ICU API overload before slicing a buffer.

For reproducible defect reports, provide the original encoded data, the first observable transformation, the expected result and the tested API versions. The same logical text can produce different font glyphs, and an accidental scalar replacement can be invisible when those glyphs look alike.

Interoperability and evidence matrix

Each row covers one independent contract. Use code-point equality when assessing text integrity, supported locale algorithms for boundary or sorting work and assistive technology feedback for reading comprehension. Avoid substituting one test for another.

Verification notes and regression acceptance

1. Normalization. Use Normalizer2 as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: NFC/NFD/NFKC/NFKD. Record false positives and separate presentation differences from changed data.
2. Graphemes. Use BreakIterator character as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: text boundary rather than scalar length. Record false positives and separate presentation differences from changed data.
3. Words. Use BreakIterator word as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: locale/dictionary-dependent. Record false positives and separate presentation differences from changed data.
4. Collation. Use Collator as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: locale-aware sort and compare. Record false positives and separate presentation differences from changed data.
5. Unicode properties. Use UnicodeSet / UCharacter as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: character database classification. Record false positives and separate presentation differences from changed data.
6. Encoding. Use Converter APIs as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: bytes versus Unicode text. Record false positives and separate presentation differences from changed data.

Include canonical equivalence, supplementary emoji, combining sequences, directional controls and locale-sensitive cases where applicable. Keep all expected outputs versioned rather than changing them automatically during upgrades. For byte-to-text errors use explicit strict or replacement policies; for identity checks use the actual standard named by the product.

Handling platform differences

An updated Unicode data version can introduce new assignments and properties, while font and OS support may lag. Test the actual deployed environments and record differences as supported variation, expected tailoring or defects. Preserving original input is critical to distinguishing these outcomes.

Primary technical specifications and implementation references

The sources below provide authoritative normative algorithms and API documentation. Conformance to one specification does not imply that all aspects of encoding, rendering, identity, accessibility and security are correct without end-to-end tests.

• ICU User Guide — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• ICU Normalization — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• ICU Boundary Analysis — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• ICU Collation — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UTS #10 Unicode Collation Algorithm — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UAX #29 Unicode Text Segmentation — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.

Related published research

Continue within the Text APIs & Implementation Patterns virtual silo or read closely related published Unicode technical references below.

• javascript intl segmenter
• python unicodedata
• unicode collation algorithm
• unicode line breaking scripts

This research reference is designed to support practical diagnostics and verifiable implementation decisions rather than optimizing separate article keywords.