Unicode Word Boundary Rules: UAX #29, Scripts, Apostrophes and Segmenter APIs

A Unicode word boundary is not the same as a dictionary word

Unicode Standard Annex #29 specifies default word boundary rules that divide a string into segments usable for text navigation, selection, search and analysis. These are algorithmic boundaries over Unicode properties; a natural-language word requires additional morphology, dictionaries and context in many writing systems. A program that splits on ASCII spaces or uses an ASCII-only regular-expression word class does not implement the Unicode default segmentation algorithm.

Word boundaries are different from extended grapheme boundaries. Grapheme boundaries help avoid splitting combining marks or emoji clusters during editing; word boundaries group runs of letters, numbers and certain punctuation for navigation. A correct word-navigation implementation often needs both: word stepping should never leave the caret inside a grapheme cluster even if a lower-level algorithm counts UTF-16 code units.

The Word_Break property is separate from General_Category

The UCD provides Word_Break values such as ALetter, Hebrew_Letter, Numeric, Katakana, ExtendNumLet, MidLetter, MidNum, MidNumLet, Single_Quote, Extend, Format, ZWJ, Regional_Indicator and WSegSpace. The segmentation rules examine property sequences, not just a character's visual type or UCD General_Category.

For example a combining acute accent may be classified for word breaking so that it does not create a boundary within the letter it modifies; a single quote can behave differently depending on adjacent letters and scripts. The same punctuation appearing in isolation is not automatically part of a word. Conformance requires the property data and rule priorities of the chosen Unicode version.

Selected default WB rule families

The table summarizes families instead of reproducing exact normative regular expressions. WB-number assignments and property definitions should always be checked against the actual versioned UAX #29 text and WordBreakProperty.txt. A simplified “letters plus underscores are words” pattern cannot capture these cases.

Apostrophes, periods and numeric separators

Words such as can't illustrate why a word boundary engine needs context. Under the default rules the apostrophe can remain internal when flanked by appropriate letters; it should not be treated as a universally separating delimiter. Numeric punctuation similarly participates conditionally in digit sequences: a decimal-like 3.14 differs from a stray period before another word. Ellipses, quotes and punctuation after a word require their own boundaries.

A URL, identifier or filename is not guaranteed to be segmented according to an application's desired tokenization. For example, a hyphenated identifier or snake_case variable may use domain-specific navigation and code-editor behavior. Use a tailored tokenizer or identifier grammar on top of Unicode segmentation where the product requires syntax-aware selection.

Thai, Lao, Chinese and Japanese need more than generic WB rules

Many scripts do not separate every lexical word with spaces. The UAX #29 default rule set supplies boundaries in these strings, but dictionary-based or statistical segmentation may be necessary for useful Thai, Lao, Chinese or Japanese word selection. A user-facing editor and a search index may implement different tailorings, and neither should be represented as an error in Unicode encoding.

Internationalization libraries may incorporate locale-specific knowledge. Always distinguish the default word boundary test data from a locale-tailored Intl.Segmenter result; the API is permitted to tailor word segmentation according to the supplied locale and runtime data. Word segmentation quality should be evaluated with language speakers and representative corpora, not a handful of English contractions.

Intl.Segmenter word mode and isWordLike

JavaScript offers Intl.Segmenter with granularity: "word". Each segment record includes the original UTF-16 starting index, its segment substring, and an isWordLike boolean identifying whether that segment is considered word-like by the implementation. Spaces and punctuation can be returned as segments as well; filtering blindly by truthiness of segment does not distinguish words from separators.

const seg = new Intl.Segmenter("en", { granularity: "word" });
const text = "Hello, café 3.14!";
for (const part of seg.segment(text)) {
  console.log(part.segment, part.index, part.isWordLike);
}
const words = [...seg.segment("Hello world")]
  .filter(part => part.isWordLike)
  .map(part => part.segment);
console.log(words); // ["Hello", "world"]

Do not assume the index offsets represent scalar counts or grapheme-cluster counts: they are offsets into a JavaScript UTF-16 string. When displaying a highlight range, the corresponding original substring is valid for that same string, but offsets cannot be transferred unchanged to an NFC-normalized copy or another encoding representation.

Graphemes and word segments answer separate UI questions

A word boundary is a navigation step; an extended grapheme cluster is closer to a visible editing unit. A word could consist of many grapheme clusters. An emoji ZWJ sequence can be one grapheme but might not be classified as an ordinary lexical word. A cursor moving one word at a time should still be snapped to appropriate grapheme boundaries to avoid splitting an emoji sequence or a combining mark.

The distinction matters for copyable decorated Unicode strings as well: the US Fancy Text tool can generate mathematical letters and combining sequences whose code-point identities differ from plain text. Word selection in a destination editor can vary without implying that the copied bytes or scalars were corrupted.

Python requires an appropriate segmentation library

Python's standard string split() without an argument splits on runs of whitespace and does not implement UAX #29 word segmentation. The built-in re module's \b and \w describe regular-expression word boundaries and character categories, not the full UAX #29 WB algorithm. For conformity across scripts and punctuation, use a maintained Unicode segmentation implementation or ICU wrapper with an explicit runtime and data version.

sample = "can't read café 3.14"
print(sample.split())
# Whitespace tokens are not the UAX #29 word boundary result.
import re
print(re.findall(r"\b\w+\b", sample))
# A regex engine's word-class boundary is not a UAX #29 assertion.

Streaming, edits and incremental boundary calculation

A WB decision can depend on characters on either side of a potential boundary, and sequences of ignored marks may intervene. An incremental tokenizer receiving arbitrary network chunks cannot finalize every word boundary at the end of the current chunk. For correctness, retain enough state, use an implementation with streaming support or reprocess a contextual window when new text arrives.

Editing a string also invalidates adjacent cached boundaries: inserting an apostrophe or a number separator can join or split previously recognized word-like segments. Word-boundary caches should be associated with the original scalar text and segmentation version, not reused across normalization or input edits without verification.

Useful word boundary regression cases

For rigorous conformance, use Unicode's version-matched WordBreakTest.txt test corpus and corresponding property data. Record language/locale and runtime versions when comparing browser segmentation with ICU or backend results.

Standards and implementation references

1. UAX #29 — Unicode Text Segmentation
2. UAX #44 — Unicode Character Database
3. UCD — WordBreakTest.txt
4. MDN — Intl.Segmenter
5. ICU — Boundary Analysis

Independent Unicode interoperability research. Segmentation is a default text-boundary algorithm, not a complete linguistic parser.