Unicode Sentence Boundary Rules: UAX #29, Abbreviations and Segmentation Limits

Sentence segmentation is a heuristic text-boundary task

Unicode Standard Annex #29 defines default sentence boundary rules to approximate where one sentence ends and the next begins. The algorithm uses classes assigned to code points plus limited context; it is not a semantic parser and cannot always resolve abbreviations, titles, quotations, decimal numbers or domain-specific punctuation. Human readers know that a period after “Dr.” may not end a sentence, but a general-purpose segmenter cannot infer every abbreviation from Unicode properties alone.

A sentence-break implementation should be judged against its declared purpose: UI sentence navigation, reading assistance, search snippets or NLP training data. Each can require different tailoring. UAX #29 provides a standard baseline for interoperability, while higher-level language tools may add dictionaries, learned patterns and task-specific punctuation rules.

Sentence_Break values and data sources

The Unicode Character Database defines Sentence_Break values including ATerm (abbreviation terminator), STerm (sentence terminator), Close (closing punctuation), Sp (space), Sep (separator), CR, LF, Extend, Format, Upper, Lower, OLetter and Numeric. The rules combine these classes to find boundaries rather than treating every ASCII period as a hard split.

These classes are not identical to Unicode General_Category or an English punctuation list. A full implementation uses versioned SentenceBreakProperty.txt from the UCD and the precedence-ordered SB rules in UAX #29. The rules can skip certain format and extending characters for matching without deleting them from the resulting text.

Representative rules in the SB algorithm

These are explanatory summaries, not literal normative rule expressions. Implementers must consult the actual rules for the Unicode version they support. An abbreviation dictionary may override default behavior for particular languages or products, and such tailoring must be tested separately rather than mislabelled as Unicode conformance failure.

“Dr.”, “e.g.” and the ambiguity of periods

The string Dr. Smith arrived. contains a period that often should not end a sentence. The string It ended. Smith left. contains a period that usually should. Both involve uppercase letters following a terminator, so case and punctuation alone cannot resolve every case. UAX #29 includes context heuristics, but domain-specific abbreviations and personal names require additional knowledge.

Repeated periods and ellipses present further ambiguity, particularly in chat messages and source-code documentation. A punctuation-only splitter such as text.split(/[.!?]/) loses the punctuation itself, fragments decimal values, mishandles quoted text and fails to account for the user's language. Instead, preserve original segments and their offsets so the text can be reconstructed exactly.

A decimal point is not necessarily an ending

Sentence boundary rules explicitly account for a period adjacent to numeric characters in appropriate contexts. A number like 3.14 is not reasonably two sentences because a dot occurs between its digits. However punctuation inside version strings, URL paths or legal citations may also need application-specific handling. A generic Unicode sentence segmenter is not a lexer for every structured data format.

A newline can be a formatting boundary without constituting a grammatical sentence ending. Conversely, a sentence can end at the end of a buffer without an explicit terminator. Define whether the product needs linguistic sentences, display lines or paragraphs before selecting a segmentation algorithm.

Intl.Segmenter sentence mode

The JavaScript internationalization API supplies Intl.Segmenter with granularity: "sentence". The returned segments include trailing punctuation and often spaces according to the implementation's boundaries; each entry has a UTF-16 index into the source. A consumer should preserve the segments verbatim rather than stripping all trailing whitespace and then assuming the original text can be reconstructed.

const text = "Ready. Go now!";
const segmenter = new Intl.Segmenter("en",{granularity:"sentence"});
const segments = [...segmenter.segment(text)];
console.log(segments.map(x=>x.segment));
console.log(segments.map(x=>x.index));
console.log(segments.map(x=>x.segment).join("") === text); // true
// Check exact segmentation fixtures against your runtime and locale.

Locale-specific segmentation results can vary by runtime and ICU/CLDR version; the API does not expose which UAX #29 sentence-break data release is used. A test suite should record locale and platform, and should distinguish fixed version-matched UAX #29 conformance fixtures from user-experience expectations for conversational prose.

Terminator followed by closing punctuation or spaces

A sentence may finish inside a quotation: She said “Done.” Then left. A simple scanner that breaks immediately after the period may detach the closing quotation mark from the preceding sentence. UAX #29 accounts for classes such as Close and Sp when deciding a boundary after a terminator. The exact result can still depend on context and tailoring, so retain original punctuation and offsets.

Unicode quotation marks, apostrophes and bracket characters have different properties that an ASCII-only parser misses. A semantic NLP pipeline may additionally recognize direct speech, quoted abbreviations or dialogue structure. The Unicode default is an interoperable starting point, not a complete grammar model.

Language-specific sentence punctuation and scripts

Not every script uses a Western full stop, nor does every language distinguish sentences using the same capitalization behavior. Unicode property data includes many punctuation characters outside ASCII. Some writing systems lack case, so heuristics depending on Upper or Lower cannot have the same effect across all scripts.

Test multilingual strings containing Devanagari danda, CJK punctuation, Arabic script and emoji. Neither the presence of a space nor the shape of a glyph defines a universal sentence boundary. Tailoring should target actual user language, and text should be preserved rather than rewritten to fit an English punctuation convention.

Whitespace and regex splits are not UAX #29

Python's ordinary str.split() separates whitespace-delimited fields. Its standard re library can match sentence-ending punctuation, but this is not a UAX #29 sentence boundary implementation. When strict conformance is required, use ICU or another Unicode sentence-break implementation and test its versioned behavior. A regular-expression heuristic remains useful for a controlled domain only when its limitations are explicit.

text = "Dr. Smith arrived. Version 3.14 shipped."
print(text.split("."))
# Fragments "Dr" and numeric "3"/"14" are not useful sentences.
# Use a versioned UAX #29/ICU segmenter for production
# default sentence boundaries; add domain-specific tailoring separately.

Incremental boundary detection and normalization

Chunks from a file or network can end just after a period while the next chunk begins with characters needed to decide whether to break. A streaming implementation must defer certain decisions until it has sufficient following context. It cannot unconditionally emit a new sentence when a period appears at the end of the current chunk.

Sentence boundary indices refer to a particular original string representation. Normalizing text, converting newline conventions or removing combining marks may change the offsets. Preserve byte and scalar integrity separately from the segmentation result; if content changes, rerun or incrementally update boundaries rather than using stale indices.

Sentence boundary fixtures for a real product

The official SentenceBreakTest.txt suite is the foundation for checking UAX #29 conformance. Real-world quality evaluation should additionally compare output against curated, language-specific samples and domain-aware abbreviation conventions.

Standards and implementation references

1. UAX #29 — Unicode Text Segmentation
2. UAX #44 — Sentence_Break property
3. UCD — SentenceBreakTest.txt
4. MDN — Intl.Segmenter
5. ICU — Sentence Boundary Analysis

Independent Unicode interoperability research. Segmentation is a default text-boundary algorithm, not a complete linguistic parser.