Unicode Collation Algorithm: Weights, Tailoring and Stable Sort Keys

Collation is language-aware ordering, not code-point arithmetic

Sorting Unicode text by the numeric value of its code points gives a deterministic binary order, but not the order that people expect from a dictionary, contact list, library catalog or locale-specific index. The Unicode Collation Algorithm (UCA), specified by Unicode Technical Standard #10, defines a systematic way to compare strings using collation elements and weighted sort keys. The Default Unicode Collation Element Table (DUCET) supplies a default mapping; language-specific tailoring can refine that mapping.

For example, a naive bytewise comparison can place Z before a because their UTF-8 bytes differ in ASCII order. A user-facing collator may consider case a secondary distinction and place “a” before “Z.” Neither answer is universally correct: binary identifiers and natural-language indexes have different comparison contracts.

Primary, secondary, tertiary and quaternary distinctions

The UCA constructs a sequence of collation elements whose weights can be compared at successive levels. Primary weights generally distinguish base letters; secondary weights usually capture accent distinctions; tertiary weights often cover case and variant distinctions. Some configurations use a quaternary level, for example to resolve differences involving variable punctuation. An optional identical tie-breaker may differentiate strings that collate the same at earlier levels.

These are common roles, not guarantees for every writing system or tailoring. Language-specific contractions and scripts can change the weights and relationships. A collator set to primary strength may consider accent or case variants equal for sorting, while a higher strength may distinguish them. “Compare equal” under one strength is not the same statement as “the strings contain identical Unicode scalar values.”

The DUCET baseline and CLDR root are not every locale

UCA defines a default collation with DUCET, but actual application libraries commonly use CLDR-based collation data and tailorings. Alphabetic order differs across locales: Swedish traditionally places å, ä and ö after z, whereas other locales integrate accented forms near their base letters. A country code, language tag and optional collation keyword may affect results. A default collator's behavior should not be described as “Unicode's universal alphabetical order.”

DUCET and tailored collations must be versioned. The collation weights of strings and the binary form of sort keys can change between Unicode or CLDR releases. Persisting opaque sort keys in a database without recording the generating library and version risks inconsistent ordering after an upgrade; rebuild indexes as required by the storage system.

From input scalars to ordered collation elements

Conceptually, UTS #10 normalizes the input (or applies an equivalent canonical-equivalence-preserving optimization), maps text into collation elements, constructs level-weight sort keys and compares those keys. Some collation elements represent multiple input characters (contractions); a single input character may expand into multiple collation elements. Looking up an independent weight for each code point and concatenating the results is insufficient for all locales.

Canonical equivalence matters. U+00E9 and U+0065 U+0301 should compare equivalently under a conforming, appropriately configured UCA implementation even though they occupy different UTF-8 bytes. Libraries may avoid fully materializing an NFD copy when they can achieve an equivalent result through a fast canonical-decomposition path. The observable collation behavior is the standard's concern, not the internal allocation strategy.

Intl.Collator for browser sorting and searching

JavaScript provides the Intl.Collator API, whose results are driven by the runtime's internationalization data. It offers usage: "sort" or "search", sensitivity, numeric and ignorePunctuation options. This is an application API informed by CLDR/ICU rather than a promise that the exact UCA internal sort keys are exposed.

const names = ["z", "ä", "å", "ö", "a"];
const sv = new Intl.Collator("sv", {usage:"sort"});
const en = new Intl.Collator("en", {usage:"sort"});
console.log([...names].sort(sv.compare));
console.log([...names].sort(en.compare));
// Locale-sensitive results; assert them against the chosen runtime data.
const num = new Intl.Collator("en", {numeric:true});
console.log(["item10","item2"].sort(num.compare));
// ["item2", "item10"]

Do not check whether compare(a,b) returns precisely -1, 0 or 1: the ECMAScript contract requires the sign of a negative, zero or positive result, not a particular nonzero magnitude. Store a created collator and reuse it when sorting many strings rather than creating a new one for every comparison.

Search versus sort, and primary-equivalent keys

For browser search using Intl.Collator with sensitivity: "base", an accented spelling can compare equal to its unaccented counterpart in some locales. Such behavior is not the same as removing diacritic code points from the stored name. In German a search collator may treat certain variants differently from a Swedish collator. CLDR locale tailoring can also affect contractions or letter identities; no single static “accent-insensitive” transform is sufficient for every language.

usage: "search" optimizes comparison equivalence for finding matches; it does not deliver substring search or a full text index. Unicode word segmentation, phrase matching and grapheme-safe highlighting remain additional responsibilities.

Python locale sorting versus a real UCA implementation

Python's basic string comparison uses Unicode code point order; sorted() without a key does not perform UCA collation. locale.strxfrm() and locale.strcoll() rely on the process's C locale environment, which is global and can differ between machines and operating systems. They should not be described as a portable UCA implementation. ICU (including PyICU bindings where available) offers a more explicit, versioned locale and collation interface.

items = ["a", "Z", "ä"]
print(sorted(items))  # Unicode code-point order, not locale collation
# With an ICU binding, choose a locale explicitly and retain
# the ICU/CLDR version used to produce persistent sort keys.
# Avoid manipulating process-global locale inside concurrent services.

Do not use collator equality for authentication

A collation may treat two distinct strings as equivalent at a chosen strength. That makes it useful for user-visible sorting but potentially dangerous if applied inadvertently to passwords, opaque tokens, file-system paths or cryptographic inputs. Define identity normalization and casing independently from presentation sorting; select database collations and unique constraints with awareness of the resulting equivalence class.

Case folding, confusables detection and accent-sensitive lexical matching each address different problems. A primary-strength collator does not demonstrate that two strings are bytewise identical, canonically equivalent, visually confusable or safe as identifiers.

Collation acceptance tests and upgrades

A good regression suite records the Unicode/CLDR/ICU version, requested locale and options, comparison sign or equivalence, and data fixtures. If a server and browser disagree, inspect their collation profiles before concluding the underlying Unicode strings changed.

Standards and implementation references

1. UTS #10 — Unicode Collation Algorithm
2. UTS #35 — Locale Data Markup Language, Collation
3. CLDR — Locale data and tailoring
4. MDN — Intl.Collator
5. UAX #15 — Unicode normalization
6. ICU User Guide — Collation

Independent technical reference; exact comparison behavior depends on the applicable algorithm, data version, locale and application policy.