Canonical-Equivalent Unicode Text Matching: NFC, NFD and Search Semantics

Different scalar streams can be canonically equivalent

Unicode canonical equivalence means that two sequences are considered equivalent according to the Unicode character model even if their code points and UTF encoding bytes differ. A textbook example is U+00E9 LATIN SMALL LETTER E WITH ACUTE and U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT. JavaScript's strict equality says these two sequences differ, because it compares code units, but normalizing each to NFC or NFD produces matching scalar sequences.

Canonical equivalence is a carefully defined relation, not a test of whether two strings look alike in one font. For example, Greek capital Alpha and Latin capital A can appear visually similar but are not canonically equivalent. Mathematical bold A and ordinary A are compatibility-related characters, not canonically equivalent. Identical screenshots, the same grapheme-cluster count or a base-strength collation match do not establish canonical equivalence.

Normalization-based equivalence checks

Unicode Standard Annex #15 defines NFC and NFD. NFC canonically decomposes, orders marks and recomposes eligible pairs; NFD canonically decomposes and orders without recomposing. For valid Unicode scalar sequences, the canonical-equivalence test NFC(a) === NFC(b) (or the equivalent with NFD) is appropriate when the strings are compared as exact sequences of code points after normalization. It is important that both operands use the same normalization form and that the normalization implementation has a documented UCD version.

const composed = "\u00E9";
const decomposed = "e\u0301";
console.log(composed === decomposed);                    // false
console.log(composed.normalize("NFC") ===
            decomposed.normalize("NFC"));               // true
console.log(composed.normalize("NFD") ===
            decomposed.normalize("NFD"));               // true
console.log("𝐀".normalize("NFC") === "A".normalize("NFC")); // false

Normalization is not reversible with respect to original scalar sequences: after both strings become U+00E9, the normalized copy does not reveal which spelling the author originally entered. Preserve the original in systems where forensic history, editing round trips or cryptographic signatures require the original byte representation.

Canonical combining-class reordering

Some equivalent sequences differ not in precomposed letters but in the order of combining marks. Canonical reordering uses Canonical_Combining_Class (ccc) from the Unicode Character Database and stably reorders eligible non-starters. U+0307 COMBINING DOT ABOVE has ccc 230; U+0323 COMBINING DOT BELOW has ccc 220. Thus q + dot-above + dot-below and q + dot-below + dot-above normalize to the same NFD sequence.

const before = "q\u0307\u0323";
const after = "q\u0323\u0307";
console.log(before === after); // false
console.log(before.normalize("NFD") === after.normalize("NFD")); // true

Do not sort marks by numeric code point: marks with equal nonzero ccc retain their relative order. And do not strip all combining marks and call the result “canonical normalization”; removing accents is a different, potentially lossy comparison policy.

Algorithmic Hangul equivalence

Canonical equivalence includes algorithmic decomposition and recomposition of modern Hangul syllables. U+AC00 HANGUL SYLLABLE GA decomposes to U+1100 HANGUL CHOSEONG KIYEOK and U+1161 HANGUL JUNGSEONG A. Their NFC values match, but their raw UTF-8 encodings differ. A Latin-only composition lookup cannot perform correct Unicode-wide canonical matching.

const precomposed = "\uAC00";
const conjoiningJamo = "\u1100\u1161";
console.log(precomposed === conjoiningJamo); // false
console.log(precomposed.normalize("NFD") ===
            conjoiningJamo.normalize("NFD")); // true

Why equal-looking clusters can still differ

Extended grapheme clusters are designed to approximate editing units perceived by people, not to define canonical equivalence. An emoji formed from a ZWJ sequence may render as one glyph even though it contains multiple scalars. Some emoji use variation selectors that affect presentation. UAX #29 grapheme segmentation, font shaping and UAX #15 normalization are distinct. A grapheme-cluster match cannot replace normalization-based scalar comparison.

Likewise, compatibility symbols such as circled digits and fullwidth Latin letters are not necessarily canonically equivalent to ordinary characters. A search index using NFKC may deliberately collapse such distinctions, but this must be documented as compatibility-insensitive matching—not mistakenly described as canonical equivalence.

Collation equality and canonical equality have different widths

UTS #10 requires canonical-equivalent sequences to collate equivalently when applying Unicode collation. However, a locale-aware collator at a limited strength may also equate additional strings that are not canonically equivalent. For example, a base-sensitive search collator may treat café and cafe as equal in English even though the accent is meaningful and their NFC strings remain distinct.

const baseSearch = new Intl.Collator("en", {
  usage: "search", sensitivity: "base"
});
console.log(baseSearch.compare("café","cafe") === 0); // true
console.log("café".normalize("NFC") ===
            "cafe".normalize("NFC")); // false

Choose the desired relation explicitly: canonical-equivalent matching for normalization consistency, case folding for caseless matching, collation for locale-sensitive search and ordering, or compatibility normalization when an identifier policy requires it. One test cannot safely substitute for the others.

Substring search and mapping back to original text

Normalizing a string can change its length in UTF-16 code units and in scalars. Finding a match in an NFC or NFD copy and using its offset to slice an unnormalized source can select the wrong range when earlier text contains decomposed accents or supplementary code points. A correct highlighting system needs a mapping between normalized match positions and original text ranges or a search engine that already tracks source spans.

Offsets should also be mapped to grapheme-cluster boundaries for UI selection. If a match occurs inside a combining sequence, slicing only the mark can leave visually broken output or an orphaned diacritic. Normalization identifies semantic equality, not caret movement or display selection semantics.

Python implementation and Unicode data versions

import unicodedata as ud
def canonically_equal(a, b):
    return ud.normalize("NFC",a) == ud.normalize("NFC",b)
assert canonically_equal("\u00E9","e\u0301")
assert canonically_equal("q\u0307\u0323","q\u0323\u0307")
assert canonically_equal("\uAC00","\u1100\u1161")
assert not canonically_equal("𝐀","A")
assert not canonically_equal("café","cafe")
print("UCD:", ud.unidata_version)

Pin or record the UCD version for reproducibility across the browser, Python, ICU and database layers. Unicode normalization stability protects established assignments, but newly assigned code points and runtime version differences still warrant testing. For full conformance, use the published version-matched NormalizationTest.txt dataset rather than a sample of Latin vowels.

Equality, database constraints and signatures

A database's bytewise unique constraint can allow two canonically equivalent names if their bytes differ. Conversely, a collation-based unique constraint may collapse more variants than NFC would. Define a normalized lookup column or consistent comparison policy, retain the source string when needed, and avoid mixing bytewise and locale-aware equality rules in different application paths.

Digitally signed text must be signed in the exact byte representation required by its protocol. Two canonically equivalent strings normally have different raw UTF-8 digests unless an explicit normalization rule is applied before both signing and verification. Normalization at an undocumented layer is a source of signature mismatch, not a universal security fix.

Regression vectors for canonical matching

Also test malformed UTF-8 and unpaired UTF-16 surrogate boundaries separately: decoding failure is not a canonical-equivalence problem. All comparison fixtures should be checked on the exact decoded scalar streams, not merely how the browser displays them.

Standards and implementation references

1. UAX #15 — Unicode Normalization Forms
2. UAX #44 — Unicode Character Database
3. UTS #10 — Unicode Collation Algorithm
4. UAX #29 — Unicode Text Segmentation
5. MDN — String.normalize()
6. Python — unicodedata

Independent technical reference. Preservation, comparison, segmentation and presentation use different Unicode rules.