Unicode Normalization: NFC, NFD and Canonical Equivalence

Canonical equivalence is not byte identity

Unicode permits certain abstract text to be encoded by more than one sequence of scalar values. For example, é can be U+00E9 LATIN SMALL LETTER E WITH ACUTE or U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT. These sequences differ as strings and in UTF-8 byte representation, yet Unicode defines them as canonically equivalent. Unicode Normalization Forms provide a deterministic representation for comparing or storing canonically equivalent strings when an application elects to normalize them.

NFD stands for Normalization Form D: canonical decomposition followed by canonical ordering. NFC stands for Normalization Form C: canonical decomposition, ordering and canonical recomposition. A normalized representation is not a claim that the source had the same code points as the result. The choice of form is a data-contract decision and not an operation mandated for every system.

NFC and NFD in the four-form landscape

A common mistake is to use NFKC when NFC was intended. Mathematical bold A (U+1D400) and ordinary A (U+0041) are not canonically equivalent, so NFC and NFD preserve the mathematical style distinction. NFKC can fold the styled character to A because compatibility decomposition is a different, deliberately broader equivalence relation. The US Fancy Text tool gives a practical setting to inspect how an encoded styled letter survives NFC but may lose its distinctive code point after compatibility normalization.

Recursive decomposition and UCD mappings

Canonical decomposition follows mappings recorded in the Unicode Character Database (UnicodeData.txt), applying them recursively until there are no further canonical mappings. Hangul syllables are handled algorithmically instead of by enumerating every syllable in decomposition data. NFD is the result of complete canonical decomposition plus stable canonical ordering of combining marks; it is not simply a one-pass mapping from precomposed letters to one base and one accent.

For U+00E9, NFD yields U+0065 U+0301. For U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE, NFD yields U+0041 U+030A. U+212B ANGSTROM SIGN also has a canonical mapping ultimately yielding A plus the combining ring. These mappings demonstrate that visual similarity by itself is not the rule: canonical equivalence is specified by the UCD and its stability policies.

const hex = s => [...s].map(x =>
  "U+" + x.codePointAt(0).toString(16).toUpperCase()).join(" ");
console.log(hex("\u00E9".normalize("NFD"))); // U+65 U+301
console.log(hex("\u00C5".normalize("NFD"))); // U+41 U+30A
console.log(hex("\u212B".normalize("NFD"))); // U+41 U+30A
console.log(hex("𝐀".normalize("NFD")));     // U+1D400

Canonical Combining Class and deterministic ordering

Not every sequence of marks is already in canonical order. Unicode assigns a Canonical_Combining_Class (ccc) property to character values. Most scalars have ccc 0 (starters); many combining marks have nonzero values. The canonical ordering algorithm reorders non-starters into nondecreasing combining-class order where applicable, while retaining relative order for equal class values. It does not indiscriminately sort the entire Unicode string by code point.

For a concrete sample, U+0307 COMBINING DOT ABOVE has ccc 230 and U+0323 COMBINING DOT BELOW has ccc 220. When the marks follow q in that order, NFD reorders the dot-below mark ahead of the dot-above mark. Using a base with no relevant precomposed canonical composition makes the reordering visible in scalar values without implying a glyph-perfect appearance.

const input = "q\u0307\u0323";
const output = input.normalize("NFD");
console.log([...input].map(x => x.codePointAt(0).toString(16)));
// ["71","307","323"]
console.log([...output].map(x => x.codePointAt(0).toString(16)));
// ["71","323","307"]

Marks with ccc 0 can interrupt a combining sequence. Equal combining classes preserve relative order and can affect whether later composition is allowed. A reliable implementation follows the normative canonical ordering algorithm, not ad-hoc mark sorting by glyph position or numerical code point value.

How NFC recomposition works

NFC first creates the canonical decomposed, ordered sequence, then composes eligible starter and non-starter pairs. Composition is constrained by canonical composition exclusions and blocking rules. In particular, not every apparently related base-plus-mark sequence becomes one precomposed scalar. Unicode's algorithm relies on normative composition data and certain special rules, not on the existence of a font glyph.

The familiar e plus U+0301 composes to U+00E9. But q plus combining dot below and combining dot above need not compose into a single precomposed Latin letter. NFC can therefore produce multiple scalars even when the entire sequence forms one user-perceived grapheme cluster.

const one = "\u00E9";
const two = "e\u0301";
console.log(one === two);                      // false
console.log(two.normalize("NFC") === one);      // true
console.log(one.normalize("NFD") === two);      // true
console.log("q\u0323\u0307".normalize("NFC").length); // 3

Normalization is idempotent: applying NFC to an already NFC-normalized string does not change it, and likewise for NFD. If an application accepts both forms, comparing the normalized outputs provides an equality check for canonical equivalence. This check is not a substitute for locale-sensitive search, case folding, confusable detection, or mathematical notation semantics.

Hangul syllables are handled algorithmically

Modern Hangul syllables in U+AC00..U+D7A3 are decomposed into conjoining Jamo according to rules in the Unicode Standard. U+AC00 HANGUL SYLLABLE GA has canonical decomposition U+1100 HANGUL CHOSEONG KIYEOK and U+1161 HANGUL JUNGSEONG A. NFC recombines that valid L+V pair into U+AC00. Syllables with trailing consonants add a third Jamo component; the arithmetic is specified and should not be guessed for unrelated Hangul sequences.

const composed = "\uAC00";     // 가
const parts = "\u1100\u1161";
console.log([...composed.normalize("NFD")].map(x =>
  x.codePointAt(0).toString(16))); // ["1100","1161"]
console.log(parts.normalize("NFC") === composed); // true

Hangul shows why the normalization engine must understand Unicode algorithms as well as stored decomposition records. A replacement table that only covers Latin accented letters cannot claim NFC/NFD conformance.

Concatenating normalized chunks can break normalization

Normalizing each piece of a string separately does not guarantee that concatenation is still normalized. For instance, NFC(e) is e and NFC(U+0301) is the combining mark, but their concatenation e plus U+0301 is not NFC because those scalars would compose into U+00E9. Incremental streaming normalizers must retain enough context across chunk boundaries to handle both canonical ordering and recomposition.

const part1 = "e".normalize("NFC");
const part2 = "\u0301".normalize("NFC");
const joined = part1 + part2;
console.log(joined === joined.normalize("NFC")); // false
console.log(joined.normalize("NFC") === "\u00E9"); // true

Similarly, a new combining mark can require reordering with marks at the end of an earlier chunk. UAX #15 specifies normalization-safe processing methods and stream-safe text considerations. Do not assume a generic fixed byte chunk boundary can be used as a normalization boundary.

Search, databases and canonical comparison

Database equality can depend on collation and normalization assumptions. Two canonically equivalent strings may compare unequal under binary collation. Alternatively, accent-insensitive collation may equate far more strings than NFC equality would. A useful data model keeps the user's submitted form for faithful display and a separately versioned normalized key where the business rules require it.

Normalize consistently at system boundaries if canonical equality is intended. Avoid a state where a frontend uses NFC while a backend compares unnormalized UTF-8 bytes and a downstream index uses compatibility folding. A cryptographic signature or bytewise checksum changes when normalization changes scalar values, even if rendered text appears unchanged. Sign the exact byte format specified by the protocol.

Python unicodedata and version sensitivity

Python's unicodedata module exposes normalization, character names and combining-class values from the Unicode Character Database version bundled with that Python runtime. ICU-based services, browsers and Python installations can bundle different UCD releases, so record the library version when diagnosing edge cases. Normalization stability policies constrain changes, but versioned conformance tests remain advisable for new data.

import unicodedata as ud
assert ud.normalize("NFD", "\u00E9") == "e\u0301"
assert ud.normalize("NFC", "e\u0301") == "\u00E9"
assert ud.combining("\u0307") == 230
assert ud.combining("\u0323") == 220
assert ud.normalize("NFD", "q\u0307\u0323") == "q\u0323\u0307"
assert ud.normalize("NFC", "\u1100\u1161") == "\uAC00"
print(ud.unidata_version)

For production algorithms, use the official NormalizationTest.txt from the matching Unicode version rather than a handful of handpicked examples. Verify Unicode stability and composition exclusions rather than deriving a mapping from character names. Unicode normalization is not a spelling corrector, language detector or transliterator.

Conformance vectors and common category mistakes

A conformance-oriented test harness should check that canonically equivalent inputs normalize to equal results; that each form is idempotent; that the decomposition and composition algorithms follow the same UCD version; and that the code preserves supplementary scalars, malformed-input policies and boundary context. Displaying the correct accent in a browser does not establish normal-form conformance.

Standards and technical references

1. UAX #15 — Unicode Normalization Forms
2. UAX #44 — Unicode Character Database
3. Unicode Character Database — NormalizationTest.txt
4. MDN — JavaScript String.normalize
5. Python unicodedata documentation
6. Unicode Standard Chapter 3 — Conformance and normalization

Independent Unicode technical reference. Canonical equivalence is distinct from byte identity and compatibility folding.