Unicode NFKC and NFKD: Compatibility Decomposition, Information Loss and Safe Use

Compatibility equivalence is intentionally broader than canonical equivalence

Unicode distinguishes canonical equivalence, which expresses essentially the same abstract character, from compatibility equivalence, where symbols can have a distinct presentation or specialized use. NFKD is compatibility decomposition plus canonical ordering; NFKC performs that full decomposition, orders combining marks, then applies canonical composition. The K means “compatibility,” not “case folding” or a particular keyboard layout. NFKC and NFKD can remove distinctions that NFC and NFD retain.

As specified in Unicode Standard Annex #15, normalization uses data from UnicodeData.txt and algorithmic Hangul decompositions. A compatibility mapping may be marked with a tag such as <wide>, <circle>, <super>, <font> or <fraction>. The tag describes the character's type of decomposition in the database; it is not a command to perform a narrow, tag-specific transformation. Full compatibility decomposition includes recursive canonical decomposition as needed.

Exact transformations worth testing

For the last but one example, NFKC produces fraction slash U+2044, not ASCII SOLIDUS U+002F. Unicode compatibility decomposition is not equivalent to replacing every unconventional symbol with the nearest ASCII-looking sequence.

NFKC versus NFKD, and why composition follows decomposition

Both K forms begin with the same compatibility decomposition. NFKD leaves the canonically ordered decomposed result; NFKC recomposes eligible sequences. For example, NFKD of U+00E9 is U+0065 U+0301, while NFKC of U+00E9 is U+00E9. Neither form is a universal transliterator or ASCII converter: decomposed sequences can retain accents, and CJK compatibility ideographs have specific mappings rather than a language translation.

const cp = s => [...s].map(c => "U+" + c.codePointAt(0).toString(16).toUpperCase());
for (const input of ["Ａ", "𝐀", "①", "²", "fi", "½", "é"]) {
  console.log(input, cp(input.normalize("NFKD")), cp(input.normalize("NFKC")));
}
console.log("½".normalize("NFKC") === "1\u20442"); // true
console.log("é".normalize("NFKD") === "e\u0301");  // true

Normalization cannot reverse a many-to-one mapping

After NFKC, three distinct strings containing A, Ａ and 𝐀 can all contain exactly U+0041. Without out-of-band metadata, no algorithm can infer which source the author originally chose. The same risk applies to circled digits, modifier letters and many mathematical alphabets. A product should never promise that normalized output can be converted back into its original style on demand.

For mathematical notation, the distinction between ordinary and bold/Fraktur/double-struck variables may encode actual mathematical meaning. Compatibility folding that turns distinct variables into one plain letter can make an equation invalid. For human-facing decorative text, the US Fancy Text tool can generate copyable characters that illustrate why preserving the raw string matters; testing only what the result looks like does not verify its code points.

Identifier profiles, confusables, and overnormalization

Unicode compatibility forms can be useful in precisely specified identifier and search profiles, but NFKC alone is not a security validator. It does not replace mixed-script restrictions, protocol-specific allowed-character sets, reserved-word checks or confusable detection. UTS #39 and identifier specifications supply additional constraints. An application should define whether a normalized string or an original string is used for authorization and uniqueness, and maintain consistency from validation through storage and lookup.

Applying NFKC after a validation filter can introduce characters that the filter would have rejected had they appeared in their normalized form. Instead, define an explicit canonicalization pipeline, reject disallowed input at the appropriate points, and ensure the same representation is used in security-sensitive comparisons. Conversely, globally normalizing prose can remove deliberate typography or distinctions in scholarly text.

Normalization at chunk boundaries

Full decomposition and canonical ordering can cross concatenation boundaries, and recomposition can combine a base letter from one chunk with a mark in the next. NFKC applied to each chunk separately is not in general equal to NFKC applied to the concatenation. Streaming implementations must retain context or use a Unicode-defined normalization boundary algorithm. Splitting text by arbitrary network chunk size and normalizing each chunk independently does not provide a conforming whole-string transformation.

Python and runtime-version constraints

import unicodedata as ud
samples = {"fullwidth": "Ａ", "bold": "𝐀", "circled": "①",
           "fraction": "½", "ligature": "fi"}
for key, text in samples.items():
    normalized = ud.normalize("NFKC", text)
    print(key, [f"U+{ord(c):04X}" for c in normalized], normalized)
assert ud.normalize("NFKC", "𝐀") == "A"
assert ud.normalize("NFKD", "\u00E9") == "e\u0301"
assert ud.normalize("NFKC", "½") == "1\u20442"
print("UCD version:", ud.unidata_version)

Python bundles a particular Unicode Character Database version; JavaScript engines and ICU versions may differ in update cadence. Normalization stability limits changes for older assigned characters, but data-version pinning remains necessary for conformance testing, new scalar assignments and reproducible forensic work.

A conformance matrix rather than an ad-hoc “cleaner”

For serious implementations, test against the version-matched UCD NormalizationTest.txt and the rules of UAX #15. Store original data when exact user-authored character identity matters. A lossy folding index can coexist with a faithful display field without rewriting original text.

Standards and implementation references

1. UAX #15 — Unicode Normalization Forms
2. UAX #44 — Unicode Character Database
3. Unicode FAQ — normalization and compatibility caveats
4. UTS #39 — Unicode Security Mechanisms
5. MDN — String.prototype.normalize
6. Python — unicodedata

Independent reference. Unicode normalization, character identity, display and application-specific equivalence must be evaluated separately.