Unicode Case Folding: Full, Simple and Turkic Caseless Matching

Case folding supports caseless comparison—not human-facing lowercasing

Unicode case folding maps text into a comparison representation for operations such as case-insensitive matching. It is distinct from uppercasing, lowercasing and titlecasing used for presentation. The default full fold is largely language independent, though Unicode publishes optional Turkic-specific mappings. Applying a default lowercase function and comparing its output is not equivalent to full case folding.

The Unicode Character Database has CaseFolding.txt, with statuses C (common), F (full), S (simple) and T (special Turkic). A simple mapping results in at most one code point for a code point input; full mappings may expand to multiple scalars. Case folding does not include arbitrary accent stripping, transliteration, compatibility decomposition or visual-confusable detection.

One-to-many folding: German sharp s

U+00DF LATIN SMALL LETTER SHARP S (ß) provides an important counterexample to “lowercase both sides.” Under default full case folding, ß becomes ss. Meanwhile "straße".toLowerCase() remains straße. A full case-folded comparison can match STRASSE and straße while a naive lowercasing comparison may not.

import unicodedata as ud
a, b = "Straße", "STRASSE"
assert a.lower() != b.lower()
assert a.casefold() == b.casefold() == "strasse"
print("UCD version:", ud.unidata_version)

Another character, U+1E9E LATIN CAPITAL LETTER SHARP S (ẞ), has case-folding behavior that should be tested distinctly from lowercase presentation. Simple-case transformations and full-case expansions are different algorithms; never assume that every uppercase character maps to exactly one lowercase scalar.

Greek sigma and context

Greek has U+03A3 CAPITAL SIGMA (Σ), U+03C3 SMALL SIGMA (σ) and U+03C2 SMALL FINAL SIGMA (ς). Full default case folding maps sigma variants to a common form useful for caseless matching. Human-facing lowercasing applies contextual rules to choose final sigma near the end of a word. Hence case folding can be context-insensitive even where lowercasing is context-sensitive.

Default folding versus optional Turkic tailoring

Turkish and Azerbaijani distinguish dotted İ (U+0130), dotless ı (U+0131), Latin I (U+0049) and i (U+0069). Default full case folding of U+0130 produces i followed by U+0307 COMBINING DOT ABOVE. A Turkic-sensitive fold can instead map U+0049 to U+0131 and U+0130 to U+0069 in the appropriate profile. Those are different equivalence relations.

CaseFolding.txt includes T mappings to support this alternate behavior. An application must decide whether it is implementing default language-neutral matching, Turkic-sensitive behavior, or a locale-sensitive search/collation profile. Silently switching depending on the server process locale is an integration bug waiting to occur.

Normalization and canonical caseless matching

Case folding does not by itself guarantee canonical-normalized results after every transformation. A robust comparison profile can incorporate Unicode-defined canonical caseless matching, potentially normalizing and folding more than once according to the specified algorithm. For a common pragmatic search key, normalize("NFC") after a full fold is useful, but this simple pipeline should not be advertised as equivalent to every Unicode-defined caseless matching relation or to NFKC_Casefold identifier normalization.

Compatibility case folding used in some identifier profiles is a separate operation that incorporates compatibility distinctions. A system using NFKC on one side and default casefold on the other can produce inconsistent uniqueness or access-control results. Define the profile and Unicode data version explicitly, store original text separately and test all equivalent variants.

Why JavaScript lowercasing is not full casefold

JavaScript exposes toLowerCase() and toLocaleLowerCase(), but does not expose a general ECMAScript String.prototype.caseFold() API. A developer requiring the normative Unicode full fold should use a maintained implementation backed by versioned UCD data, or a Unicode library exposing that algorithm. Lowercasing alone misses full-fold expansions such as the sharp-s example.

console.log("Straße".toLowerCase());        // straße
console.log("STRASSE".toLowerCase());       // strasse
console.log("Straße".toLowerCase() === "STRASSE".toLowerCase());
// false: this is not a complete caseless comparison
console.log("Σ".toLowerCase());             // σ
console.log("ΟΣ".toLowerCase());            // ος (final sigma context)

Python full folding and code-point inspection

samples = ["Straße","STRASSE","Σ","σ","ς","İ","I","ı","i"]
for s in samples:
    print(s, s.casefold(), [f"U+{ord(c):04X}" for c in s.casefold()])
assert "ß".casefold() == "ss"
assert "Σ".casefold() == "σ"
assert "ς".casefold() == "σ"
assert "İ".casefold() == "i\u0307"
assert "I".casefold() == "i"  # default, not Turkic

Python casefold() supplies Unicode full case folding based on the runtime's data version. Python's lower() should be used for lowercase presentation; the two have overlapping but nonidentical purposes. A deployment that combines Python, ICU and JavaScript must verify its comparison results across versions and profiles.

Database keys, regex engines, and identity risks

Not every regex engine's ignore-case matching implements Unicode full folding, and some regular-expression specifications use simple case folding or context-specific rules. Likewise, an SQL case-insensitive collation is not necessarily the same as Unicode default folding. For identifiers, apply an agreed normalization and folding profile at registration, login, storage and lookup, then compare the same canonical representation.

Case folding is not a cure for homograph attacks. Cross-script confusables and invisible characters remain possible. UTS #39 and identifier profiles are separate from caseless matching. Do not use casefolded display text as a replacement for the original if preserving the user's spelling is important.

Implementation regression vectors

To validate a full library, use version-matched UCD CaseFolding.txt rather than approximating the mapping with a static English alphabet. Test full versus simple folding and optional T mappings explicitly, including one-to-many expansions and supplementary scalar sequences.

Standards and implementation references

1. Unicode Standard Chapter 5 — Case Mappings and Case Folding
2. Unicode FAQ — Case Mapping versus Case Folding
3. UCD — CaseFolding.txt
4. UAX #44 — CaseFolding property data
5. Python str.casefold
6. UTS #39 — Unicode Security Mechanisms

Independent technical reference. Normalization and casing rules are algorithmic data transformations, not substitutes for visual or language-specific validation.