Accent-Insensitive Unicode Search: Collation, Folding and Data Integrity

Define what “accent-insensitive” means for your users

An accent-insensitive search may consider café and cafe matching strings. That seems simple for many Latin words, but the actual definition depends on language, storage requirements and the chosen comparison algorithm. Accents can distinguish unrelated words, and marks in non-Latin scripts are not necessarily optional decoration. A method that removes every Unicode combining mark from stored content can destroy meaningful letters or vowel information.

Two broad architectures are useful: a locale-aware collation comparator that may ignore secondary differences for a particular search, and a separately maintained folded search index built by an explicit mapping pipeline. Neither should overwrite the original content when fidelity matters.

NFC/NFD are not accent removal

Canonical normalization converts canonically equivalent sequences into comparable forms: U+00E9 and U+0065 U+0301 can be normalized to the same NFD or NFC representation. NFD exposes some accents as separate combining marks, but it does not remove them. NFKD performs compatibility decomposition as well and may flatten styled characters or mathematical symbols in addition to decomposing marks; that is often inappropriate for a general-purpose text archive.

Locale-aware base-strength comparison

Intl.Collator(locale, {usage: "search", sensitivity: "base"}) provides a browser API to compare strings while usually ignoring differences in case and many accents. sensitivity: "accent" includes accents but ignores case in the API's model. The exact equivalence relation depends on the locale: a Swedish collator must not be assumed to treat å like a in every setting.

const compare = new Intl.Collator("en", {
  usage:"search", sensitivity:"base"
});
console.log(compare.compare("café", "cafe") === 0); // true
console.log(compare.compare("CAFÉ", "cafe") === 0); // true
const exactAccent = new Intl.Collator("en", {
  usage:"search", sensitivity:"accent"
});
console.log(exactAccent.compare("café", "cafe") === 0); // false

A collator compares whole strings. To implement substring or prefix search, an application also needs text indexing, Unicode-aware boundary handling, and a way to locate match offsets back in the original displayed content. A global search-and-replace algorithm cannot substitute for that work. Also note that a positive/negative return from compare() has implementation-dependent magnitude; compare only its sign.

Decomposition plus selective mark handling

A widely used limited search-key strategy is to normalize to NFD, remove certain combining accent marks, lowercase or case-fold the result, and store the folded version alongside the original. The mark-selection policy matters. A regular expression restricted to U+0300..U+036F captures many familiar Latin accents but does not cover all Unicode combining marks. In contrast, removing the entire \p{M} category can change meaning across writing systems, and will remove legitimate enclosing or spacing marks.

Even for Latin text, NFD mark removal does not transliterate every accented-looking letter. Characters such as ø, ł and đ need explicit linguistic mapping decisions if the product intends to equate them with o, l and d. Neither Unicode canonical normalization nor a Unicode property escape promises those transformations.

function latinSearchKey(input) {
  return input.normalize("NFD")
    .replace(/[\u0300-\u036f]/g, "")
    .toLowerCase();
}
console.log(latinSearchKey("café") === "cafe"); // true
console.log(latinSearchKey("łódź"));            // łodz
// Intentionally NOT a multilingual or UCA-compliant search function.

Case folding is another independent operation

JavaScript's toLowerCase() does not implement Unicode full case folding. In German, full fold transforms ß to ss, while default lowercasing leaves ß unchanged. If a search feature promises case-insensitive Unicode matching across multiple locales, it must choose and document either full case folding, a suitable locale-aware collator or an application-specific query normalization algorithm.

Do not combine full fold, compatibility normalization, diacritic stripping and confusable skeletons without specifying order and purpose. A name that matches under a permissive search profile might intentionally remain distinct under the site's account-identifier profile or under mathematical notation rules.

Search highlighting and offset mapping

Folding can change string length. Precomposed é takes one scalar; its NFD equivalent contains two. Full folding of sharp s expands to two letters, and normalization of some symbols can expand into multiple code points. Consequently, the index position of a match in a folded string does not necessarily equal its UTF-16 code-unit position in the original string. Naive slice(start, end) highlighting can select the wrong characters or split surrogate pairs.

A production search index should either keep an explicit folded-to-original offset map or ask a search engine to return offsets in the original representation according to its normalization analyzer. For UI ranges, handle user-perceived grapheme clusters rather than splitting combining sequences. Preserve user-authored text for display, accessibility and export.

Collations and indexes in storage engines

Database collations can be accent-insensitive, case-insensitive or both; their behavior differs by engine, collation, version and locale. A database index that treats strings equal under a collation can also affect uniqueness constraints. Do not apply a search-oriented accent-insensitive collation as the sole authentication identity rule without reviewing whether it collapses permitted identifiers.

A robust architecture separates (1) original display string, (2) normalized or folded search key, and (3) a strict identity key if the application needs it. Version the search-key algorithm and rebuild derived indexes when normalization, Unicode, ICU or CLDR data changes.

Reproducible Latin-only example in Python

import unicodedata as ud
def latin_only_key(text):
    decomposed = ud.normalize("NFD", text)
    return "".join(ch for ch in decomposed
                   if not (0x0300 <= ord(ch) <= 0x036F)).casefold()
assert latin_only_key("CAFÉ") == "cafe"
assert latin_only_key("naïve") == "naive"
assert latin_only_key("łódź") == "łodz"
print("Using UCD", ud.unidata_version)
# This intentionally limited key is not a language-universal solution.

Accent-sensitive distinctions and accessibility

Accents and vowel marks have semantic roles. In Vietnamese, removing tone marks or vowel signs can collapse many distinct words. In Arabic and Indic scripts, deleting marks indiscriminately may alter reading, meaning or grammatical interpretation. A permissive search feature can offer suggested or approximate matches without changing the underlying stored data or silently rewriting the words users authored.

For a copyable styled-text application, accent-insensitive indexing should be a feature of lookup, not a destructive transformation of clipboard output. The US Fancy Text tool offers a useful test scenario: compare exact source scalars to any indexed key so mathematical styles, diacritics and legitimate combining sequences remain preserved at display/export time.

Adversarial search fixtures

Test the expected matching relation separately from the preserved display string and database uniqueness policy. Every match should have a known reason: canonical equivalence, case folding, accent-sensitive collation, transliteration, or some other explicitly configured transformation.

Standards and implementation references

1. UTS #10 — Unicode Collation Algorithm
2. UAX #15 — Unicode Normalization Forms
3. UAX #29 — Grapheme and word boundaries
4. MDN — Intl.Collator
5. MDN — Unicode property escapes
6. Python unicodedata

Independent technical reference; exact comparison behavior depends on the applicable algorithm, data version, locale and application policy.