Unicode Confusables and Lookalikes: UTS #39 Skeletons, Scripts and Collision Analysis

Unicode supports many writing systems, so different encoded characters may have very similar glyph shapes. A security review cannot treat appearance as identity: Latin a U+0061 and Cyrillic а U+0430 are distinct scalars even when they look the same in a particular font. Unicode Security Mechanisms (UTS #39) defines versioned confusable detection tools, not a replacement spelling for ordinary text.

Character identity and perceived glyph are separate concepts

A text string contains Unicode scalar values, whereas a font chooses glyph outlines. Similar glyphs can arise within one script, across scripts, from mathematical alphabets or through typographic styles. The risk is largest when people must decide whether one identifier belongs to a known account, organization or origin.

The string pаypal can contain a Cyrillic U+0430 in place of a Latin a; the shape may look convincing to some readers. Comparing bytes, scalar arrays or code-point-aware strings detects the difference. Neither exact equality nor screenshot similarity alone is a complete anti-spoofing defense.

Understand UTS #39 confusable mapping and skeletons

UTS #39 provides a confusables data file and a process that derives a comparison skeleton so potentially confusable strings can be compared. This is a detection representation rather than a preferred display spelling, transliteration standard or universal canonical form.

The skeleton may combine several transformations and mappings and is not equivalent to simply applying normalize('NFKC') or lowercasing. Implement the normative process with a maintained Unicode security library and pin the data version; handwritten one-character replacement tables miss many-to-many relationships.

Single-script and mixed-script confusables differ

UTS #39 distinguishes confusable pairs whose resolved script sets intersect from those with disjoint script sets. A name can be ambiguous even if every letter belongs to Latin, while Latin-Cyrillic substitution can create a mixed-script confusable.

Security tooling that flags only mixed-script text misses single-script lookalikes. Conversely, mixed-script text appears routinely in multilingual prose and scientific notation. A restriction-level result is a risk signal requiring context, not evidence of malicious intent.

Whole-script lookalikes require additional analysis

An identifier entirely in Cyrillic may visually resemble an identifier entirely in Latin without containing any internal script mixture. UTS #39 discusses whole-script confusables and how they fit within the mixed-script classification for string pairs.

A detector that counts scripts within each candidate and allows every single-script identifier will not identify all lookalikes. Compare against protected identifiers and accepted registration policy, then decide whether a warning, restricted display or review is appropriate.

Script and Script_Extensions need contextual interpretation

Many digits and punctuation marks have Script=Common; marks can have Script=Inherited. UTS #39 uses resolved script sets rather than treating every punctuation mark as a foreign script. Some characters have meaningful Script_Extensions containing several scripts.

Counting all raw Script property changes inflates alarms for normal usernames. A proper implementation uses Unicode script data, special handling for shared characters and the prescribed mixed-script analysis. Record Unicode data version so future additions can be audited.

Normalization helps equality but does not prevent spoofing

NFC and NFD address canonical equivalence; NFKC addresses additional compatibility characters. None of these generally converts Greek capital alpha U+0391 into Latin A U+0041. Removing combining marks or compatibility characters indiscriminately can also corrupt meaningful names.

Use normalization when required by the account identifier policy and preserve the original user-facing spelling when appropriate. Confusable matching, source identity and search aliases should remain different fields or stages in a well-instrumented pipeline.

Risk depends on field purpose and trust boundaries

In a public forum post, multilingual writing should be broadly accepted and quoted faithfully. In a login handle, payment recipient name or software package identifier, similarity to a protected label can cause real confusion. Different resources warrant different profiles and collision checks.

Choose an allowlist or review process for high-value identifiers, but do not apply a strict ASCII-only policy indiscriminately to natural-language messages. Include transparent user feedback and explain which character was flagged, rather than issuing a generic ban on unfamiliar scripts.

A skeleton collision is not proof of impersonation

UTS #39 intentionally favors detecting potential confusion and can flag legitimate strings. Unrelated names can yield similar skeletons, and visual outcomes depend on fonts, context and viewers. A collision must not automatically trigger account seizure or a claim that two people have the same identity.

Use collision results to trigger measured safeguards: display full verified account context, protect known names or require additional review. Preserve logged original code points so reviewers can distinguish accidental collisions from deliberate abuse without relying on screenshots.

Construct deterministic test cases without teaching misuse

Build a small local test corpus with Latin a, Cyrillic а, Greek α, fullwidth forms and a few single-script near-neighbors. Validate exact code points, apply a maintained confusables implementation and record which pairs collide under the chosen Unicode release.

For each positive match, test at least one legitimate multilingual counterexample and verify that the tool presents a warning rather than silently rewriting user input. A security component is successful when it reduces confusion without destroying valid names.

Reproducible technical example

Use the following minimal experiment to inspect representation rather than guessing from appearance. It does not implement a complete Unicode security algorithm, determine malicious intent, or measure platform-wide font support. Each of those tasks requires its own documented conformance or decision model.

const latin = 'a';
const cyrillic = '\u0430';
const greek = '\u03B1';
const inspect = s => [...s].map(c=> 'U+'+c.codePointAt(0).toString(16).toUpperCase());
console.log({latin:inspect(latin),cyrillic:inspect(cyrillic),greek:inspect(greek)});
console.log(latin===cyrillic); // false
console.log(latin.normalize('NFKC')===cyrillic.normalize('NFKC')); // false
// Consult UTS #39 confusables data; NFKC is NOT a confusable skeleton.

Record the input Unicode scalars, the expected application behavior, and which layer first diverges: transport, parsing, normalization, security classification, layout, or accessibility. Use specific tool versions and preserve the original input for comparison after any policy decision.

Evidence matrix: distinguish text, appearance and policy

A similar-looking output is not proof of identical text. Equally, two strings with different scalar sequences may be intentionally equivalent under a narrowly defined product matching policy. When evaluating risk, separate exact identity, confusable matching, account verification and readable display.

Validation checklist and failure classification

1. Latin a. Build a reproducible specimen for U+0061 and separately capture source scalars, normalization choices, policy settings and rendered output. The interpretation is: separate scalar identity. Do not conflate a visual similarity with a binary equality or a security verdict.
2. Cyrillic а. Build a reproducible specimen for U+0430 and separately capture source scalars, normalization choices, policy settings and rendered output. The interpretation is: can resemble Latin a. Do not conflate a visual similarity with a binary equality or a security verdict.
3. Greek α. Build a reproducible specimen for U+03B1 and separately capture source scalars, normalization choices, policy settings and rendered output. The interpretation is: different script and character. Do not conflate a visual similarity with a binary equality or a security verdict.
4. UTS #39 skeleton. Build a reproducible specimen for derived comparison value and separately capture source scalars, normalization choices, policy settings and rendered output. The interpretation is: risk-signal, not replacement spelling. Do not conflate a visual similarity with a binary equality or a security verdict.
5. Identifier profile. Build a reproducible specimen for field-specific restrictions and separately capture source scalars, normalization choices, policy settings and rendered output. The interpretation is: must allow legitimate language context. Do not conflate a visual similarity with a binary equality or a security verdict.

Keep fixtures with positive cases, false positives and context-dependent exceptions. When an implementation returns a finding, show which Unicode characters and rules contributed to it. Avoid silently rewriting untrusted identifiers, and do not apply machine-identifier restrictions globally to multilingual prose.

Boundaries of automated conclusions

Standards offer useful algorithms and properties, not perfect attribution of a user's intent. Visual confusability, supported script combinations, locale, account reputation and actual network origin need independent evidence. A system should fail safely in high-trust contexts while preserving valid Unicode for normal communication.

Primary specifications and data sources

The references provide normative definitions and authoritative technical data where available. UTS #39 and UTS #46 are separate technical standards; conformance to the core Unicode Standard alone does not imply conformance to either. Product-specific warnings and enforcement thresholds remain implementation decisions.

• UTS #39 Unicode Security Mechanisms (v18) — primary standard or data source. Check the exact release version and the scope of its conformance claims; implementation policy and user-interface warnings are separate choices.
• Unicode Confusables Data — primary standard or data source. Check the exact release version and the scope of its conformance claims; implementation policy and user-interface warnings are separate choices.
• UAX #24 Unicode Script Property — primary standard or data source. Check the exact release version and the scope of its conformance claims; implementation policy and user-interface warnings are separate choices.
• Unicode Normalization UAX #15 — primary standard or data source. Check the exact release version and the scope of its conformance claims; implementation policy and user-interface warnings are separate choices.
• Unicode Character Database UAX #44 — primary standard or data source. Check the exact release version and the scope of its conformance claims; implementation policy and user-interface warnings are separate choices.

Related published research

This article belongs to the Identifiers, Confusables & Security virtual silo. Nearby reference articles explain the adjacent Unicode, rendering and text-processing issues in greater depth.

• mixed script text processing
• transliteration vs script conversion
• unicode normalization nfkc nfkd
• bcp47 language tags unicode

Technical information is presented for defensive review, reproducible diagnostics and trustworthy text interchange. The original encoded content should remain available when any derived display or comparison representation is generated.