Unicode identifier validation cannot be reduced to a regular expression such as 'all Unicode letters are allowed'. UAX #31 defines properties and approaches for identifiers; UTS #39 provides security mechanisms and restriction guidance. Each application must still specify whether the identifier is a program symbol, username, tenant slug or domain name, and choose an explicit compatibility and migration contract.
Begin with field purpose and threat model
A display name is primarily human-readable content; a unique login handle is an account locator; a programming identifier has parser constraints; a DNS label follows IDNA rules. A single global Unicode policy would either over-restrict natural-language writing or under-protect high-value identifiers.
Write a policy declaration with allowed syntax, normalization, case handling, script restrictions, confusable comparison, length metric and user feedback. Keep the field's canonical identity separate from its decorated display label so security decisions do not depend on font or CSS.
Use UAX #31 identifier properties correctly
UAX #31 describes Unicode identifier conventions, including ID_Start/ID_Continue and stability-oriented XID_Start/XID_Continue properties. XID properties account for closure under NFKC-related transformations useful in identifier syntax contexts. They do not automatically define a safe account username profile.
A programming language may additionally allow underscores or impose its own grammar and normalization. An application's use of XID properties does not imply it must apply NFKC to all user text. Specify the Unicode data version and supplemental syntax rules.
Normalization is a policy choice with collisions
NFC can make canonically equivalent spellings compare identically; NFKC can collapse compatibility variants such as mathematical bold characters. NFKC is not the same as UTS #39 confusable mapping and does not eliminate Latin/Cyrillic lookalikes.
Before enforcing a new normalization rule, measure whether existing distinct account handles would collapse to the same comparison key. Preserve original labels and establish an explicit collision-remediation process rather than silently merging accounts.
UTS #39 provides security profiles and restriction levels
UTS #39 defines identifier profiling mechanisms and levels for assessing script mixture, including Highly Restrictive and more permissive modes. Their purpose is to mitigate potentially confusable identifiers while allowing meaningful internationalized syntax where appropriate.
Choosing Highly Restrictive may be appropriate in some registration flows, but it is not a universal requirement for comments, research manuscripts or multilingual messaging. Document allowed combinations and explain to users which rule rejects a proposed name.
Join_Control characters need contextual restrictions
U+200C ZERO WIDTH NON-JOINER and U+200D ZERO WIDTH JOINER can have legitimate linguistic and emoji roles. Security profiles may restrict use to particular contextual patterns rather than allowing them everywhere or banning them indiscriminately.
An account field may disallow emoji while a comment field preserves them. A script that legitimately relies on join controls needs appropriately implemented ContextJ or related context checks. Do not treat all Default_Ignorable_Code_Point characters as identical.
Confusable skeletons are review keys, not user identities
Use an authoritative UTS #39 implementation to derive risk-relevant comparison data. Skeleton collisions can be checked against protected accounts or high-value reserved words. They should not replace original spelling in the database or appear as the authoritative display form.
The detector can return false positives and may differ by Unicode data release. Version derived keys and store exact source scalars for auditing. A skeleton collision alone should trigger the policy's review path, not a claim that the account holder is impersonating someone.
Case folding differs from locale-sensitive casing
Unicode default case folding is useful for some caseless comparisons, while locale-specific casing (for example Turkish I) may have different behavior. Full case folding can produce multiple code points and is not simply ASCII lowercase applied character by character.
Declare which comparison regime is used and whether locale is part of the contract. Inconsistent client/server folding produces duplicate handle registration or login lookup failures. Treat case transformation and script confusability as different processing steps.
Length, truncation and database uniqueness matter
JavaScript string length counts UTF-16 code units; UTF-8 size counts bytes; UAX #29 grapheme count approximates user-perceived characters. Database collations can implement additional equivalence rules. The allowed size of a handle must explicitly name its metric.
Never truncate an input inside a surrogate pair, combining sequence or security comparison key without defined behavior. Apply uniqueness constraints to a deliberately computed stable key, then test collisions under the actual database collation and normalization settings.
Version and migrate policies safely
Unicode property assignments and confusables data evolve. A registration rule that accepted a handle under one release may reject it after a library upgrade. Without versioned enforcement, existing users may be locked out or newly created names may collide with stored ones.
Store policy version, keep immutable account IDs, audit changes before rollout, reserve already-issued handles appropriately, and run migration tests on the actual corpus. Explain affected cases to users rather than silently rewriting canonical account references.
Design a compliance suite with valid and invalid cases
Use positive fixtures for Latin, Cyrillic and Japanese names that fit the selected profile, plus negative or review cases for unexpected bidi controls, disallowed joiners, mixed-script lookalikes and normalization collisions. Include canonical equivalent forms and non-BMP characters.
Test server-side authoritative decisions and client-side validation separately. A passing regex property test proves only syntax, not the absence of confusables or identity spoofing. Record expected status, reason and standard/data version for each fixture.
Reproducible example and scope
This example isolates one specific aspect of text representation, validation or accessible content. It is not a full production security validator or a comprehensive accessibility test: field-specific policy, Unicode data releases, browser behavior and assistive-technology feedback require independent verification.
// Illustrative syntax property check, NOT a complete identifier policy:
const syntactic = /^\p{XID_Start}\p{XID_Continue}*$/u;
console.log(syntactic.test('alpha')); // true
console.log(syntactic.test('alpha_1')); // false in this minimal pattern;
// UAX #31 profiles can explicitly permit additional characters.
const original='๐';
console.log(original.normalize('NFKC')); // A (compatibility mapping)
// Apply server-enforced, versioned profile + UTS #39 checks separately.Capture the source code points, context, any intentional transformation and observed outcome. Exact Unicode identity, comparison-key equivalence and user comprehension are separate measurements. Preserve the original value when generating derived search, display or inspection forms.
Evidence and interoperability comparison
| Case | Specimen | Interpretation |
|---|---|---|
| Syntax | XID_Start / XID_Continue | grammatical eligibility, not security |
| Normalization | NFC vs NFKC | can change uniqueness relationships |
| Confusables | UTS #39 skeleton | possible collision, not canonical identity |
| Mixed scripts | restriction level | product-defined risk threshold |
| Join controls | U+200C/U+200D | need context-dependent policy |
| Storage | immutable ID + comparison key | do not overwrite original human spelling |
Do not treat a successful string normalization or a valid HTML structure as evidence that every possible attack has been prevented or that every screen reader will pronounce the result correctly. Match each claim to an applicable specification and observable test.
Testing protocol and limitations
- Syntax. Build a controlled test of
XID_Start / XID_Continue, retaining the actual Unicode scalar sequence and relevant browser, screen-reader or protocol version. Expected interpretation: grammatical eligibility, not security. Document both correct behavior and plausible false positives. - Normalization. Build a controlled test of
NFC vs NFKC, retaining the actual Unicode scalar sequence and relevant browser, screen-reader or protocol version. Expected interpretation: can change uniqueness relationships. Document both correct behavior and plausible false positives. - Confusables. Build a controlled test of
UTS #39 skeleton, retaining the actual Unicode scalar sequence and relevant browser, screen-reader or protocol version. Expected interpretation: possible collision, not canonical identity. Document both correct behavior and plausible false positives. - Mixed scripts. Build a controlled test of
restriction level, retaining the actual Unicode scalar sequence and relevant browser, screen-reader or protocol version. Expected interpretation: product-defined risk threshold. Document both correct behavior and plausible false positives. - Join controls. Build a controlled test of
U+200C/U+200D, retaining the actual Unicode scalar sequence and relevant browser, screen-reader or protocol version. Expected interpretation: need context-dependent policy. Document both correct behavior and plausible false positives. - Storage. Build a controlled test of
immutable ID + comparison key, retaining the actual Unicode scalar sequence and relevant browser, screen-reader or protocol version. Expected interpretation: do not overwrite original human spelling. Document both correct behavior and plausible false positives.
For system-level verification, add adversarial input and legitimate multilingual exceptions. Test both the algorithmic result and the human decision it supports. A high-quality report includes exact Unicode scalars, environment versions, expectations, policy choice and clear error classification.
Interpreting unintended changes
When a result differs, determine whether the change arises during byte decoding, normalization, security classification, rendering, assistive technology or the clipboard. A fix targeting the wrong layer can silently destroy valid language text or leave the original ambiguity unresolved.
Primary standards and implementation documentation
Use the following primary references to verify normative requirements and data-version implications. A Unicode Technical Standard, core Unicode conformance, browser accessibility behavior and application policy are distinct mechanisms with different obligations.
- UAX #31 Unicode Identifiers โ primary specification or accessibility reference; verify its applicable version and distinguish normative language from implementation-specific recommendations.
- UTS #39 Unicode Security Mechanisms โ primary specification or accessibility reference; verify its applicable version and distinguish normative language from implementation-specific recommendations.
- UAX #15 Unicode Normalization โ primary specification or accessibility reference; verify its applicable version and distinguish normative language from implementation-specific recommendations.
- UAX #24 Unicode Script Property โ primary specification or accessibility reference; verify its applicable version and distinguish normative language from implementation-specific recommendations.
- Unicode Character Database UAX #44 โ primary specification or accessibility reference; verify its applicable version and distinguish normative language from implementation-specific recommendations.
Related published research
The Identifiers, Confusables & Security virtual silo connects these technical notes to the broader Unicode interoperability research library. Links below point to other published documents with adjacent concerns.
- unicode confusables lookalikes
- mixed script identifier detection
- invisible character spoofing
- unicode normalization nfkc nfkd
Keep user-facing content independently intelligible, and maintain an explicit distinction between decorative output, its encoded characters and any alternate form supplied for accessibility or security review.