Mixed-Script Identifier Detection: Script_Extensions, Resolved Script Sets and False Positives

A naïve mixed-script detector marks almost every multilingual phrase as suspicious and can also miss identifiers that resemble trusted names. Unicode Script, Script_Extensions and the resolved-script-set machinery of UTS #39 are designed to support more principled analysis. The application must still choose whether it is validating a username, programming symbol, hostname or prose.

Distinguish a script property from a language label

UAX #24 defines Script and Script_Extensions properties for Unicode characters. A character with Script=Latin is not necessarily English, and a character used in both Japanese and Chinese contexts can participate in several language systems.

BCP 47 language tags identify language preferences but are not a substitute for script analysis of the actual characters. Similarly, an editor's chosen font or glyph run cannot be used as a security classification for source scalars.

Common and Inherited are not independent foreign scripts

Digits, punctuation and spaces frequently belong to Script=Common, while combining marks can have Script=Inherited. Counting all values literally would make 'hello123!' look mixed-script merely because numbers and punctuation occur beside Latin letters.

UTS #39 resolves script sets with special treatment of Common and Inherited values. Interpret Script_Extensions rather than just single Script values when context permits. A production detector should use maintained Unicode security library algorithms instead of improvised ASCII-rejection rules.

Resolved script sets make identifiers more meaningful

The script set for a string is evaluated with reference to acceptable scripts for each character, and UTS #39 defines a resolved script set for detection. Characters shared by several scripts may be compatible with an otherwise single-script word, while incompatible scripts can indicate a true mixture.

This is not identical to simply collecting the union of non-Common scripts. Study the UTS #39 process for intersection and composite CJK relationships before implementing it. A simple union heuristic is useful for diagnostics, not for claiming formal conformance.

Understand script restriction levels

UTS #39 describes restriction levels including ASCII-Only, Highly Restrictive, Moderately Restrictive, Minimally Restrictive, Unrestricted and related concepts as defined by the current standard. These express different degrees of allowed script mixing, not one global policy.

An account-registration system may choose stronger restrictions than a multilingual display field, but it should state the supported repertoire and explain rejected input. A legitimate Japanese name can include Han, Hiragana and Katakana without being a security anomaly.

Apply checks only after defining identifier syntax

A programming language may allow one repertoire under UAX #31 plus its own syntax constraints; a chat display name may allow emoji and spaces; an IDNA label follows IDNA processing and registry policy. The detector cannot substitute one shared set of rules for all these fields.

Choose field-specific normalization, allowable syntax, required scripts, risk checks and storage policy. Keep the original input and the validated representation when their roles differ. Do not silently alter source text merely to pass the restriction-level check.

A single-script identifier can still be confusable

An all-Cyrillic string may resemble an all-Latin brand, or a Latin-only identifier may contain near-lookalikes. Mixed-script detection will not detect every case, so protected-name collision checks and UTS #39 confusable analysis should be separate stages.

Conversely, two scripts in one string do not prove spoofing. A thoughtful system correlates a suspicious mixture with collision targets and account context, yielding warnings or review where appropriate.

JavaScript property escapes are insufficient for formal detection

JavaScript supports expressions such as \p{Script=Greek} and \p{Script_Extensions=Greek} in Unicode-aware regular expressions. These are useful for individual code-point inspection, but do not automatically calculate the full UTS #39 resolved script set.

Do not assume every character belongs to exactly one user language, and do not treat regex character-class matches as word segmentation. To classify a grapheme, code point-by-code point results may need to be combined with UAX #29 segmentation and language-aware policy.

Data versions and migration can change verdicts

Unicode adds characters and updates property or security data. An older runtime may not know a script extension added in a later release. Systems with distributed validators should use compatible data versions to prevent a client passing a name that the server rejects.

Log the Unicode and UTS #39 data versions with risk verdicts, especially for persistent user handles. Plan migration rather than silently enforcing a new profile retroactively against existing users.

Evaluate with a multilingual corpus

Include Latin + digits, Cyrillic text, Greek notation, Japanese Han/Hiragana/Katakana, Arabic with combining marks and Latin-Cyrillic near-lookalikes. Record expected identification, restriction level and protected-name collision independently.

Audit false positives for each locale and field. Security should be proportionate: warn about potential confusion without claiming that all mixed scripts are malicious or that all single-script inputs are safe.

Reproducible technical example

Use the following minimal experiment to inspect representation rather than guessing from appearance. It does not implement a complete Unicode security algorithm, determine malicious intent, or measure platform-wide font support. Each of those tasks requires its own documented conformance or decision model.

const samples = ['hello123!', 'a\u0430', '漢字かなカナ'];
for (const s of samples) {
 const scalars=[...s].map(c=>'U+'+c.codePointAt(0).toString(16).toUpperCase());
 console.log({s,scalars});
}
console.log(/\p{Script_Extensions=Greek}/u.test('α'));
// NOT a UTS #39 implementation: use a Unicode security library to
// compute the resolved script set, restriction level and confusable alerts.

Record the input Unicode scalars, the expected application behavior, and which layer first diverges: transport, parsing, normalization, security classification, layout, or accessibility. Use specific tool versions and preserve the original input for comparison after any policy decision.

Evidence matrix: distinguish text, appearance and policy

A similar-looking output is not proof of identical text. Equally, two strings with different scalar sequences may be intentionally equivalent under a narrowly defined product matching policy. When evaluating risk, separate exact identity, confusable matching, account verification and readable display.

Validation checklist and failure classification

1. Latin + digits. Build a reproducible specimen for hello123 and separately capture source scalars, normalization choices, policy settings and rendered output. The interpretation is: Common digits do not imply a second writing system. Do not conflate a visual similarity with a binary equality or a security verdict.
2. Latin/Cyrillic. Build a reproducible specimen for a + U+0430 and separately capture source scalars, normalization choices, policy settings and rendered output. The interpretation is: requires script and confusable review. Do not conflate a visual similarity with a binary equality or a security verdict.
3. Japanese compound. Build a reproducible specimen for Han/Hiragana/Katakana and separately capture source scalars, normalization choices, policy settings and rendered output. The interpretation is: legitimate composite writing system. Do not conflate a visual similarity with a binary equality or a security verdict.
4. Combining marks. Build a reproducible specimen for Script=Inherited and separately capture source scalars, normalization choices, policy settings and rendered output. The interpretation is: handle via Script_Extensions and context. Do not conflate a visual similarity with a binary equality or a security verdict.
5. UTS #39 result. Build a reproducible specimen for restriction level and separately capture source scalars, normalization choices, policy settings and rendered output. The interpretation is: signal calibrated to identifier field. Do not conflate a visual similarity with a binary equality or a security verdict.

Keep fixtures with positive cases, false positives and context-dependent exceptions. When an implementation returns a finding, show which Unicode characters and rules contributed to it. Avoid silently rewriting untrusted identifiers, and do not apply machine-identifier restrictions globally to multilingual prose.

Boundaries of automated conclusions

Standards offer useful algorithms and properties, not perfect attribution of a user's intent. Visual confusability, supported script combinations, locale, account reputation and actual network origin need independent evidence. A system should fail safely in high-trust contexts while preserving valid Unicode for normal communication.

Primary specifications and data sources

The references provide normative definitions and authoritative technical data where available. UTS #39 and UTS #46 are separate technical standards; conformance to the core Unicode Standard alone does not imply conformance to either. Product-specific warnings and enforcement thresholds remain implementation decisions.

• UTS #39 Unicode Security Mechanisms — primary standard or data source. Check the exact release version and the scope of its conformance claims; implementation policy and user-interface warnings are separate choices.
• UAX #24 Script Property — primary standard or data source. Check the exact release version and the scope of its conformance claims; implementation policy and user-interface warnings are separate choices.
• UAX #31 Unicode Identifiers — primary standard or data source. Check the exact release version and the scope of its conformance claims; implementation policy and user-interface warnings are separate choices.
• Unicode Character Database UAX #44 — primary standard or data source. Check the exact release version and the scope of its conformance claims; implementation policy and user-interface warnings are separate choices.
• MDN Unicode Property Escapes — primary standard or data source. Check the exact release version and the scope of its conformance claims; implementation policy and user-interface warnings are separate choices.

Related published research

This article belongs to the Identifiers, Confusables & Security virtual silo. Nearby reference articles explain the adjacent Unicode, rendering and text-processing issues in greater depth.

• mixed script text processing
• unicode confusables lookalikes
• bcp47 language tags unicode
• unicode property escapes regex

Technical information is presented for defensive review, reproducible diagnostics and trustworthy text interchange. The original encoded content should remain available when any derived display or comparison representation is generated.