Unicode Script and Script_Extensions: Accurate Script Analysis

What a Unicode script property is

Unicode identifies writing-system associations through two related properties: Script (abbreviation sc) and Script_Extensions (scx). They are specified by Unicode Standard Annex #24 and maintained in Unicode Character Database files. The properties allow software to recognize, for example, Latin, Greek, Cyrillic, Arabic, Han, Hiragana, Katakana, and other script associations without assigning each code point to a spoken language.

Script is not language. English, Vietnamese, Turkish and many other languages use Latin script, while a single language such as Japanese conventionally combines multiple scripts. A locale tag such as ja-JP or en-US provides linguistic context; neither can be derived reliably from one code point’s Script property.

Script is also not a font name or an instruction to render text in a certain direction. Directionality, shaping, glyph choice, line breaking and language-specific typography are handled by other properties and algorithms.

Script values and the three implicit classes

Script assigns one value to every Unicode code point. Most values identify a particular writing system; three special values express cases where a specific script association is not appropriate:

There are many more explicit script values, and their set expands as new scripts are encoded. Applications should not build a fixed enumeration of “all possible scripts” from an obsolete Unicode version. The short names follow ISO 15924 conventions where applicable, but the Unicode Script property and ISO 15924 solve different classification problems and are not perfectly interchangeable.

Consider a (U+0061): its Script value is Latin. By contrast, an ASCII digit such as 3 has Script=Common. The combining acute accent U+0301 normally has Script=Inherited. The class Inherited does not mean that the stored property dynamically changes to Latin after a Latin vowel: it remains Inherited in the UCD, while text-processing algorithms may resolve a script run using context.

Script_Extensions and shared script usage

Script_Extensions is a set-valued property that identifies scripts in which a code point is commonly used. It handles shared characters more precisely than a single Script value. For instance, U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK has Script=Common but Script_Extensions associates it with both Hiragana and Katakana. A text-processing algorithm that classifies it as an arbitrary punctuation symbol solely because Script=Common will lose useful writing-system information.

A code point’s Script_Extensions value often consists of its single Script value, but not always. For some marks and shared characters it contains several scripts; for other generally reusable symbols it may remain {Common} or {Inherited}. Do not assume Script_Extensions always eliminates implicit values, or that every Common character has a finite explicit list of scripts.

Another subtlety documented in UAX #24 is that a code point can have a primary Script value and a broader Script_Extensions set because its usage crosses script boundaries. Do not assume that an extension set with more than one explicit member implies the primary Script is always Common or Inherited. A primary script may be assigned on predominant-use grounds.

Data files, aliases, and versioning

The core data files are Scripts.txt (primary Script values) and ScriptExtensions.txt (nondefault extension sets), distributed with the UCD. PropertyValueAliases.txt lists standardized short and long aliases, and PropertyAliases.txt defines property names. These are versioned datasets, so regression tests should record the exact Unicode release. A hardcoded snapshot becomes stale when Unicode adds scripts or adjusts derived property data.

Files use code-point ranges with hexadecimal endpoints and property value columns. They also include comments and default-value conventions. Read the UAX #44 description of each file rather than treating every missing line as “no property.” A Unicode text tool can maintain a compact interval table derived from the official release files, but it must preserve default rules and correctly handle supplementary scalar values.

When using JavaScript Unicode property escapes, the engine’s UCD version is determined by the JavaScript engine rather than by the webpage. For C/C++ software using ICU, use the ICU version and documented property APIs; for server-side languages, pin the relevant library and data version in automated tests.

JavaScript Unicode property escape examples

Modern JavaScript supports \p{Script=...} and \p{Script_Extensions=...} in Unicode-aware regular expressions. Use the u flag, and explicitly state which property you need:

const greekPrimary = /\p{Script=Greek}/u;
const greekMembership = /\p{Script_Extensions=Greek}/u;
const hiraganaPrimary = /\p{Script=Hiragana}/u;
const hiraganaMembership = /\p{Script_Extensions=Hiragana}/u;

console.log(greekPrimary.test("α"));          // true
console.log(hiraganaPrimary.test("あ"));      // true
console.log(hiraganaPrimary.test("ー"));      // false
console.log(hiraganaMembership.test("ー"));   // true

const latinPrimary = /^\p{Script=Latin}+$/u;
console.log(latinPrimary.test("abc"));        // true
console.log(latinPrimary.test("abc123"));     // false:
                                                // digits are Common

With \p{Script_Extensions=Hiragana}, the prolonged sound mark matches as part of Hiragana’s repertoire; with Script=Hiragana, it does not. Treat such differences as intentional semantics, not inconsistent Unicode data.

Script detection on whole strings is not equivalent to a single ^...+$ test. Common punctuation, inherited marks, whitespace, and mixed-script orthography make a single “entire text is script X” Boolean too crude for layout engines and language identification.

Script-run classification and text-processing design

Many shaping engines process text in script runs: contiguous regions whose characters are compatible with a shaping script. A robust implementation can use Script_Extensions to identify candidates, propagate or resolve Common and Inherited positions with contextual rules, and then split runs when the permissible script set changes. This is more subtle than attaching every Common code point to the immediately preceding letter.

For example, a string mixing Han ideographs, Hiragana, Katakana and shared punctuation is ordinary Japanese text. Treating every script transition as malicious, invalid, or a language boundary is incorrect. Similarly, a combining mark may need to travel with the previous base through shaping; breaking between the base and mark simply because one has Script=Latin and the other Inherited can damage rendering.

Run segmentation also interacts with bidirectional layout and font fallback. The Unicode Bidirectional Algorithm determines directional ordering, not script membership. OpenType shaping systems use script and language tags to select appropriate substitutions/positioning. A font supporting a code point is not proof it can shape the entire adjacent sequence correctly.

A practical script classifier should therefore produce annotated spans with explicit uncertainty rather than a guessed language label for an entire document. The design should specify the behavior for unknown data, private-use code points, isolated marks, mixed scripts, and Unicode-version changes.

Mixed scripts, identifier security, and false positives

Script properties are useful building blocks in identifier-spoofing defenses, but do not detect all confusables. Latin a (U+0061) and Cyrillic а (U+0430) can look nearly identical, yet are distinct code points and have different Script values. A malicious identifier can exploit these similarities. UTS #39 describes confusable detection and identifier restriction strategies that combine script and character-property information.

At the same time, rejecting all mixed-script strings is inappropriate for many legitimate texts and identifiers: Latin names embedded in Japanese writing and shared mathematical symbols are routine. Security policies must be calibrated to their specific field (domain name, user handle, programming-language symbol, search term, or free-form text), not copied wholesale from a single regex.

• Data validation: Restrict scripts for a narrowly defined identifier profile if warranted; allow multilingual prose as its own class.
• Display: Preserve original code points while optionally annotating suspicious confusable identifiers in sensitive contexts.
• Comparison: Do not substitute case folding or normalization for a full confusable skeleton policy.
• Auditing: Include mixed-script tests with legitimate examples so security measures do not disproportionately break normal international text.

Diagnostic test cases

Version-specific Script_Extensions sets should be tested against the canonical UCD data files rather than approximated from appearance or letter names.

Standards and source files

1. UAX #24 — Unicode Script Property (Unicode 18.0.0), normative treatment of Script and Script_Extensions.
2. UAX #44 — Unicode Character Database, UCD files and versioning conventions.
3. UCD Scripts.txt, primary Script assignments.
4. UCD ScriptExtensions.txt, nondefault set-valued assignments.
5. UTS #39 — Unicode Security Mechanisms, script usage in identifier and spoofing security.
6. The Unicode Standard 18.0.0, core text properties and definitions.

Independent technical interpretation based on published Unicode standard annexes and property datasets. Implementations should verify the supported UCD version.