Unicode Input Validation: Scalar Well-Formedness, Allowed Repertoires and Security Profiles

Unicode input validation is a series of decisions at different layers. Byte decoding establishes that input can represent a Unicode scalar sequence; normalization decides whether equivalent spellings are accepted as the same value; field policy restricts which sequences are allowed; and security review may flag confusables or unusual formatting controls. Treating everything outside ASCII as invalid conflates these concerns and excludes legitimate languages.

Decode encoded input using a defined error policy

UTF-8 rejects certain byte sequences, including overlong encodings and invalid scalar representations. A strict decoder can reject malformed byte input; a tolerant decoder may replace failures with U+FFFD. That replacement is a lossy data transformation, not evidence that the original bytes were valid.

Log encoding errors separately from application policy rejections. A string containing a valid non-ASCII scalar should not be rejected merely because a byte parser was written for ASCII. Preserve raw bytes for permitted diagnostics but avoid unnecessary sensitive-data retention.

Unicode scalar validity differs from permitted repertoire

A Unicode scalar value excludes the surrogate range U+D800–U+DFFF and values above U+10FFFF. JavaScript strings can still contain unpaired UTF-16 surrogates, so a boundary accepting JavaScript input may need an explicit well-formedness check.

A valid scalar value may be disallowed in a constrained identifier field for reasons of policy. Keep errors distinct: 'malformed UTF-16' is different from 'character not permitted in a username' and from 'visually confusable name'.

Define a profile per field type

A natural-language comment field should accept broad Unicode; a programming identifier may use UAX #31 XID properties; a domain label follows IDNA; a payment reference might require a restricted repertoire. Do not substitute one profile for every product surface.

Store the policy version and provide user-facing feedback explaining what is permitted. Where input is corrected or transformed, describe exactly which characters or normalization rules changed it; otherwise users cannot reproduce the result.

Normalization and comparisons need a stable policy

NFC makes canonically equivalent forms compare identically when applied to both sides. NFKC can change mathematical styled letters and some compatibility symbols, sometimes producing new collisions. Case folding is another distinct operation.

Do not apply NFKC as a supposedly harmless filter to a Fancy Text output intended for copying. For account handles, create a declared comparison key and check uniqueness after the chosen transformations without rewriting the visible original unintentionally.

Reject contextually prohibited controls, not every format scalar

U+200C and U+200D can be important in legitimate writing systems and emoji. U+202E and bidi overrides can be risky in filenames or high-value identifiers. Variation selectors can be needed for standard emoji presentation.

Use contextual rules and UTS #39/UAX #31 guidance where relevant, and make invisible controls inspectable in security-sensitive fields. A blanket strip of all category Cf code points destroys some valid multilingual data.

Confusable analysis is separate from permitted-character syntax

An identifier can pass an XID syntax rule and still resemble a protected name. UTS #39 provides confusable skeleton and script-restriction concepts, but they are risk signals that need a field-specific policy and collision targets.

A skeleton is not a canonical display string or a general transliteration. Keep original code points and the result of the risk analysis in separate places. Review legitimate collisions before enforcing account-level penalties.

Validate length using an explicit unit

For a REST request quota, encoded byte length may be relevant; JavaScript length counts UTF-16 units; the user interface may prefer grapheme clusters for visible text limits. Each metric can disagree for emoji, combining accents and multi-script content.

An input validator should name the metric and avoid truncating or slicing at an unsafe boundary. A server-side byte limit cannot be reliably enforced by an unrelated client-side grapheme count alone.

Prevent unsafe rendering at the output boundary

A validated Unicode string containing the characters '<' or '&' can still be dangerous if inserted as executable HTML. Use context-appropriate escaping or DOM textContent for untrusted content. Unicode validation is not an XSS sanitizer.

Similarly, HTML escaping does not prevent confusable name spoofing. Implement output encoding, identifier security and user verification as separate defenses with their own tests.

Build a clear decision result and audit trail

Instead of returning a bare boolean, a validation function can report well-formedness, normalized comparison value (when relevant), rejected scalars, contextual-control findings, length metrics, confusable alerts and policy version.

Use non-destructive previews and avoid automatically rewriting submitted text. Tests should include both valid Arabic or Japanese sentences and constrained machine identifiers so the system can distinguish legitimate multilingual input from actual field violations.

Reproducible technical example

This controlled specimen demonstrates a single subsystem, not a comprehensive conformance claim. Use exact Unicode scalar values, record language/runtime versions and distinguish byte decoding, normalized equality, layout rendering and application validation. A visual match does not guarantee a reversible text interchange.

const value='A👩\u200D💻';
const wellFormed=typeof value.isWellFormed==='function'?value.isWellFormed():true;
console.log({wellFormed,units:value.length,bytes:new TextEncoder().encode(value).length});
const identifier=/^\p{XID_Start}\p{XID_Continue}*$/u;
console.log(identifier.test('alpha')); // syntax-only illustration
// Enforce policy on the server, including context, normalization and collisions.
// Never substitute this regex for UTS #39 or for HTML sanitization.

For reproducible defect reports, provide the original encoded data, the first observable transformation, the expected result and the tested API versions. The same logical text can produce different font glyphs, and an accidental scalar replacement can be invisible when those glyphs look alike.

Interoperability and evidence matrix

Each row covers one independent contract. Use code-point equality when assessing text integrity, supported locale algorithms for boundary or sorting work and assistive technology feedback for reading comprehension. Avoid substituting one test for another.

Verification notes and regression acceptance

1. UTF-8 decoding. Use strict vs replacement as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: determines lossless bytes-to-scalars conversion. Record false positives and separate presentation differences from changed data.
2. Scalar well-formedness. Use unpaired surrogate as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: not a valid Unicode scalar. Record false positives and separate presentation differences from changed data.
3. Identifier syntax. Use XID_Start/XID_Continue as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: field-specific starting point. Record false positives and separate presentation differences from changed data.
4. Normalization. Use NFC or NFKC as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: declared equality transformation. Record false positives and separate presentation differences from changed data.
5. Confusable check. Use UTS #39 as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: review signal, not exact identity. Record false positives and separate presentation differences from changed data.
6. Output escape. Use textContent as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: protects HTML boundary, not user-name similarity. Record false positives and separate presentation differences from changed data.

Include canonical equivalence, supplementary emoji, combining sequences, directional controls and locale-sensitive cases where applicable. Keep all expected outputs versioned rather than changing them automatically during upgrades. For byte-to-text errors use explicit strict or replacement policies; for identity checks use the actual standard named by the product.

Handling platform differences

An updated Unicode data version can introduce new assignments and properties, while font and OS support may lag. Test the actual deployed environments and record differences as supported variation, expected tailoring or defects. Preserving original input is critical to distinguishing these outcomes.

Primary technical specifications and implementation references

The sources below provide authoritative normative algorithms and API documentation. Conformance to one specification does not imply that all aspects of encoding, rendering, identity, accessibility and security are correct without end-to-end tests.

• Unicode Core Specification — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UAX #31 Unicode Identifiers — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UTS #39 Unicode Security — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UAX #15 Unicode Normalization — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• WHATWG Encoding Standard — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• WHATWG HTML Standard — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.

Related published research

Continue within the Text APIs & Implementation Patterns virtual silo or read closely related published Unicode technical references below.

• unicode identifier security profiles
• unicode confusables lookalikes
• displaying untrusted unicode safely
• unicode safe string truncation

This research reference is designed to support practical diagnostics and verifiable implementation decisions rather than optimizing separate article keywords.