Unicode Regression Test Cases: Normalization, Grapheme, Bidi, Emoji and Encoding Fixtures

A Unicode-aware application needs more than a dozen visually unusual strings. Regression coverage must target the contract of each subsystem: encoded bytes, normalization form, grapheme segmentation, bidi ordering, emoji sequence recognition, locale-sensitive operations, security validation and accessible reading. Versioned official Unicode test files help make algorithmic assertions precise and repeatable.

Divide the corpus by subsystem, not by appearance

Encoding tests focus on valid and malformed UTF-8 sequences; normalization tests compare exact expected scalar arrays; grapheme tests assert boundary positions; bidi tests evaluate levels or display ordering; font tests assess shaping and fallback.

Putting all cases into one screenshot test hides the cause of failures. A good suite names the expected property, input bytes or scalars, chosen Unicode version and actual observation point.

Use official normalization conformance data

The Unicode Character Database publishes NormalizationTest.txt with positive and negative cases for NFC, NFD, NFKC and NFKD, plus invariants relating them. A bespoke handful of accented strings is helpful for onboarding but cannot establish full algorithm conformance.

A test framework should import the official file matching the implementation's Unicode data version. Record whether the application normalizes at ingestion, index creation or search comparison. Exact output comparisons must not be replaced by visual snapshots.

Use GraphemeBreakTest and WordBreakTest as separate suites

UAX #29 distributes test data for grapheme, word and sentence boundary rules. Fixtures should assert UTF-16 or scalar offsets as appropriate to the target API and include combining marks, CRLF, emoji ZWJ and regional-indicator pairs.

A boundary test cannot prove glyph coverage or typography. For edited text, add integration tests that apply Backspace, selection and copy/paste at those boundaries; these interactions can fail even when the segmentation algorithm passes.

Use LineBreakTest for line-break eligibility

UAX #14 LineBreakTest defines expected break opportunities under the default Unicode line-breaking algorithm. Higher-level CSS overflow-wrap and language tailoring may intentionally alter layout behavior.

Test algorithmic opportunities separately from actual browser wrapping positions that depend on font metrics and viewport. For CJK punctuation, include strict and loose line-breaking policies where the product exposes them.

Use BidiTest and BidiCharacterTest for UAX #9

Bidirectional algorithm tests record embedding levels and reordering expectations for paragraphs containing strong, weak, neutral and formatting characters. These vectors are more reliable than comparing one screenshot of Arabic and Latin text.

A browser's choice of font and line-break width can change painted positions while preserving algorithmic correctness. Keep the direction, paragraph boundaries, isolates and line wrapping constraints explicit.

Emoji tests need RGI data and grapheme tests

UTS #51 publishes emoji-test.txt and sequence data for recommended interchange. RGI status and grapheme boundaries are separate properties; a joined sequence can form one grapheme without belonging to the recommended emoji repertoire.

For compatibility testing, also run platform rendering tests on supported devices. Do not compare native emoji artwork pixel-for-pixel across OSes and call different graphics a data-corruption defect.

Test security-sensitive identifiers with false positives

UTS #39 provides confusables data and restriction mechanisms. Include mixed-script lookalikes, an entirely single-script non-Latin name, legitimate Japanese script combinations and invisible directional controls.

A security harness must assert both detections and allowed exceptions under the application's chosen profile. A Unicode validity pass is not evidence of safe account identity, while mixed-script text alone does not prove spoofing.

Include encoding and platform round trips

Test UTF-8/UTF-16 conversion, JSON escape handling, CSV quoted fields, HTML character references, clipboard text/plain and text/html and storage/retrieval. Preserve before/after exact scalar arrays and relevant MIME types.

Bad results should be classified as decoding failure, normalization change, HTML entity double-decoding, clipboard preference or font fallback. A screenshot may conceal a dropped variation selector or a newly inserted U+FFFD.

Record versions and make every fixture reproducible

Store Unicode version, ICU/CLDR version, browser and operating system, target fonts, normalization setting and expected input/output bytes. Include references to official source test files and automation commands in the suite.

When a runtime upgrade legitimately changes a dictionary word boundary or new emoji repertoire, review the expected output with its new data version. Avoid silently rewriting fixtures to make the CI badge green.

Prioritize product impact and cover negative paths

Focus integration tests on editing, copy/paste, character limits and identifiers that directly affect users. Include invalid byte sequences, unpaired surrogates, incomplete ZWJ sequences and unexpectedly long combining-mark chains.

A good regression report includes exact input, expected behavior, observed difference, involved algorithm and safe reproduction steps. This enables specific fixes without applying destructive generic 'Unicode cleanup' to valid content.

Reproducible technical example

This controlled specimen demonstrates a single subsystem, not a comprehensive conformance claim. Use exact Unicode scalar values, record language/runtime versions and distinguish byte decoding, normalized equality, layout rendering and application validation. A visual match does not guarantee a reversible text interchange.

import assert from 'node:assert/strict';
const cps=s=>[...s].map(c=>c.codePointAt(0));
assert.deepEqual(cps('e\u0301'.normalize('NFC')),[0xE9]);
assert.deepEqual(cps('👩\u200D💻'),[0x1F469,0x200D,0x1F4BB]);
assert.equal([...[...new Intl.Segmenter('en',{granularity:'grapheme'}).segment('🇺🇸')]].length,1);
// Supplement these vectors with the matching version of the
// official NormalizationTest.txt, GraphemeBreakTest.txt and BidiTest.txt.

For reproducible defect reports, provide the original encoded data, the first observable transformation, the expected result and the tested API versions. The same logical text can produce different font glyphs, and an accidental scalar replacement can be invisible when those glyphs look alike.

Interoperability and evidence matrix

Each row covers one independent contract. Use code-point equality when assessing text integrity, supported locale algorithms for boundary or sorting work and assistive technology feedback for reading comprehension. Avoid substituting one test for another.

Verification notes and regression acceptance

1. NormalizationTest.txt. Use UAX #15 as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: NFC/NFD/NFKC/NFKD invariants. Record false positives and separate presentation differences from changed data.
2. GraphemeBreakTest.txt. Use UAX #29 as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: cluster boundary conformance. Record false positives and separate presentation differences from changed data.
3. LineBreakTest.txt. Use UAX #14 as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: break opportunity rules. Record false positives and separate presentation differences from changed data.
4. BidiTest.txt. Use UAX #9 as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: embedding/reordering data. Record false positives and separate presentation differences from changed data.
5. emoji-test.txt. Use UTS #51 as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: versioned RGI repertoire. Record false positives and separate presentation differences from changed data.
6. confusables.txt. Use UTS #39 as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: spoofing review input, not final verdict. Record false positives and separate presentation differences from changed data.

Include canonical equivalence, supplementary emoji, combining sequences, directional controls and locale-sensitive cases where applicable. Keep all expected outputs versioned rather than changing them automatically during upgrades. For byte-to-text errors use explicit strict or replacement policies; for identity checks use the actual standard named by the product.

Handling platform differences

An updated Unicode data version can introduce new assignments and properties, while font and OS support may lag. Test the actual deployed environments and record differences as supported variation, expected tailoring or defects. Preserving original input is critical to distinguishing these outcomes.

Primary technical specifications and implementation references

The sources below provide authoritative normative algorithms and API documentation. Conformance to one specification does not imply that all aspects of encoding, rendering, identity, accessibility and security are correct without end-to-end tests.

• UAX #15 Normalization — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UAX #29 Text Segmentation — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UAX #14 Line Breaking — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UAX #9 Bidirectional Algorithm — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UTS #51 Unicode Emoji — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UTS #39 Unicode Security — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.

Related published research

Continue within the Text APIs & Implementation Patterns virtual silo or read closely related published Unicode technical references below.

• javascript intl segmenter
• unicode input validation
• unicode copy paste round trips
• unicode line breaking scripts

This research reference is designed to support practical diagnostics and verifiable implementation decisions rather than optimizing separate article keywords.