Python unicodedata: Unicode Properties, Normalization, Names and Safe Text Handling

Python's standard-library unicodedata module exposes character-property information and Unicode normalization. It is an excellent diagnostic tool for inspecting scalar values and canonical equivalence, but it is not a complete grapheme segmenter, confusable detector, renderer or internationalized-domain validator. A precise implementation should state which Unicode database version it uses and which transformations it performs.

Query Unicode name and category for diagnostics

unicodedata.name() returns the standard name of an encoded character when available; category() returns a two-letter general category such as Lu, Ll, Mn or Cf. A character's category helps interpret data, but does not determine whether a symbol is dangerous, visible or suitable for a given username.

Names may not exist for some code points, and callers can provide a default value to avoid an exception. Use property output as a debugging aid alongside the actual hexadecimal code point, rather than replacing user text with names in storage.

Inspect combining classes and decompositions

unicodedata.combining() exposes a canonical combining class useful for explaining how combining marks can be reordered under canonical normalization. decomposition() returns a textual description of a character's decomposition mapping, which may include compatibility tags.

For U+00E9 LATIN SMALL LETTER E WITH ACUTE, NFD produces U+0065 and U+0301. Combining class zero is not a guarantee of visible width or stand-alone glyph appearance; it simply represents a specific property relevant to normalization.

Choose NFC, NFD, NFKC or NFKD deliberately

unicodedata.normalize('NFC', text) applies canonical composition, while NFD decomposes. NFKC and NFKD additionally apply compatibility mappings and may change mathematical styled letters to ordinary Latin, among many other mappings.

A Python program must not silently apply NFKC to user-generated fancy text or mathematical notation if exact scalar identity is promised. Normalize for well-defined equality or indexing purposes, and retain the original if a reversible audit trail is needed.

Understand Python's len and UTF-8 encoding metrics

Python 3 str indexing and len() operate on Unicode code points in the normal scalar-based representation of valid decoded text, unlike JavaScript's UTF-16 code-unit string length. The UTF-8 byte length is computed with len(text.encode('utf-8')).

Neither count identifies extended grapheme clusters. The string 'e' plus U+0301 has length two in Python, even when displayed as a single accented letter. Avoid using len as a visual-character limit in an editor or social post.

Surrogates and malformed input still require handling

Python strings can contain surrogate code points if created explicitly or through special decoding/error pathways. Default UTF-8 encoding rejects such unpaired surrogate code points. Decoding errors can be strict, replace or handled by a deliberately chosen error handler.

Do not claim a Python string is always a valid Unicode scalar sequence merely because it is a str object. Validate at data boundaries, and document whether malformed input is rejected, replaced or retained for lossless filesystem interchange using an appropriate special-purpose handler.

Bidirectional classes describe text behavior, not order changes

unicodedata.bidirectional() returns a bidi class such as L, R, AL or EN for an input character. Those are raw properties used by the Unicode Bidirectional Algorithm. The property value alone does not compute the final visual run order of a paragraph.

A full bidi implementation needs context, embedding levels, isolates and line reordering according to UAX #9. Python's unicodedata module does not offer a one-call visual layout engine. Use a maintained layout/bidi system when presentation is required.

East Asian width is a property with context

unicodedata.east_asian_width() returns properties such as F, W, H, Na, N or A; these can inform terminal-cell width estimates in some environments. Their interpretation is not identical to actual glyph pixel width or cursor advance in a browser.

Combining marks, emoji presentation and font substitution complicate display width. A terminal utility needs a deliberate width policy and may use versioned wcwidth-like tables. Do not assume that east_asian_width() returns a correct pixel measurement.

Unicode database versions affect reproducibility

unicodedata.unidata_version reports the version of the Unicode Character Database bundled with the running Python. unicodedata.ucd_3_2_0 provides a separate legacy database intended for applications such as IDNA2003 compatibility.

Do not select the legacy database as a general solution for modern text. When comparing results across CI, desktop or server Python versions, log unidata_version and pin test expectations appropriately so changes in assigned characters and properties are understood.

Build a cross-language property test harness

Prepare fixtures for U+00E9, U+0301, mathematical bold A, emoji U+1F600 and format controls. Emit code-point values, names, categories, combining classes, NFC/NFKC outputs and UTF-8 byte lengths.

Compare Python with JavaScript and ICU only for properties or algorithms genuinely intended to match under equivalent data versions. If a browser draws a different glyph, unicodedata alone cannot prove which font caused the difference.

Reproducible technical example

This controlled specimen demonstrates a single subsystem, not a comprehensive conformance claim. Use exact Unicode scalar values, record language/runtime versions and distinguish byte decoding, normalized equality, layout rendering and application validation. A visual match does not guarantee a reversible text interchange.

import unicodedata as ud
samples = ['é', 'e\u0301', '𝐀', '😀', '\u200d']
for text in samples:
    print([f'U+{ord(ch):04X}' for ch in text], len(text),len(text.encode('utf-8')))
    for ch in text:
        print(ud.name(ch,'<unnamed>'),ud.category(ch),ud.combining(ch))
    print('NFC',ud.normalize('NFC',text),'NFKC',ud.normalize('NFKC',text))
print('Unicode database:',ud.unidata_version)

For reproducible defect reports, provide the original encoded data, the first observable transformation, the expected result and the tested API versions. The same logical text can produce different font glyphs, and an accidental scalar replacement can be invisible when those glyphs look alike.

Interoperability and evidence matrix

Each row covers one independent contract. Use code-point equality when assessing text integrity, supported locale algorithms for boundary or sorting work and assistive technology feedback for reading comprehension. Avoid substituting one test for another.

Verification notes and regression acceptance

1. Name. Use unicodedata.name as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: scalar identification. Record false positives and separate presentation differences from changed data.
2. Category. Use unicodedata.category as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: property, not a safety verdict. Record false positives and separate presentation differences from changed data.
3. NFC. Use unicodedata.normalize as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: canonical composition. Record false positives and separate presentation differences from changed data.
4. NFKC. Use compatibility normalization as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: may change mathematical alphabets. Record false positives and separate presentation differences from changed data.
5. Code-point count. Use len(text) as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: not visible grapheme count. Record false positives and separate presentation differences from changed data.
6. UTF-8 bytes. Use len(text.encode('utf-8')) as a starting fixture. Define exact input code points, the expected transformation or boundary and the tested runtime or Unicode data version. Correct interpretation: transport/storage metric. Record false positives and separate presentation differences from changed data.

Include canonical equivalence, supplementary emoji, combining sequences, directional controls and locale-sensitive cases where applicable. Keep all expected outputs versioned rather than changing them automatically during upgrades. For byte-to-text errors use explicit strict or replacement policies; for identity checks use the actual standard named by the product.

Handling platform differences

An updated Unicode data version can introduce new assignments and properties, while font and OS support may lag. Test the actual deployed environments and record differences as supported variation, expected tailoring or defects. Preserving original input is critical to distinguishing these outcomes.

Primary technical specifications and implementation references

The sources below provide authoritative normative algorithms and API documentation. Conformance to one specification does not imply that all aspects of encoding, rendering, identity, accessibility and security are correct without end-to-end tests.

• Python unicodedata Documentation — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UAX #44 Unicode Character Database — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UAX #15 Unicode Normalization — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UAX #9 Bidirectional Algorithm — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.
• UAX #29 Text Segmentation — authoritative technical resource. Verify the selected revision, data version and specific API behavior for the target environment before making universal compatibility claims.

Related published research

Continue within the Text APIs & Implementation Patterns virtual silo or read closely related published Unicode technical references below.

• javascript strings code points
• unicode normalization nfc nfd
• unicode normalization nfkc nfkd
• mixed script text processing

This research reference is designed to support practical diagnostics and verifiable implementation decisions rather than optimizing separate article keywords.