Unicode Combining Marks and Diacritics: Ordering, Rendering, and Normalization

Combining marks, diacritics, and base characters

Unicode represents many writing-system distinctions using combining marks: code points that modify, annotate, enclose, or otherwise combine with nearby text. Familiar examples include accents, vowel signs and script-specific marks, but the concept is broader than Latin diacritics. A combining mark is encoded as textual data, not as a positioning instruction to “draw a tiny symbol above the previous glyph.” Its rendering depends on the script, font, shaping engine, base character and surrounding text.

A diacritic is a linguistic or typographic term for a distinguishing mark, often applied to a letter. Not every Unicode combining mark is an accent, and not every visible accent-like symbol is encoded as a combining mark: spacing modifier letters and symbols can also look like diacritics. Therefore the questions “is this a combining mark?” and “is this a diacritic?” require different criteria.

Unicode supports both precomposed characters and decomposed sequences. For example, U+00E9 LATIN SMALL LETTER E WITH ACUTE represents é as one code point. The sequence U+0065 U+0301 represents a base e followed by COMBINING ACUTE ACCENT. These sequences are canonically equivalent under Unicode normalization and often look identical, even though they contain different numbers of code points and produce different bytes in UTF-8.

General categories and canonical combining classes

The General_Category property groups marks into three atomic classes. Each class describes broad character behavior but does not fully determine how a mark is visually drawn:

The canonical combining class (Canonical_Combining_Class, abbreviated ccc) is a different property: an integer that governs canonical ordering of certain non-starter marks in Unicode normalization. Common values include 230 for COMBINING ACUTE ACCENT (U+0301) and 220 for COMBINING DOT BELOW (U+0323). The value 0 frequently indicates a starter or mark without reordering class, but ccc=0 does not imply “not a mark”. Some characters in Mn or Mc have canonical combining class zero.

Canonical combining class is not a universal stacking index and should not be used to place glyphs physically above or below letters. Two marks can share a class, and the standard’s stable-ordering rules do not promise that their final visual positions follow the numeric order of those classes. Rendering engines instead use font metrics, mark-to-base and mark-to-mark attachment, substitutions and complex-script layout.

Canonical equivalence, NFC, NFD, and mark ordering

Unicode Standard Annex #15 defines four normalization forms. NFD recursively applies canonical decomposition and canonical ordering. NFC starts with canonical decomposition and reordering, then composes eligible sequences. NFKD and NFKC additionally apply compatibility mappings, which can erase formatting distinctions such as certain mathematical styled alphanumerics and compatibility glyphs.

Canonical ordering requires that eligible marks be placed in a standard nondecreasing order of canonical combining class within the applicable sequence. For instance, q U+0307 U+0323 (q, DOT ABOVE class 230, DOT BELOW class 220) becomes q U+0323 U+0307 under NFD. This is a structural equivalence rule; it does not mean the user typed an incorrect sequence or that all visually alike combinations are canonically equivalent.

const a = "\u00E9";            // precomposed e-acute
const b = "e\u0301";           // base e + combining acute
console.log(a === b);         // false
console.log(a.normalize("NFD") === b.normalize("NFD"));
// true

const unordered = "q\u0307\u0323";
const ordered = unordered.normalize("NFD");
console.log([...ordered].map(c =>
  "U+" + c.codePointAt(0).toString(16).toUpperCase()));
// ["U+71", "U+323", "U+307"]

console.log("e\u0301".normalize("NFC") === "\u00E9");
// true

Canonical composition is subject to defined composition exclusions and blocking rules; NFC does not necessarily turn every base-plus-mark sequence into one precomposed code point. It also does not generally erase legitimate distinctions, whereas compatibility normalization deliberately maps some distinct presentations to common forms. The appropriate form depends on the data contract, not on which output is shorter.

The character U+034F COMBINING GRAPHEME JOINER has canonical combining class zero and can be used in specialized text processing to affect canonical ordering and grapheme behavior. It should not be inserted casually as an invisible visual fix: invisible control-like marks can complicate equality, searches and interoperability.

Grapheme clusters, font anchors, and dotted circles

A base letter followed by one or more combining marks typically belongs to one extended grapheme cluster, according to UAX #29. That is why a cursor may move past e U+0301 as one unit. However, “one grapheme cluster” does not imply a single code point or a single glyph. The renderer may produce multiple positioned glyphs or a substituted composite glyph.

When a combining mark appears without a suitable base, some fonts or editors display a dotted circle to make the intended attachment position visible. The dotted circle can be a renderer-inserted cue and should not be assumed to be an actual U+25CC code point in the stored text. Conversely, if a document literally contains U+25CC, that is textual data.

Positioning can fail even if both characters are supported individually by fonts. A fallback font might contain the base letter but not a compatible anchor for the mark; a shaping engine can choose separate fallback fonts; or a script’s font features can be incomplete. Resulting detached accents or overlapping shapes are rendering problems rather than proof of invalid Unicode.

Many Indic scripts use combining vowel signs and other dependent marks whose visual positions do not correspond to the linear encoding order. Text processing must not “fix” their storage order according to where glyphs appear. Script shaping rules should receive the underlying logical character sequence.

Practical Python and JavaScript examples

Python’s unicodedata API exposes both the broad category and canonical combining class. Use both fields when investigating a mark, but do not infer complete display behavior from them.

import unicodedata as ud

for c in ["e", "\u0301", "\u0323", "\u034f", "\u093e"]:
    print(
        f"U+{ord(c):04X}",
        ud.name(c, "?"),
        "gc=" + ud.category(c),
        "ccc=" + str(ud.combining(c))
    )

assert ud.combining("\u0301") == 230
assert ud.combining("\u0323") == 220
assert ud.normalize("NFD", "\u00E9") == "e\u0301"
assert ud.normalize("NFD", "q\u0307\u0323") == "q\u0323\u0307"

The output property data come from the Unicode version bundled with Python, available as unicodedata.unidata_version. For JavaScript, a Unicode property escape tests General_Category=Mark without implementing canonical combining-class data:

const isMark = /^\p{Mark}$/u;
console.log(isMark.test("\u0301")); // true (Mn)
console.log(isMark.test("\u093E")); // true (Mc)
console.log(isMark.test("e"));      // false (Ll)

const graphemeSeg = new Intl.Segmenter("en", {
  granularity: "grapheme"
});
console.log([...graphemeSeg.segment("e\u0301")].length);
// 1 extended grapheme cluster

Neither Intl.Segmenter nor a regex over \p{Mark} is an accent-insensitive search engine. Segmentation detects boundaries; mark-category filtering may destroy meaningful distinctions and may miss spacing modifier characters. Search rules must be designed separately.

Context-sensitive problems and failure cases

• Nonspacing versus width: Mn does not guarantee the glyph occupies zero rasterized pixels. It describes character behavior, not the drawn glyph’s bounding box.
• Combining without a base: A leading mark is legal text data but may produce a degenerate grapheme cluster or a dotted-circle rendering. Validators should choose an application policy rather than silently deleting it.
• Same canonical class: Unicode normalization generally preserves relative ordering of equal-class marks; do not sort equal classes arbitrarily in a homemade normalization algorithm.
• Interaction with ZWJ and variation selectors: These affect shaping or presentation in appropriate sequences and should not be removed simply because they are invisible or categorized as marks/format characters.
• Copy and paste across programs: An editor may preserve decomposed text, normalize to NFC, or transform formatting. Compare canonicalized sequences only when the application’s equivalence policy demands it.
• Precomposed forms: The mere existence of a visually sensible base-plus-accent sequence does not imply Unicode defines a precomposed scalar for it.
• Language-specific meaning: A mark can distinguish words or grammatical structures. Accents are often semantically essential, not optional decoration.

Detailed diagnostics and Unicode test vectors

Conformance testing should also use the version-specific NormalizationTest.txt and GraphemeBreakTest.txt datasets, not only a few handpicked European-language examples. Such files include cases intended to expose errors in canonical ordering, boundary handling and special classes.

Search normalization and security considerations

A simplistic “remove all accents” approach often decomposes with NFD, removes every Mn, then optionally recomposes. This can be useful for a carefully scoped, secondary search key, but it is not a universally safe normalization of user data. It changes spelling and can merge strings that users or applications need to keep distinct. It may also fail to remove modifier symbols, affect scripts unevenly, and mishandle precomposed forms if the chosen normalization is incomplete.

For reliable systems, preserve the original input, store a documented normalization form if the protocol requires one, and derive optional search-folded indexes separately. Avoid modifying passwords, signatures, code identifiers, domain names or user-visible names through unreviewed accent stripping. In security-sensitive identifier processing, use an established Unicode profile and documented confusable-handling rules rather than heuristic mark deletion.

Standards and reference datasets

1. UAX #15 — Unicode Normalization Forms, decomposition, composition, canonical order and exclusions.
2. UAX #29 — Unicode Text Segmentation, extended grapheme cluster boundaries.
3. UAX #44 — Unicode Character Database, General_Category and canonical combining class data.
4. NormalizationTest.txt, normative conformance data.
5. GraphemeBreakTest.txt, segmentation test data.
6. The Unicode Standard 18.0.0, the core character model.

Independent technical guide grounded in Unicode annexes and data. Font rendering differs by font and platform; Unicode data versions may differ across runtimes.