Mojibake: Causes, Byte-Level Diagnosis and Safe Recovery

Mojibake is a decoding mismatch, not a Unicode font

Mojibake is apparently nonsensical text produced when a byte sequence is decoded under a character encoding different from the one used to produce it, or when a later re-encoding preserves the corrupted interpretation. The encoded bytes and the decoded Unicode scalar sequence are different layers. A font can display the resulting wrong characters perfectly: replacing the font cannot reconstruct the intended bytes.

For example, U+00E9 LATIN SMALL LETTER E WITH ACUTE is encoded as C3 A9 in UTF-8. Interpreting those two bytes as Windows-1252 yields U+00C3 LATIN CAPITAL LETTER A WITH TILDE and U+00A9 COPYRIGHT SIGN: é. The symptom arises because the wrong decoder ran, not because UTF-8 has multiple meanings for U+00E9. An application that subsequently saves é as UTF-8 now has the bytes C3 83 C2 A9 and has persisted the damage.

Follow bytes from source to final display

A reliable diagnostic separates at least six stages: producer's Unicode string; producer's chosen encoding scheme; transmitted or stored octets; recipient's declared or assumed encoding; decoded scalar sequence; and rendered glyphs. Corruption can enter at one stage and be propagated faithfully by all later stages. A mismatch in an HTTP header, database connector, CSV import or legacy client is common. Other visible anomalies come from normalization or font fallback and require different treatment.

Capture raw bytes before they pass through any replacement decoder. If the source bytes still contain C3 A9, a strict UTF-8 decode can recover é. If the only copy is the text é, reverse-transcoding may be possible under a known mapping, but cannot be promised from appearance alone. Once bytes were replaced with U+FFFD or dropped, information may be irrecoverable.

Common signatures and their limits

A symptom such as é is suggestive but not conclusive. These are actual assigned Unicode characters; an author may have intended them. A one-step repair should be proposed only when the encoding history is supported by provenance, raw bytes or a controlled sample. Large datasets may contain sections produced through different encodings and need record-level or field-level inspection.

Reproduce and reverse a controlled error

In Python, the byte path can be demonstrated without relying on screenshots. The reverse operation works here because every corrupted character is mappable back into the same single-byte character set and the resulting byte sequence is valid UTF-8. It is not a general text-cleaning recipe.

original = "café"
encoded = original.encode("utf-8")
assert encoded.hex(" ") == "63 61 66 c3 a9"
bad_text = encoded.decode("cp1252")
assert bad_text == "café"
recovered = bad_text.encode("cp1252").decode("utf-8")
assert recovered == original
print(encoded.hex(" "), bad_text, recovered)

In a real incident, validate every repaired record and keep an untouched original snapshot. If someone has already applied .encode("utf-8") to the corrupted string, the bytes will differ from the original UTF-8 source. Repeating "UTF-8 decode" on a Unicode string is a category error: decoders consume bytes, while encoders produce bytes.

Double encoding and multiple accidental passes

Repeated mistakes form chains. For example: UTF-8 bytes C3 A9 are read as Windows-1252 to produce é; saving that Unicode string as UTF-8 creates C3 83 C2 A9. A subsequent wrong Windows-1252 decode of those bytes yields a still more distorted result. The number of passes and exact legacy charset matter; stripping "Ã" characters by regular expression is neither safe nor reversible.

Each proposed repair should be expressed as a concrete inverse of a known operation: identify the codec used to turn bytes into the mistaken characters, re-encode the exact characters using that codec, then decode using the intended codec. The inverse may fail with an encode error when the intermediate text includes characters outside that codec, or with a decode error when the reconstructed bytes are not well-formed.

ISO-8859-1, Windows-1252, and WHATWG labels

Windows-1252 and ISO-8859-1 coincide for many common Latin letters but differ in the C1 range 0x80..0x9F. The byte 0x80 represents a control under literal ISO-8859-1, but a euro sign under Windows-1252. Modern browsers following the WHATWG Encoding Standard intentionally map some labels historically written as ISO-8859-1 to Windows-1252 for web compatibility. Python's latin-1 codec, however, implements the true ISO-8859-1 byte mapping. This difference alone can make a web-browser reconstruction disagree with a backend script.

Therefore document the decoder implementation and label interpretation, not merely the declared word "Latin-1." Some bytes are undefined in Windows-1252, which means lossless back-conversion cannot be assumed for arbitrary UTF-8 sequences that were misread with that codec. A reverse transform that works for café may fail for an emoji.

The replacement character is evidence of data loss

U+FFFD REPLACEMENT CHARACTER often appears when a decoder encounters ill-formed input under a replacement policy. Once a decoder has emitted U+FFFD, the original offending bytes are generally not recoverable from that character alone. A font's missing-glyph box is visually different in origin: the underlying scalar might remain correct, with only presentation unsupported.

Compare byte dumps, decoding exceptions and scalar values to distinguish byte damage, lossy recovery, normalization, and glyph absence. An automatic repair that silently removes U+FFFD can make a database superficially cleaner while permanently concealing the corruption.

A byte-first investigation with decision points

Begin by preserving a raw export and computing a checksum. For each suspect field, inspect its octets in hex, the protocol's charset declaration, and the first decoding operation. Attempt strict decoding with the documented codec. If it succeeds, compare scalar values, not only rendered output. Test an independent byte-level fixture containing ASCII, an accented Latin character, a CJK character and a supplementary scalar to expose mismatched behavior.

If a field already contains wrong Unicode text, compare alternative hypotheses against known originals and record whether the candidate round trip is invertible. Mixed-encoding datasets should not be processed by one blind global substitution. If the source was UTF-8 but a downstream parser folded compatibility characters, the problem is normalization rather than mojibake; if valid code points merely show tofu, the problem is font coverage.

JavaScript and browser debugging

In JavaScript, TextDecoder explicitly selects a decoding label, and fatal: true rejects invalid sequences rather than replacing them. The browser's TextEncoder emits UTF-8 bytes from a JavaScript string. Use both operations to reconstruct controlled cases; avoid relying on implicit document encodings.

const data = new TextEncoder().encode("café");
const utf8 = new TextDecoder("utf-8", {fatal:true}).decode(data);
const legacy = new TextDecoder("windows-1252").decode(data);
console.log(utf8);   // café
console.log(legacy); // café
console.log(Array.from(data, b => b.toString(16).padStart(2, "0")));

For text generators, the US Fancy Text tool can supply known styled Unicode sequences to compare before and after a copy-paste or CSV transport. A mismatch must still be classified by bytes, normalization and glyph coverage instead of attributing every altered character to mojibake.

Recovery acceptance tests

Standards and technical references

1. WHATWG Encoding Standard — encoding labels and decoding
2. RFC 3629 — UTF-8 well-formedness
3. Unicode Standard Chapter 3 — Encoding Forms
4. MDN — TextDecoder and fatal decoding
5. Python codecs reference

Independent technical research. Use strict byte-level evidence to diagnose encoding defects before transforming data.