Character Encoding Troubleshooting Handbook: Mojibake, URL Encoding, and Base64 Explained

é”±, 锟斤拷, %E4%B8%AD%E6%96%87, \u4e2d\u6587.

Most developers have met all four — but they have completely different causes, and completely different fixes. Worse, two of them (percent-encoding and Unicode escaping) aren't errors at all; they're just normal data representations that people routinely try to "fix" as bugs.

This handbook sorts them by symptom: first decide whether something is actually broken, then jump to the fix.

Locate your symptom first

One-line rule: if re-reading the bytes with a different encoding restores the original text, it's an encoding problem. If the content is an encoded representation (percent signs, \u), it's fine — just decode it.

Encoding primer (needed to read mojibake)

Three concepts get conflated constantly. They do different jobs:

Mojibake in one sentence: bytes encoded with rule A were decoded with rule B. The bytes are fine; the interpretation is wrong.

"中" --UTF-8--> E4 B8 AD --read as GBK--> "涓" (mojibake)

⚠️ Common misconception: "mojibake means the data is corrupted." Usually the bytes are perfectly intact — only the interpretation is wrong, which means most mojibake is recoverable.

For the full picture of how ASCII, Unicode, and UTF-8 relate, see ASCII vs Unicode vs UTF-8 explained.

Type 1: Real mojibake (encoding mismatch)

Symptom table

Key takeaway:

• Seeing ? or 锟斤拷 means data is already lost — go back to the source and re-export.
• Other forms usually mean the bytes are intact, so re-reading with the correct encoding recovers the text.

How to fix

# 1. Detect the real encoding (don't guess)
file -I data.csv
# or
enca -L zh_CN data.csv

# 2. Convert (GBK -> UTF-8)
iconv -f GBK -t UTF-8 data.csv > data.utf8.csv

# 3. If conversion fails, some chars can't be mapped; //IGNORE skips them (loses data)
iconv -f GBK -t UTF-8//IGNORE data.csv > data.utf8.csv

# Read with the correct encoding
with open('data.csv', encoding='gbk') as f:      # key: be explicit
    content = f.read()

# Already mis-decoded (mojibake): reverse the damage
broken = 'æ–‡å—'
fixed = broken.encode('latin-1').decode('utf-8')  # -> readable text

Preventing it

Golden rule: UTF-8 everywhere, and always state the encoding explicitly — never rely on system defaults.

MySQL gotcha: utf8 is not real UTF-8 (3 bytes max, can't store emoji). Always use utf8mb4.

Type 2: URL percent-encoding (not mojibake)

Chinese appearing as %E4%B8%AD%E6%96%87 in a URL is percent-encoding — standard behavior, not an error.

• 中 in UTF-8 is E4 B8 AD → encoded as %E4%B8%AD
• One Chinese character = 3 bytes = 3 %XX groups

When to handle it manually

encodeURIComponent('中文')      // '%E4%B8%AD%E6%96%87'
decodeURIComponent('%E4%B8%AD') // '中'

// encodeURI vs encodeURIComponent:
encodeURI('https://a.com/中文')           // keeps : / ? — for a whole URL
encodeURIComponent('https://a.com/中文')  // encodes everything — for a param value

from urllib.parse import quote, unquote
quote('中文')                 # '%E4%B8%AD%E6%96%87'
unquote('%E4%B8%AD%E6%96%87') # '中文'

Use the URL encoder/decoder to see exactly how each character is encoded.

👉 Full details: Chinese URL encoding — what the percent signs mean.

Type 3: Unicode escaping (not mojibake)

\u4e2d\u6587 is a Unicode escape — legal in JSON, Java .properties, and JS strings. It decodes to readable text.

If you want literal characters

json.dumps(data, ensure_ascii=False)   # disable escaping

JSON.stringify({name: '中文'})  // JS emits readable text by default

# Convert Java properties to UTF-8
native2ascii -reverse -encoding UTF-8 app.properties app.utf8.properties

👉 For JSON, see Chinese turning into \uXXXX escapes.

Type 4: Base64 and Chinese

Base64 handles bytes only — it knows nothing about character sets. So every Chinese issue happens before or after the Base64 step, when text converts to bytes and back.

中文 --(UTF-8)--> bytes --(Base64)--> 5Lit5paH

Why it breaks: one side encoded with UTF-8, the other decoded bytes back using GBK.

Doing it right

// Encode: explicitly go to UTF-8 bytes first
const bytes = new TextEncoder().encode('中文');
const b64 = btoa(String.fromCharCode(...bytes));

// Decode: Base64 -> bytes -> interpret as UTF-8
const bin = atob(b64);
const bytes2 = Uint8Array.from(bin, c => c.charCodeAt(0));
new TextDecoder('utf-8').decode(bytes2);               // readable

import base64
base64.b64encode('中文'.encode('utf-8'))               # explicit UTF-8
base64.b64decode(token).decode('utf-8')                # explicit UTF-8

⚠️ btoa('中文') throws (Character Out Of Range) because it only handles Latin-1. Always convert to UTF-8 bytes first.

Verify quickly with the Base64 encoder/decoder.

👉 See Base64 decoding to garbled Chinese and the complete guide to Base64 with Chinese.

General debugging workflow

Four steps, in order:

1. Decide whether anything is actually broken %XX or \uXXXX? → not a bug, just decode. ? or 锟斤拷? → data is gone, re-export from source.

2. Inspect bytes, not the display Your terminal and editor each apply their own encoding. Look at raw bytes:

hexdump -C file.txt | head
xxd file.txt | head

"中" is e4 b8 ad in UTF-8 and d6 d0 in GBK — the bytes tell you the real encoding.

3. Re-interpret with the correct encoding (don't convert) If the bytes are intact, re-reading recovers the text. Converting (iconv) is for content that's already decoded correctly. Doing them in the wrong order causes further damage.

4. Standardize on UTF-8 and always be explicit Write encoding='utf-8' at every read/write boundary. Never trust defaults.

Troubleshooting toolkit

Every tool above runs locally in your browser — safe for sensitive content. Open the Network panel (F12) and verify it yourself.

FAQ

Q: Why is the same file fine on my machine but garbled on a colleague's? Different default encodings. Windows Notepad historically defaulted to GBK; Linux/macOS default to UTF-8. The fix is always to state the encoding explicitly and never depend on defaults.

Q: Can 锟斤拷 be recovered? Almost never. It comes from the EF BF BD replacement character introduced during a UTF-8 → GBK conversion; the original bytes were discarded. Re-export from the data source.

Q: Why doesn't Base64 know about Chinese? Because Base64 operates on bytes only — it can't tell whether you fed it text or an image. "Chinese" only exists before Base64 (characters → bytes) and after (bytes → characters).

Q: Should a space in a URL be + or %20? In a query string, servers generally accept both; + is a legacy convention. In the path portion you must use %20. Safest bet: use encodeURIComponent, which produces %20.

Q: MySQL throws Incorrect string value when storing emoji. You're using utf8 instead of utf8mb4. MySQL's utf8 holds 3 bytes max and cannot store 4-byte emoji. Switch database, table, and connection charset to utf8mb4.

Summary

The core skill is telling representation apart from corruption:

1. %XX, \uXXXX → an encoded representation; decode it, nothing is broken
2. ?, 锟斤拷 → data is already lost; go back to the source
3. Other mojibake → bytes are intact; re-read with the right encoding
4. Prevention → UTF-8 everywhere + always state the encoding explicitly

Next time you hit mojibake, match it against the table at the top instead of guessing your way through iconv.

Triage notes

The symptom table at the top carries an "is it a bug" column because two of the four high-frequency patterns (percent-encoding and Unicode escapes) are not errors at all, yet get "fixed" as bugs — classify real versus apparent first, then talk remedies; that is the spine of the whole handbook. Reversibility splits true mojibake in half again: once ? or 锟斤拷 appears, the bytes are gone and only a re-export from the source helps, while the remaining forms can be rescued by re-reading them. Step 3 of the flow distinguishes "re-reading" from "converting encodings" deliberately: iconv is for content already correctly decoded, and running it in the wrong order damages the data a second time — which is why the summary says match the table first and stop iterating on iconv.