ToolVault

é”±, 锟斤拷, %E4%B8%AD%E6%96%87, \u4e2d\u6587.

Most developers have met all four — but they have completely different causes, and completely different fixes. Worse, two of them (percent-encoding and Unicode escaping) aren't errors at all; they're just normal data representations that people routinely try to "fix" as bugs.

This handbook sorts them by symptom: first decide whether something is actually broken, then jump to the fix.

Locate your symptom first

| What you see | Is it a bug? | Cause | Jump to | |---|---|---|---| | Chinese becomes ???? | ✅ Yes | Written with an encoding that can't represent Chinese | Type 1 | | Chinese becomes é”± / å¼ ä¸ | ✅ Yes | UTF-8 bytes read as GBK/Latin-1 | Type 1 | | 锟斤拷 / æ–‡å— | ✅ Yes (often irreversible) | Converted back and forth between encodings | Type 1 | | Chinese becomes %E4%B8%AD%E6%96%87 | ❌ No | URL percent-encoding — legal | Type 2 | | Chinese becomes \u4e2d\u6587 | ❌ No | Unicode escaping — legal | Type 3 | | Base64 decodes to garbled Chinese | ✅ Yes | Charset mismatch before/after encoding | Type 4 | | Space becomes + or %20 | ❌ No | URL encoding convention for spaces | Type 2 |

One-line rule: if re-reading the bytes with a different encoding restores the original text, it's an encoding problem. If the content is an encoded representation (percent signs, \u), it's fine — just decode it.

Encoding primer (needed to read mojibake)

Three concepts get conflated constantly. They do different jobs:

| Concept | What it is | Analogy | |---|---|---| | Unicode | A character set — every character gets a number ("中" = U+4E2D) | The words in a dictionary, with IDs | | UTF-8 | An encoding — turns numbers into bytes ("中" → E4 B8 AD) | Writing the ID in Morse code | | GBK | A different encoding — 2 bytes for Chinese | A different Morse code |

Mojibake in one sentence: bytes encoded with rule A were decoded with rule B. The bytes are fine; the interpretation is wrong.

"中" --UTF-8--> E4 B8 AD --read as GBK--> "涓" (mojibake)

⚠️ Common misconception: "mojibake means the data is corrupted." Usually the bytes are perfectly intact — only the interpretation is wrong, which means most mojibake is recoverable.

For the full picture of how ASCII, Unicode, and UTF-8 relate, see ASCII vs Unicode vs UTF-8 explained.

Type 1: Real mojibake (encoding mismatch)

Symptom table

| Garbled form | Cause | Recoverable? | |---|---|---| | ???? | Written as ASCII/Latin-1, which can't represent Chinese — replaced with ? | ❌ No — original bytes are gone | | é”± | UTF-8 bytes read as Latin-1 (ISO-8859-1) | ✅ Yes | | å¼ ä¸ | UTF-8 bytes read as Latin-1, then stored as UTF-8 | ✅ Yes | | 涓枃 | UTF-8 bytes read as GBK | ✅ Yes | | 锟斤拷 | Round-tripped UTF-8 → GBK → back; EF BF BD replacement chars appeared | ❌ No |

Key takeaway:

• Seeing ? or 锟斤拷 means data is already lost — go back to the source and re-export.
• Other forms usually mean the bytes are intact, so re-reading with the correct encoding recovers the text.

How to fix

# 1. Detect the real encoding (don't guess)
file -I data.csv
# or
enca -L zh_CN data.csv

# 2. Convert (GBK -> UTF-8)
iconv -f GBK -t UTF-8 data.csv > data.utf8.csv

# 3. If conversion fails, some chars can't be mapped; //IGNORE skips them (loses data)
iconv -f GBK -t UTF-8//IGNORE data.csv > data.utf8.csv

# Read with the correct encoding
with open('data.csv', encoding='gbk') as f:      # key: be explicit
    content = f.read()

# Already mis-decoded (mojibake): reverse the damage
broken = 'æ–‡å—'
fixed = broken.encode('latin-1').decode('utf-8')  # -> readable text

Preventing it

Golden rule: UTF-8 everywhere, and always state the encoding explicitly — never rely on system defaults.

| Layer | What to do | |---|---| | Files | UTF-8 (preferably without BOM) | | Database | utf8mb4 on database, table, and connection | | HTTP | Content-Type: text/html; charset=utf-8 | | HTML | <meta charset="utf-8"> | | Editor | Save as UTF-8; turn off "auto-detect encoding" |

MySQL gotcha: utf8 is not real UTF-8 (3 bytes max, can't store emoji). Always use utf8mb4.

Type 2: URL percent-encoding (not mojibake)

Chinese appearing as %E4%B8%AD%E6%96%87 in a URL is percent-encoding — standard behavior, not an error.

• 中 in UTF-8 is E4 B8 AD → encoded as %E4%B8%AD
• One Chinese character = 3 bytes = 3 %XX groups

When to handle it manually

| Situation | Action | |---|---| | Building URL parameters | Encode them, or Chinese/special chars break the URL | | Want to read %XX | Just decode it | | Space as + vs %20 | Both work in query strings; + is a legacy convention |

encodeURIComponent('中文')      // '%E4%B8%AD%E6%96%87'
decodeURIComponent('%E4%B8%AD') // '中'

// encodeURI vs encodeURIComponent:
encodeURI('https://a.com/中文')           // keeps : / ? — for a whole URL
encodeURIComponent('https://a.com/中文')  // encodes everything — for a param value

from urllib.parse import quote, unquote
quote('中文')                 # '%E4%B8%AD%E6%96%87'
unquote('%E4%B8%AD%E6%96%87') # '中文'

Use the URL encoder/decoder to see exactly how each character is encoded.

👉 Full details: Chinese URL encoding — what the percent signs mean.

Type 3: Unicode escaping (not mojibake)

\u4e2d\u6587 is a Unicode escape — legal in JSON, Java .properties, and JS strings. It decodes to readable text.

| Where you see it | Meaning | |---|---| | In JSON | Legal — the JSON spec allows it | | Java .properties | Default format (ISO-8859-1 legacy) | | Python serialized output | Because ensure_ascii=True is the default |

If you want literal characters

json.dumps(data, ensure_ascii=False)   # disable escaping

JSON.stringify({name: '中文'})  // JS emits readable text by default

# Convert Java properties to UTF-8
native2ascii -reverse -encoding UTF-8 app.properties app.utf8.properties

👉 For JSON, see Chinese turning into \uXXXX escapes.

Type 4: Base64 and Chinese

Base64 handles bytes only — it knows nothing about character sets. So every Chinese issue happens before or after the Base64 step, when text converts to bytes and back.

中文 --(UTF-8)--> bytes --(Base64)--> 5Lit5paH

Why it breaks: one side encoded with UTF-8, the other decoded bytes back using GBK.

Doing it right

// Encode: explicitly go to UTF-8 bytes first
const bytes = new TextEncoder().encode('中文');
const b64 = btoa(String.fromCharCode(...bytes));

// Decode: Base64 -> bytes -> interpret as UTF-8
const bin = atob(b64);
const bytes2 = Uint8Array.from(bin, c => c.charCodeAt(0));
new TextDecoder('utf-8').decode(bytes2);               // readable

import base64
base64.b64encode('中文'.encode('utf-8'))               # explicit UTF-8
base64.b64decode(token).decode('utf-8')                # explicit UTF-8

⚠️ btoa('中文') throws (Character Out Of Range) because it only handles Latin-1. Always convert to UTF-8 bytes first.

Verify quickly with the Base64 encoder/decoder.

👉 See Base64 decoding to garbled Chinese and the complete guide to Base64 with Chinese.

General debugging workflow

Four steps, in order:

1. Decide whether anything is actually broken %XX or \uXXXX? → not a bug, just decode. ? or 锟斤拷? → data is gone, re-export from source.

2. Inspect bytes, not the display Your terminal and editor each apply their own encoding. Look at raw bytes:

hexdump -C file.txt | head
xxd file.txt | head

"中" is e4 b8 ad in UTF-8 and d6 d0 in GBK — the bytes tell you the real encoding.

3. Re-interpret with the correct encoding (don't convert) If the bytes are intact, re-reading recovers the text. Converting (iconv) is for content that's already decoded correctly. Doing them in the wrong order causes further damage.

4. Standardize on UTF-8 and always be explicit Write encoding='utf-8' at every read/write boundary. Never trust defaults.

Troubleshooting toolkit

| Need | Tool | |---|---| | URL percent-encode / decode | URL encoder/decoder | | Base64 encode / decode | Base64 | | Unicode code points ↔ characters | Unicode encoder/decoder | | HTML entity encode / decode | HTML entity encoder | | Base32 / Base58 | Base32/Base58 encoder | | Diff two pieces of text | Text diff |

Every tool above runs locally in your browser — safe for sensitive content. Open the Network panel (F12) and verify it yourself.

FAQ

Q: Why is the same file fine on my machine but garbled on a colleague's? Different default encodings. Windows Notepad historically defaulted to GBK; Linux/macOS default to UTF-8. The fix is always to state the encoding explicitly and never depend on defaults.

Q: Can 锟斤拷 be recovered? Almost never. It comes from the EF BF BD replacement character introduced during a UTF-8 → GBK conversion; the original bytes were discarded. Re-export from the data source.

Q: Why doesn't Base64 know about Chinese? Because Base64 operates on bytes only — it can't tell whether you fed it text or an image. "Chinese" only exists before Base64 (characters → bytes) and after (bytes → characters).

Q: Should a space in a URL be + or %20? In a query string, servers generally accept both; + is a legacy convention. In the path portion you must use %20. Safest bet: use encodeURIComponent, which produces %20.

Q: MySQL throws Incorrect string value when storing emoji. You're using utf8 instead of utf8mb4. MySQL's utf8 holds 3 bytes max and cannot store 4-byte emoji. Switch database, table, and connection charset to utf8mb4.

Summary

The core skill is telling representation apart from corruption:

1. %XX, \uXXXX → an encoded representation; decode it, nothing is broken
2. ?, 锟斤拷 → data is already lost; go back to the source
3. Other mojibake → bytes are intact; re-read with the right encoding
4. Prevention → UTF-8 everywhere + always state the encoding explicitly

Next time you hit mojibake, match it against the table at the top instead of guessing your way through iconv.