Online Unicode Encoder/Decoder: Handle Unicode Character Conversion

What Is Unicode

Unicode is an international standard that assigns a unique number (code point) to nearly every character in all writing systems. It solved compatibility issues with older encodings (ASCII, GB2312, Shift-JIS) — use an online unicode encoder/decoder to convert between these representations, or a base64 decoder when you need binary-to-text encoding instead.

Unicode Representation Forms

Why You Need Unicode Encoding/Decoding

1. Cross-language compatibility: Character transfer between different programming languages
2. JSON safety: Ensure special characters transmit correctly in JSON
3. Database storage: Handle character encoding issues in databases
4. URL encoding: Encode non-ASCII characters in URLs
5. Log analysis: Process log data containing Unicode characters
6. Internationalization: Handle encoding conversion for multi-language text

UTF-8 vs UTF-16

How to Use an Online Tool

Using ToolVault's Unicode Encoder/Decoder:

1. Paste source text on the left (CJK, emoji, and mixed scripts are fine)
2. Pick an encode format:
   • \uXXXX: UTF-16 code-unit escapes; astral characters become high+low surrogate pairs (same habit as JSON.stringify)
   • \u{XXXXX}: full code-point escapes (ES6 style, easy to read)
   • U+XXXX: standard code-point list for comparing against specs
   • \xHH: UTF-8 byte escapes for hex dumps and packet captures
3. Click Encode for escapes; Decode accepts a mix of the forms above (including surrogate pairs)
4. The header shows three lengths at once: UTF-16 units, code points, and UTF-8 bytes
5. The breakdown table lists each character's U+…, UTF-16 escapes, and UTF-8 bytes

Lone surrogates (a bare \uD83D with no low surrogate) or illegal UTF-8 bytes surface as clear errors instead of silently producing half a character.

Unicode Handling in Code

JavaScript

// Character → code points (iterate scalar values; do not walk charCodeAt alone)
function toUnicode(str) {
  return Array.from(str).map((char) => {
    const cp = char.codePointAt(0);
    return 'U+' + cp.toString(16).toUpperCase().padStart(cp > 0xffff ? 5 : 4, '0');
  });
}

// Code point → character
function fromUnicode(codePoint) {
  return String.fromCodePoint(parseInt(codePoint.replace(/^U\+/i, ''), 16));
}

// ES6 braced escapes (do not invent an illegal \\u1F600)
function toBracedEscapes(str) {
  return Array.from(str)
    .map((char) => {
      const cp = char.codePointAt(0);
      return '\\u{' + cp.toString(16).toUpperCase() + '}';
    })
    .join('');
}

// JSON-style: BMP as \\uXXXX; astral as a surrogate pair
function toSurrogateEscapes(str) {
  return Array.from(str)
    .map((char) => {
      const cp = char.codePointAt(0);
      if (cp <= 0xffff) {
        return '\\u' + cp.toString(16).toUpperCase().padStart(4, '0');
      }
      const adj = cp - 0x10000;
      const hi = 0xd800 + (adj >> 10);
      const lo = 0xdc00 + (adj & 0x3ff);
      return (
        '\\u' + hi.toString(16).toUpperCase().padStart(4, '0') +
        '\\u' + lo.toString(16).toUpperCase().padStart(4, '0')
      );
    })
    .join('');
}

Python

import unicodedata

# Character → Unicode code point
def to_unicode(char):
    return f"U+{ord(char):04X}"

# Unicode code point → character
def from_unicode(code_point):
    return chr(int(code_point.replace('U+', ''), 16))

# Get character information
def char_info(char):
    return {
        'char': char,
        'code_point': f"U+{ord(char):04X}",
        'name': unicodedata.name(char, 'UNKNOWN'),
        'category': unicodedata.category(char)
    }

Common Unicode Ranges

The three encoding forms and why UTF-8 won the web

UTF-8 (variable 1-4 bytes, ASCII-transparent), UTF-16 (2 or 4 bytes, what JavaScript/Java/C# strings use internally), UTF-32 (fixed 4 bytes, rare outside special cases). UTF-8 won the web because it is backwards-compatible with ASCII (English text is byte-identical, no migration cost for the 1990s internet) and self-synchronizing (a parser can find the character boundary from any position — corruption stays local). The costs of the winner: CJK characters take 3 bytes (vs 2 in UTF-16 — Chinese storage is 50% larger in UTF-8), and emoji take 4. The practical implication for tools: byte-length vs character-length vs code-point-length are three different numbers in every encoding, and the "how long is this string" question must name which one it means (our encoder shows all three).

Normalization: NFC, NFD, and the equality that fails

The same visual character can have multiple code point sequences: é as one precomposed character (U+00E9, NFC form) or e plus combining accent (U+0065 U+0301, NFD form). They render identically and compare as different strings — the bug behind "the username works when typed but not when pasted" and duplicate database entries. Normalization (to NFC, the composed standard for the web) fixes comparison; the API boundaries that need it: user input vs stored values, filename comparisons, search. The deeper layer: Unicode itself periodically adds characters (emoji versions), so "normalized and equal" is version-dependent for the newest code points — pin the Unicode version in long-lived systems, or equality drifts across library upgrades.

Code points, characters, and the grapheme gap

A "character" as humans perceive it (one emoji face with skin tone, a Korean syllable block, a family emoji with ZWJ) is a grapheme cluster — potentially many code points. Every layer below lies a little: code points count emoji with modifiers as several; UTF-16 code units split astral-plane characters in two (string length 11 for what users see as 6); bytes inflate again. Text truncation, cursor movement, and column alignment want grapheme boundaries (Intl.Segmenter in JavaScript, grapheme clusters in Unicode libraries) — counting anything else produces the "half an emoji" artifact users screenshot for bug reports. When a tool reports string length, the honest ones name the unit; the decoder shows code points explicitly because that is what the machine sees, and the gap to what the eye sees is the lesson.

Surrogate pairs: the invisible JS length trap

'😀'.length is 2 in JavaScript, not 1: engines count UTF-16 code units, and astral characters become a high+low surrogate pair (for example \uD83D\uDE00). A charCodeAt loop that emits \uXXXX one unit at a time will break decoding if a pair is split. Prefer codePointAt, for…of, or Array.from — or let the tool encode by scalar value. Conversely, valid JSON often contains paired \uD83D\uDE00; a lone \uD83D should be rejected. Our decoder errors clearly on unpaired surrogates instead of quietly leaving half an emoji behind.

FAQ

Why Does the Same Character Have Different Sizes in Different Encodings?

Different encodings use different byte counts: Chinese characters are 3 bytes in UTF-8, 2 bytes in UTF-16, and 2 bytes in GBK. This is due to different encoding scheme designs.

What's the Relationship Between Unicode and UTF-8?

Unicode is the character set standard (assigning numbers to characters). UTF-8 is an encoding implementation of Unicode (defining how to represent those numbers as bytes). UTF-8 is the most widely used encoding on the Internet.

Why Do Some Chinese Characters Display as Garbled?

Common causes: encoding declaration mismatch (file saved as UTF-8 but read as GBK), missing font support, encoding conversion errors during transmission. Ensure consistent UTF-8 encoding across the entire pipeline; when characters look wrong, paste them into a unicode encoder/decoder to check the underlying code points.

When Should I Use \uXXXX vs \u{XXXXX}?

Use \uXXXX when you need to match legacy JSON or existing escape dumps (emoji become two escapes). Use \u{1F600} when you want the real code point visible, or when writing modern JavaScript sources. The tool encodes and decodes both; switch to UTF-8 \xHH when you are comparing against a hex dump.

Triage notes

Any 'how long is this string' question must settle the unit first — byte length, code-point length, and UTF-16 code-unit length are three different numbers, and the tool reporting all three at the top plus a per-character breakdown table is precisely that argument shipped as a product. The surrogate-pair section anchors on 😀 having length 2: engines count code units, and an unpaired surrogate makes this site's decoder fail explicitly rather than quietly leave half an emoji — echoing the code-point/grapheme section's conclusion that counting at the wrong layer is what produces the 'half an emoji' users screenshot and report. The NFC/NFD section is titled 'the equality that fails' for a reason: é has two code-point sequences that render identically but compare unequal, the culprit behind 'typed by hand it works, pasted it does not' username bugs.

This guide is part of the ToolVault toolkit. More developer tools at the homepage.