DevToolkit Pro

What Is Unicode

Unicode is an international standard that assigns a unique number (code point) to nearly every character in all writing systems. It solved compatibility issues with older encodings (ASCII, GB2312, Shift-JIS).

Unicode Representation Forms

| Form | Example | Description | |------|---------|-------------| | Code point | U+4F60 | Standard notation | | UTF-8 | \xe4\xbd\xa0 | Variable-length (1-4 bytes) | | UTF-16 | \u4f60 | 2 or 4 bytes | | Unicode escape | \u4F60 | Used in JavaScript/Java | | HTML entity | 你 | Decimal entity | | HTML entity (hex) | 你 | Hexadecimal entity |

Why You Need Unicode Encoding/Decoding

1. Cross-language compatibility: Character transfer between different programming languages
2. JSON safety: Ensure special characters transmit correctly in JSON
3. Database storage: Handle character encoding issues in databases
4. URL encoding: Encode non-ASCII characters in URLs
5. Log analysis: Process log data containing Unicode characters
6. Internationalization: Handle encoding conversion for multi-language text

UTF-8 vs UTF-16

| Feature | UTF-8 | UTF-16 | |---------|-------|--------| | Byte count | 1-4 bytes | 2-4 bytes | | ASCII compatible | ✅ Fully | ❌ No | | Network transmission | ✅ Standard | ❌ Rarely used | | Byte order | None (no BOM) | Yes (big/little endian) | | Space efficiency | Efficient for English | Efficient for CJK |

How to Use an Online Tool

Using DevToolkit Pro's Unicode Encoder/Decoder:

1. Enter text or Unicode encoded string
2. Choose encode/decode mode
3. Supports multiple output formats:
   • \uXXXX JavaScript escape
   • &#XXXX; HTML entity
   • U+XXXX Unicode standard notation
4. Real-time conversion results

Unicode Handling in Code

JavaScript

// Character → Unicode code point
function toUnicode(str) {
  return Array.from(str).map(char =>
    'U+' + char.codePointAt(0).toString(16).toUpperCase().padStart(4, '0')
  );
}

// Unicode code point → Character
function fromUnicode(codePoint) {
  return String.fromCodePoint(parseInt(codePoint.replace('U+', ''), 16));
}

// JavaScript escape sequence
function toEscaped(str) {
  return Array.from(str).map(char =>
    '\\u' + char.codePointAt(0).toString(16).padStart(4, '0')
  ).join('');
}

Python

import unicodedata

# Character → Unicode code point
def to_unicode(char):
    return f"U+{ord(char):04X}"

# Unicode code point → Character
def from_unicode(code_point):
    return chr(int(code_point.replace('U+', ''), 16))

# Get character info
def char_info(char):
    return {
        'char': char,
        'code_point': f"U+{ord(char):04X}",
        'name': unicodedata.name(char, 'UNKNOWN'),
        'category': unicodedata.category(char)
    }

Common Unicode Ranges

| Range | Content | Example | |-------|---------|---------| | U+0000-U+007F | Basic Latin | A, b, 1, ! | | U+0080-U+00FF | Latin Supplement | á, ñ, ü | | U+0400-U+04FF | Cyrillic | А, Б, В | | U+4E00-U+9FFF | CJK Unified Ideographs | 中, 文, 字 | | U+3040-U+309F | Hiragana | あ, い, う | | U+30A0-U+30FF | Katakana | ア, イ, ウ | | U+1F600-U+1F64F | Emoji | 😀, 😂, ❤️ |

FAQ

Why Does the Same Character Have Different Sizes in Different Encodings?

Different encodings use different byte counts: Chinese characters are 3 bytes in UTF-8, 2 bytes in UTF-16, and 2 bytes in GBK. This is due to different encoding scheme designs.

What's the Relationship Between Unicode and UTF-8?

Unicode is the character set standard (assigning numbers to characters). UTF-8 is an encoding implementation of Unicode (defining how to represent those numbers as bytes). UTF-8 is the most widely used encoding on the Internet.

Why Do Some Chinese Characters Display as Garbled?

Common causes: encoding declaration mismatch (file saved as UTF-8 but read as GBK), missing font support, encoding conversion errors during transmission. Ensure consistent UTF-8 encoding across the entire pipeline.

This article is brought to you by DevToolkit Pro. More developer tools at the homepage.