Characters, code points and bytes
Three distinct things, routinely conflated.
A character is what a person perceives: the letter é, the emoji 👍, the Chinese character 語.
A code point is the number Unicode assigns to a character. é is U+00E9, or 233. 👍 is U+1F44D. Unicode has room for about 1.1 million code points and has assigned roughly 150,000 of them.
An encoding maps code points to bytes for storage or transmission. This is where the trouble lives, because the same code point can be represented by different byte sequences, and the same byte sequence can be interpreted as different characters.
The critical consequence: a file, a string or a network response is meaningless without knowing its encoding. There is no way to inspect bytes and determine with certainty what they were meant to say. Every guess is a heuristic.
What UTF-8 actually does
UTF-8 is a variable-length encoding. Code points below 128 - the ASCII range - use one byte with the same value ASCII always used. Higher code points use two, three or four bytes.
| Range | Bytes | Covers |
|---|---|---|
| U+0000 - U+007F | 1 | ASCII: English letters, digits, basic punctuation |
| U+0080 - U+07FF | 2 | Latin with accents, Greek, Cyrillic, Hebrew, Arabic |
| U+0800 - U+FFFF | 3 | Chinese, Japanese, Korean, most remaining scripts |
| U+10000 - U+10FFFF | 4 | Emoji, historic scripts, rare characters |
Backward compatibility with ASCII is why UTF-8 won. A file containing only English text is byte-identical whether you call it ASCII or UTF-8, so decades of existing software and data kept working. UTF-8 is now used by over 98% of web pages, and the correct default for essentially everything.
Where é comes from
The garbled text has a name - mojibake, from Japanese - and each variety points at a specific mistake.
é instead of é. The text was encoded as UTF-8 and decoded as Latin-1 or Windows-1252. In UTF-8, é is the two bytes 0xC3 0xA9. Read as Latin-1, one byte at a time, those are à and ©. This is by far the most common case.
’ instead of a curly apostrophe. Identical cause, three bytes instead of two. Common in text pasted from Microsoft Word, which uses typographic quotes.
? or �. The text was converted to an encoding that cannot represent the character. This is destructive - the original is gone, and no amount of re-decoding recovers it.
Emoji becoming ???? in MySQL. The column uses utf8, which in MySQL is a three-byte encoding that cannot store four-byte characters. The correctly named utf8mb4 is the real UTF-8, and has been the recommended setting for over a decade.
Diagnosing an encoding problem
Text passes through many layers, and each one can misdeclare the encoding. Work through them in order.
- The source file. Check what your editor saved it as. UTF-8 without a byte-order mark is correct for almost everything.
- The database connection. The column, the table, the database and the connection each have a character set, and they must agree. A correct column read over a Latin-1 connection produces mojibake.
- The HTTP response. The
Content-Typeheader should saycharset=utf-8. This overrides anything in the HTML. - The HTML declaration.
<meta charset="utf-8">must appear within the first 1024 bytes of the document. - The template engine and the application layer. Some frameworks default to something other than UTF-8, particularly older ones.
The layer that matters is the first one that gets it wrong; everything after that is inheriting the damage. Working backwards from the display to the source usually locates it within a few minutes.
Why string length is not what you think
Encoding also breaks the intuition that a string has one length.
The emoji 👍 is one character, one code point, four bytes in UTF-8, and two units in JavaScript, because JavaScript strings are UTF-16 and characters above U+FFFF are stored as a surrogate pair. So "👍".length is 2.
It gets worse. A family emoji is built from several emoji joined by zero-width joiners and can be seven or more code points. Flags are two regional indicator characters. An accented letter may be one code point or two, depending on normalisation.
Practical consequences: a 280-character limit implemented with .length charges two for every emoji; reversing a string by splitting on characters corrupts anything outside the basic plane; and truncating at a byte or code-unit boundary can split a character in half and produce a replacement character.
The fix is to use grapheme-aware operations where the count is user-visible - Intl.Segmenter in modern JavaScript - and to normalise strings with normalize('NFC') before comparing them, so that the two representations of é compare as equal.
Rules that prevent all of this
- UTF-8 everywhere. Files, databases, connections, HTTP headers, HTML declarations. Every layer, no exceptions.
- In MySQL, use
utf8mb4. The encoding namedutf8is not UTF-8 and cannot store emoji. - Declare the charset in the header and the HTML. Belt and braces, and the header wins.
- Do not guess encodings. Detection heuristics are wrong often enough to cause silent corruption. Require the encoding to be specified.
- Normalise before comparing. Two visually identical strings can have different byte sequences.
- Test with real data. Add an emoji, a Chinese character and an accented name to your test fixtures. Almost every encoding bug reaches production because the test data was ASCII.
Encoding problems are cheap to prevent and expensive to fix after the fact - once text has been converted to ?, the original characters are unrecoverable.
Frequently asked questions
The text was written as UTF-8 and read as Latin-1 or Windows-1252. The é is two bytes in UTF-8, and reading them one at a time as single-byte characters produces à followed by ©.
MySQL's utf8 is a three-byte encoding that cannot store characters above U+FFFF, which includes all emoji. utf8mb4 is the real four-byte UTF-8 and is what you should use.
Generally no. UTF-8 does not need one, and a BOM causes problems in shell scripts, PHP files and JSON. The exception is CSV files intended for Excel on Windows, which uses the BOM to detect UTF-8.
JavaScript strings are UTF-16, and characters above U+FFFF are stored as two code units. Use Intl.Segmenter or spread the string into an array of code points when you need a user-visible character count.
No. Conversion to ? is lossy - the original bytes were discarded. Recover from a backup or the original source. Mojibake such as é, by contrast, is usually recoverable because the bytes survived.