Text, Unicode and UTF-8

"How long is this string?" has four correct answers: the number of bytes, the number of code units, the number of code points, and the number of characters a reader sees. Lantern's catalogue holds titles in dozens of scripts, and Noor has met each answer the hard way: a title that fails to match its own search, a length check that cuts an emoji in half, a column that takes three times the bytes it was sized for.

All three bugs come from treating one layer as another. This topic separates the four layers of a string, shows how UTF-8 turns numbers into bytes, and prices each choice a language or a database makes about which layer to count.

Four Layers of One String

A grapheme is what a reader sees as one character. A code point is Unicode's number for one abstract character. A code unit is the fixed-size chunk an encoding works in: 8 bits in UTF-8, 16 bits in UTF-16. And bytes are what is actually stored. For plain English text all four counts agree, and so much code gets away with confusing them.

They come apart quickly. The letter é can be one code point, or two: a plain e followed by a combining accent. The family emoji of a man, a woman and a girl is one grapheme built from five code points: three people joined by two invisible zero-width joiners. In UTF-16 those five code points take 8 code units, and in UTF-8 they take 18 bytes. Every language's length function counts one of these layers, and they do not all count the same one.

Unicode, the Numbering

Unicode assigns a number to every character in every script, from U+0000 to U+10FFFF, a space of 1,114,112 code points. Unicode 17.0, the version Python 3.15 ships, assigns 159,801 characters, most of them Chinese, Japanese and Korean ideographs. The numbers are the agreement: U+00F1 is ñ on every machine in the world.

A number is not yet a byte. An encoding is the rule for writing code points as bytes, and there are several, each with a different price. Mixing up the numbering and the encoding is the root of most text bugs, because two programs can agree completely on Unicode and still disagree on the bytes.

How UTF-8 Writes a Code Point

UTF-8 uses 1 byte for code points up to U+007F, identical to ASCII, 2 bytes up to U+07FF, 3 bytes up to U+FFFF, and 4 bytes for the rest. The first byte's leading bits say how many bytes the character takes: a leading 0 means one, 110 means two, 1110 means three, 11110 means four. Every continuation byte starts with the bits 10. The code point's own bits are poured into the free slots.

So é, U+00E9, has 11 significant bits and becomes the two bytes C3 and A9 in hexadecimal. The kanji for forest, U+68EE, needs 16 bits and becomes three bytes, E6, A3 and AE. Because continuation bytes are marked, a reader dropped anywhere in a UTF-8 stream can step back to the start of the current character, and a search for an ASCII byte such as a slash or a quote can never match the inside of another character: every byte of a multi-byte character has its top bit set, and no ASCII byte does.

Lantern's Titles

Measured in UTF-8 on Python 3.15, "The Dispossessed" is 16 bytes for 16 characters. "Cien años de soledad" is 21 bytes for 20 characters, because ñ takes two. The Russian title "Война и мир" is 20 bytes for 11 characters. The Japanese title ノルウェイの森 is 21 bytes for 7 characters, three bytes each. And the Hindi title गोदान shows the fourth layer: it is 5 code points and 15 bytes, but a reader sees 3 characters, because two of the code points are vowel signs that attach to the consonant before them.

Title	Graphemes	Code points	UTF-8 bytes
The Dispossessed	16	16	16
Cien años de soledad	20	20	21
Война и мир	11	11	20
ノルウェイの森	7	7	21
गोदान	3	5	15

A title field sized as "100 characters, so 100 bytes" holds 100 English characters and only 33 Japanese ones. A cut at an arbitrary byte can land inside a character: cutting the Japanese title after 10 bytes leaves three whole characters and one stray byte, which decodes as the replacement character, a question mark in a diamond, or raises an error, depending on how the reader was told to handle bad input. Python's standard library counts code points and has no function that counts graphemes. The grapheme counts here follow the Unicode segmentation rules and were checked with the segmenter built into JavaScript.

Normalization and Matching

The letter ñ can be stored as one code point, U+00F1, or as n followed by a combining tilde, U+0303. The two look identical on every screen and compare unequal, because they are different code points: 2 bytes in one form and 3 in the other. A record catalogued in the decomposed form never matches a query typed on a kiosk in the composed form. Lantern has to normalize both sides to one form, usually the composed form called NFC, before indexing and again before searching.

Case needs the same care. Lowercasing is not enough for case-insensitive matching, because some characters do not map one to one: the German ß uppercases to SS, so "Straße" and "STRASSE" lowercase to different strings. Case folding, which Python offers as a separate string method, maps both to "strasse" and makes them match.

What Each Choice Costs

UTF-8 costs 1 byte per character for English, 2 for Cyrillic, Greek and accented Latin, and 3 for most Chinese and Japanese, where UTF-16 costs 2 for all of them. Finding the thousandth code point in UTF-8 means scanning from the start, because characters differ in width, so indexing by code point is O(n) and languages choose which layer to index. Go and Rust index bytes. JavaScript and Java index UTF-16 units. CPython stores each string at 1, 2 or 4 bytes per code point, depending on its widest character, and keeps indexing O(1).

CPython's choice has a price of its own. On 3.15 a string of 1,000 ASCII letters takes 1,041 bytes. Change one letter to 森 and the whole string moves to 2 bytes per character, 2,058 bytes. Change it to an emoji instead and it moves to 4 bytes per character, 4,060 bytes: one emoji roughly quadruples the memory of the whole string. A service holding millions of short strings pays for its widest character in each one.