Unicode and UTF-8

Why this matters

A computer stores every piece of text as numbers. When you type a name into a form, the program turns each letter into a number and writes those numbers to a file or a database. When someone else opens that file, their program has to turn the numbers back into letters. If the two programs disagree about the rules, the name comes out wrong. A name like José can turn into José, and a price like €5 can turn into €5. This kind of garbled text is called mojibake.

Engineers meet this problem in web pages, database rows, email subjects and exported spreadsheets. It often shows up only for names, prices or words from another language, so it can slip past a test that uses plain English. The fix is short, but you need a clear model of what text is inside the computer first.

This lesson covers three ideas. A code point is the number that identifies one character. UTF-8 is one set of rules for writing code points as bytes. Mojibake is what you see when bytes are read with a different set of rules. The JSON lesson earlier in this module shows where text sits inside a data file. This lesson looks at the text itself.

Characters, code points and bytes

A character is what a person sees as one letter, digit or symbol: A, é, €, a smiley face. Computers do not store characters directly. They store numbers.

A code point is the number that the Unicode standard gives to one character. Unicode is a large shared table that assigns a unique number to each character in most of the world's writing systems, plus many symbols and emoji. Code points are written as U+ followed by a hexadecimal number. Hexadecimal is base 16, so it uses the digits 0 to 9 and the letters A to F. U+0041 is the capital letter A. U+00E9 is é.

A byte is 8 bits, which is a number from 0 to 255. Files, network connections and memory move bytes, not characters. So for text to be saved or sent, each code point must become one or more bytes. The set of rules that does this is an encoding.

Here is a first example. The script loops over three characters, prints each one's code point, and prints the bytes that UTF-8 uses for it. The .encode("utf-8") method turns a string into bytes using the UTF-8 rules, and .hex(" ") prints each byte as two hexadecimal digits.

text = "é€😀"
for ch in text:
    raw = ch.encode("utf-8")
    print(ch, f"U+{ord(ch):04X}", raw.hex(" "), len(raw))

The output is:

é U+00E9 c3 a9 2
€ U+20AC e2 82 ac 3
😀 U+1F600 f0 9f 98 80 4

Read one line at a time. The letter é has the code point U+00E9. UTF-8 stores it as the two bytes c3 and a9. The euro sign takes three bytes and the grinning face takes four. Each character has exactly one code point, but the number of bytes depends on the character.

The ord() function in Python returns a character's code point as a number. The ord name comes from "ordinal", which means a position in a sequence.

Why one character can be several bytes

Before Unicode, many systems used a single byte per character. The ASCII table covers 128 characters, including the English letters, digits and common punctuation. Each one fits in one byte with a value from 0 to 127. That works for English, but not for é, ñ, ü, Greek, Hindi or Chinese. Early systems solved this with many competing code pages, where the same byte meant different letters on different machines.

UTF-8 solves the problem with one rule that covers every code point. It uses a variable number of bytes:

Two properties make UTF-8 useful. The first is that every ASCII character uses the same single byte it always did. A file that contains only English letters is therefore identical in ASCII and UTF-8, and old tools keep working. The second is that the first byte of each multi-byte character tells the reader how many bytes follow, so a program can find character boundaries without reading the whole file from the start.

The byte that starts a multi-byte character has its top bits set to a pattern that marks its length. The bytes after it always start with the pattern 10. You do not need these bit patterns to use UTF-8 day to day. What matters is that the rule is fixed and every program that follows it reads the same text.

Here is the same idea applied to a word. The word café has four characters, but five bytes, because the é takes two:

word = "café"
print(len(word))
print(len(word.encode("utf-8")))

The output is:

4
5

The first number counts code points, which is what Python's len() function counts for a string. The second counts bytes. Both numbers are correct. They answer different questions, and mixing them up causes many of the bugs in this lesson.

Counting: characters are not always one code point

Code points are not always the same as what a reader sees as one character. Some visible characters are built from two or more code points.

The letter é can be stored in two ways. One way is a single code point, U+00E9. The other is the letter e followed by a combining accent, U+0065 plus U+0301. Both look identical on screen. Here is a check:

import unicodedata
 
a = "é"
b = "é"
print(a, b)
print(a == b, len(a), len(b))
print(unicodedata.normalize("NFC", b) == a)

The output is:

é é
False 1 2
True

The two strings look the same, but a == b is False, because the code points differ. The unicodedata.normalize function rewrites a string into one standard form. "NFC" means "composed form", where the accent is merged into a single code point when one exists. After normalization, the two strings are equal. The NFC name is an abbreviation for one of the standard normalization forms, and you do not need to remember the details. The practical rule is that user input from different keyboards or operating systems can arrive in either form, so compare text after normalizing both sides.

Emoji show the same effect in a larger way. A thumbs-up with a medium skin tone is a single symbol on screen, but it is two code points:

thumbs = "👍🏽"
print(thumbs)
print(len(thumbs), len(thumbs.encode("utf-8")))
print([f"U+{ord(c):04X}" for c in thumbs])

The output is:

👍🏽
2 8
['U+1F44D', 'U+1F3FD']

The first code point is the thumbs-up. The second is a skin-tone modifier. Together they make one symbol on screen, using eight bytes. A family emoji made of several people joined by invisible connector code points is longer still. Software that needs the number of symbols a reader sees, for example to cut a string to 10 visible characters, must use a grapheme library. A grapheme is the unit a reader perceives as one character. Python's built-in len() does not count graphemes.

Encodings disagree

An encoding is only a rule. If the writer and the reader use the same rule, the text survives. If they use different rules, the bytes are the same but the characters change.

Take the text café €5. Encoded as UTF-8, it becomes 10 bytes:

original = "café €5"
raw = original.encode("utf-8")
print(raw)

The output is:

b'caf\xc3\xa9 \xe2\x82\xac5'

In this output, \xc3 is how Python prints the byte c3. Now suppose a program reads those same bytes using the older Windows encoding called cp1252, which is a single-byte encoding for Western European text. Every byte is treated as one character:

wrong = raw.decode("cp1252")
print(wrong)

The output is:

café €5

Each byte of é became its own character, Ã and ©. The euro sign became three characters, â, ‚ and ¬. The text is not lost, because the original bytes are still there. The reader just used the wrong rule to turn them into characters. That is the core of mojibake.

The same bytes read as UTF-16 would look different again. UTF-16 is a different encoding that uses at least two bytes for every code point. This is why a file saved as UTF-16 and opened as UTF-8 often shows strange gaps between letters, or the text does not decode at all.

Fixing mojibake

Mojibake can often be repaired, but only if no information was lost along the way. The repair is a reversal. You undo the wrong decode by encoding the garbled text back into the bytes it came from, using the same wrong rule. Then you decode those bytes with the right rule.

wrong = "café €5"
fixed = wrong.encode("cp1252").decode("utf-8")
print(fixed)
print(fixed == "café €5")

The output is:

café €5
True

The first step, .encode("cp1252"), turns the garbled characters back into the original bytes, because cp1252 was the rule that produced them. The second step, .decode("utf-8"), reads those bytes with the correct rule. The whole chain works for this example.

Now a worked example of increasing difficulty. Imagine a name that was saved as UTF-8, read as cp1252, and then saved again as UTF-8. The text was encoded twice, so the damage has two layers. The name résumé becomes:

name = "résumé"
once = name.encode("utf-8").decode("cp1252")
twice = once.encode("utf-8").decode("cp1252")
print(once)
print(twice)

The output is:

résumé
résumé

To repair the double damage, you reverse each layer in order. The second layer is undone first, then the first. A reversal is safe only when each step is the exact inverse of the step that caused it. If the encoding step ever replaced an unknown character with ? or dropped it, that information is gone and no reversal brings it back.

Here is the full repair:

twice = "résumé"
once = twice.encode("cp1252").decode("utf-8")
name = once.encode("cp1252").decode("utf-8")
print(name, name == "résumé")

The output is:

résumé True

A useful habit is to test a repair on a short sample first, compare the result with a known good value, and only then run it on the whole file. Keep the original file. Repairs that go wrong are hard to undo if you overwrite the only copy.

Reading files and getting real errors

Most mojibake and encoding errors come from reading or writing files. A file is only bytes. The program that opens it must say which encoding to use. Some tools guess, and guesses are sometimes wrong.

Here is a file that contains the word café in UTF-8. The script takes the file name and the encoding as arguments, so you can try different rules on the same file:

import sys
 
path = sys.argv[1]
with open(path, encoding=sys.argv[2]) as f:
    print(f.read())

Run with the UTF-8 rule, the file reads correctly:

$ python3 read_text.py cafe.txt utf-8
café

Run with the ASCII rule, which only allows bytes from 0 to 127, the program fails:

$ python3 read_text.py cafe.txt ascii
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

The message is read from the right. codec is the name of the rule. can't decode byte 0xc3 says which byte could not be read. in position 3 is the byte's index, counted from zero, in the file. The byte 0xc3 is the first byte of é, which is the byte at index 3, after the three ASCII letters c, a and f.

Now try a file whose bytes are not valid in the rule cp1252 either. The byte 0x9d is not assigned a character in that encoding:

$ python3 read_text.py bad.txt cp1252
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1: character maps to <undefined>

The word charmap appears in this message because Python uses the same internal name for single-byte code-page rules.

The command-line tool iconv converts text between encodings. When a character cannot be represented in the target encoding, it stops:

$ printf 'café\n' | iconv -f UTF-8 -t ASCII
iconv: illegal input sequence at position 3

The exit code is 1, which means failure. A script that checks the exit code can stop instead of writing a broken file.

You can look at the raw bytes of a file with od, which stands for "octal dump", even though it can print hexadecimal:

$ printf 'é' | od -An -tx1
 c3 a9
$ printf 'é' | wc -c
2

The -An flag hides the address column and -tx1 prints each byte in hexadecimal. The wc -c command counts bytes. This is the quickest way to check whether a file is UTF-8 without opening it in an editor that might guess.

Byte order marks

Some editors, especially on Windows, begin a UTF-8 file with three extra bytes called a byte order mark, usually written as BOM. The bytes are ef bb bf. They tell a reader that the file is UTF-8, but a program that does not expect them sees them as an invisible first character. To try the example, create a file with that BOM first: printf '\xef\xbb\xbfname,city\n' > bom.csv.

with open("bom.csv", encoding="utf-8") as f:
    print(repr(f.readline()))
with open("bom.csv", encoding="utf-8-sig") as f:
    print(repr(f.readline()))

The output is:

'\ufeffname,city\n'
'name,city\n'

The first line starts with \ufeff, which is the code point for the zero-width no-break space that Unicode uses as a BOM. The name of the first column is \ufeffname, not name, so a lookup for the column called name fails with no visible reason. The utf-8-sig encoding removes the BOM when reading and is the simplest fix for files that came from Windows tools.

Common mistakes

Using len() to check a byte limit. A database column declared to hold 10 bytes accepts ååååå, which is five characters and ten bytes. It rejects åååååå, which is six characters and twelve bytes, even though a person would say both are short. Check the limit with len(text.encode("utf-8")), and make sure the database limit is also counted in bytes. Some databases count characters instead, so read the documentation of the column type.

Relying on the default encoding. Python's open() uses the platform default when you do not give an encoding. On Linux and macOS that is usually UTF-8. On Windows it has been an older code page for many years. The same script can work on one machine and fail on another. Always pass encoding="utf-8" when you open a file you control.

Silencing errors with errors="ignore". The ignore option skips any byte it cannot read. The program runs, but the output is missing characters. In the earlier example, b"caf\xe9".decode("utf-8", errors="ignore") returns caf, with no warning. Prefer errors="replace", which puts a visible replacement mark � where a character was lost, or fix the source of the bad bytes.

Repairing text twice. Running a repair on text that was already fixed damages it again. café repaired once is café. Repaired again, it fails or becomes something else. Check the output before running a second repair, and keep the original.

Comparing strings without normalizing. Two strings that look identical can differ in their code points, as the é example showed. A search for café will not find café written with a combining accent unless both sides are normalized to the same form.

Trusting a file name extension. A file named data.csv is not guaranteed to be UTF-8. Excel, for example, can save CSV files in a local code page. Check the bytes with od or the file command, or ask the sender which encoding they used.

Practice

Exercise 1: Count the bytes

How many bytes does the string ñ take in UTF-8? Write the bytes in hexadecimal.

Show the answer

It takes 2 bytes, c3 b1. The code point is U+00F1, which falls in the range U+0080 to U+07FF, so UTF-8 uses two bytes.

Exercise 2: Length in characters and bytes

What does this code print?

word = "naïve"
print(len(word), len(word.encode("utf-8")))

Show the answer

It prints 5 6. The word has five code points, because ï is one code point. Its UTF-8 form has six bytes, because ï (U+00EF) takes two bytes.

Exercise 3: Predict the mojibake

What does "résumé".encode("utf-8").decode("cp1252") print?

Show the answer

It prints résumé. Each é becomes the two bytes c3 a9, and cp1252 reads those as the characters Ã and ©.

Exercise 4: Repair a file

A file contains the text é, which was saved by a program that wrongly read UTF-8 text as cp1252. What is the original text, and what two lines of Python recover it?

Show the answer

The original text is é. The two lines are:

broken = "é"
print(broken.encode("cp1252").decode("utf-8"))

The output is é. The encode("cp1252") step gives back the bytes c3 a9, and decode("utf-8") reads them correctly.

Exercise 5: Explain the error

Your script fails with UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data. The file is known to contain café saved as Latin-1, which uses the byte e9 for é. What is the fix?

Show the answer

The file is not UTF-8. The byte e9 alone is é in Latin-1, but UTF-8 expects é as the two bytes c3 a9. Open the file with encoding="latin-1", or convert it once to UTF-8 with a tool such as iconv -f ISO-8859-1 -t UTF-8, and then use UTF-8 everywhere else.

Summary

• A code point is the number Unicode gives a character, and a byte is a number from 0 to 255 that storage and networks move.
• UTF-8 stores code points as one to four bytes, and every ASCII character takes one byte as it always did.
• len() counts code points in Python, while len(text.encode("utf-8")) counts bytes, and limits in databases may use either.
• Mojibake happens when bytes are read with a different encoding than the one that wrote them, and it can be repaired only when no characters were lost.
• Always name the encoding when you open or write a file, and avoid errors="ignore", which hides missing characters.