Code points and emoji in Python strings

Why this matters

A program often needs to answer a simple question: how long is this text? Or: what is the third character? For plain English letters the answer is easy, because each letter takes one box in memory. Emoji, accented letters and text from other languages break that rule. A single thing you see on screen can be built from two or more code points, and different tools count the text in different ways.

This lesson covers how Python stores text, why len gives a number that can differ from what you see, and how to work with text one real character at a time. It uses the word "code point" for the number that Unicode gives each character. It does not cover the other string tasks in this module, such as CSV splitting, substring search and regex.

Three words you need

A character is what a reader sees as one symbol: the letter a, the letter é, or a smiling face.

A code point is the number that the Unicode standard gives to one character. The standard has over a million code points. The letter a is code point 97, which is written as U+0061 in Unicode notation. The smiling face 😀 is code point U+1F600. The U+ means "Unicode code point" and the digits after it are in hexadecimal, which is a base-16 way of writing numbers.

A byte is one box of 8 bits, and a file or a network connection stores text as bytes. A UTF-8 encoding is the most common way to turn code points into bytes. A plain letter takes one byte in UTF-8, and other characters take two, three or four.

In Python, a str is a sequence of code points. Each item of the sequence is one code point, which the built-in ord turns into a number and chr turns back into a character. That is different from some other languages, where a string is a sequence of smaller units, so the length counts those units instead.

Length counts code points in Python

Start with a word that uses only plain letters, and then try the smiling face.

The output is:

4
1
3

Each character is one code point, so the lengths are 4, 1 and 3. The emoji counts as one character, which matches what you see on screen. This is the main difference from the languages that count smaller units. Python's len counts code points, so it gives the number a reader expects for most text.

The length still does not count bytes. A file or a network message needs bytes, so use encode to get them:

The output is:

4
b'caf\xc3\xa9'
5

The emoji takes four bytes in UTF-8. The accented é takes two bytes, so caf plus é gives 5 bytes, while len("café") is 4. The two numbers measure different things, and both are correct. Use len for the number of characters, and count bytes only when you write data to a file or send it over a network.

The next loop prints each code point of a😀é with its number, and the number of bytes it takes in UTF-8:

The output is:

U+0061 'a' 1
U+1F600 '😀' 4
U+00E9 'é' 2

The loop visits each code point once, so the emoji appears as a single item. The byte count column adds up to 7 for this string, while len gives 3.

Visible characters can be several code points

Some things you see as one character are built from several code points. The loop above still visits each code point separately, which is correct for code points but may not match what a reader thinks of as one symbol.

The clearest case is an accented letter written with a combining mark. The letter é can be stored as one code point, U+00E9, or as two code points: the letter e followed by U+0301, a mark that adds an accent. The screen shows the same letter for both. In Python, write the second form with an escape so that you know which bytes you are testing:

The output is:

1 2
False

The two strings look the same on screen, but they hold different code points, so == returns False. The standard library's unicodedata module can convert one form to the other. The function unicodedata.normalize("NFC", s) puts the characters in the composed form, which uses one code point for é:

The output is:

True

If a program must treat the two forms as equal, normalize both strings first, and then compare them. This module does not go further into normalization, but the rule is simple: decide which form your data uses, and normalize text that comes from outside.

A family emoji is an even bigger case. The family of three is built from three person emoji joined with a zero width joiner, which is code point U+200D. A zero width joiner is a character with no width that tells the screen to draw the joined symbols as one picture. Here is the family, with each code point named:

The output is:

5 18
U+1F468 MAN
U+200D ZERO WIDTH JOINER
U+1F469 WOMAN
U+200D ZERO WIDTH JOINER
U+1F467 GIRL

The string has five code points and takes 18 bytes in UTF-8, but a screen that supports it draws one family. The line that prints U+200D shows no visible mark, because the character has no width. The unicodedata.name function gives the official name of each code point, which helps when you debug text you cannot see.

The standard library does not split text into visible characters, which are called grapheme clusters. A third-party package named regex supports the pattern \X, which matches one visible character, and it is the usual tool for that job. This lesson does not run it, because it is not in the standard library. When you need the count of what a reader sees, look for a package of that kind, and check its documentation for the version you install.

Cutting text without breaking a character

A slice cuts a string at code point positions. Because a Python string is a sequence of code points, a slice never splits an emoji in the middle:

The output is:

'b'
'a😀'

The slice [:2] keeps the two code points a and the emoji, and the emoji stays whole. The cut is at code point 2, so it cannot land between the halves of the emoji, as it could in a language that counts smaller units.

Splitting happens when you cut bytes, because a byte cut can land in the middle of a multi-byte character. The example below cuts the UTF-8 bytes of a😀b after the first two bytes:

The output is:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 1: unexpected end of data
'a'

The first two bytes hold the letter a and the first byte of the emoji, which is not a complete character. Decoding fails. The second line shows one way out: errors="ignore" drops the incomplete part. That drops data silently, so use it only when losing a character is acceptable.

Walking a string one code point at a time

A for loop over a string already visits one code point at a time, so you do not need an index to walk the text. The loop below prints each code point, its number and its name, and it stays correct for emoji:

The output is:

index 0  U+0061  LATIN SMALL LETTER A
index 1  U+1F600  GRINNING FACE
index 2  U+00E9  LATIN SMALL LETTER E WITH ACUTE

The index counts code points, so the emoji takes one index, and the é is at index 2. Compare this with a language that counts smaller units, where the same emoji would take two positions and the loop would need to step by the size of each character. In Python the step is always one code point.

To count the code points in a string, len is enough. The method str.encode gives bytes, and len of the bytes gives the byte count, as shown earlier. For a count that matches the visible text, use a grapheme cluster tool, as described in the earlier section.

Reversing text safely

A common exercise is to reverse a string. Reversing code points keeps each emoji whole, because each emoji is one item of the string:

The output is:

True
'x́e'

The first line shows that the emoji stays whole. The second line shows the limit of this method. Reversing e followed by the combining accent U+0301 and then x gives x, then U+0301, then e, so the accent now sits on the x. Reversing by code point does not respect visible characters. If you need a reverse that keeps each visible character together, split the text into visible characters first, reverse the list of those, and join them back. That needs the grapheme cluster tool described above.

A worked example: a title with a length limit

A common task is to shorten a title so that it fits in a box that shows at most a set number of characters. The limit is a count of characters, so the code should count code points. Python's slice does that directly:

The output is:

[Party time 🎉]
21 12

The title has 21 code points. The result is 12 code points long, and the party popper emoji, U+1F389, is whole. The bracket marks show the exact start and end of the output.

A second example uses two emoji in a row, which shows the count in code points:

The output is:

[go 🚀] length=4 utf-16 units=5

The truncated string has 4 code points: g, o, a space and one rocket. Its length is 4 in Python. The last part of the output counts the UTF-16 units, which is 5, because the rocket takes two units in UTF-16. This is the count some other languages use for the same text, and it shows why a number from one language does not always match another. The division by 2 turns the byte count of UTF-16 into a unit count.

The limit in the box may be a count of bytes instead, for example in a database column that holds a fixed number of bytes. In that case cut the encoded bytes, and then decode with errors="ignore", as shown in the cutting section, and say in the code that a character can be lost at the end.

Common mistakes

Mistake 1: cutting bytes as if they were characters. A slice of encode("utf-8") can stop in the middle of a multi-byte character, and decoding that part raises UnicodeDecodeError, as shown earlier. Cut the string, not its bytes, when the limit counts characters.

Mistake 2: using len as the number of visible characters. len counts code points. For an emoji with a skin tone or a family emoji, the count is larger than the number of symbols you see. Use the grapheme cluster tool when the count must match the screen.

Mistake 3: comparing visible characters with ==. Two strings that look the same can have different code points, as the é example showed. The == operator returns False. Normalize both strings with unicodedata.normalize first, when the data may come in either form.

Mistake 4: assuming two languages count the same way. The text café has length 4 in Python and 5 bytes in UTF-8. The cut go 🚀 has 4 code points and 5 UTF-16 units. A test that checks a length in one system will fail in another. When data crosses between systems, write down the unit that each side counts.

Practice

Each exercise has an answer under it. Try the exercise first, run your code, and only then read the answer.

1. What does len("👍") return, and how many bytes does the same string take in UTF-8? The thumbs up sign is U+1F44D.

Show the answer

len("👍") returns 1, because the thumbs up sign is one code point. In UTF-8 it takes 4 bytes, which you can check with len("👍".encode("utf-8")).

2. The word naïve can be written with the letter ï as one code point, U+00EF. How many code points does it have, and how many UTF-16 units?

Show the answer

It has 5 code points and 5 UTF-16 units, because ï fits in one unit. Every letter here is in the Basic Multilingual Plane, which covers code points up to 65535, so each code point is one unit in UTF-16.

3. Write a function count_code_points(s) that returns the number of code points in s. Then say why len already does this job.

Show the answer

def count_code_points(s):
    return len(s)

The function returns the number of items in the string, and each item is one code point. The function exists only to name the job. In a language that counts smaller units, the function would need a loop over the code points instead.

4. For the string "a😀b", what is s[2]? Explain the answer in one sentence.

Show the answer

It is b, because the emoji is one code point and sits at index 1. Index 2 is the b after it. In a language that counts 16-bit units, index 2 would be the second half of the emoji.

5. Reverse the string "ab😀" and check that the emoji is still whole. Write the expected result as a Python string literal.

Show the answer

The reverse is "😀ba", which is the emoji first, then b, then a. The expression "ab😀"[::-1] == "😀ba" evaluates to True.

Summary

• A Python str is a sequence of code points, so len counts code points, and an emoji counts as one.
• UTF-8 bytes are a different count. Use encode when you write data to a file or a network connection.
• A visible character can be several code points, such as e plus a combining accent. Use unicodedata.normalize to compare forms, and a grapheme cluster tool when the count must match the screen.
• Slices cut at code point positions, so they never split an emoji. Cutting bytes can split a character, and decoding then fails.
• Reversing by code point keeps emoji whole but can move a combining accent onto the wrong letter.