Code points and emoji

Why this matters

A program stores text as numbers. To count letters, cut a word or reverse a string, the program has to know which numbers belong to one character. For plain English letters the rule is simple: one byte per letter. Once the text has an accented letter or an emoji, a plain count of bytes gives a wrong answer. A program that cuts text at a byte position can split a character in half and print a broken symbol.

This lesson explains what "length" means for text in Java and in C++. It then shows how to read a string one real character at a time. The code is C++. The Java behaviour is described with results I checked by running the same text in Java, so you can see the difference.

The earlier lessons in this module assume each character takes one byte, which is true only for basic English letters. This lesson explains where that assumption breaks.

Four words you need

A code point is one numbered symbol in Unicode. Unicode is the large list of symbols that modern computers use. Each symbol has a number, and that number is its code point. The letter e is code point 101, and people write it as U+0065, using hexadecimal. Hexadecimal is counting in base 16, with the digits 0 to 9 and the letters A to F. You do not need to do arithmetic in hexadecimal here. It is only a shorter way to write a number. The grinning face emoji is U+1F600.

A byte is 8 bits, which is a number from 0 to 255. Memory is measured in bytes.

An encoding is the rule that turns code points into bytes. UTF-8 is the most common encoding. It uses one byte for the basic English letters, digits and punctuation, which are code points 0 to 127. Larger code points take two, three or four bytes. The C++ compiler g++ reads source files as UTF-8 by default, so a string literal written with an accented letter or an emoji is stored as UTF-8 bytes.

A grapheme cluster is what a reader sees as one character. It can be built from several code points. A thumbs up with a skin tone is two code points. A family emoji is several code points joined together. The last example in this lesson shows one.

Length in Java and in C++

Java's String works in UTF-16. In UTF-16 each unit is 16 bits, and most characters fit in one unit. Characters with large code points need two units, called a surrogate pair. The method length() in Java returns the number of 16-bit units, not the number of characters. For the grinning face emoji, Java's length() returns 2. Java's codePointCount method returns 1 for the same string. I checked both results with javac 25 on this machine.

C++ std::string stores bytes. Its size() method returns the number of char values, and a char is one byte. The same emoji takes 4 bytes in UTF-8, so size() returns 4. The same is true for the accented letter é, which takes 2 bytes.

So the same text can have three different lengths: 2 in Java, 4 in C++ bytes, and 1 code point. None of these is wrong. Each one counts something different. The right choice depends on the task. A byte count is right when you need the size of a file on disk. A code point count is right when you need the number of symbols.

Here is a program that prints both numbers for a short string.

The output is:

bytes: 10
code points: 6
text[3] as hex: C3

The string has 10 bytes: c, a and f take 1 byte each, é takes 2 bytes, the space takes 1 byte, and the emoji takes 4 bytes. That adds up to 10. The code points are c, a, f, é, the space and the emoji, which is 6.

The last line shows the danger. text[3] is the first byte of é, which is C3. It is not the letter é. If you print that byte by itself, the terminal shows a broken symbol.

The mask 0xC0 keeps the top two bits of a byte. The value 0x80 is the pattern 10000000. Together, the test asks whether the byte starts with 10. A byte that starts with 10 is a continuation byte, the second or later byte of a character. It cannot be the first byte, so it is skipped.

Walking a string one code point at a time

Counting continuation bytes gives the number of characters. It does not give you the characters. To read them, you need to decode each character from its bytes.

A UTF-8 character starts with a lead byte, and the lead byte tells you how many bytes follow. The pattern of the first bits decides the length.

The bits after the leading pattern hold the value of the code point. Each byte after the lead byte starts with 10, and its last six bits add to the value. Decoding means reading the lead byte, reading the right number of follow-up bytes, and joining the bits.

This program does that. It returns each character as a code point and the number of bytes it used. Then it walks the whole string.

The output is:

bytes: 19
U+0063
U+0061
U+0066
U+00E9
U+0020
U+1F600
U+0020
U+1F44D
U+1F3FD
code points: 9
family bytes: 18, code points: 5

Read the first part of the code slowly. The function decode_at looks at one byte. If the byte is below 0x80, it is a basic English character and uses one byte. Otherwise the lead byte pattern tells it how many follow-up bytes to read. Each follow-up byte must start with 10. If a byte does not fit, the function returns U+FFFD, the replacement character, and moves on by one byte. This is what a careful decoder does with broken input. It does not crash.

The function code_points calls decode_at in a loop and moves forward by the length each call reports. It never reads a character in the middle of another one.

The output shows that the thumbs up U+1F44D and the skin tone U+1F3FD are two separate code points. They are shown as one picture on most screens. The program does not join them. That is the job of a grapheme library, which knows the rules for joining code points into one visible character.

The last line is the family emoji. It is three people joined by two invisible joiners called zero width joiners, shown as U+200D. The text has 18 bytes and 5 code points. A reader sees one picture. A program that counts code points says 5. A program that counts bytes says 18. Neither count is the number of pictures a reader sees.

Finding emoji in a string

Once you can list code points, you can make decisions about them. Emoji are spread across several ranges of code points. The range from U+1F300 to U+1FAFF holds most of the common pictographs, including faces, animals, food and symbols such as the party popper.

The function below counts how many code points fall in that range. Add it to the previous program, below code_points and above main. It uses the same includes.

std::size_t count_pictographs(std::string_view s) {
    std::size_t n = 0;
    for (char32_t cp : code_points(s)) {
        if (cp >= 0x1F300 && cp <= 0x1FAFF) ++n;
    }
    return n;
}

Calling it on "ok \U0001F600 \U0001F389 x \U0001F44D\U0001F3FD" returns 4. The grinning face, the party popper, the thumbs up and the skin tone are all in the range. A range check like this is a rough guess. It misses some symbols outside the range, and it counts the skin tone modifier as a symbol of its own. For exact rules about emoji, use a library that implements the Unicode text segmentation rules.

Cutting text to a number of characters

Apps often show a short preview, such as the first few characters of a title. The C++ method substr cuts by byte positions, so a cut that looks correct on English text can split an accented letter or an emoji. To cut by characters, the program has to step over whole characters, using the decoder from the previous section.

Add this function below code_points in the walker program.

std::string first_n_code_points(std::string_view s, std::size_t n) {
    std::size_t i = 0;
    for (std::size_t count = 0; count < n && i < s.size(); ++count) {
        i += decode_at(s, i).length;  // step over one whole character
    }
    return std::string(s.substr(0, i));
}

Then add these lines at the start of main in the same program.

std::string preview = "café \U0001F600 \U0001F44D\U0001F3FD";
std::string byte_cut = preview.substr(0, 4);
std::cout << "byte cut ends with byte " << std::format("{:02X}", static_cast<unsigned char>(byte_cut.back())) << "\n";
std::string char_cut = first_n_code_points(preview, 4);
std::cout << "code point cut: " << char_cut << " (" << char_cut.size() << " bytes)\n";

The output is:

byte cut ends with byte C3
code point cut: café (5 bytes)

The first four bytes are c, a, f and 0xC3, which is the lead byte of é. The byte cut stops after the lead byte, so the last byte of the cut is half of a character. The character cut counts four whole characters, c, a, f and é, so it keeps both bytes of é. The result is 5 bytes long. The loop moves by whole characters, so the result can never end inside one.

Common mistakes

Indexing bytes as characters. The string "café" has five bytes but four characters. A program that writes text[3] expects the letter é, but gets the first of its two bytes. If you print it, you get a broken symbol. Use a decoder, or a library that works on code points, when the position matters.

Putting a non-ASCII character in a char. A char holds one byte. A single character such as é takes two bytes, so the literal does not fit. The compiler warns about it and keeps only part of the value. Here is the real output from g++ -std=c++20 -Wall for char c = 'é';:

warning: multi-character character constant [-Wmultichar]
warning: overflow in conversion from 'int' to 'char' changes value from '50089' to '-87' [-Woverflow]

The fix is a wider type. char32_t holds one code point, and a literal with the U prefix gives one. char32_t c = U'\U0001F600'; compiles without a warning. The narrow literal '\U0001F600' in a char gives this warning:

warning: multi-character character constant [-Wmultichar]
warning: overflow in conversion from 'int' to 'char' changes value from '-257976192' to '-128' [-Woverflow]

Expecting size() to match what the user sees. The family emoji in the example has a size() of 18 and one visible picture. Do not use size() to decide how many symbols a user will see. Use it to decide how much memory the text uses, or where the next byte is.

Reading past the end. The decoder checks i + extra >= s.size() before reading the follow-up bytes. Without the check, the last lead byte of a broken file would read past the end of the string. A string index beyond the size is undefined behaviour. With -D_GLIBCXX_ASSERTIONS, the standard library reports it. For example, w[5] on the string "abc" stops the program. The message ends with:

Assertion '__pos <= size()' failed.

Practice

Exercise 1: Count bytes and code points

Count the bytes and the code points in "héllo wörld". Use the function count_code_points from the answer.

Show the answer

Bytes: 13. Code points: 11. The letters é and ö take 2 bytes each, and the other nine characters take 1 byte each, so the byte count is 9 + 2 + 2 = 13. The code point count is the number of characters, 11.

#include <cstddef>
#include <iostream>
#include <string>
 
std::size_t count_code_points(const std::string& s) {
    std::size_t n = 0;
    for (char c : s) {
        if ((static_cast<unsigned char>(c) & 0xC0) != 0x80) ++n;
    }
    return n;
}
 
int main() {
    std::string text = "héllo wörld";
    std::cout << "bytes: " << text.size() << "\n";
    std::cout << "code points: " << count_code_points(text) << "\n";
}

Exercise 2: Count the pictographs

Use count_pictographs on the string "ok \U0001F600 \U0001F389 x \U0001F44D\U0001F3FD". Predict the result before you run it.

Show the answer

The result is 4. The string holds the grinning face, the party popper, the thumbs up and the skin tone modifier. All four are in the range from U+1F300 to U+1FAFF.

Exercise 3: Count the code points of a four-person emoji

The string "\U0001F468‍\U0001F469‍\U0001F467‍\U0001F466" shows a family of four. How many bytes and how many code points does it have?

Show the answer

It has 25 bytes and 7 code points. Each person emoji takes 4 bytes, so there are 16 bytes for the people. Each zero width joiner takes 3 bytes, and there are 3 of them, which is 9 bytes. The total is 25. The code points are 4 people plus 3 joiners, which is 7.

Exercise 4: Predict the length of a combining accent

The string "café" shows the word café written as e followed by a combining acute accent, U+0301. The accent sits on top of the e. How many bytes and how many code points does it have?

Show the answer

It has 6 bytes and 5 code points. The letters c, a, f and e take 4 bytes. The combining accent U+0301 takes 2 bytes in UTF-8. The code points are the four letters and the accent, which is 5. A reader sees 4 characters, because the accent sits on the e.

Exercise 5: Explain why the lead byte matters

A decoder reads the byte 0xC3 and then the byte 0xA9. Which code point do they make, and why does the decoder not need to look at any bytes before 0xC3?

Show the answer

They make U+00E9, the letter é. The byte 0xC3 starts with 110, so it is a lead byte for a two-byte character. The next byte 0xA9 starts with 10, so it is a continuation byte. The decoder only needs the lead byte, because a lead byte always marks the start of a new character. Every byte before it belongs to an earlier character, and the decoder has already moved past those.

Summary

• A code point is one numbered symbol. A grapheme cluster is what a reader sees as one character, and it can use several code points.
• Java's length() counts 16-bit units. C++ std::string::size() counts bytes. The same text can give different numbers.
• In UTF-8, a byte that starts with 10 is a continuation byte. Counting the other bytes counts code points.
• A lead byte tells the decoder how many bytes make up the character. Reading the lead byte first is how a program walks real characters.
• Use char32_t and U'' literals for one code point, and a grapheme library when you need what a reader sees.