Bit masks in Python

Why this matters

A program often needs many yes-or-no settings: a light that is on or off, a file that can be read or written, a network packet that has a flag set. One way to store them is one bool variable per setting. Another way is one integer where each bit is one setting. The second way uses one number, can be saved to a file in one step, and matches how hardware and file formats store flags.

A bit mask is a number used to pick out or change particular bits. The bitwise operators (&, |, ^, ~) and the shift operators (<<, >>) are covered in the bitwise operators lesson. This lesson uses them for one job: setting, clearing, toggling and testing single bits, and combining several bits into one mask.

A mask is a number with one bit chosen

Bits are counted from 0 at the right. Bit 0 is the rightmost column, bit 1 is the next column to the left, and so on. A mask for bit n is a number with a 1 in column n and 0 everywhere else. The easy way to build it is 1 << n, which moves the single 1 from column 0 to column n:

Each mask has exactly one 1, so each mask names exactly one setting. The four operations below all use a mask in the same way.

• Test a bit. (settings & mask) != 0 is True when the bit is 1.
• Set a bit. settings | mask turns the bit on and leaves the others alone.
• Clear a bit. settings & ~mask turns the bit off and leaves the others alone.
• Toggle a bit. settings ^ mask flips the bit: 0 becomes 1, and 1 becomes 0.

The test uses != 0 and not == 1. The reason is in the common mistakes section.

Worked example one: a light with four settings

Suppose a smart light stores four settings in the low four bits of one integer. Bit 0 is power, bit 1 is dim mode, bit 2 is colour mode, and bit 3 is a timer. Each name becomes a constant with a fixed mask. Python has no keyword for constants, so the convention is a name in capital letters that the program does not change.

POWER = 1 << 0   # bit 0 = 1
DIM = 1 << 1     # bit 1 = 2
COLOUR = 1 << 2  # bit 2 = 4
TIMER = 1 << 3   # bit 3 = 8

Next come four small functions, one for each operation. Each function is one line, so the expression is easy to check against the rules above:

def is_set(settings, mask):
    return (settings & mask) != 0
 
def set_bit(settings, mask):
    return settings | mask
 
def clear_bit(settings, mask):
    return settings & ~mask
 
def toggle(settings, mask):
    return settings ^ mask

The name set_bit is used instead of set, because set is a built-in type in Python, and a program that uses the built-in later would fail with a confusing error.

A helper prints the value as eight binary columns, so each step is easy to see. The 0xFF in the helper keeps only the lowest eight bits of the number. The 0x prefix means the number is written in hexadecimal (base 16), and 0xFF is the same as 255, which is eight 1s in binary. The format code 08b writes the number in binary with eight digits, padded with zeros on the left:

def bits(n):
    return format(n & 0xFF, "08b")

Now run a short sequence of steps on one value. The full program has the constants, the functions and the helper above it:

The output is:

start        00000000 0
set POWER    00000001 1
set TIMER    00001001 9
isSet DIM    False
isSet TIMER  True
toggle DIM   00001011 11
toggle DIM   00001001 9
clear TIMER  00000001 1

Trace the binary by hand for two of the steps. Start with s equal to 00001001, which is 9.

• Test DIM. The mask is 00000010. The AND is 00001001 & 00000010, which is 00000000. The result is 0, so != 0 is False.
• Clear TIMER. The mask is 00001000, and ~mask is ...11110111. The AND is 00001001 & ...11110111, which is 00000001. Bit 3 is now 0, and bit 0 is still 1.

Toggle DIM twice and you get the original value back. The first toggle sets bit 1 (9 becomes 11), and the second toggle clears it again (11 becomes 9). That is the reason toggle is useful: applying it twice undoes it.

Worked example two: several bits in one mask

A mask can have more than one 1. For example, POWER | COLOUR has bits 0 and 2 set, so it is 101, which is 5. This mask names two settings at once, and there are two different questions you might ask about it.

• Are all of them on? Use (value & both) == both.
• Is any of them on? Use (value & both) != 0.

The program below tests both questions on two values, then clears both bits from a third value:

The output is:

all of POWER|COLOUR in 5? True
any of POWER|COLOUR in 5? True
all of POWER|COLOUR in 4? False
any of POWER|COLOUR in 4? True
clear both from 15: 00001010 10

Walk through the second value, u = 4 (binary 0100). Its AND with both is 0100 & 0101, which is 0100. That is not equal to both (which is 0101), so "all of" is False. It is not zero, so "any of" is True, because bit 2 is on.

For the clear, 15 is 1111, and ~both is ...11111010. The AND is 1010, which is 10. Both named bits are cleared and the other two are unchanged.

A mask built from a list of positions works the same way. Start with 0 and OR in one 1 << pos for each position. The |= operator does mask = mask | (1 << pos), so the mask grows one bit at a time:

The output is:

41
0b101001

The three positions give the bit values 1, 8 and 32. Their sum is 41, and the binary text 101001 has 1s in columns 0, 3 and 5. The bin function writes the binary digits with the prefix 0b.

Worked example three: permission letters

Unix file permissions are a classic bit mask. A file has three permissions: read, write and execute. Each one is a bit: read is 4, write is 2 and execute is 1. A value of 7 means all three, and 5 means read and execute with no write.

The function below turns a mask into the three letters rwx. It loops over the three positions. For each position it builds the mask 1 << (2 - i), which is 4 then 2 then 1, and shows the letter if the bit is set:

The output is:

rwx r-x rw- ---
add exec: 7 rwx
remove write: 5 r-x

The value 6 is rw-, which is read and write. Setting execute gives 7, which is rwx. Clearing write gives 5, which is r-x. Each change touches one bit, and the other bits keep their value.

Shifts and the size of an integer

Building masks with 1 << n has a limit in many languages, because an int there has 32 bits, and bit 31 is its sign bit. Python integers have no fixed size, so none of those limits apply:

The output is:

1 << 31 = 2147483648
1 << 32 = 4294967296
1 << 40 = 1099511627776

No bit is treated as a sign bit, and the shift count is not wrapped. A mask above bit 31 needs no special literal. The arithmetic and bitwise lessons explain the negative case, where Python shows a negative number with an endless row of leading 1 bits.

A fixed-size format still has a size. A file header or a hardware register may have 32 bits, and the mask must fit in those bits. In that case, check the highest bit you use, or mask the value with & 0xFFFFFFFF before you write it.

Masks or a set type

Python's standard library has a type for a set of named flags: enum.Flag. It holds named constants, and it combines them with the same operators as a mask. The program below does the same job as the light example, with names instead of numbers:

The output is:

set: Setting.POWER|TIMER
has DIM? False
after remove: Setting.POWER
after add: Setting.POWER|COLOUR

The printed names are listed in the order the flags are declared in the class, and the value inside is still the same bit pattern.

The two approaches have different trade-offs.

• With enum.Flag, a setting name that does not exist is an error, and printing the value shows the names. Code is easier to read.
• With a plain integer mask, the whole set is one number. It is easy to save, send over a network or write to a hardware register. A wrong bit position is not caught, so the mistake shows up only when the program runs.

Use plain integer masks when the number itself must match an outside format, such as a file header or a device register. Use enum.Flag when the flags exist only inside your program and readable names matter more than the number. A plain set of names is a third choice, for unordered groups of values that do not need bit operations at all.

Common mistakes

Mistake 1: negating before shifting

~1 << 2 is read as (~1) << 2, because the ~ is a unary operator and binds tighter than the shift. The result is not a mask for bit 2. Python gives no error, and the wrong value appears later in the program. Put the shift in parentheses before the ~:

The output is:

~1 << 2 = -8
~(1 << 2) = -5

The second line is the mask with bit 2 cleared. Everything except bit 2 is 1 in the result, which is -5 in Python's notation for negative numbers.

Mistake 2: clearing with OR

To clear a bit you must AND with the inverted mask. Using | leaves the target bit as it was and sets all the other bits to 1. Here is the wrong line next to the right one, with the value 1010 (decimal 10) and the mask for bit 1:

The output is:

wrong clear  x | ~mask: -1
right clear  x & ~mask: 8

The wrong line produces -1, which in Python's notation has every bit set. The right line produces 8, which is 1000, so bit 1 is cleared and bit 3 is still set.

Mistake 3: toggling when you meant to set

Toggle flips the bit. If the bit is already 1, toggling turns it off. Use | when you want the bit on no matter what it was before:

The output is:

on ^ DIM when already set: 0

The value 2 has bit 1 set. Toggling bit 1 gives 0, and that is the opposite of the setting you wanted.

Mistake 4: a test without brackets

In Python, & ranks above ==, so flags & 4 == 4 means (flags & 4) == 4. That works, but other languages read the same text differently, and a reader who knows them may be misled. Write the brackets: (flags & 4) == 4.

Mistake 5: checking a multi-bit mask with the wrong test

If the mask has several bits, == 1 and != 0 answer different questions. In the example below, bit 0 and bit 2 are both on, so POWER is set, but the test == 1 is False, because t & both is 5 and not 1:

The output is:

(t & both) == 1 gives False
(t & both) == both gives True

For "all of these bits", compare with the mask itself. For "any of these bits", compare with 0. Testing against 1 only works for the mask 1, which is bit 0.

Practice

Each exercise has its answer below it. Work on paper before you read the answer.

Exercise 1. Write a Python expression for a mask with bits 0 and 5 set, and give its value.

Show the answer

The expression is (1 << 0) | (1 << 5). Bit 0 is 1 and bit 5 is 32, so the value is 33.

Exercise 2. The value n is 22, which is 10110 in binary. What does n & ~(1 << 2) give?

Show the answer

Bit 2 of 22 is 1, because 22 is 16 + 4 + 2. Clearing bit 2 removes the 4, which leaves 16 + 2, which is 18. The result is 18.

Exercise 3. The value flags is 6. Toggle bit 0, then set bit 3. What is the final value?

Show the answer

Toggling bit 0 changes 6 (0110) to 7 (0111). Setting bit 3 with | 8 gives 15 (1111). The final value is 15.

Exercise 4. Write a function has_all(flags, required) that returns True only when every bit set in required is also set in flags.

Show the answer

The output is True False. For flags 7 and required 5, the AND is 5, which equals required, so the result is True. For flags 4 and required 5, the AND is 4, which does not equal 5, so the result is False.

Exercise 5. A student wants to clear bit 3 and writes m = ~1 << 3. What is wrong, and what should the line be?

Show the answer

The ~ applies to the 1 before the shift, so ~1 is -2 (all bits set except bit 0), and shifting that left by 3 gives -16, not a mask for bit 3. The fix is m = ~(1 << 3), which gives -9, with only bit 3 cleared. The student should also use value & m to clear the bit, not value | m.

Summary

• A mask is a number with a 1 in the column you care about. 1 << n builds the mask for bit n.
• Test with (x & mask) != 0, set with x | mask, clear with x & ~mask and toggle with x ^ mask.
• Wrap the shift in parentheses before ~: ~(1 << n), not ~1 << n.
• For several bits, use (x & mask) == mask to check all of them and (x & mask) != 0 to check any of them.
• Python integers have no fixed size, so a mask above bit 31 needs no special literal. A fixed-size format still has a size to respect.
• Use an integer mask when the number must match an outside format, and enum.Flag when readable names matter more.