Augmented assignment and fixed-size numbers in Python

Why this matters

An augmented assignment is a short way to change a variable using itself. The forms are +=, -=, *=, /=, //=, %= and **=. Most loops that add up a total use one of them, and most of the time they do exactly what they look like they do.

In Python, an augmented assignment never changes the type of a value in a hidden way. x += 1 computes x + 1 and stores the result, and the result keeps the type Python gives it. An int plus an int is an int, with no size limit. The surprises of small integer types come from a different place, the types that have a fixed size. Python has several of them in the standard library: ctypes.c_int8, ctypes.c_int16, ctypes.c_int32 and array.array. When a value is stored in one of those, the rules of this lesson apply.

The earlier lesson on augmented assignment covered the order of operations. This lesson covers what happens to a value when it is stored in a box that has a fixed size, and what a program should do about it.

A value has a fixed number of bits

Every integer type that has a fixed size uses a set number of bits. A bit is one binary digit, either 0 or 1. The more bits a type uses, the more different values it can hold.

An 8-bit signed type can hold 256 different values. Those values are the whole numbers from -128 up to 127. Here are the signed types from the ctypes module, with their ranges:

The range of a type is the lowest and highest value it can hold. A value outside the range does not fit into the box. The ctypes type does not stop the program when that happens, and it does not keep the value as is. It keeps only the bits that fit, and the result wraps around. Wrapping means that going past the top brings you back to the bottom, like the numbers on a clock. Going up from 127 by one gives -128.

A plain Python int has no range at all:

The output is:

1267650600228229401496703205376

The plain integer keeps every digit, so nothing wraps. The rest of this lesson uses fixed-size types to show what wrapping does.

What the augmented form really does

For a plain integer, x += y means x = x + y. The stored value is the sum, with its own type, and nothing is cut.

The output is:

plain int: 128

A ctypes value is a box with a fixed size. Its value attribute is the number inside the box. When you assign to value, the box keeps only the bits that fit:

The output is:

c_int8 after += 1: -128

The augmented form does the same as the plain form here. It computes c.value + 1, which is 128, and then stores 128 in the 8-bit box. The box keeps the low 8 bits, which read as -128. No error is shown.

The same result comes from the plain form, so the two are equal:

The output is:

c_int8 after = + 1: -128

The rule has two steps. First Python works out the sum. Then the box stores the sum, and the box decides what is kept. For a plain integer the box is unlimited, so nothing is lost. For a fixed-size box, the stored value is the sum with the extra bits removed.

Worked example 1: a counter that never reaches its goal

Suppose a loop counts up to 200 with an 8-bit counter. The loop test is count.value < 200. The natural guess is that the loop stops at 200. It does not, because the counter can never be 200. It is an 8-bit signed value, and its largest value is 127.

The next program has a cap on the number of steps, so that it finishes. Without the cap, the loop would run forever.

The output is:

counter after 1001 steps: -23

The count goes from 0 to 127, then wraps to -128, then climbs again. Each full cycle of 256 steps brings the value back to where it started. After 1001 steps there have been three full cycles, which is 768 steps, plus 233 more steps. An 8-bit value holding 233 wraps to 233 minus 256, which is -23. Every value an 8-bit signed counter can hold is at most 127, so the test count.value < 200 is True on every step, and the loop never ends. Python gives no sign that anything went wrong.

The fix is to use a type that holds the largest value you need. For a count up to 200, use a plain int, which has no range, or a 16-bit type such as ctypes.c_int16, which holds values up to 32767.

Here is the same loop with a plain integer:

The output is:

count = 200

Worked example 2: a total of readings

Suppose a sensor sends its readings as 8-bit values, and you add them up. Three readings of 100 should give a total of 300. The loop below adds them to an 8-bit total and to a plain integer total:

The output is:

8-bit total = 44
correct total = 300

Each step of the first loop computes a sum and then stores it in the 8-bit box. After the second reading the total is 200, which is already outside the range, so it wraps to -56. The third step adds 100 to -56, which gives 44. The plain integer total never goes outside any range, so it gives 300.

The rule to remember is that the type of the total decides the range, so a total must be at least as large as the biggest value it can reach. Use a plain int for a total that can grow, and use a fixed-size box only when the total is known to stay inside its range.

Worked example 3: decimal parts and large values

A plain int in Python does not cut off the decimal part when you use += or *=. The value takes the type of the result, so a float stays a float:

The output is:

i after i *= 0.55: 5.5
int(i): 5
j after j += 2.9: 2.9 2

The first line computes 10 times 0.55, which is 5.5. The value stays a float. The function int cuts off the decimal part toward zero and gives 5. The second line adds 2.9 to 0 and gets 2.9. The function int gives 2. The truncation happens only when you ask for it. Python does not cut the value for you.

A 32-bit box keeps only its 32 bits, so a large value wraps in the same way as the 8-bit box:

The output is:

k c_int32 after k += 3000000000: -1294967296

The value 3000000000 is too large for 32 bits. The box keeps the low 32 bits, and the result is 3000000000 minus 4294967296, which is -1294967296.

A 16-bit box behaves the same way. The next program multiplies a 16-bit value by a number that is too big for it:

The output is:

s after s *= 40000: -25536

The 16-bit box gives -25536 because 40000 minus 65536 is -25536.

Worked example 4: a health bar that comes back

A game stores a player's health in an 8-bit box. Health starts at 100. A hit of 250 damage should leave the player at -150, which means the player has no health left, and the program should show that. The plain integer version gives that value.

The output is:

hp after the hit: 106
int hp after the hit: -150

The 8-bit version shows 106. The calculation is 100 minus 250, which is -150. That is below the lowest 8-bit value of -128, so it wraps. Adding 256 to -150 gives 106. The player with no health is now shown with 106 health, and Python gives no error. The plain integer version gives the value the game designer expected.

The table below traces the same steps, one row for each step, the way you would trace them on paper:

Worked example 5: a character code that wraps at the top

A character is also a number underneath. Python's ord function gives the code number of a character, and chr gives the character for a code. Adding 1 to the code of z gives 123, which is the code of {. That is a different character, but the number is still valid, so nothing goes wrong. The function chr only accepts codes up to 1114111, which is the largest Unicode code point, so the top of the range is where the error appears.

The output is:

letter after z + 1: { code 123

The ord function gives the number, so the program prints the code of the next character after the sum. A program that moves letters forward must check which characters it reaches, because the code after z is a symbol, not the next letter of a loop.

Going past the top of the range does not wrap in Python. It is an error:

The output is:

Traceback (most recent call last):
  File "/path/to/main.py", line 1, in <module>
    print(chr(0x110000))
          ~~~^^^^^^^^^^
ValueError: chr() arg not in range(0x110000)

The file path in the first line depends on where you save the file.

A checklist for any fixed-size value

When you read code that stores a number in a fixed-size type, check three things in order:

1. Write down the type of the box, such as c_int8 or array('b').
2. Look up the smallest and largest values for that type in the table above.
3. Work out the largest value the box can reach in the program. If that value can pass the top or the bottom of the range, the wrap can happen.

The right side can also cause a cut. If it is a float and you store it with int(), the decimal part is removed. If it is a number larger than the box, the high bits are removed. Plain integers are not checked against a range at all, so the checklist applies only to the fixed-size boxes.

Why small types are still used

If the wrap rule is so easy to hit, why use a fixed size at all? The answer is memory. The array module stores many values of one type in a compact block. The type code 'b' means one signed byte per value, and 'i' means a value of the C int size. The next program prints the size in bytes of several type codes:

The output is:

b: 1 h: 2 i: 4 q: 8

An array of one million 'b' values takes about one million bytes, and an array of one million 'i' values takes about four million bytes. When a program holds a large amount of data, such as an image or a sound clip, the compact type saves a lot of memory.

The cost is that you must know the range of the values before you choose the type. Choose a small type only when you know the largest value it will ever hold. Otherwise use a plain int.

The array module does not wrap silently. It refuses a value that does not fit:

The output is:

Traceback (most recent call last):
  File "/path/to/main.py", line 4, in <module>
    arr[0] += 1
    ~~~^^^
OverflowError: signed char is greater than maximum

This is the opposite choice from ctypes. An array raises an error at the moment the value does not fit, and a ctypes value wraps without a message. Choose the one that matches what your program should do when the value is out of range.

When a fixed-size type is safe

Plain integers are safe for counters and totals in almost every program, because they grow as needed. A fixed-size type is safe only when the result fits the type for every input the program can receive. For a value that is read from a file or a network, check the range when you read it, not after you have used it in arithmetic.

Strings are not numbers, and += on a string joins the text. Nothing is narrowed in that case:

The output is:

n=4

Common mistakes

Expecting an error from a ctypes value. A ctypes value wraps and prints no message. The same silent wrap happens with a plain Python int if you store it in one, so a program that stores numbers in fixed-size boxes and gives a strange number has probably overflowed one of them. Read each assignment to a fixed-size value and ask what the largest possible value is.

Using a small type for a running total. A total grows with every item added, so it soon passes the range of a small type. Use a plain int for the total, and convert to a fixed size at the very end only if you must.

Assuming a decimal part is rounded. int(2.9) gives 2, not 3. The function int removes the fraction. If you need rounding, use round, but remember that round rounds halves to the nearest even whole number. round(2.5) is 2, and round(3.5) is 4.

Storing a value that is out of range in an array. The array module raises OverflowError when a value does not fit. Check the range before the assignment, or use a ctypes type if a wrap is what you want.

Mixing up the box and the value. The result of b + 1 is a plain integer, but the stored value takes the type of the box. For a ctypes box, the box decides the range.

The output is:

-56 200 255

The same number 200 is -56 in a signed 8-bit box and 200 in an unsigned 8-bit box. The number -1 becomes 255 in the unsigned box, because the box keeps the low 8 bits.

Practice

Exercise 1. Predict the output of this program.

import ctypes
 
n = ctypes.c_int8(-128)
n.value -= 1
print(n.value)

Show the answer

The output is 127. The value -128 minus 1 is -129. That is below the smallest 8-bit value, so it wraps to the largest one, which is 127.

Exercise 2. Predict the output of this program.

import ctypes
 
s = ctypes.c_int16(30000)
s.value += 5000
print(s.value)

Show the answer

The output is -30536. The sum 35000 is above the largest 16-bit signed value, which is 32767. Subtracting 65536 from 35000 gives -30536.

Exercise 3. Predict the output of this program. Explain why the answer is not 8.

total = 0
total += 7.99
print(int(total))

Show the answer

The output is 7. The sum 0 plus 7.99 is 7.99. The function int removes the fraction, so the result is 7. The answer is not 8 because the fraction is removed, not rounded.

Exercise 4. Predict the output of this program.

c = "x"
c = chr(ord(c) + 2)
print(c)

Show the answer

The output is z. The character x has the code number 120. Adding 2 gives 122, which is the code for z.

Exercise 5. Write a function sum_bytes(values) that returns the correct total of a list of 8-bit values, without wrapping. Test it with [100, 100, 100], which should give 300, and with [-100, -100, 50], which should give -150.

Show the answer

The output is 300 -150. The total is a plain integer, so it holds any size of sum. Each value is added as it is, and nothing wraps.

Summary

• For a plain Python int, x += y stores the sum with no size limit and no cut.
• A fixed-size box, such as ctypes.c_int8, keeps only the bits that fit, so a value outside the range wraps around without an error.
• An array with a small type code raises OverflowError instead of wrapping.
• Assigning a float to a plain variable keeps the decimal part. int() removes it, and round() rounds it.
• Use a plain int for totals and counters that can grow, and use a fixed-size type only when the largest value is known.