Debugging lab: dict and set pitfalls in Python

Why this matters

Most bugs with dicts and sets are not crashes at first. The program runs, prints something, and the number is wrong. Or the program runs for a while and then fails on a line that looked harmless. Reading code and guessing is slow. Running it, reading the exact output, and matching that output to a cause is faster.

This lab gives you six small programs. Each one has a bug in one of three families.

• An accidental insert: a read that adds a key to the dict without you meaning to.
• An invalid iterator: a loop or a stored iterator that breaks after the dict changes.
• An assumed order: code that expects a set to print or walk its items in a particular order.

Two extra cases cover a dict literal that silently keeps the wrong value, and a None value that breaks arithmetic.

The neighbouring lessons in this module explain the rules in detail. The lesson on iteration order and changing a dict in a loop covers the loop rules, and the lesson on custom keys covers __eq__ and __hash__. This lab only asks you to find the fault and fix it.

How to work through a debugging lab

Use the same four steps for each case.

1. Write down what you expect the program to print.
2. Run it. Copy the exact output.
3. Compare the output with your expectation. Find the first line that differs.
4. Find the line that causes the difference, and say why in one sentence.

Some terms help when reading the output.

• A key is the name you look up in a dict, such as "doc2". A value is what the dict stores under that key.
• An iterator is an object that walks through a collection one item at a time. It keeps its place and the size the collection had when it was made. If the collection changes size in a way the iterator did not expect, the iterator refuses to continue.
• A traceback is the report Python prints when an error stops the program. Its first lines show the call that failed, and its last line names the error type and a short message. The last line is the one to read first.

Case 1: the read that writes

A program checks how many tags a document has. The author wants to read the count and never change the dict. The dict is a defaultdict(list), a dict that builds a default value for a missing key, so a missing key gets an empty list. Look at the code and predict the output.

count for doc2: 0 stored: True
count for doc3: 0 stored: False
keys now: ['doc1', 'doc2']

The first count is right, since doc2 has no tags. The problem is stored: True. The author asked for a count and got a new key in the dict. The dict now has doc2, which the author never added.

The cause is tags["doc2"]. A defaultdict creates the missing value when you read it, not only when you write it. So the read stored an empty list under doc2, and then len read that new list.

The defaultdict is useful when you want to create a missing entry, such as the first list for a new word. For a read only, use get with a default. The second half of the program does this. tags.get("doc3", []) returns an empty list for doc3 and stores nothing, so stored: False is printed.

The same risk applies to setdefault, which stores the default when the key is missing. Use get when you only want to read.

Case 2: the None that breaks arithmetic

A program keeps scores in a dict. It looks up the score for "bo", who has not played yet, and adds a bonus to it.

score = {"ana": 5}
bo = score.get("bo")
print("bo with bonus:", bo + 1)

TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

dict.get returns None, Python's value for "no value", when the key is missing. The name bo now holds None, and None + 1 has no meaning. The error names both types, so the cause is easy to see on the line that calls get.

If the program had used square brackets instead, the error would come one step earlier:

score = {"ana": 5}
print(score["bo"])

KeyError: 'bo'

The fix is to say what a missing score means, and to put that default in the call:

bo with bonus: 1

If a missing score should stop the program or be handled differently, check for None before you use the value, with if bo is None:.

Case 3: the iterator that went stale

Here the program makes an iterator over the keys of a dict. It then adds a new key, and calls next on the old iterator.

iterator made, map changed
RuntimeError: dictionary changed size during iteration

A dict keeps a count of its size. The iterator saves that size when it is made. On each next call, it compares the saved size with the current one. If they differ, the dict changed in a way the iterator cannot follow, and it raises RuntimeError. The message says what happened.

Here the change was an added key, "lime". Changing the value of a key that already exists is different. The next experiment shows that.

value update, next: 3
RuntimeError: dictionary changed size during iteration

The update to "ana" did not break the iterator, because no key was added or removed. Adding "cy" did break it. The fix for a stored iterator is to make a new iterator after the change. A plain for loop does this for you, but it cannot stop the error when the loop itself changes the dict.

The same bug in a loop

A for loop over a dict uses an iterator internally, so the same rule applies. This version removes entries with a value of 0 while it walks the keys.

stock = {"apple": 0, "pear": 4, "fig": 0, "kiwi": 2, "lime": 1}
for item in stock:
    if stock[item] == 0:
        del stock[item]

RuntimeError: dictionary changed size during iteration

The safe way to remove matching entries is to collect the keys first, then delete them after the loop:

after safe removal: {'kiwi': 2, 'lime': 1, 'pear': 4}

A comprehension that builds a new dict is also safe, and it is the shortest form when you do not need to change the original dict in place.

Case 4: the order you did not ask for

A program adds five fruits to a dict and to a set. It then prints the dict, prints the set, and takes the first key from the dict as if the order were sorted.

dict (insertion order): {'pear': 4, 'apple': 5, 'fig': 3, 'banana': 6, 'cherry': 6}
first key: pear
sorted keys: ['apple', 'banana', 'cherry', 'fig', 'pear']
set of ints: {17, 3, 100, 40, 8}

The dict keeps insertion order, so pear comes first, because it was added first. That order is a promise of the language, and the program can rely on it.

The set of integers does not follow the order in which the values were written, and it does not follow sorted order either. The order comes from the slots (the positions in the set's internal table) the values landed in, and the Python documentation does not promise any order for a set. The output above is what this Python version printed on this run. Another Python version, or the same program with more values, can print a different order.

A set of strings is a stronger case. Its order can change from one run of Python to the next, because by default the hash (the number Python computes from a value to place it) of each string is randomized each time the interpreter starts. So a test that compares the printed text of a set with a fixed string can pass once and fail the next day.

The fix depends on what the program needs.

• If it needs the order of insertion, use a dict, as fruit does. dict.fromkeys(items) builds an ordered set from a list when you need unique items in order.
• If it needs keys in sorted order, use sorted(...) before you print them.
• If it needs only membership checks, the order does not matter, so do not depend on it.

Case 5: a dict literal that keeps the wrong value

Python does not check for repeated keys in a dict literal. The later value wins, and no error is raised. The program below writes a repeated key by mistake, then tries to change a read-only view of a dict.

duplicate literal: {'a': 2}
TypeError: 'mappingproxy' object does not support item assignment
frozenset of duplicates: frozenset({5}) 1
AttributeError: 'frozenset' object has no attribute 'add'

The first line is the surprise. The literal {"a": 1, "a": 2} has no error, and it keeps only the last value, 2. A typo in a long literal can hide this for a long time, because the program still runs. If a repeated key is a mistake in your data, check for it when you build the dict, for example by comparing len(pairs) with the number of distinct keys.

A frozenset silently drops the repeated element, which is correct for a set. The MappingProxyType is a read-only view of a dict. Assigning to it raises TypeError. A frozenset has no add method at all, so calling it raises AttributeError. Use these when you want a value that cannot change after it is built.

Case 6: the membership check that adds people

A program keeps a friend set for each person. For each pair of names, it asks whether the first person knows the second. A person who has never appeared has no friend set yet. The author uses a defaultdict(set) to avoid a KeyError, then checks membership (whether a name is in a collection, written with in) in the result.

ana knows bo? True
dev knows ana? False
eli knows fay? False
people in dict: ['ana', 'dev', 'eli']
size: 3

The answers are right, but the dict now has three people when only one was ever added. The names dev and eli appeared only in questions, and each question stored an empty friend set for them. Anything that later prints the keys, counts people, or saves the dict will include those names.

This is the same accidental insert as case 1. The fix is to read with get and an empty default, which stores nothing:

ana knows bo? True
dev knows ana? False
eli knows fay? False
people in dict: ['ana']
size: 1

Checking a fix

After a change, run the same input again and compare more than the printed answer. Cases 1 and 6 both printed correct answers while the dict held extra keys. A useful check after any change to a dict or set is to print its size and its keys, as the programs above did, and compare them with what you expect.

Run the fixed program on the same input as before. If the output differs in a line that did not change, the fix changed something else. Remove the change and try again on a smaller input, so the difference is easy to see.

Common mistakes

The table below lists the messages you are likely to see in this module and the usual cause of each.

Two more mistakes do not raise an error at all. A program that reads with a defaultdict keeps extra keys and prints a wrong size. A program that prints a set in a test expects one order and gets another after a change to the values. Those are the mistakes to look for when the output is wrong and no error appears.

Practice

Try each exercise before you open its answer.

Exercise 1. This code runs on the same dict twice. What does the last line print?

m1 = {}
m1.setdefault("x", 1)
m1.setdefault("x", 5)
print(m1["x"])

Show the answer

It prints 1. The first call finds no key "x", stores 1, and returns it. The second call finds the key, so it returns the stored 1 and does not change the value. The second default, 5, is never stored.

Exercise 2. A loop goes over the keys of a dict, and the body adds a key with d[k + "2"] = 1. Explain why the program fails, and what the smallest correct change is.

Show the answer

Adding a key changes the dict's size, so the iterator behind the loop sees a size it did not expect. On its next step, it raises RuntimeError: dictionary changed size during iteration. The smallest correct change is to collect the new keys in a list while the loop runs, and add them to the dict after the loop ends. Looping over list(d) also works, because the loop then walks a copy.

Exercise 3. The output of a set looks random. Write one line that prints the same set of strings in sorted order.

Show the answer

sorted: ['a', 'b', 'c']

The sorted function returns a new list, and s itself is not changed.

Exercise 4. This dict literal runs with no error. What does it print, and what is the risk?

pairs = {"a": 1, "b": 1, "a": 3}
print(pairs)

Show the answer

It prints {'a': 3, 'b': 1}. The repeated key "a" keeps its last value, 3, and the first value, 1, is lost. No error appears. To catch this in real data, compare the number of pairs in the input with the number of distinct keys in the dict.

Exercise 5. A function receives a dict wrapped in types.MappingProxyType and calls pop("a") on it. What happens, and why?

Show the answer

It raises AttributeError: 'mappingproxy' object has no attribute 'pop'. A MappingProxyType is a read-only view, and it has only the read methods. The fix is to copy the dict first with dict(view) and call pop on the copy.

Exercise 6. A program counts words with counts[w] = counts[w] + 1. It fails on the first occurrence of each word. Give the line that fixes it, and say what the dict holds afterwards.

Show the answer

The first time a word appears, counts[w] raises KeyError. The fix is counts[w] = counts.get(w, 0) + 1. The first time a word appears, the default of 0 is used, and the dict ends up holding each word once with its count.

{'x': 2, 'y': 1}

Summary

• An accidental insert comes from a defaultdict read or from setdefault, which store a value even when you only meant to read. Use get with a default for reads.
• dict.get returns None for a missing key. Arithmetic on that None raises TypeError, so give get a default.
• Adding or removing a key while a loop or an iterator is walking the dict raises RuntimeError. Collect the keys first, or loop over a copy.
• A set has no promised order, and a set of strings can change order between runs. Use sorted(...) before printing, and use a dict when insertion order matters.
• A dict literal with a repeated key keeps the last value without an error, so check the data when a repeat is a mistake.
• Read the last line of a traceback first. It names the error and the message, and the line above it names the code that failed.