Leaks and detection in Python

Why it matters

A leak is memory that the program keeps but never uses again. One leak in a short script does no harm, because the process ends and the operating system takes the memory back. A leak in a server that runs for weeks does harm. The memory use climbs a little each hour, then the program slows down or stops with a MemoryError.

Lesson 4 described what a leak is in Python. This lesson is about finding one. Python has no new and delete for you to forget, so a leak in pure Python code is always an object that is still reachable from a name, a container or another object, even though the program no longer needs it. The tools in this lesson show you what is alive, how much memory it uses where it was made, and what still holds it.

Lesson 8 covered the design choices that prevent leaks, such as weak references. This lesson covers what to do when one is already in the code.

What a leak looks like from outside

Most leaks show up as one of two things. The process uses more and more memory over time, and it may get slower as the operating system swaps memory to disk. Or the process stops with MemoryError.

Here is a small simulation of that growth. A module-level list keeps every buffer, and a second loop clears the list. The tracemalloc module, from the standard library, counts the memory that Python objects use. Its numbers are in bytes.

Type nothing into the Input box for this program.

memory grew while kept: True
memory returned after clear: True

The first check is true, because 50 buffers of 10,000 bytes each are about half a megabyte, which is well above 400,000 bytes. The second check is true, because cache.clear() removed the last reference to each buffer, and CPython freed them. The exact byte counts depend on the Python version and the machine, so the checks print only whether the numbers are in the expected range.

In a real process, the same shape means the list is never cleared. The memory keeps growing, and nothing in the program reports an error until the limit is reached.

Step 1: count the objects by class

The first tool is a count of live objects by class. The gc module can list the objects that the cyclic collector tracks, with gc.get_objects(). Counting the ones whose type has a given name tells you how many of that class exist.

Most objects of ordinary classes are tracked, so this count works well for your own classes. Some built-in types, such as bytearray, are not tracked, so use a class of your own when you want to watch a count.

The next program counts instances of a class Session before and after it keeps 200 of them in a module-level list.

Type nothing into the Input box for this program.

before: 0
after 200 sessions kept: 200

A count is most useful when you take two of them, a while apart, and compare. The class whose count keeps rising is the suspect. A count that stays flat for a class is not part of the leak, even if the class is large in memory.

The tracemalloc module gives a second view. After tracemalloc.start(), it records where each allocation was made, by file and line. Take a snapshot at two times and compare them. The lines that grew are the places where memory is being kept. The code looks like this. It is not run here, because its output contains file paths that differ on every machine.

import tracemalloc
 
tracemalloc.start()
# ... run a batch of requests ...
first = tracemalloc.take_snapshot()
 
# ... run more requests ...
second = tracemalloc.take_snapshot()
 
for stat in second.compare_to(first, "lineno")[:3]:
    print(stat)

Each line of the output names a file, a line number, the size change and the number of blocks. The line with the largest growth is where the kept memory was created. That tells you where the object is made, but not who holds it. The next step answers that.

Step 2: find what holds the object

The gc.get_referrers(obj) function returns the objects that refer to obj. It is the Python form of asking "what still points at this?". Its result includes some temporary references, such as the frame of the function that asked, so filter the result to the objects you expect.

The next program keeps one session in a module-level list, and then asks which objects refer to it.

Type nothing into the Input box for this program.

the list is one of the holders: True
the session is referenced from the list: True

The first line says the module-level list is a holder. The name registry lives in the module, so it is a root, and the list keeps the session alive for as long as the program runs. A holder that is a module-level name, a long-lived object or a cache is the usual cause of a leak.

Use get_referrers on one object you care about, not on everything. It walks the references of the whole program, so it can be slow on a large program.

Step 3: check one object with a weak probe

Sometimes you want to test one suspect object directly. A probe does this. Keep a weak reference to the object, which does not keep it alive. Remove the strong references that you know about. Then run the collector. If the weak reference still returns the object, something else still holds it.

The next program uses a module-level list as the holder. It adds an object to the list, drops its own name, and then clears the list.

Type nothing into the Input box for this program.

alive while the registry holds it: True
alive after the registry is cleared: False

While the list holds the object, the probe still sees it. After clear(), nothing holds it, and the probe returns None. The probe confirms that the list was the only thing keeping the object alive.

Here is a case where the probe stays alive after a holder is cleared, because another name still holds the object:

Type nothing into the Input box for this program.

still alive because another name holds it: True
gone once the last name is removed: True

If a probe stays alive after the holder you know about is cleared, use gc.get_referrers on the object to find the other holder. Then follow the holders up until you reach a name that should not be there.

Use the probe on one object you believe should be freed. Do not use it as a count of everything in the program. For that, use the object count from step 1.

Step 4: fix the holder

Finding the holder is the detection. The fix depends on why the holder exists. A list that is used as a cache needs a size limit or an expiry rule, so entries leave when they are old or when the cache is full. Lesson 8 showed a weak cache, where the entries leave on their own when nothing else uses them. A list of callbacks needs a remove call when the callback is no longer needed. A dictionary of extra data about objects should use weak keys, as lesson 8 showed with WeakKeyDictionary.

Calling gc.collect() in the program does not fix a leak. It asks the collector to search for cycles, and the collector cannot free an object that a module-level name still holds. The fix is always to remove or limit the reference.

Memory that Python code cannot forget to free

In C and C++, a program can call new and never call delete. The memory is then lost until the process ends, and a sanitizer such as AddressSanitizer can report the allocation with its line number. Python code does not do this. The interpreter frees an object when its last reference goes, and the cyclic collector frees cycles. So a leak in pure Python is always an object that is still reachable.

A C extension, which is a library written in C that Python code calls, can still forget to free memory. Tools for that case work at the level of the C code, and they are outside this lesson.

Common mistakes

Reading a quiet result as proof. A tracemalloc comparison shows only the allocations made after tracemalloc.start(). Memory allocated before the start is not counted. Start the tracing before the code you want to measure, and check the code path that creates and drops the objects.

Taking one snapshot and calling it a leak. A large count is normal for some classes, and the interpreter itself creates many objects. A leak is a count that keeps rising across several snapshots while the program does steady work. Compare two or more.

Missing a leak in a module-level name. A module-level list or dictionary keeps its items alive for the whole run of the program. Tools report such items as reachable, and they are not counted as garbage. Read the code for module-level collections yourself.

Making memory bigger to hide the leak. Giving the process more memory lets it run longer before it fails. The leak keeps growing, and the failure arrives later. Find the holder first.

Expecting gc.collect() to fix or prove anything. It is a request to search for cycles. It cannot free objects that are still reachable. Use it in a probe for a test, not as a remedy.

Using gc.get_referrers on a whole program in a loop. It walks every reference, so it can be slow. Call it on one suspect object at a time.

Practice

Exercise 1: read the count

A program defines class Session: pass, then runs count_sessions() before a loop, then runs the loop for i in range(3): kept.append(Session()), then runs count_sessions() again. The first call returns 0. What does the second call return, and why?

Show the answer

It returns 3. Each pass of the loop creates one Session, and the list kept holds all three, so each one is still alive when the count runs.

Exercise 2: fix the loop

Change the loop in exercise 1 so that each Session is freed before the next pass, and the count after the loop is 0. Write the changed loop.

Show the answer

Do not keep the object. Write the loop as for i in range(3): Session(). Each object has no name and no container that holds it, so CPython frees it when the statement ends, and the count after the loop is 0.

Exercise 3: a count comparison

Two object counts were taken from a running server, 10 minutes apart. The class Session had 40 instances in the first and 9200 in the second. Every other class stayed about the same. Where do you look next, and why?

Show the answer

Look for the code that creates Session objects and the code that removes them. A count that rises steadily while nothing else changes points to objects that are added but never removed. Use gc.get_referrers on one of the sessions to find the list, dictionary or callback that holds it, and check whether logout or the end of a request removes it.

Exercise 4: the probe stays alive

In the probe program, the last line prints True even though the registry was cleared. Name two things that could cause this, and say how to find out which one applies.

Show the answer

Another strong reference could still point at the object. Two common causes are a name in another function that still holds it, and an attribute of another long-lived object, such as a callback list. Call gc.get_referrers on the object, and follow each holder up until you find the one that should have let go.

Exercise 5: when tracing starts

A program calls tracemalloc.start() after it has already built a large list. A comparison of two snapshots shows that the list does not grow. Why does the comparison miss the list?

Show the answer

tracemalloc records allocations only from the time it starts. The list was built before the start, so its memory is not in the traced snapshots. Start tracing before the code you want to measure.

Summary

• A Python leak is an object that is still reachable from a name, a container or another object, even though the program no longer needs it.
• Count live objects by class with gc.get_objects(), and take the counts at two times to see which class keeps growing.
• tracemalloc shows where memory was allocated, by file and line. Compare two snapshots to find the lines that keep memory.
• gc.get_referrers(obj) shows what still points at one object. Follow the holders until you reach a name that should not be there.
• A weak probe checks whether one suspect object is still held after you remove your own references.
• Fix the holder by limiting it, removing entries when the work ends, or using a weak reference. gc.collect() does not fix a leak.