Context managers and the with statement

Why this matters

A program often borrows something from the operating system. It might open a file, take a lock so two parts of the program do not change the same data at once, or open a network connection. These borrowed things are called resources. The operating system keeps a limited supply of them. A file that stays open uses a slot, and a lock that stays taken blocks every other part of the program that wants it.

The difficulty is the word "always." You must give each resource back on every path through the code: the normal path, the path where the function returns early, and the path where something raises an error. Beginners usually remember the normal path and forget the others. A function that opens a file, reads a line, and then returns early because the line is empty has left the file open. Nothing crashes at once. The leak shows up later, when the program runs out of slots.

Python solves this with the with statement. You write the resource in a with line, and Python calls the resource's cleanup method when the indented block ends. It does this on every path: a normal end, a return, a break, and an exception. A class that works with with is called a context manager. The rest of this lesson shows how to use one, how to write one, and what happens when the body and the cleanup both fail.

This lesson assumes you know classes, __init__, try and except, and the idea of an exception, which is an error that moves control out of the current code.

The with statement for a file

The most common resource is a file. The built-in function open returns a file object, and the file object is a context manager. Its cleanup method closes the file. The next example writes a file and reads it back. Each step uses its own with block.

The output is:

read: first line
read: second line

The writer closes the file when its block ends, before the reader opens it. The program never calls close. If the file were not closed, the data might sit in a memory buffer and never reach the disk, and the reader would see an empty file. Closing the file flushes the buffer, so the data is complete when the reader opens it.

The name after as is the value that the context manager gives back. For a file that is the file object itself. You can use it inside the block, and not after it, because the file is closed by then.

The block and the cleanup on every exit

Now the case that breaks manual cleanup: an early return. The class below is a small resource. It prints when it is opened and when it is closed, so you can see the order of the calls.

The output is:

acquire early
returning early
release early
acquire early
not stopped
release early

The class has two special methods. The method __enter__ runs when the with line starts, and its return value is the name after as. The method __exit__ runs when the block ends. It receives three values that describe any exception that is passing through, or None for each of them when there is no exception.

The early return leaves the function before the last line. The release still runs, because Python calls __exit__ on that path too. With manual cleanup, you would need a second release call before the return, and a third one if another exit appeared later. The with statement puts the release in one place, so it cannot be forgotten.

The return value of __exit__ matters in one case. If it returns a true value, Python swallows an exception that is passing through the block. Returning False, as this class does, lets the exception continue. Return False unless you have a clear reason to hide an error.

Several resources in one with line

A with line can name more than one resource, separated by commas. Python opens them from left to right and closes them in the reverse order. The resource opened last is closed first, so it can still use the ones before it.

The output is:

acquire outer
acquire inner
inside the block
release inner
release outer
after the block

inner was opened after outer, so it is released first. The line print("after the block") runs after both releases, because the with block is finished when the indented part ends.

When the body and the cleanup both fail

The hardest case is an error in the block and a second error in the cleanup. A cleanup can fail, for example when a file cannot be flushed to the disk. Python keeps the first error visible, and the second error becomes the one that is raised, with the first one attached as its context.

The output is:

close failing
caught: close failed
context: ValueError('body failed')

The except block catches the RuntimeError from the cleanup, because that is the error that was raised last. The original ValueError is kept in the attribute __context__, so it is not lost. When the program does not catch the error, Python prints both, one after the other, with the text "During handling of the above exception, another exception occurred".

Python has no list of suppressed errors. The original error is available through __context__, and that is the place to look when a cleanup error hides a real problem.

When the body finishes normally and only the cleanup fails, there is no earlier error to keep:

The output is:

close failing
caught: close failed context: None

The context is None, because there was no earlier error. The cleanup error is the only one.

Writing a context manager with a generator

A class with __enter__ and __exit__ is one way to write a context manager. A shorter way is a generator function marked with @contextmanager from the contextlib module. The function runs up to the yield when the with line starts, and the code after the yield runs when the block ends. A try and finally around the yield makes sure the cleanup runs on every path.

The output is:

open C
use C
close C

The finally block runs when the block ends normally, and it also runs when an exception passes through the yield. Use the generator form when the resource is simple and the cleanup is a few lines. Use a class when the resource has several methods that the caller uses.

Locks and the with statement

A lock is a resource that makes sure only one part of a program changes some data at a time. The threading module's Lock is a context manager. Its with form takes the lock when the block starts and releases it when the block ends, even if an exception is raised inside.

The output is:

caught: failed while locked
locked? False

The exception passed through the block, and the lock was released before the except block ran. Without the with form, the code would call lock.acquire(), change the data, and then call lock.release(). The exception would skip the release, and the lock would stay taken. Every other part of the program that waited for the lock would wait forever.

Common mistakes

Opening a resource without with

A file opened with open and never closed stays open until Python frees the file object. In CPython, the usual version of Python, an object is freed when nothing refers to it, so a file in a function is usually closed when the function returns. That behaviour is not guaranteed, and other Python implementations may close the file later. The safe habit is to use with every time.

f = open("notes.txt")
print(f.readline())

The code works, but the file stays open until the object is freed. The fix is to write with open("notes.txt") as f:, so the close happens at the end of the block.

Using the name after the block

The name after as still exists after the block, because Python has no block scope for names. The object, however, is closed. Reading from a closed file raises an error. The next example shows it:

The output is:

closed? True
ValueError: I/O operation on closed file.

The name f is still there, but the file is closed. Do the work inside the block, and do not keep using the name after the block ends.

An object without __exit__

An object that has no __enter__ and __exit__ methods cannot be used with with. Python raises an error when the with line starts:

The output is:

TypeError: 'Plain' object does not support the context manager protocol (missed __exit__ method)

The message names the missing method. A method called close alone does not make a with line work. Add __enter__ and __exit__, or use the @contextmanager form.

Relying on __del__ for timing

A class can define __del__, which Python calls when an object is freed. The time of that call is not promised by the language, and it depends on the Python program in use. Use with and an explicit cleanup method for any resource where the timing matters, such as a file, a lock or a network connection. __del__ is a last resort.

Debugging walkthrough: a resource that stays open

Suppose a program runs out of open files, or a lock is never released, and nothing crashes at once. Use this routine.

1. Search the code for the calls that open the resource, such as open(, acquire( or a connection function. Count them.
2. For each call, check whether it sits in a with line. A call that does not is the first suspect.
3. For a context manager you wrote, check that __exit__ does the cleanup and that it runs on every path. Add a print to __enter__ and __exit__ to see the order.
4. Move the resource into a with line, or into a function that holds it for a short time.
5. Run the program again and check the count of open files or the lock state.

Variations and edge cases

A context manager can return a value that is not the object itself. The open function returns the file object, but another resource might return a connection or a cursor. The name after as is whatever __enter__ returns.

The contextlib module also has suppress, which swallows a named exception type inside a block. It is useful for one specific case, such as a file that may not exist, and it should be used with care, because it hides the error.

Edge case: a return inside a with block releases the resource before the value leaves the function. The value is computed first, then __exit__ runs, and then the caller gets the value. Make sure the value does not depend on the resource after it is closed.

Interview-style questions

What does the with statement do? It calls __enter__ when the block starts and __exit__ when the block ends, on every path, including an exception.

What is a context manager? An object with __enter__ and __exit__ methods, or a generator function marked with @contextmanager.

What happens when both the block and the cleanup fail? The cleanup's error is raised. The block's error is kept in the __context__ attribute of that error.

Why is with better than a try and finally written by hand? It cannot be forgotten, and it is shorter. The cleanup code lives with the resource, not with each caller.

When does Python close a file that was opened without with? When the file object is freed, which is usually when nothing refers to it any more. The timing is not guaranteed by the language.

Real-world uses

A web application opens a database connection at the start of a request and returns it to the pool at the end, using with. A logging library writes each message to a file inside a lock, so two threads cannot mix their lines. A test framework uses a context manager to change a setting for one test and restore it afterwards, even when the test fails.

Practice

Each exercise is followed by its answer. Try to answer before reading the answer.

Exercise 1

Predict the output of this program, then run it. It uses the Guard class from this lesson, with __enter__ printing acquire and __exit__ printing release.

Show the answer

acquire a
acquire b
release b
acquire c
release c
release a

The inner with block for b ends first, so b is released before c is opened. The outer block for a ends last, so a is released at the end.

Exercise 2

Write a context manager called Lamp with __enter__ that prints lights on and __exit__ that prints lights off. Use it in a with line, with a block that prints reading. Predict the output.

Show the answer

Output:

lights on
reading
lights off

The block prints reading between the two calls.

Exercise 3

A context manager's __exit__ raises RuntimeError("stuck"). The block prints body and finishes normally. The except block prints the message and the value of __context__. What is printed?

Show the answer

body
caught stuck context None

The body did not fail, so there is no earlier error to keep. The cleanup error is raised on its own, and its __context__ is None.

Exercise 4

Write a function count_lines(path) that opens a file with with, counts its lines, and returns the count. Test it with a file that has three lines.

Show the answer

The output is 3. The return inside the with block closes the file before the function returns the count.

Exercise 5

A teammate writes lock.acquire(), then changes some data, then lock.release(). Explain what goes wrong if the change raises an error, and rewrite the code with with.

Show the answer

If the change raises an error, the line lock.release() never runs. The lock stays taken, and every other part of the program that waits for it waits forever. The fix is to use the lock as a context manager:

The output is data changed, then locked? False.

The lock is released when the block ends, whether the change finishes or raises an error.

Summary

• The with statement runs the start code when the block begins and the cleanup code when it ends, on every path, including exceptions and early returns.
• A class with __enter__ and __exit__ is a context manager. A generator marked with @contextmanager is a shorter way to write one.
• Several resources in one with line are released in the reverse order they were opened.
• If the block and the cleanup both fail, the cleanup's error is raised, and the block's error is kept in __context__.
• Files, locks and connections should always be used with with. Do not rely on __del__ for timing.