Callbacks and passing functions in Python

Why this matters

Some functions need to do a job that depends on a rule the caller chooses. A sort needs to know how to order two items. A file copier needs to report progress to whoever started the copy. A button needs to run whatever code the screen designer attached to it. In each case the function should not hard-code the rule. The caller should be able to hand the rule in.

This idea is called a callback. A callback is a function that you pass to another function, and that function calls it back at a later point. Python makes this easy, because a function is a value. You can store it in a variable, pass it as an argument and call it through the variable. Many of Python's built-in helpers, such as sorted and max, accept callbacks, so this pattern appears often in real code.

If you know Java, a Python function value does the job that an interface and a lambda do there. You do not need to declare an interface first.

The idea: a note with a phone number

Here is an everyday example. You order food at a busy restaurant. The host says, "We will call you when your table is ready." You leave your phone number and go for a walk. The restaurant does not wait for you to come back and check. It calls you at the right moment, and you walk back and sit down.

Map this onto code. The restaurant is a function. The phone number you leave is the callback. The function does its own work, and at the right moment it calls the callback. The function decides when to call. The caller decides what happens when the call comes.

Two words are worth fixing in your memory now.

• A function is a named block of code that does one job. The lesson on parameters covered functions.
• A parameter is a name in a function header that receives a value. A callback is a parameter whose value is a function rather than a number or text.

What a function value is

In C, a function has an address in memory, like a house has a street address. A function pointer is a variable that stores that address, and C programmers use it for callbacks. Python does not expose addresses like this. In Python, the name of a function is a variable that refers to a function object. Writing the name alone gives you the function. Writing the name with parentheses calls it.

Output:

<class 'function'>
5
5

The line f = add does not call add. It gives the same function a second name. Programmers call this property "first-class": a function can be stored in a variable, passed to another function and returned from one, like any other value.

Example 1: passing an operation to a function

Start with a calculator that takes two numbers and applies an operation to them. The operation is the callback. The function calculate does not know whether the operation is addition or multiplication. The caller chooses.

Output:

9
18
3
1024

Inside calculate, the line op(a, b) calls whatever function the caller passed in. The name op is the parameter, and it holds a function.

The call calculate(6, 3, add) passes the function add. There are no parentheses after add. Writing add(1, 2) would call the function right away and pass its result instead. The common mistakes section shows that error.

The call calculate(6, 3, lambda a, b: a - b) passes a lambda. A lambda is a short function with no name, written in one line. Its form is the word lambda, the parameter names, a colon and one expression. The value of that expression is what the function returns. The lambda lambda a, b: a - b does the same job as def subtract(a, b): return a - b. Lambdas suit short functions that you use once.

The function power uses a loop. The underscore in for _ in range(b) is a name that says "I do not need this value". The loop runs b times.

The point of the example is that calculate never changes. You can pass in a new operation without editing calculate. That is the main benefit of a callback: the function that calls the rule stays the same, and the rule can vary.

Example 2: a callback that reports progress

The next example adds up a list of numbers and reports progress after each one. The reporting rule is not part of the adding. A console program might print a line. A graphical program might move a progress bar. A test might check the numbers. The adding function should work with all of them.

Output:

1 of 6 added
2 of 6 added
3 of 6 added
4 of 6 added
5 of 6 added
6 of 6 added
sum = 108
finished 6 items
quiet sum = 108

The function sum_with_progress calls on_progress once per item, with two numbers: how many items are done and how many there are in total. It never asks what the callback does with them. The function enumerate gives a pair for each item, the position and the value. Positions start at 0, so i + 1 counts from 1.

The first call passes a lambda that prints a line for each item. The second call passes only_last, a normal function that prints only when the last item is done. Both calls use the same sum_with_progress and get the same total, 108. Only the reporting differs.

The second callback uses a normal def because it has an if statement. The body of a lambda must be a single expression, and an if statement is not one. A normal function has no such limit.

The names done and count in the lambda are new names for the values that arrive. They do not have to match the names used inside sum_with_progress. Here the lambda uses count so that it does not look like it is reading the outer total.

Example 3: callbacks stored and called later

Examples 1 and 2 called the callback during one function call. Callbacks are often different. The receiver stores the callback, and the call happens much later, maybe after the user clicks something. This is the basis of event handling in graphical programs.

The next program has a Button class. Code can attach any number of handlers to a button. When the button is clicked, the handlers run in the order they were added.

Output:

handlers registered, nothing has run yet
Save clicked
writing file
save count is 1
Save clicked
writing file
save count is 2

The method on_click adds the function to a list. Nothing runs at that point, which is why the line handlers registered prints before the first click. The method click calls each handler with handler().

The function make_save_handler creates a variable count and then defines save inside itself. The inner function save keeps access to count after make_save_handler has returned. A function that remembers variables from the place where it was created is called a closure. Python builds this in, so you do not have to set anything up.

The line nonlocal count says that count belongs to the outer function. Without it, the line count += 1 would try to create a new local variable. The mistakes section shows that error.

This example differs from example 1 in one way. The button keeps the callback and may run it many times, long after on_click has returned.

The common kinds of callback

Java has a set of standard callback types, such as Runnable and Predicate. Python has no such list. Any function, or any object you can call with parentheses, works as a callback. The table below gives the job each common kind of callback does, and the Python form of it.

The next program uses the built-in tools that take callbacks. The function keep_if keeps the items for which test returns True. The function for_each_item calls action on each item. The function compose builds a new function that runs one function and then another.

Output:

[8, 14]
hello ada
hello linus
hello grace
11

The line [item for item in items if test(item)] is a list comprehension. It builds a new list from a loop with a condition, in one line. The test lambda n: n % 2 == 0 is true for even numbers, so the result is [8, 14].

Python has no built-in compose, so the function writes its own. The expression compose(double, add_one)(5) first doubles 5 to get 10, then adds one to get 11. The returned lambda does not change double or add_one, so both still work on their own.

The built-in functions sorted, max, filter and map also take callbacks. The next program uses them.

Output:

['fig', 'kiwi', 'apple', 'banana']
['banana', 'apple', 'fig', 'kiwi']
banana
['fig']
['APPLE', 'KIWI', 'BANANA', 'FIG']

The key argument is a callback that returns the value to compare. The function len is itself a function, so key=len passes it directly. The second sorted compares the last letter of each word, which w[-1] reads. The function max returns the item with the largest key. The function filter keeps the items for which the callback returns True. The function map calls the callback on each item. Both filter and map return an iterator, which gives items one at a time, so list() collects them into a list. str.upper is the upper-case method of the string type, passed in without parentheses.

Python type hints can describe a callback. The next function declares that op takes two integers and returns an integer. Type hints are notes. Python does not check them while the code runs.

Output:

20

The type Callable[[int, int], int] means "a function that takes two integers and returns an integer".

Common mistakes

Mistake 1: a lambda reads the variable when it runs, not when it is made. Inside a loop, every lambda below refers to the same variable i. When the lambdas run, i has its last value.

Output:

[2, 2, 2]

The fix is to give each lambda its own copy of the value with a default parameter, written i=i. The default is filled in when the lambda is created.

Output:

[0, 1, 2]

The same thing happens when a variable changes after a function is made:

Output:

21

Mistake 2: calling the function when you meant to pass it. The call calculate(6, 3, add(1, 2)) runs add right away. The third argument becomes the number 3, and calculate then tries to call the number:

def calculate(a, b, op):
    return op(a, b)
 
 
def add(a, b):
    return a + b
 
 
print(calculate(6, 3, add(1, 2)))

Output, last line only:

TypeError: 'int' object is not callable

Remove the parentheses and the arguments and pass add. Or pass a lambda, such as lambda x, y: add(x, y), which calls add later.

Mistake 3: a callback with the wrong number of parameters. The function calculate calls the callback with two values, so the callback must take two parameters. A lambda with one parameter fails when it is called:

def calculate(a, b, op):
    return op(a, b)
 
 
print(calculate(6, 3, lambda a: a * 2))

Output, last line only:

TypeError: <lambda>() takes 1 positional argument but 2 were given

Count the parameters in the lambda and in the call. The message names <lambda> because the function has no name.

Mistake 4: a lambda cannot hold a statement. A lambda body is one expression. An if statement or a return is not an expression, so this line is a syntax error:

def f():
    x = lambda done, total: if done == total: print(done)

Output, last line only:

SyntaxError: invalid syntax

Write the callback with def instead, as only_last did in example 2.

Mistake 5: a None callback. The value None is not a function. Calling it fails:

def calculate(a, b, op):
    return op(a, b)
 
 
print(calculate(6, 3, None))

Output, last line only:

TypeError: 'NoneType' object is not callable

Check for None at the top of the function when callers might pass it. The practice exercise shows the check.

Mistake 6: changing an outer variable without nonlocal. A function that assigns to a name makes that name local to itself. Here count is assigned inside bump, so Python treats it as a new local variable. Then the first read of count fails:

Output, last line only:

UnboundLocalError: cannot access local variable 'count' where it is not associated with a value

Add nonlocal count as the first line of bump, as make_save_handler does in example 3.

Mistake 7: an error inside a callback stops the caller. Python has no checked exceptions, so nothing forces the caller to handle an error from a callback. The error goes up through the function that called the callback. It stops the program unless some caller catches it. Here the callback divides by zero:

def calculate(a, b, op):
    return op(a, b)
 
 
print(calculate(6, 0, lambda a, b: a / b))

Output, last line only:

ZeroDivisionError: division by zero

Practice

Exercise 1: count matching words. Write a function count_matches(words, check) that takes a list of words and a function check, and returns how many words pass the check. Test it with the list "apple", "kiwi", "banana", "fig". Count the words longer than four letters, then count the words that start with the letter f.

Show the answer

Expected output: 2 and then 1. The words longer than four letters are apple and banana. The only word starting with f is fig.

def count_matches(words, check):
    return sum(1 for w in words if check(w))
 
 
words = ["apple", "kiwi", "banana", "fig"]
print(count_matches(words, lambda s: len(s) > 4))
print(count_matches(words, lambda s: s.startswith("f")))

The expression 1 for w in words if check(w) produces a 1 for each word that passes, and sum adds the 1s.

Exercise 2: predict the order. Using the Button class from example 3, what does this code print?

go = Button("Go")
go.on_click(lambda: print("first"))
go.on_click(lambda: print("second"))
go.click()

Show the answer

It prints three lines: Go clicked, then first, then second. The handlers run in the order they were added, and click prints the button's name before it runs any of them.

Exercise 3: keep the earlier value. The code below prints 21, not 11, because the function reads base when it runs. Change it so the function prints 11, and base still holds 20 at the end.

base = 10
 
 
def show():
    return base + 1
 
 
base = 20
print(show())

Show the answer

Copy the value into a new variable before base changes. The function reads the copy, and base can hold any value.

base = 10
shown_base = base
 
 
def show():
    return shown_base + 1
 
 
base = 20
print(show(), base)

This prints 11 20. The function reads shown_base, which holds 10, so the result is 11. The variable base ends at 20, but the function never reads it.

Exercise 4: apply a function twice. Write apply_twice(x, f) that calls f on x, then calls f again on the result, and returns the final value. Test it with lambda n: n + 3 on the input 5, and with lambda n: n * n on the input 2.

Show the answer

Expected output: 11 and then 16. Five plus three is eight, and eight plus three is eleven. Two squared is four, and four squared is sixteen.

def apply_twice(x, f):
    return f(f(x))
 
 
print(apply_twice(5, lambda n: n + 3))
print(apply_twice(2, lambda n: n * n))

Exercise 5: guard against a None callback. Write calculate(a, b, op) so that a None callback raises a clear error with the message op must not be None. Call it once with a function that multiplies, then once with None. Write down what the first call prints and what the second call raises.

Show the answer

The first call prints 18, because 6 times 3 is 18. The second call raises a ValueError with the message op must not be None. The check runs before the callback is used, so the error points at the start of the function rather than at the failing call inside it.

def calculate(a, b, op):
    if op is None:
        raise ValueError("op must not be None")
    return op(a, b)
 
 
print(calculate(6, 3, lambda x, y: x * y))
print(calculate(6, 3, None))

Output, with the traceback lines above the last one left out:

18
ValueError: op must not be None

Summary

• A callback is a function passed to another function, which the other function calls later. It lets the caller choose a rule without changing the function that uses it.
• In Python a function is a value. You can store it, pass it and return it, so no interface is needed.
• Lambdas hold one expression. Use def for anything longer, such as a function with an if statement.
• A function that reads an outer variable sees its value at the time it runs. Use a default parameter or a copy to keep an earlier value.
• A None callback fails with TypeError when it is called. Check for it at the start of the function when callers might pass it.