Shadowing in Python

Why it matters

A program that runs without errors can still give wrong answers. One common cause is a name that means one thing in one place and another thing in a different place. The code looks right when you read it quickly, because the same word appears in both places. Python, however, uses the closer name and ignores the one further away.

This is called shadowing. A name in a smaller area hides a name with the same spelling in a bigger area. The bigger name is still there, but the code cannot reach it using only that name.

Shadowing bugs are hard to find because nothing crashes. The function runs, prints a number, and the number is zero when it should be 465. This lesson shows how shadowing happens in Python, which cases Python treats differently from other languages, and how to fix each one. The variable scope lesson explains where a name is visible. This lesson is about what happens when two visible names share the same spelling.

The idea: the nearest name wins

Think of two people named Asha in the same family. At home, when someone says "Asha", everyone knows they mean the daughter who lives there. Her cousin, also named Asha, lives in another city. The cousin is still real, but at home the name points to the daughter. If you want the cousin, you must say something more specific, such as "the Asha from Pune".

Python works the same way. When a function uses a name, Python searches four places in this order: the function itself, any function that contains it, the module (the whole file), and the built-in names that Python provides, such as print and len. It stops at the first place that has the name. The order is often remembered as LEGB, from the first letters of Local, Enclosing, Global and Built-in.

A parameter is a name in a function's header that receives a value when the function is called. A local variable is a name that the function assigns to. A module-level variable is a name assigned at the top of a file, outside every function. A name in a smaller area, such as a parameter, shadows a name in a bigger area, such as a module-level variable.

Here is the simplest case. A module-level variable and a parameter have the same name:

Output:

inside: 81
outside: 0

Inside show_score, the name score is the parameter, so it hides the module-level score. The module-level variable is still 0 after the call. Nothing was overwritten. The parameter only hid the other name while the function ran.

Attributes do not clash with parameters

An attribute is a value stored on an object, and you reach it with a dot. In self.score, the attribute is score and the object is self. In Java, a field and a parameter can share a name, and the bare name inside a method points to the parameter. Python handles this differently. A bare name inside a method never looks at an attribute, so a parameter called score cannot hide self.score.

Python still has a version of the Java bug, because the assignment writes to the bare name. Here the method copies the parameter into itself:

Output:

0

The method was called with 81, but score = score only changes the local name score. It never reaches self.score. Python does not warn about this, so the object keeps its old value.

The fix is to put the dot on the left side of the assignment:

Output:

81

self is the object the method was called on. Python passes it in automatically as the first argument. The name self is a convention, not a keyword, but every Python programmer uses it, so you should too.

Reaching a hidden module-level name

Python has no this for this job, because an attribute is already marked by self.. A module-level name that a parameter or local hides is reached another way. The built-in function globals() returns a dictionary for the whole module. A dictionary is a collection of key and value pairs, and you look up a value by its key. The key here is the name, as text.

Output:

local total: 3
global total: 100
total from module: 100

Inside show, the plain name total means the local. The expression globals()["total"] means the module-level variable. This works, but it is harder to read than a different name, so the next section shows the usual fix.

Fixing shadowing by renaming

When two names are close in meaning and spelling, rename the one in the smaller area. A parameter lasts only for one call, and a module-level variable lasts for the whole run. A good parameter name says what value the caller is giving.

The same program with a new parameter name:

Output:

inside: 81 outside: 0

The parameter is now points, so nothing hides score. Both names are visible in the same line, and each one means what it says.

Shadowing a module-level variable with a local

The next case looks like the Java example with a local variable and a field. In Python, a method that makes a local variable with the same name as an attribute does not shadow the attribute. The example below stores the totals on the object, and the local total is only a plain name.

Output:

inside method: 465
after broken call: 0 0

The method adds the prices correctly into the local total, and it prints 465. The object was never changed. The bug is that the method never writes the answer to self.total. The shared spelling had no effect here.

Shadowing does happen with a module-level variable. In this version, the function's local total hides the module-level total:

Output:

inside function: 465
after broken call: 0 0

The fix uses a different name for the local, and the function writes its result to the module-level names. The global line says that total and items refer to the module-level variables, so the assignments change them:

Output:

after fixed call: 465 3

The local is named subtotal, not sum. A local named sum would hide the built-in function sum() for the rest of that function. The name subtotal avoids that.

Most Python code avoids global when it can, because any function that declares it can change the value. The class version at the start of this section keeps the totals inside the object, which is usually the clearer design.

Class attributes and the class name

In Java, a static field can be reached as Main.limit even when a local hides it. Python has a different rule. A name written in the body of a class, such as limit below, is not visible to the methods by its bare name. A method reaches it with Config.limit or self.limit.

Output:

local limit: 3
class limit: 10

Inside report, the plain name limit means the local. The expression Config.limit means the class attribute. Because a method never sees the class body by a bare name, the following version fails:

class Config:
    limit = 10
 
    def show(self):
        print(limit)
 
Config().show()

Output:

Traceback (most recent call last):
  File "main.py", line 7, in <module>
    Config().show()
    ~~~~~~~~~~~~~~^^
  File "main.py", line 5, in show
    print(limit)
          ^^^^^
NameError: name 'limit' is not defined. Did you mean: 'self.limit'?

The traceback shows the file path where the code runs, so your path will differ. The message NameError means Python looked for limit in the function, then in the module and the built-in names, and found it in none of them. The suggestion self.limit points to the fix.

Nested functions and nonlocal

Java has inner classes, which can have fields with the same name as the outer class. Python has nested functions, which are functions defined inside another function. A nested function can read a name from the function around it. If it assigns to the same name, it creates a new local, which shadows the outer name.

Output:

inner
outer

The assignment inside inner made a new local label. The outer label was not changed. To change the outer name, use nonlocal:

Output:

inner
inner

nonlocal label says that label refers to the name in the nearest enclosing function. Java's Outer.this is different: it reaches an enclosing object, not an enclosing local variable.

Names Python accepts without an error

Java refuses to compile a local variable that reuses the name of another local or parameter in the same method. Python accepts all of these cases. The program runs, and the wrong answer appears later, or never appears at all.

A loop variable reused in a nested loop:

Output:

outer i after inner loop: 1
outer i after inner loop: 1
outer i after inner loop: 1

The inner loop sets i to 0, then 1. The inner loop runs after the outer loop sets i, so the print sees the inner loop's last value, 1. This is a silent error.

A lambda has its own parameter, and that parameter can share a name with a variable outside it. A lambda is a short function written in one line, such as lambda name: name.upper():

Output:

RAVI
ana
inside: 4

The lambda's name is separate from the module-level name, so the module-level value is still ana. Assigning to a parameter inside a function is also allowed. The caller's value of 1 is replaced by 4 inside show, and the caller never sees the change.

Built-in names are a special case. Python provides functions such as sum, len, max and list before your program starts. If your program assigns to one of these names at the module level, the function is replaced for the rest of the file:

sum = 0
for n in [1, 2, 3]:
    sum += n
print("total:", sum)
print(sum([1, 2, 3]))

Output:

total: 6
Traceback (most recent call last):
  File "main.py", line 5, in <module>
    print(sum([1, 2, 3]))
          ~~~^^^^^^^^^^^
TypeError: 'int' object is not callable

The first print works, because sum is an integer at that point. The second call fails, because sum is now the integer 6, and an integer cannot be called like a function. The message TypeError means a value was used in a way its type does not allow.

Common mistakes

Assuming the outer name changed. The most common shadowing bug is a function that computes a value into a local, then ends without storing it in the outer name. The printed answer looks right inside the function, and the outer name looks wrong afterwards. In the module-level example above, the module-level total stayed 0. When a value looks wrong after a call, search the function for an assignment to a name with the same spelling.

Writing score = score where you meant self.score = score. Python does not warn about this. Check that the attribute has self. in front of it on the left side of the =.

Renaming only one side. After the parameter becomes new_score, a line that still says score = score is wrong. Check that the attribute is on the left and the parameter is on the right.

Forgetting global or nonlocal. Any assignment to a name makes that name local to the function, unless a global or nonlocal line says otherwise. Without the line, the code fails with an error:

total = 0
 
def add_prices(prices):
    subtotal = 0
    for price in prices:
        subtotal += price
    total += subtotal
 
add_prices([120, 45, 300])
print(total)

Output:

Traceback (most recent call last):
  File "main.py", line 9, in <module>
    add_prices([120, 45, 300])
    ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "main.py", line 7, in add_prices
    total += subtotal
    ^^^^^
UnboundLocalError: cannot access local variable 'total' where it is not associated with a value

The line total += subtotal is an assignment, so Python treats total as local to add_prices. It then tries to read the local before it has a value. The fix is the global total line from the example above.

Reusing a loop name in nested loops. Python gives no error for this, as the example above showed. Pick names such as row and col for grids, so each loop has its own name.

Naming a variable after a built-in. Names such as sum, list, max, str and input are built-in functions. Assigning to them replaces the function for the rest of the file. Use names such as subtotal or values instead.

Choosing names that differ by one letter. A name such as total and totals is easy to confuse, even when Python accepts both. Use names that describe what each one holds, such as cart_total for the module-level value and subtotal for the local.

Practice

Exercise 1. The class below has a bug. The method set_name does nothing useful. Fix it in two ways: once with self, and once by renaming the parameter. Explain which way you prefer and why.

class Player:
    def __init__(self):
        self.name = "guest"
 
    def set_name(self, name):
        name = name

Show the answer

Both fixes work. With self, the method body is self.name = name. With a rename, the method is def set_name(self, new_name): followed by self.name = new_name. The rename is usually easier to read, because a reader does not have to check which side of = is the attribute. Both forms are common in Python code.

Exercise 2. Find the bug in this function. The function should add bonus to the module-level points. Fix it so that the module-level value changes.

points = 10
 
def add_bonus(bonus):
    points = bonus
    points += 5

Show the answer

The local points hides the module-level points. The function adds 5 to the local, and the module-level value stays at 10. The fix declares global points and adds the bonus to it: global points, then points += bonus. If the extra 5 should also be added, write points += bonus + 5.

Exercise 3. Write a module-level variable rate set to 2, and a function scale(factor) that returns factor multiplied by rate. Call it with scale(10) and print the result. Then explain why a parameter named rate would make the function harder to write.

Show the answer

The printed result is 20. A parameter named rate would hide the module-level rate, so the function would need globals()["rate"] to reach the value. Naming the parameter factor keeps rate visible by its plain name, and the code reads more clearly.

Exercise 4. Predict what this code prints. Say whether Python reports an error.

for n in range(3):
    for n in range(2):
        pass
    print(n)

Show the answer

Python reports no error. The inner loop sets n to 0 and then 1, so the outer loop prints 1 three times. Java would stop with a compile error, so Python's silence is the more dangerous behaviour. Use different names for the two loops, such as row and col.

Exercise 5. A function declares count = 0 in an outer function, and an inner function tries to add one to it. Run the code below and read the error. Then fix it so that the outer function prints 1.

def outer():
    count = 0
 
    def inner():
        count = count + 1
 
    inner()
    print(count)
 
outer()

Show the answer

The assignment count = count + 1 makes count local to inner, and the local is read before it has a value. The error is UnboundLocalError. The fix adds nonlocal count as the first line of inner. The printed result is 1.

Summary

• Shadowing happens when a name in a smaller area, such as a parameter or local, has the same spelling as a name in a bigger area, such as a module-level variable.
• Python searches the function, the enclosing functions, the module, and the built-in names, in that order, and stops at the first match.
• A bare name inside a method never sees an attribute, so a parameter cannot hide self.name. Assigning to the bare name still leaves the attribute unchanged, so write self.name.
• Python accepts reused names and built-in names without an error. The result is a wrong answer, or an error later in the program.
• Use global or nonlocal when an assignment must change a name in an outer area, and rename the local when you do not want the change.