Refactoring a long Python script

Why this matters

A program often starts as one long block of lines. It reads the data, works out the results and prints the report, all in one place. This is easy to write at first. After a few hundred lines it becomes hard to read, and a change in one part can break another part you did not mean to touch.

The fix is to split the long block into smaller functions, each with a name that says what it does. Programmers call this refactoring. A refactoring changes the shape of the code but keeps its behaviour, so the program prints the same output as before.

This lesson shows one safe way to do it. You change one small piece at a time, run the program, and compare the output with the output you saved before the change. If the output differs, you know which change caused it. The lesson on parameters and return values shows how to write a function like the ones created here. The lesson on variable scope explains which names a function can see, and that matters in step 2 and in the common mistakes below.

Words used in this lesson

A function is a named block of code. You define it with def, give it a name, and call it by that name. Its lines run each time you call it.

A parameter is a name in the function's header that receives a value. In def sum_of(marks):, the name marks is a parameter.

A return value is the value a function hands back to its caller, written with return.

A script is a file of Python code that you run with python3 grades.py. The file in this lesson is a script.

Refactoring is changing the structure of code without changing what it does for the user.

The starting program

The program below is the starting point. It reads five students and their scores from lists, works out an average and a letter grade for each, prints one row per student, and then prints a summary. Everything is at the top level of the script, with no functions.

Run it, and you get this output:

=== Term 1 grade report ===
Name     Average  Grade  Result
------------------------------
Ana        84.33  B      pass
Ravi       58.33  F      fail
Mei        95.00  A      pass
Tomas      46.33  F      fail
Liu        71.00  C      pass
------------------------------
Class average: 71.00
Passed: 3 of 5
Best student: Mei

This output is the reference. Save it to a file before you change anything, because every later step must produce exactly the same text. The lines that matter most are the ones with numbers. A changed average, a changed grade, or a changed count would show that a step broke something.

The line that starts with print(f"...") uses an f-string. The part {names[i]:<8} prints the name left-aligned in eight spaces. The part {average:7.2f} prints the number with two digits after the point, right-aligned in seven characters. You do not need to understand every detail of the format to refactor the program. You only need to keep the line exactly as it is.

The rule for every step

Each step follows the same four actions.

1. Make one small change. Move one block of lines into a function, or move one decision into a function. Do not change two things at once.
2. Run the script. Python reads the whole file before it runs, so a mistake in the indentation or a misspelled name shows up here.
3. Save the output to a file.
4. Compare the new output file with the saved reference. If they match, keep the change and go on to the next step. If they differ, undo the change and find out why before you go on.

Here is the command pattern used in the steps below. The > sign sends the output into a file instead of the screen. The second command compares the two files.

python3 grades.py > reference.txt
python3 grades.py > out.txt
cmp reference.txt out.txt

The cmp command compares two files byte by byte. It prints nothing when the files are the same. It prints a message when they differ. On Linux and macOS, cmp is available in the terminal. On Windows, the fc command does the same job. The file names here are examples, so use the names you chose for your own steps.

Two things can change the output without any bug in the code. A line can move to a different place. A text can change by one space. The comparison catches both, because it compares every character.

Step 1: give the header a name

The first three lines of the output form a header. The header has nothing to do with the rows, so it is the easiest block to move. Create a function called print_header, and move the three print lines into it.

def print_header():
    print("=== Term 1 grade report ===")
    print("Name     Average  Grade  Result")
    print("------------------------------")

Python does not need the word static here. A function defined at the top level of a file is ready to call. The def line and the indented body are all it needs. The function takes no parameters, because it prints the same text every time.

In the main part of the script, delete the three lines and put one call in their place.

print_header()

Run the script and compare the output with the reference. The header is still first, and the rest is unchanged, so cmp prints nothing.

The change is small, but it already makes the script easier to read. A reader now sees print_header() and knows what happens without reading three lines.

Step 2: name the sum of one student's marks

Inside the row loop, the program adds up the three marks for one student. The loop is clear, but it is a separate job. Move it into a function called sum_of. The function takes one list of marks as its parameter and returns the total.

def sum_of(marks):
    total = 0
    for mark in marks:
        total = total + mark
    return total

The loop here uses the form for mark in marks, which reads the marks one by one without an index. It has the same effect as the index loop in the starting program. Python also has a built-in function sum that does the same job. This lesson writes its own version so that you can see each step.

In the row loop, replace the inner for loop with one line that calls the function. The line that computes average still divides by the number of marks, so keep that line too.

total = sum_of(scores[i])
average = total / len(scores[i])

Here is a point that trips up many people. The variable total is still needed later in the row loop, because class_total = class_total + total uses it. Do not remove the variable when you move the loop. Keep the assignment and set it from the function. Run the script. The output still matches the reference.

Python has one rule about loop variables that is different from Java. A for loop does not end the life of the variables it sets. The variable keeps its last value after the loop ends. Here is a short example.

Output:

2 20

The loop ended, but i and last still exist. A variable that a function creates is different. It exists only while that function runs. Common mistake 1 below shows what happens when code outside the function uses such a variable.

Step 3: name the letter grade

The chain of if and elif lines decides a letter from an average. This is a clear job with one input and one answer, so it fits a function called letter_grade. The function takes the average and returns a string.

def letter_grade(average):
    if average >= 90:
        return "A"
    elif average >= 80:
        return "B"
    elif average >= 70:
        return "C"
    elif average >= 60:
        return "D"
    return "F"

Python writes elif where other languages write else if. Each return stops the function at once. So the last line, return "F", runs only when none of the earlier conditions was true. That is why the function has no else before the final return.

In the main part of the script, the long chain is replaced by one line.

grade = letter_grade(average)

Run the script. The grades in the table are the same as before. Check the Mei row, which has an average of 95.00 and should still read A, and the Ana row, which has 84.33 and should still read B.

Notice that the function boundary changes the shape of the code, but not the behaviour. The if chain used to set a variable. Now it returns a value. The two are equal here because the caller stores the result in grade. If you had changed one threshold in the chain by accident, the output would show the wrong letter, and cmp would report the difference.

Step 4: name the summary

The last part of the script prints the separator line, the class average, the pass count and the best student. These four lines form a summary. Move them into a function called print_summary. The function needs four values that the main part calculated: the list of names, the pass count, the class average and the index of the best student. Those become its parameters.

def print_summary(names, pass_count, class_average, best_index):
    print("------------------------------")
    print(f"Class average: {class_average:.2f}")
    print("Passed: " + str(pass_count) + " of " + str(len(names)))
    print("Best student: " + names[best_index])

In the main part, the four lines are replaced by two lines. The class average is still calculated there, because it uses class_total and the number of scores. Then the main part calls the summary function.

class_average = class_total / (len(names) * len(scores[0]))
print_summary(names, pass_count, class_average, best_index)

The separator line moved from the main part into the function. Its position in the output does not change, because it still prints after the last row and before the class average. Run the script, and compare with the reference.

The finished program

After four steps, the main part no longer contains the details of the work. It reads like a list of jobs. The main part is placed in a function called main, and the last two lines of the file call it. This is a common layout for scripts. The check if __name__ == "__main__": means that main() runs when you start the file directly with python3. If another file imports this one, the check is false and main() does not run.

The loop is still in main, and that is correct for this program. The loop keeps running totals (class_total, pass_count, best_index) that are shared by every row. A further step could move the row printing into its own function, but it would need many parameters, and that makes the code harder to read. A refactoring stops when the code reads clearly, not when no lines are left in main.

The four functions are short. Each does one job, and its name tells you what that job is. A reader who wants to change how a grade is chosen goes straight to letter_grade and does not need to read the whole program.

Common mistakes

Mistake 1: a variable that a function keeps to itself

When you move a loop into a function, the variables it set exist only inside that function. In step 2, total is needed for class_total, and the code that uses it must be in the same function. Here is a version where the loop was moved into sum_of, and main tries to print total after the loop.

def sum_of(marks):
    total = 0
    for mark in marks:
        total = total + mark
    return total
 
def main():
    scores = [[78, 85, 90], [55, 62, 58]]
    class_total = 0
    for row in scores:
        class_total = class_total + sum_of(row)
    print(class_total)
    print(total)
 
main()

Running it prints the first line and then stops with this error:

428
Traceback (most recent call last):
  File "grades.py", line 15, in <module>
    main()
    ~~~~^^
  File "grades.py", line 13, in main
    print(total)
          ^^^^^
NameError: name 'total' is not defined

The last line gives the error type and the message. NameError means that Python could not find a name. The name is total, and the line is 13 in the function main. Python prints the full path of the file on the File lines. This lesson shortens it in every error message. The fix is to keep the variable in main and set it from the function, as step 2 did.

Mistake 2: a function that does not return a value

When you turn an if chain into a return chain, the function must return a value on every path through the code. Python does not stop with an error when a path has no return. The function ends and gives back None, which is Python's word for no value. The program fails later, at the first line that tries to use that value. A small version of letter_grade with two cases shows this.

def letter_grade(average):
    if average >= 90:
        return "A"
    elif average >= 80:
        return "B"
 
print(letter_grade(75) + "!")

Running it prints this error:

Traceback (most recent call last):
  File "grades.py", line 7, in <module>
    print(letter_grade(75) + "!")
          ~~~~~~~~~~~~~~~~~^~~~~
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

TypeError means that a value has the wrong type for the operation. Here the function gave None, which has type NoneType, and the program tried to add text to it. The fix is to add a final return "F" after the chain, so that every path returns a letter.

Mistake 3: an indentation error after a move

Python uses indentation to mark the body of a function. When you move lines into a def, every moved line must be indented by the same amount. If the body is not indented, Python stops before it runs anything.

def print_header():
print("=== Term 1 grade report ===")
 
print_header()

The error is:

  File "grades.py", line 2
    print("=== Term 1 grade report ===")
    ^^^^^
IndentationError: expected an indented block after function definition on line 1

The message names the line above the problem. Here it is line 1, the def line. The fix is to indent the body by four spaces.

Mistake 4: changing two things at once

Suppose you move print_header and also change the text of one heading in the same step. If the output differs, you do not know which change caused it. Keep each step to one change, and run the comparison before the next change. That way a difference always points to the last thing you did.

Mistake 5: no saved reference

If you do not save the first output, you have nothing to compare with. You can compare against your memory of the output, but memory is not exact. Save the output of the original script once, before any change, and keep the file.

Mistake 6: moving a call to a different place

Moving print_header() below the loop does not break the program, but the header now prints after the five rows. The output is different, so the comparison catches it. The order of the calls in the main part decides the order of the output.

Practice

Exercise 1. Write a function is_passing that returns True when a grade is not "F". Use it in the loop of the finished program in place of the test grade == "F", and check that the output still matches the reference.

Show the answer

def is_passing(grade):
    return grade != "F"

In the loop, write if is_passing(grade): with the pass lines under it, and else: with result = "fail". The output stays the same. The function returns True for "B" and False for "F".

Exercise 2. The program below has an error after a refactoring. Find it, and fix it with the smallest change.

def sum_of(marks):
    total = 0
    for mark in marks:
        total = total + mark
    return total
 
def main():
    scores = [[78, 85, 90], [55, 62, 58]]
    class_total = 0
    for i in range(len(scores)):
        class_total = class_total + sum_of(scores[i])
    print(total)
    print(class_total)
 
main()

Show the answer

The error is NameError: name 'total' is not defined at print(total). The name total exists only inside sum_of, and main never sets it. The smallest fix is to print class_total instead, which is the value the program wants. If the program needs the last row total, write total = sum_of(scores[i]) inside the loop, as step 2 did, before the class_total line.

Exercise 3. In the finished program, what happens if you swap the calls print_header() and the first loop? Does the output change? Explain how you would check.

Show the answer

Yes, the output changes. The loop prints the rows, and print_header prints the header. If the header call comes after the loop, the header prints after the rows. To check, save the output again and compare it with the reference using cmp. The comparison shows a difference, which tells you the order changed.

Exercise 4. A refactored program prints Passed: 4 of 5 where the reference shows Passed: 3 of 5. The rows match. Which step is most likely to be wrong, and why?

Show the answer

The rows match, so the grade letters and the averages are right. The pass count is wrong. Look at the code that adds to pass_count. The likely cause is that the pass test changed when it moved into a function, for example a >= became >, or the increment line was lost when the if was rewritten. Check that the increment still runs in the pass branch and only there.

Exercise 5. Write a function max_of that takes a list of numbers and returns the largest value. Use it in a short program that prints the largest of [4, 17, 9]. Predict the output before you run it.

Show the answer

The output is 17. The function starts with the first value as the best, then checks every value and keeps the largest. Python also has a built-in function max that gives the same answer for this list.

Summary

• Refactoring changes the shape of the code, not the output, so save the original output before you start.
• Make one small change per step, then run the script and compare the output with cmp.
• Move a block into a function when it has a clear job, and give the function a name that says what the job is.
• Keep a variable in the main part when a later line still needs it, and set it from the function.
• A function that returns a value must return one on every path. Python does not report a missing return at once. It gives None, and the error appears later.