Checking AI code with tests

Why this matters

An assistant can write a function in seconds, and the function often looks correct. Looking is not enough. A test is a small program that calls your code with an input you chose, then compares the result with an answer you decided in advance. If the test passes, the code gave your answer for that input. If it fails, the test shows you where the code went wrong.

Most beginners write the test after the code. The test then tends to check whatever the code already does, including its mistakes. Writing the test first keeps the expected answer in your hands.

Lesson 03 covers general checks, including a few edge-case tests and a worked example with a median. This lesson covers the order of work (test first), how to read a failing test, and the ways a test can pass when the code is wrong.

Words used in this lesson

• Test: code that calls a function and checks what it returned.
• Assertion: a line that states what must be true, such as "this value must equal 2024". If the statement is false, the test fails.
• Test runner: a program that finds the test functions, runs them and reports the results. This lesson uses unittest, which comes with Python.
• Expected value: the answer you decided on before running anything.
• Red and green: a failing test is red and a passing test is green. The first run, before any code exists, is the red step.
• False pass: a test that passes even though the code is wrong.
• Exit code: the number a command returns when it ends. 0 means success, and any other number means something went wrong.

The example in this lesson is the leap year rule. A year is a leap year when it divides evenly by 4. The exception is a century year, one that divides evenly by 100. A century year is a leap year only when it also divides evenly by 400. So 2024 is a leap year, 2023 is not, 1900 is not, and 2000 is. The 1900 case is the one most first attempts get wrong.

The idea, built step by step

Step 1: write the cases before you ask for code

Make a folder for the work. In it, create test_leap.py:

import unittest
from leap import is_leap_year
 
 
class TestIsLeapYear(unittest.TestCase):
    def test_year_divisible_by_4_is_leap(self):
        self.assertTrue(is_leap_year(2024))
 
    def test_year_not_divisible_by_4_is_not_leap(self):
        self.assertFalse(is_leap_year(2023))
 
    def test_century_not_divisible_by_400_is_not_leap(self):
        self.assertFalse(is_leap_year(1900))
 
    def test_century_divisible_by_400_is_leap(self):
        self.assertTrue(is_leap_year(2000))
 
 
if __name__ == "__main__":
    unittest.main()

Each test name states the rule it checks. The four cases cover the normal rule and the century exception. You wrote all of this without any code from an assistant, using only the rule.

Run the tests now, before leap.py exists:

$ python3 -m unittest test_leap

The output, shortened here, is:

ERROR: test_leap (unittest.loader._FailedTest.test_leap)
...
ModuleNotFoundError: No module named 'leap'
...
FAILED (errors=1)

This is a red for the wrong reason. The test file could not even load, because leap.py is missing. Nothing here tells you anything about leap years, so this run is not a useful check of the code.

Next, create leap.py with a stub that always answers False:

def is_leap_year(year):
    return False

Run the tests again with python3 -m unittest -v test_leap. The output is trimmed to the lines that matter, and the time after Ran will differ on your machine:

test_century_divisible_by_400_is_leap ... FAIL
test_century_not_divisible_by_400_is_not_leap ... ok
test_year_divisible_by_4_is_leap ... FAIL
test_year_not_divisible_by_4_is_not_leap ... ok
 
FAIL: test_century_divisible_by_400_is_leap
  self.assertTrue(is_leap_year(2000))
AssertionError: False is not true
 
FAIL: test_year_divisible_by_4_is_leap
  self.assertTrue(is_leap_year(2024))
AssertionError: False is not true
 
Ran 4 tests in 0.001s
FAILED (failures=2)

This is a good red. The code runs, the test calls the function, and the two positive cases fail with a message that names the wrong answer. The two negative cases pass only because the stub answers False, which is what those tests expect. A stub that always says False would pass the negative tests forever, which is why the positive cases are needed too.

Step 2: give the assistant the test, not only the task

When you ask for code, paste the test file and say the test is the goal. A useful request looks like this:

Write leap.py with a function is_leap_year(year) so that every test in test_leap.py passes.
Do not change the test file. Return only the code.

A common first answer from an assistant looks like this:

def is_leap_year(year):
    return year % 4 == 0

Save it as leap.py and run the tests:

FAIL: test_century_not_divisible_by_400_is_not_leap
  self.assertFalse(is_leap_year(1900))
AssertionError: True is not false
 
Ran 4 tests in 0.000s
FAILED (failures=1)

The test caught the missing exception before you trusted the code. The message True is not false says the function returned True and the test expected False. The test name says which rule was broken.

Step 3: read the failure and make the smallest fix

The failing line is the assertFalse for 1900. The fix is to check the century rule first:

def is_leap_year(year):
    if year % 400 == 0:
        return True
    if year % 100 == 0:
        return False
    return year % 4 == 0

Run the tests again:

test_century_divisible_by_400_is_leap ... ok
test_century_not_divisible_by_400_is_not_leap ... ok
test_year_divisible_by_4_is_leap ... ok
test_year_not_divisible_by_4_is_not_leap ... ok
 
Ran 4 tests in 0.000s
OK

Only the code changed. The expected values in the test file are the decision you made before any code existed, so they should stay as they are.

Step 4: check that the test file did not change

An assistant sometimes edits the test so that its own code passes. Keep a saved copy of your test file, and compare the two after each round of changes. Here the working copy was changed so that 1900 expects True:

$ diff -u saved/test_leap.py test_leap.py

The output, with the two header lines left out, is:

@@ -10,7 +10,7 @@
         self.assertFalse(is_leap_year(2023))
 
     def test_century_not_divisible_by_400_is_not_leap(self):
-        self.assertFalse(is_leap_year(1900))
+        self.assertTrue(is_leap_year(1900))
 
     def test_century_divisible_by_400_is_leap(self):
         self.assertTrue(is_leap_year(2000))

The minus line is the old expected value and the plus line is the new one. This edit would make the wrong year % 4 == 0 code pass the test, and the correct code would fail it. Any change to the test file needs a reason you can state. If you did not ask for it, put the old version back.

Worked example 2: tests that pass for the wrong reason

The next file is a weak test suite. It is run against the wrong year % 4 == 0 code from Step 2.

import unittest
from leap import is_leap_year
 
 
class TestWeak(unittest.TestCase):
    def test_2024_is_leap(self):
        self.assertTrue(is_leap_year(2024))
 
    def test_1900_has_a_result(self):
        self.assertIsNotNone(is_leap_year(1900))
 
    def test_matches_the_rule(self):
        for year in [1900, 2000, 2023, 2024]:
            self.assertEqual(is_leap_year(year), year % 4 == 0)
 
    def test_1900_swallowed(self):
        try:
            self.assertTrue(is_leap_year(1900))
        except Exception:
            pass
 
 
if __name__ == "__main__":
    unittest.main()

Run it with python3 -m unittest -v test_weak against the wrong code:

test_1900_has_a_result (test_weak.TestWeak.test_1900_has_a_result) ... ok
test_1900_swallowed (test_weak.TestWeak.test_1900_swallowed) ... ok
test_2024_is_leap (test_weak.TestWeak.test_2024_is_leap) ... ok
test_matches_the_rule (test_weak.TestWeak.test_matches_the_rule) ... ok
 
Ran 4 tests in 0.000s
OK

All four pass, and all four are wrong. Each one fails to check what it seems to check:

1. test_2024_is_leap checks one year, and the wrong rule happens to get it right.
2. test_1900_has_a_result uses assertIsNotNone, which accepts True and False alike. It only checks that something came back.
3. test_matches_the_rule computes the expected value with year % 4 == 0, the same wrong rule. The test copies the bug, so it can never see it.
4. test_1900_swallowed wraps the check in a try block. A failed assertion raises an AssertionError, which is an exception, and the except block catches it and does nothing. The test passes whatever the function returns.

The fix is to replace these four with the explicit expected values from test_leap.py. Write each expected value from the rule, never from the code's output. Do not catch errors in a test unless you then check them.

Worked example 3: a test that never runs

This file has one test with the wrong name. The method starts with check_, not test_:

import unittest
from leap import is_leap_year
 
 
class TestNames(unittest.TestCase):
    def check_century_1900(self):
        self.assertFalse(is_leap_year(1900))
 
 
if __name__ == "__main__":
    unittest.main()

Run it:

$ python3 -m unittest -v test_names

----------------------------------------------------------------------
Ran 0 tests in 0.000s
 
NO TESTS RAN

On this machine, which runs Python 3.14.4, the exit code after that output is 5. The runner only runs methods whose names start with test, so it found nothing to run. This is a silent failure: no test failed, so the run looks clean at a glance. Two habits catch it. Compare the number in "Ran N tests" with the number of cases you wrote, and check the exit code when a script runs the tests.

Rename the method to test_century_1900 and run the file again with the wrong year % 4 == 0 code in leap.py:

test_century_1900 (test_names.TestNames.test_century_1900) ... FAIL
 
FAIL: test_century_1900 (test_names.TestNames.test_century_1900)
  self.assertFalse(is_leap_year(1900))
AssertionError: True is not false
 
Ran 1 test in 0.001s
FAILED (failures=1)

Once the name was right, the test ran and caught the bug.

Common mistakes

Copying the code's output into the expected value

The most common error is running the code, copying what it printed, and pasting that as the expected value. The test then passes every time, because it expects whatever the code does. Write each expected value from the rule first. For a leap year, 1900 is not a leap year, and you know that before you run anything.

Testing only the easy case

A suite with only 2024 checks one case, and the simple rule passes it. Include the exception cases, which for this rule are 1900 and 2000.

Assertions that accept anything

assertIsNotNone, assertTrue(True) and a check that only prints a value do not confirm an answer. Use assertEqual, assertTrue or assertFalse with a specific expected value.

Catching errors inside the test

A try block with an empty except makes the test pass whatever happens. If you need to test that code raises an error, use a check built for that, such as assertRaises. Do not hide failures.

Importing a name that does not exist

When the test file imports a name the module does not define, that test file fails to load. For example, an import of is_leap from a file that defines is_leap_year prints this:

ImportError: cannot import name 'is_leap' from 'leap' (/path/to/leap.py)

The name in the test and the name in the code must match. Check the import line first when a suite fails before any test runs.

Editing the test to match the code

When a test fails, the code is the first suspect. Changing the expected value in the test until it passes removes the check. Use the diff from Step 4 to see every change to the test file, and only accept a change you can justify from the rule.

Variations and edge cases

Most languages have a test runner that reports how many tests ran and how many failed. The steps in this lesson work the same way there: write the expected values first, run the suite before the code exists, read each failure message, and compare the test count with the number of cases you wrote.

Some results cannot be compared with one exact value, such as a random pick from a list. In that case, test a property instead, such as "the result is one of the items in the list" or "the result is between 0 and 100". A property test can still fail for the wrong reason, so write it with the same care as an exact-value test.

When the assistant writes the tests as well as the code, do not trust the tests on their own. Read each expected value against the rule. Then test the suite against a version of the code you know is wrong, like the year % 4 == 0 version above. If the suite passes that version, the suite is too weak.

How this appears in interviews and real work

Many teams ask for a test with each change, and reviewers read the test to see what the change promises. A test that was written after the code and copies its output tells a reviewer very little. A test that was written from the rule, and that fails on the old code, tells the reviewer that the change does what it claims.

In a technical interview, you may be asked to write a few tests for a small function. Interviewers often look for the boundary cases, such as the first value of a range and the value just outside it, and for a test that names what it checks.

Practice

Exercise 1. Before any code exists, write the test cases for is_adult(age), which should return True for an age of 18 or more and False below 18. List each case as input and expected value.

Show the answer

is_adult(0)   -> False
is_adult(17)  -> False
is_adult(18)  -> True
is_adult(40)  -> True

The pair 17 and 18 matters most, because that is where the rule changes. The value 0 checks a plain low input. The expected values come from the rule, not from any code.

Exercise 2. The test run prints Ran 4 tests and FAILED (failures=1), with the line AssertionError: True is not false for a call to is_leap_year(1900). What did the function return, what did the test expect, and which test is it?

Show the answer

The function returned True. The test expected False. The failing test is test_century_not_divisible_by_400_is_not_leap, because 1900 is a century year that is not divisible by 400, so the correct answer is False.

Exercise 3. A test suite contains this method. Is it at risk of a false pass, and why?

def test_matches(self):
    for year in [1900, 2000, 2023, 2024]:
        self.assertEqual(is_leap_year(year), year % 4 == 0)

Show the answer

Yes. The expected value is computed with year % 4 == 0, which is the wrong rule for 1900. Any code that uses the same rule will pass, including the wrong code. The fix is to write the expected value for each year from the calendar rule, as a fixed list, such as 1900 expecting False.

Exercise 4. Running python3 -m unittest -v test_names prints Ran 0 tests and NO TESTS RAN. What is the exit code, and what change makes the test run?

Show the answer

The exit code is 5. Rename the method from check_century_1900 to test_century_1900. The runner only runs methods whose names start with test. After the rename, the run prints Ran 1 test, and with the wrong code it fails with AssertionError: True is not false.

Exercise 5. The diff below compares a saved test file with the working copy. Which line changed, and would the wrong year % 4 == 0 code pass the changed test?

@@ -10,7 +10,7 @@
-        self.assertFalse(is_leap_year(1900))
+        self.assertTrue(is_leap_year(1900))

Show the answer

The expected value for 1900 changed from False to True. The wrong year % 4 == 0 code returns True for 1900, so with this edit it would pass. The correct code would now fail. A change like this should be reverted unless you can justify it from the calendar rule.

Summary

• Write the expected values from the rule before you ask an assistant for code.
• Run the test before the code exists, and check that it fails for the reason you expect.
• Give the assistant the test file, and tell it not to change the tests.
• Read each failure message: it names the wrong answer and the answer the test wanted.
• Compare the test file with a saved copy after the assistant changes things.
• Watch for false passes: expected values copied from the code, assertions that accept anything, errors caught inside a test, and test names that the runner never finds.