Comprehensions, generators and ranges

Why this matters

Much of everyday code asks the same three questions about a list. Which items pass a test? What does each item turn into? What do all the items add up to? A plain loop answers these with a counter, an if, and a variable that changes as the loop runs. Python offers shorter ways: a comprehension builds a new list in one line, a generator expression does the same without building the list, and built-in functions such as sum, max and map cover the rest. Each style is correct. The short form is clearer when each step is simple, and harder to read when the steps depend on each other in tricky ways.

This lesson shows these styles on the same jobs so you can choose between them. It also covers ranges, which are numbered sequences such as the numbers 1 to 100. A range is the Python version of a counting loop.

The module's other lessons cover the containers themselves: lists, dicts, sets and heaps. This lesson is about processing what those containers hold. Grouping items into a dict is covered in the grouping lesson, so Counter appears only once here, in the third worked example.

Worked example 1: a loop, then the same job in one line

Here is a small job. You have a list of test scores. Add up only the scores of 70 or more.

The loop below visits each score, checks it, and adds it to a running total.

The output is:

loop total: 345
one-line total: 345

The one-line version does three things in order. for s in scores visits each score. if s >= 70 decides which scores pass. sum adds the passing scores and gives back the final answer. The expression s for s in scores if s >= 70 is a generator expression. It produces the passing scores one at a time, and sum takes them as they come.

The loop keeps its state in total, which changes on every pass. The one-line version keeps no variable that changes. Each step only passes its output to the next step.

Three parts of every pipeline

A pipeline is a chain of steps where the output of one step is the input of the next. Every pipeline has the same three parts.

A pipeline can have any number of intermediate steps, but it must end with a terminal step. Without one, nothing happens, as the next section shows.

Laziness: nothing runs until the end

A generator expression, map and filter are lazy. Lazy means work is postponed until the result is needed. When you write one of these, Python records the steps but does not run them yet. The work starts when something asks for the next item, such as list, sum or a for loop.

The program below prints a line each time an item reaches a step. The helper show prints the name it receives and passes it on unchanged. The lambda in the code is a short function with no name, written on one line.

The output is:

pipeline built, nothing printed yet
filter sees Ada
filter sees Ben
filter sees Chen
result: ['ADA', 'BEN']

The line "pipeline built" appears first, even though the show calls are written earlier in the file. The lines from show appear only after list(pipeline) runs. The name Chen passed through show but was removed by filter, because Chen has four letters and the filter keeps only names of length three.

The variable pipeline is the chain itself, not a finished list. You can keep it in a variable and decide later whether to run it. That is useful for large or endless sequences, because only the items you ask for are computed.

Idea 1: map changes each item

A map takes each item and gives back a new value for it. The result still has one output for each input. The type can change: a list of words can become a list of word lengths.

The output is:

lengths: [4, 6, 3, 6]
total letters: 19
longest: stream
empty longest: 0

The map(len, words) step gives back the length of each word: 4, 6, 3 and 6. Every word produced exactly one length. A comprehension does the same job and is often easier to read: [len(w) for w in words].

Idea 2: reduce combines items into one value

A reduction takes a running result and one item at a time, and combines them. It is the Python version of a running total. The built-in sum is a reduction for addition. max and min are reductions for comparison.

Some reductions have no safe answer for an empty sequence. The largest word has no meaning for an empty list, because there is no largest of nothing. Python reports this kind of problem as an exception, an error that stops the program unless code catches it with try and except. Python gives two ways to handle that. The first is the default argument, which returns a value when the sequence is empty. The second is to ask for the first item with next, which takes a default too.

The output is:

empty max with default: none
caught: max() iterable argument is empty
empty next with default: none

The sum of an empty list is 0 without any extra work, because sum starts from 0. The max function has no such starting value, so it raises ValueError unless you give it a default. Use default when an empty input is a normal case.

The words stream and reduce both have six letters, and max returned stream, the first of the two. Python's max returns the first item that has the largest key, so this is a fixed rule. If the order matters in your own code, sort the list first and then take the item you need.

Idea 3: collect gathers the items into a container

A comprehension gathers the items of a sequence into a new list, dict or set. Each form has its own brackets.

The output is:

['PUNE', 'DELHI', 'PUNE', 'MUMBAI']
[Delhi, Mumbai, Pune]
Pune count: 2 2
Delhi length: 5 map size: 3

Four recipes appear here.

• [c.upper() for c in cities] builds a list with one new item for each city.
• set(cities) removes the repeats, and sorted puts the remaining cities in alphabetical order. ", ".join(...) puts the text between the items. It needs strings, so the items must already be text.
• sum(1 for c in cities if c == "Pune") counts the matches. The method cities.count("Pune") gives the same answer in one call, and it is the simpler choice for a plain count.
• {c: len(c) for c in set(cities)} builds a dict. Each item becomes a key, and its length becomes the value. The dict has three entries, not four, because set removed the second Pune.

The sum of a generator expression that yields 1 for each match is a common way to count. The len function counts the items in an existing list.

Idea 4: ranges of numbers

Many loops count from one number to another. A range does the same job. range(a, b) gives the numbers from a up to, but not including, b. Python does not have a separate function for the inclusive form. To include the end, write b + 1.

The output is:

1 to 100 sum: 5050
1 to 99 sum: 4950
squares: 1 4 9 16 25
evens: [0, 2, 4, 6, 8]

The difference between range(1, 101) and range(1, 100) is the most common source of an off-by-one error. The sum from 1 to 100 is 5050, while range(1, 100) stops at 99 and gives 4950. The third argument of range is the step. range(0, 10, 2) counts up by 2.

A range is a lazy sequence. It does not build a list of all its numbers, so range(10**9) takes almost no memory. Its items are produced only when you ask for them.

For an endless sequence, use itertools.count, and cut it with itertools.islice. An endless sequence is safe only when a step such as islice, next or a break stops it. Without one, the program never finishes.

The output is:

[0, 2, 4, 6, 8]

The call count(0, 2) gives 0, 2, 4, 6, and so on, forever. The call islice(..., 5) takes the first five values and stops.

A note on C++. The same idea appears in C++20 as ranges, where std::views::filter and std::views::transform can be joined with the | operator to form a pipeline. C++ views are also lazy, and a view is a lens over a container that you still own, so you can walk through it again. A Python generator is used up after one pass, as the next section shows. This lesson's samples are all Python, so the C++ version is described in words only.

Worked example 2: a file with bad rows

Real input is often messy. The next example takes lines of the form name,score. Some lines are blank, one has a score that is not a number, and the job is to average only the good rows.

The output is:

average: 80.00
scores: [82, 67, 91]

Each step removes or changes rows, and each one has a single job.

• if line.strip() drops empty lines. The strip method removes spaces at both ends, so a line of spaces is also dropped.
• line.split(",") splits each line at the comma. Each line becomes a list of two text parts, such as ["Asha", "82"].
• len(parts) == 2 keeps only the lines with exactly two parts.
• parts[1].isdigit() keeps only the rows where the score is all digits. The method isdigit returns True for text made only of digit characters.
• int(parts[1]) turns each score text into a number.
• The conditional expression sum(scores) / len(scores) if scores else 0 gives the average, or 0 when there are no good rows. Dividing by an empty list would raise ZeroDivisionError, so the check comes first.

The bad row Ravi,not-a-number was removed before the number conversion ran. Without that filter, int would have raised ValueError on the text not-a-number.

Worked example 3: the three most common words

This example is harder. It counts how often each word appears, then prints the three most frequent words. Ties are broken alphabetically so that the output is always the same.

The output is:

['the=4', 'and=2', 'bird=1']

The work happens in three stages.

1. text.split() splits the sentence into a list of words. With no argument, it splits on any run of spaces.
2. Counter(...) builds a dict from each word to the number of times it appears. The Counter class is the grouping step from the grouping lesson, packaged as one call.
3. sorted(..., key=...) orders the pairs by count from highest to lowest. The key (-kv[1], kv[0]) sorts by the negated count first, and then by the word. The slice [:3] keeps the first three.

A Counter also has a method most_common(3). It breaks ties by the order the words first appear, so here it gives cat=1 where the sort above gives bird=1.

Worked example 4: when a loop is clearer

Some jobs do not fit a comprehension. The job below finds the length of the longest run of numbers that keep rising. In the list 3, 4, 1, 2, 3, 4, 5, 0, the rising run 1, 2, 3, 4, 5 has length 5.

A loop handles this directly. It compares each number with the one before it, and it keeps the count of the current run in a variable that changes.

The output is:

longest run: 5

A comprehension cannot do this as cleanly. Each value depends on the one before it, and a comprehension has no place to keep a running count that changes from item to item. The loop has exactly that place. Trying to force the job into a one-line expression produces code that is harder to read than the loop above.

When a plain loop is clearer

Choose a loop when one or more of these is true.

• You need the position of an item, or the item before it. A comprehension passes items along, but it does not hand you their neighbours. If you need both the item and the one before it, a loop with an index is simpler.
• The work changes several variables at once, or stops early based on what it has seen so far. The rising-run example shows this.
• The loop body does several different things, such as printing, counting and updating a dict. Squeezing that into one expression usually hides what is happening.
• The work is long, and a reader would need a comment to explain each step. A loop with clear names is easier to read than a chain of lambdas.

A useful test: if you need a comment to explain what a comprehension step is really doing, write a loop instead.

Common mistakes

Each message below is copied from a real run on Python 3.14.

Reusing a generator

A generator or a map object can be read once. After it is read, it is empty, and Python does not raise an error for the second read. Here the second list call returns an empty list:

The output is:

first: ['A', 'BB', 'CCC']
second: []

The second line prints an empty list, and the program keeps going. This is a silent mistake, and it is harder to find than an error. The fix is to keep the list, not the generator, in your variable, and to build the generator again each time you need it. A list can be read as many times as you like.

Forgetting to consume a generator

A generator does no work until something reads it. This program builds a map and never reads it, so the print inside the function never runs:

The output is:

done

The show function never runs, because nothing asks for a result. Read the generator with list(...), a for loop, or a function such as sum, or remove the steps you do not need.

Expecting an exception inside a comprehension to be handled for you

Python has no checked exceptions, so a function that raises an error inside a comprehension stops the whole comprehension. This program reads a file for each path, and one path is missing:

The output is:

caught: FileNotFoundError

The comprehension stopped at the first error, and the try block around it caught the exception. If some paths should be skipped instead of stopping the job, put the try inside a loop, where you can decide what to do for each item.

Changing an outer variable inside a lambda

A lambda body must be a single expression, so a statement such as count += x is not allowed there. Python rejects it with a SyntaxError before the program runs. The compile call below asks Python to check that line without running it. Use a loop, or a built-in such as sum, to combine values into one total:

The output is:

caught: SyntaxError
6
6

The loop changes count directly. The sum call is the reduction for the same job, and it does not need a variable at all.

Duplicate keys in a dict comprehension

A dict comprehension does not raise an error when two items give the same key. The later value replaces the earlier one, and the first value is lost:

The output is:

{'a': 2}

If the first value should be kept, skip the later ones with a check, or group the values in a list with setdefault, as the grouping lesson shows.

Calling sum on the wrong thing

sum adds numbers. Calling it on a list of text raises an error, because text cannot be added to a number:

The output is:

caught: unsupported operand type(s) for +: 'int' and 'str'

The fix is to turn each item into a number first, with sum(int(x) for x in ["1", "2"]).

Off-by-one with range

range(1, 100) stops at 99. Use range(1, 101) when you want 100 included. The sums in the range example show the gap: 5050 with range(1, 101), and 4950 with range(1, 100).

Practice

Exercise 1. Given nums = [4, 7, 10, 13], write a loop or a comprehension that prints only the odd numbers, one per line.

Show the answer

for n in nums:
    if n % 2 != 0:
        print(n)

The output is 7 and then 13. The % operator gives the remainder after division, so n % 2 != 0 is true for odd numbers.

Exercise 2. Compute the sum of the squares of the numbers from 1 to 10.

Show the answer

total = sum(i * i for i in range(1, 11))

The result is 385. Use range(1, 11) so that 10 is included.

Exercise 3. Given words = ["apple", "fig", "banana"], find the length of the longest word. Return 0 when the list is empty.

Show the answer

longest = max((len(w) for w in words), default=0)

The result is 6, for banana. The default argument gives the fallback for an empty list.

Exercise 4. Given names = ["Asha", "Anil", "Ravi", "Alia"], count the names that start with the letter A. Then say what type the count has.

Show the answer

count_a = sum(1 for s in names if s.startswith("A"))

The result is 3. The type is int, because sum of whole numbers gives a whole number. The same count comes from len([s for s in names if s.startswith("A")]).

Exercise 5. Given temps = [25, 28, 31, 34, 29], find the day number, counting from 1, of the first temperature above 30. If no day qualifies, the answer should be -1.

Show the answer

first_hot = next((i + 1 for i, t in enumerate(temps) if t > 30), -1)

The result is 3, because the temperature 31 is at index 2, and day numbers start at 1. The enumerate function gives the index with each temperature. The next call stops at the first match, and the default -1 covers the case where no day qualifies.

Summary

• A comprehension or generator expression is a chain of steps: a source, zero or more if tests or maps, and a terminal step such as sum, list or join.
• A generator, map and filter do no work until something reads them, and they can be read only once. A second read gives an empty result with no error.
• max, min and next need a default when the input may be empty, or they raise an error.
• range(a, b) stops before b, and range(a, b + 1) includes b. Most off-by-one errors come from mixing these up.
• Use a loop when you need item positions, several changing variables, early exits with state, or several separate actions for each item.