Python Generators

A generator is a function that produces values one at a time with yield, pausing between them. Nothing runs until someone asks for the next value — which is what makes generators cheap for large or infinite sequences. Output is from Python 3.13.

Lazy execution, step by step

Calling a generator function runs none of its body; each next() runs to the next yield.

def countdown(n):
    print("  starting")
    while n > 0:
        yield n
        n -= 1
    print("  done")

gen = countdown(2)
print("created", type(gen).__name__)
print("next ->", next(gen))
print("next ->", next(gen))
try:
    next(gen)
except StopIteration:
    print("StopIteration")

print(list(countdown(3)))
print(list(gen))

$ python countdown.py
created generator
  starting
next -> 2
next -> 1
  done
StopIteration
  starting
  done
[3, 2, 1]
[]

“starting” appeared only at the first next(), not when countdown(2) was called. After the last value, the function ran to its end, printed “done”, and raised StopIteration — which for loops and list() catch for you. The final list(gen) is empty: a generator is exhausted after one pass.

Generator expressions

Parentheses instead of brackets give you a lazy generator instead of a list.

import sys

squares_list = [x * x for x in range(1_000_000)]
squares_gen = (x * x for x in range(1_000_000))

print(sys.getsizeof(squares_list), "bytes for the list object")
print(sys.getsizeof(squares_gen), "bytes for the generator")
print(sum(x * x for x in range(1_000_000)))
print(next(squares_gen), next(squares_gen), next(squares_gen))

$ python genexp.py
8448728 bytes for the list object
200 bytes for the generator
333332833333500000
0 1 4

The list holds a million pointers (and getsizeof doesn’t even count the int objects they point to); the generator is a fixed-size object that remembers where it is. Exact sizes depend on the Python build. When a generator expression is the only argument, as in sum(...), it needs no extra parentheses.

Infinite sequences

A while True generator is safe as long as the consumer stops asking.

from itertools import islice

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

print(list(islice(fibonacci(), 10)))

for n in fibonacci():
    if n > 100:
        break
    last = n
print("largest below 100:", last)

$ python fib.py
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
largest below 100: 89

list(fibonacci()) would never return. islice takes a fixed number of items, and break stops on a condition; either way only the values actually requested are ever computed.

Pipelines

Chained generators process one item at a time end to end, so memory stays constant however large the input.

import csv
import io

CSV = """email,status
Ana@Example.com ,active
bo@example.com,inactive
cy@example.com,active
"""

def rows(f):
    yield from csv.DictReader(f)

def active(rows):
    for row in rows:
        print("  checking", row["email"].strip())
        if row["status"] == "active":
            yield row

def emails(rows):
    for row in rows:
        yield row["email"].strip().lower()

pipeline = emails(active(rows(io.StringIO(CSV))))
print("pipeline built")
for email in pipeline:
    print("send to", email)

$ python pipeline.py
pipeline built
  checking Ana@Example.com
send to ana@example.com
  checking bo@example.com
  checking cy@example.com
send to cy@example.com

Building the pipeline did nothing. Then each row travelled all the way through before the next was read — “send to ana” printed before bo was even checked. Swap io.StringIO(CSV) for an open file and the same code streams a file of any size.

yield from

yield from hands off to another iterable, which makes recursive generators short.

def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

print(list(flatten([1, [2, [3, 4]], [5, [6]]])))

def chain(*iterables):
    for it in iterables:
        yield from it

print(list(chain([1, 2], "ab", range(3))))

$ python yield_from.py
[1, 2, 3, 4, 5, 6]
[1, 2, 'a', 'b', 0, 1, 2]

yield from flatten(item) re-yields everything the recursive call produces, however deep. It works with any iterable — a string yields its characters, a range its numbers.

send(): values going in

yield is also an expression: send(value) resumes the generator and makes the paused yield evaluate to value.

def running_total():
    total = 0
    while True:
        value = yield total
        total += value

acc = running_total()
print(next(acc))          # advance to the first yield
print(acc.send(10))
print(acc.send(20))
print(acc.send(5))

fresh = running_total()
try:
    fresh.send(1)
except TypeError as e:
    print("TypeError:", e)

$ python send.py
0
10
30
35
TypeError: can't send non-None value to a just-started generator

Each send delivers a value in and gets the next yielded value out. A brand-new generator isn’t paused at a yield yet, so it has nowhere to put the value — hence the “priming” next() call first.

itertools

The itertools module is a toolkit of fast, lazy building blocks.

import itertools as it

print(list(it.islice(it.count(10, 5), 4)))
print(list(it.islice(it.cycle("AB"), 5)))
print(list(it.combinations("ABC", 2)))
print(len(list(it.permutations("ABC", 2))))
print(list(it.accumulate([1, 2, 3, 4])))
print(list(it.batched(range(7), 3)))    # 3.12+
print(list(it.zip_longest([1, 2, 3], "ab", fillvalue="-")))

data = [("A", 1), ("A", 2), ("B", 3), ("A", 4)]
for key, group in it.groupby(data, key=lambda pair: pair[0]):
    print(key, [n for _, n in group])

$ python itertools_tour.py
[10, 15, 20, 25]
['A', 'B', 'A', 'B', 'A']
[('A', 'B'), ('A', 'C'), ('B', 'C')]
6
[1, 3, 6, 10]
[(0, 1, 2), (3, 4, 5), (6,)]
[(1, 'a'), (2, 'b'), (3, '-')]
A [1, 2]
B [3]
A [4]

groupby surprises people: it groups consecutive equal keys, so the final ("A", 4) became a separate group. Sort by the same key first if you want one group per key. batched (3.12+) leaves a short final batch rather than padding it.

Practice

1. Write a generator evens(limit) that yields even numbers from 0 up to limit, and print list(evens(10)).

def evens(limit):
    n = 0
    while n <= limit:
        yield n
        n += 2

print(list(evens(10)))

$ python evens.py
[0, 2, 4, 6, 8, 10]

2. Add a dedupe stage to the pipeline that skips emails already seen, and feed it a CSV where one active address appears twice in different cases.

import csv
import io

CSV = "email,status\nana@example.com,active\nANA@example.com,active\ncy@example.com,active\n"

def emails(f):
    for row in csv.DictReader(f):
        if row["status"] == "active":
            yield row["email"].strip().lower()

def dedupe(items):
    seen = set()
    for item in items:
        if item not in seen:
            seen.add(item)
            yield item

print(list(dedupe(emails(io.StringIO(CSV)))))

$ python dedupe.py
['ana@example.com', 'cy@example.com']

3. Use groupby on ["apple", "avocado", "banana", "blueberry", "cherry", "apricot"] to group by first letter — first unsorted, then sorted — and compare.

from itertools import groupby

words = ["apple", "avocado", "banana", "blueberry", "cherry", "apricot"]
first = lambda w: w[0]
print({k: list(g) for k, g in groupby(words, key=first)})
print({k: list(g) for k, g in groupby(sorted(words, key=first), key=first)})

$ python groupby.py
{'a': ['apricot'], 'b': ['banana', 'blueberry'], 'c': ['cherry']}
{'a': ['apple', 'avocado', 'apricot'], 'b': ['banana', 'blueberry'], 'c': ['cherry']}

4. Write a moving_average(window) generator driven by send() that returns the average of the last window values sent.

from collections import deque

def moving_average(window):
    values = deque(maxlen=window)
    average = None
    while True:
        values.append((yield average))
        average = sum(values) / len(values)

avg = moving_average(3)
next(avg)
for reading in [10, 20, 30, 40]:
    print(reading, avg.send(reading))

$ python moving_average.py
10 10.0
20 15.0
30 20.0
40 30.0