Lesson 6 · 6 of 20

Generators and yield · Python

Process a server log line by line with Python generators: yield, next(), chained generators and generator expressions.

Python · Generators and yield in DMarlon Studio: the example's code and its output in the terminal

A function containing `yield` is a generator: calling it does not run it, it returns an object that hands out values one at a time. read_lines(LOG) does not build a list of every line; it gives one, pauses, and resumes from there when asked for the next. With a log of millions of lines, the memory use barely changes.

errors(lines) consumes another generator: it loops over the lines, splits their fields with split() and only yields those with a status of 400 or more. Chained together, errors(read_lines(LOG)) form a pipeline where each line goes through every stage before the next one is read.

Creating pipeline runs nothing yet (its type is generator). next(pipeline) runs it up to the first yield and returns that value; the for loop then carries on exactly where it stopped, which is why it does not repeat the first error. A generator can only be consumed once.

Parentheses instead of square brackets give you a generator expression: sum(1 for line in ... if ...) counts the 500 errors without building any intermediate list.

invoice_numbers has a while True, yet it never hangs: it only works out the number you ask for. The final comprehension takes three with next(numbers); the next one would be INV-2043.

Code · main.py

LOG = """2026-03-01 10:02 GET /shop 2002026-03-01 10:03 GET /cart 2002026-03-01 10:05 POST /pay 5002026-03-01 10:06 GET /shop 4042026-03-01 10:09 POST /pay 2002026-03-01 10:12 GET /admin 500"""def read_lines(text):                 # yields one line at a time    for line in text.splitlines():        yield linedef errors(lines):                    # consumes another generator    for line in lines:        date, time, method, path, status = line.split()        if int(status) >= 400:            yield f"{time} {path} -> {status}"pipeline = errors(read_lines(LOG))    # nothing has run yetprint(type(pipeline).__name__)print("First:", next(pipeline))       # runs until the first yieldfor item in pipeline:                 # continues where it stopped    print("Next: ", item)# Generator expression: like a comprehension, with ()total_500 = sum(1 for line in read_lines(LOG) if line.endswith("500"))print("Server errors:", total_500)def invoice_numbers(start):    while True:                       # infinite, but lazy        yield f"INV-{start}"        start += 1numbers = invoice_numbers(2040)print([next(numbers) for _ in range(3)])

Output · python3 main.py

generator
First: 10:05 /pay -> 500
Next:  10:06 /shop -> 404
Next:  10:12 /admin -> 500
Server errors: 2
['INV-2040', 'INV-2041', 'INV-2042']

Key points

yield turns a function into a generator.
A generator computes each value on demand (it is lazy).
next() runs to the next yield; a for loop does it for you.
A generator gets used up: to loop again, create a new one.
(expr for x in data) is a generator expression.

Exercise

Add a generator paths(lines) that yields only the path (/shop, /pay…) of each line, and use it to count how many times /pay appears.