Generators and the yield keyword

The Anatomy of a Generator Function

A generator function is defined just like a standard function, but instead of using the 'return' keyword to provide a result, it uses 'yield'. When a generator function is called, it does not actually execute the code inside the block immediately. Instead, it returns a generator object, which acts as an iterator. When you request the next value from this object, the function resumes execution from where it last left off until it encounters another yield statement. This mechanism allows the function to pause and save its entire local state—including local variables and the execution pointer—automatically. Because it holds its state in suspension rather than destroying the stack frame, it enables a sophisticated way of producing data incrementally. Understanding that 'yield' acts as a temporary pause button is fundamental to mastering how Python manages iterative data flow without requiring complex state-management classes.

# A generator function to yield numbers in a range
def countdown(n):
    print(f"Starting countdown from {n}")
    while n > 0:
        yield n  # Execution pauses here and returns the current value
        n -= 1

gen = countdown(3)
print(next(gen))  # Resumes, prints 3
print(next(gen))  # Resumes, prints 2

Lazy Evaluation and Memory Efficiency

The core philosophy behind generators is lazy evaluation. In traditional approaches, generating a list of one million items requires allocating memory for every single element simultaneously before the first item can even be processed. This is highly inefficient if you only need to iterate over the items one at a time. Generators bypass this by calculating the next item only when it is explicitly requested by the caller. By yielding items individually, the generator maintains a constant memory footprint, regardless of how many millions of items the sequence eventually produces. This design is crucial for handling large files, network streams, or computationally expensive sequences where pre-calculating everything would cause a memory overflow. When you reason about generators, visualize them as a pipeline where data is pulled through only as needed, ensuring the application remains responsive and light even when working with massive input volumes.

import sys

# A list comprehension creates the whole object in memory
list_comp = [x * 2 for x in range(1000000)]
# A generator expression creates a stream on demand
gen_exp = (x * 2 for x in range(1000000))

print(f"List size: {sys.getsizeof(list_comp)} bytes")
print(f"Generator size: {sys.getsizeof(gen_exp)} bytes")

State Maintenance via Yield

One of the most powerful features of the yield keyword is its ability to preserve the local scope of a function across multiple iterations. When a function executes a return statement, the local variables are cleaned up and the scope is destroyed. Conversely, when a generator yields, the interpreter saves the local namespace and the current instruction pointer into the generator object. When 'next()' is called again, the interpreter restores that specific context, allowing local variables to persist between steps. This makes generators an excellent tool for implementing stateful logic, such as a state machine or a custom sequence generator, without needing to create a class with instance attributes to track progress. Because this state is isolated within each generator instance, you can safely create multiple independent iterators from the same generator function, each tracking its own unique position in the logic flow.

# A generator maintaining its own internal state
def fibonacci_sequence():
    a, b = 0, 1
    while True:
        yield a  # State of 'a' and 'b' is preserved between calls
        a, b = b, a + b

fibs = fibonacci_sequence()
print([next(fibs) for _ in range(5)]) # [0, 1, 1, 2, 3]

The Yield from Syntax

When dealing with nested data structures or when you want to delegate part of the generator's task to another generator, the 'yield from' syntax becomes indispensable. Instead of manually writing a loop to iterate over another iterable and yielding its items one by one, 'yield from' creates a transparent bridge. It automatically handles the iteration protocol, including passing values, catching exceptions, and managing the state between the caller and the sub-generator. This significantly simplifies code readability and reduces the likelihood of bugs in complex recursive structures or when flattening nested collections. By using 'yield from', you essentially delegate the yielding process to another source, which is especially useful when creating complex data processing chains where one stage needs to fully consume another stage before continuing. It is a syntactical shortcut that keeps your implementation clean while retaining full performance benefits.

def sub_generator():
    yield "inner-1"
    yield "inner-2"

def main_generator():
    yield "outer-1"
    yield from sub_generator() # Delegates directly to the sub-generator
    yield "outer-2"

for val in main_generator():
    print(val)

Communication with Generators

Generators are not just one-way data producers; they can also receive input from the caller using the 'send()' method. When a generator is paused at a yield statement, the 'yield' expression itself can be assigned to a variable. When the caller calls 'gen.send(value)', that value is injected into the generator, effectively becoming the result of the yield expression where the code was suspended. This allows for bi-directional communication, turning the generator into a coroutine capable of processing incoming data or adjusting its internal logic dynamically. This advanced feature is the foundation for asynchronous programming patterns and more complex data processing pipelines where the producer needs feedback from the consumer. While the initial setup requires starting the generator with a call to 'next()' or 'send(None)' to prime it to the first yield point, this bidirectional channel provides powerful control for sophisticated data processing tasks.

def accumulator():
    total = 0
    while True:
        # Yields total, but also expects to receive a value back
        added_val = yield total
        if added_val is not None:
            total += added_val

acc = accumulator()
next(acc) # Prime the generator
print(acc.send(10)) # Adds 10, returns 10
print(acc.send(5))  # Adds 5, returns 15

Interview questions

All Python interview questions →

Check yourself

Take the full Python quiz →