Understand yield and generators in Python

In an ordinary def function, yield makes the function a generator function. Calling it returns a generator object without running the body immediately. Iteration resumes execution until the next yield, which produces a value and suspends the function while retaining its state.

Produce a sequence incrementally

def count_up_to(limit):
    value = 1
    while value <= limit:
        yield value
        value += 1

numbers = count_up_to(3)
print(next(numbers))  # 1
print(list(numbers))  # [2, 3]
print(list(numbers))  # []

Once exhausted, this generator does not restart. Call count_up_to(3) again to create another one. A generator expression, such as (value + 1 for value in range(4)), offers a compact way to compute values lazily too.

Unlike a prebuilt list, a generator does not need to hold every output value at once. It can still retain large local objects or refer to a large input, and converting it to a list materializes all remaining results.

Read a log with explicit cleanup

from contextlib import closing

def read_large_log(path):
    with open(path, encoding="utf-8", errors="replace") as source:
        for line in source:
            yield line.rstrip("\n")

with closing(read_large_log("app.log")) as lines:
    for line in lines:
        if "ERROR" in line:
            print(line)

The context manager closes the generator if the consumer exits early, which unwinds the file's with block. Do not rely solely on garbage collection to close a partially consumed generator. Memory depends on the current line and retained state; a single enormous line can still be expensive. errors="replace" is deliberately lossy and should be used only when that is acceptable for diagnostic reading.

Send values back in

def receiver():
    value = yield "ready"
    yield value

gen = receiver()
print(next(gen))       # ready
print(gen.send(42))    # 42
gen.close()

A new generator must first be advanced with next() or send(None) before receiving a non-None value. send supplies the result of the suspended yield expression. Native async coroutines use related concepts but are not interchangeable with every generator.

FAQ

Can yield and return appear together? Yes. In a synchronous generator, return ends iteration, and an optional return value becomes StopIteration.value. A normal for loop does not expose that value as another yielded item.

What does yield from do? It delegates to another iterable. For simple output it resembles a loop yielding each item, but delegation also forwards generator operations such as send and can capture a subgenerator's return value.

Does an async def containing yield behave the same way? It defines an asynchronous generator, consumed through asynchronous iteration. Some rules differ, including that it cannot return a nonempty value.

References

• Python expressions reference
• Python contextlib reference
• Python classes reference