Python - Generators

In Python, a Generator Function is a special type of function that generates values one at a time instead of returning all values at once.

A Normal Function uses the return statement to return a value and terminate the function.

A Generator Function uses the yield keyword to produce a value and temporarily pause its execution.

When a Generator Function is called, it does not immediately execute the complete function. Instead, it returns a special object called a Generator Object.

The values from the Generator Object can be retrieved one at a time.

Suppose we need to process millions of records.

If we store all records inside a list, the complete list must be stored in memory. This can consume a large amount of memory.

A Generator Function solves this problem by generating one value at a time.

Therefore, Generator Functions are useful when:

• Working with large datasets.
• Reading large files.
• Generating large sequences of numbers.
• Processing streaming data.
• Reducing memory consumption.
• Generating values only when they are required.

The yield keyword is the most important part of a Generator Function.

It is similar to the return statement because both can send a value back to the caller. However, there is an important difference.

• The return statement terminates the function completely.
• The yield statement pauses the function and saves its current state.

When the next value is requested, execution continues from the statement immediately after the previous yield.

The function display() contains three yield statements.

Because the function contains yield, Python automatically treats it as a Generator Function.

When display() is called, the function does not return all three values immediately. Instead, it returns a Generator Object.

The exact memory address displayed in the Generator Object may be different each time the program runs.

A Generator Object represents the sequence of values produced by a Generator Function.

The Generator Object is also an iterator. Therefore, we can retrieve values from it using:

• The next() function.
• A for loop.

Generator values are produced only when requested. This behaviour is known as Lazy Evaluation.

The first call to next(result) starts the Generator Function. Execution continues until the first yield statement is reached. The value 10 is produced and the function pauses.

The second call continues execution from the previous position and produces 20. The third call produces 30.

Function Call	Action	Result
display()	Creates Generator Object	No value generated yet
First next()	Executes until first yield	10
Second next()	Continues until second yield	20
Third next()	Continues until third yield	30

After all values have been generated, calling next() again raises a StopIteration exception.

10
20
StopIteration

A for loop automatically retrieves values from the Generator Object one at a time. It internally handles the StopIteration exception, so using a for loop is often easier than manually calling next().

10
20
30

Normal Function	Generator Function
Uses return.	Uses yield.
Returns a value and terminates.	Produces a value and pauses.
Does not preserve execution state after returning.	Preserves execution state between yields.
Normally executes when called.	Calling it creates a Generator Object; execution starts when iteration begins.
Usually returns the complete result.	Generates values one at a time.
May require more memory when returning large collections.	Memory efficient for large sequences.

return	yield
Used in Normal Functions.	Used in Generator Functions.
Terminates the function.	Pauses the function.
Returns a value directly.	Produces a value when requested.
Function state is not resumed after returning.	Function state is preserved and execution can continue.

Step	Description
1	A Generator Function is defined using one or more yield statements.
2	Calling the function creates a Generator Object.
3	The first value is requested using next() or iteration.
4	The function executes until it reaches yield.
5	The value is produced and execution pauses.
6	The next request resumes execution from the previous position.
7	When the function finishes, iteration stops.

A Generator Function can contain multiple yield statements. Each yield produces one value and temporarily pauses the function.

10
20
30
40

This program shows exactly where execution resumes after each yield.

Start
10
After First Yield
20
After Second Yield
30

When the first next() is called, the function starts execution. It prints Start and reaches yield 10. The value 10 is produced and the function pauses.

When the second next() is called, execution resumes after yield 10. It prints After First Yield and produces 20.

The third next() resumes the function again and produces 30.

Just like Normal Functions, Generator Functions can also accept parameters. The parameter values can be used to control which values the Generator produces.

1
2
3
4
5

A Generator Object can be converted into a list using list().

However, converting a Generator into a list generates and stores all values in memory, which reduces the memory-saving advantage of the Generator.

[10, 20, 30, 40]

A Generator Object produces each value only once. After all values have been consumed, the same Generator Object cannot automatically start again.

[10, 20, 30]
[]

To iterate again, a new Generator Object must be created by calling the Generator Function again. Each call creates an independent Generator Object.

[10, 20, 30]
[10, 20, 30]

The Generator produces Fibonacci numbers one at a time. The variables a and b store the current and next Fibonacci values.

After each yield, the Generator preserves these variable values. When execution resumes, the next Fibonacci number is calculated.

This approach is useful because a complete list of Fibonacci numbers does not need to be created before processing begins.

The Generator automatically remembers its complete state between yield statements.

Generator Feature	Description
Local Variables	Their values are preserved between yield statements.
Execution Position	The position where the function paused is remembered.
Next Request	Execution resumes from the previous position.
Completion	The Generator becomes exhausted after the function finishes.

• Values are generated only when required.
• Memory consumption is reduced.
• The execution state is automatically preserved.
• Large sequences can be processed efficiently.
• Generators work directly with for loops.
• Generator code can be simpler than creating custom iterator classes.
• Useful for processing large datasets and streaming data.