Python generators: Generator expression and function

In this article, you will learn about generator expressions and generator functions in python and the use of yield and next() with sufficient examples.

What is a generator
As the name suggests, a generator is anything that generates or outputs something. A python generator is used to generate values, one at a time.
Advantage of a generator is that it can store large number of values but its size is less.
Generator expressions and functions are classified as Generators.
A. Generator Expression
A generator expression is used to generate values.
Important feature of generator expression is that it does not return thegenerated values at the time it is defined, neither does it store these values in memory. It returns values as and when it is used.
Generator expression is a one-liner statement enclosed between parenthesis followed by the expression that results in a value and for loop as below

where expression evaluates to a value that is returned by the generator followed by the loop. This syntax resembles the syntax of list comprehension.
Below is an example of a generator expression

# create a generator expression
gen_expr = (x for x in range(5))
# print it and its type
print(gen_expr)
print(type(gen_expr))

Above example creates a generator expression that will generate values from 0 to 4 since the expression given is the same as the loop variable.
Above example prints

Notice how the generator expression is printed and its type.
Generator expression for generating even numbers till 18 would be

expr = (x * 2 for x in range(10))

Generating values
A generator expression is iterable meaning that it can be iterated. It can be used anywhere iterables can be used.

Some examples are
1. Using a generator with a for loop in place of range() function. This is because a for loop requires an iterable and a generator expression is an iterable,
2. As an argument to list constructor for initializing a list,
3. As an argument to tuple constructor for initializing a tuple,
4. As argument to string join() function etc.

Generator expression example
Below is an example which creates following different generator expressions:
a. Expression of 5 numbers and loops over it,
b. Expression from a string and initialize a list from it,
c.Expression of 5 numbers and initialize a tuple.

# create generator expression
gen_expr = (x for x in range(5))
# iterate generator expression
for num in gen_expr:
    print(num, end=' ')
print('')

# create generator expression
gen_expr = (letter for letter in 'codippa')
# create a list
list_expr = list(gen_expr)
print('List is:', list_expr)

# create generator expression
gen_expr = (x for x in range(5))
# create a tuple
tuple_expr = tuple(gen_expr)
print('Tuple is:', tuple_expr)

This prints

Remember that once a generator expression is used, it becomes empty and you can not reuse it. Example,

# create generator expression
gen_expr = (x for x in range(5))
# iterate generator expression
for num in gen_expr:
    print(num, end=' ')
print('')

# create a tuple using same expression
tuple_expr = tuple(gen_expr)
print('Tuple is:', tuple_expr)

This prints

Look, the tuple is empty. This is because the generator expression has already been iterated or consumed.
Generator expression size
As stated earlier, a generator expression does not store its elements when it is created but generates them when required.
This is the biggest benefit of a generator expression when the data set is huge as it is memory efficient.
Below example creates a list and a generator expression of 50,000 numbers and prints their size in bytes.

import sys

gen_expr = (x for x in range(50000))
numbers = 
print('Size of generator expression is {} bytes'.format(sys.getsizeof(gen_expr)))
print('Size of list is {} bytes'.format(sys.getsizeof(numbers)))

Check out the sizes below

Conditional generator expression
A generator expression can have a condition which is executed before returning a value. Syntax to include a condition in a generator expression is

The condition is written using an if statement and is executed for every value. Only when the if statement returns True, the value is returned, otherwise next value is checked.
Example, below expression returns only lower case characters from a string.

expr = (c for c in 'Python Is Good' if c.islower())

for char in expr:
    print(char, end=' ')

which prints

B. Generator functions
In the previous sections, we saw what is a generator expression. A generator function is similar to a generator expression in that it returns or generates values as and when required.
A generator function that generates numbers from 0 to 4 is given below

def gen_fun(limit):
    start = 0
    # loop from 0 to 4
    while(start < limit):
        yield start
        start += 1

This is the same as a generator expression as shown in the previous section.
Thus, following generator expression

gen = (x for x in range(10))

is equivalent to below generator function

def gen():
    n = 10    
    start = 0
    while(start < n):
        yield start
        start += 1

A generator function is different from a normal python function with differences outlined below
1. A normal function returns a value as soon as it is invoked and terminates while a generator function returns a generator object when it is invoked and it returns a value only when it is used as an iterable.
2. A normal function returns the same value when invoked multiple times while a generator function will return a new value as per its implementation every time it is called.
3. A normal function does not maintain the state of its local variables between multiple calls while generator function remembers the values of local variables between different calls.
4. A normal function returns a value with return keyword while generator function returns a value with yield keyword.

Generator function working: yield keyword
A generator function generates values and returns a different value in every call. This means that it remembers the value that it returned in the last call.
This is done with the yield keyword. When the function encounters yield followed by a variable, it stores the current value of that variable, executes the remaining function and then returns the value of that variable.
When it is called again, it starts with the previous value of that variable.

Following points should be remembered regarding a generator function.
I. Execution of generator function in subsequent calls does not start from the beginning, but from the loop that is generating values.
II. Generator function is not executed when it is created, it only executes when it is used as an iterable.

Below example will clarify both these points and its working.

# generator function
def gen(n):
    start=0
    print('Starting generator function')
    while(start < n):
        yield start
        start += 1

print('Creating generator function')
expr = gen(5)

print('Using generator function to iterate')
for num in expr:
    print(num, end=' ')

This prints

Look, when the generator function is created, the print statement inside it is not executed. When it is used in the for loop, the print statement executes.
Secondly, the print statement executes only once when the generator function is invoked for the first time.

Normally, a generator function is used directly with the loop(as shown below) rather than creating it separately. This was just for explanation.

# iterate using a generator function
for num in gen(5):
    print(num, end=' ')

next() function
A generator function can be used as an argument to next() function and it will return one value ahead of the last value returned by the generator. Example,

# define generator function
def gen(n): 
    start = 0
    while(start < n):
        yield start
        start += 1

# create generator
expr = gen(5)
print('First value from generator is', next(expr))

# use generator as iterable
for num in expr:
    print(num, end=' ')

This prints

Note that once next() is called, the generator will store the current state and in subsequent calls, it will return values after the returned value, as evident from the output.

Multiple yield statements
A generator function can have multiple yield statements, each of which is executed in different invocations of the function.
That is, when the function is called for the first time, first yield will execute. When called for the second time, second yield will execute and so on. Example, 

def gen(n):
    start = 0
    yield  start
    start += 1
    yield start
    start += 1
    yield start

for num in gen(3):
    print(num, end=' ')

This prints

Multiple yield statements are useful when there is no sequence between the different return values.

That is all on Python generators. Hit the clap if the article was useful.