Generators, iterators, lambdas and closures;

A Python generator is a piece of specialized code able to produce a series of values, and to control the iteration process. This is why generators are very often called iterators, and although some may find a very subtle distinction between these two, we’ll treat them as one.

1. An iterator is an object of a class providing at least two methods (not counting the constructor):

- __iter__() is invoked once when the iterator is created and returns the iterator’s object itself;
- __next__() is invoked to provide the next iteration’s value and raises the StopIteration exception when the iteration comes to an end.

Simple generator example:

for i in range(5):
    print(i)

1 2	for i in range(5): print(i)

The iterator protocol is a way in which an object should behave to conform to the rules imposed by the context of the for and in statements. An object conforming to the iterator protocol is called an iterator.

The Fibonacci numbers (Fib_i) are defined as follows:

Fib₁ = 1
Fib₂ = 1
Fib_i = Fib_i-1 + Fib_i-2

In other words:

the first two Fibonacci numbers are equal to 1;
any other Fibonacci number is the sum of the two previous ones (e.g., Fib₃ = 2, Fib₄ = 3, Fib₅ = 5, and so on)

class Fib:
    def __init__(self, nn):
        print("__init__")
        self.__n = nn
        self.__i = 0
        self.__p1 = self.__p2 = 1

    def __iter__(self):
        print("__iter__")
        return self

    def __next__(self):
        print("__next__")				
        self.__i += 1
        if self.__i > self.__n:
            raise StopIteration
        if self.__i in [1, 2]:
            return 1
        ret = self.__p1 + self.__p2
        self.__p1, self.__p2 = self.__p2, ret
        return ret


for i in Fib(10):
    print(i)

class Fib:

def __init__(self, nn):

print("__init__")

self.__n = nn

self.__i = 0

self.__p1 = self.__p2 = 1

def __iter__(self):

print("__iter__")

return self

def __next__(self):

print("__next__")

self.__i += 1

if self.__i > self.__n:

raise StopIteration

if self.__i in [1, 2]:

return 1

ret = self.__p1 + self.__p2

self.__p1, self.__p2 = self.__p2, ret

return ret

for i in Fib(10):

print(i)

2. The yield statement can be used only inside functions. The yield statement suspends function execution and causes the function to return the yield’s argument as a result. Such a function cannot be invoked in a regular way – its only purpose is to be used as a generator (i.e. in a context that requires a series of values, like a for loop.)

Take a look at this function:

def fun(n):
    for i in range(n):
        return i

def fun(n):

for i in range(n):

return i

It’s clear that the for loop has no chance to finish its first execution, as the return will break it irrevocably.

Moreover, invoking the function won’t change anything – the for loop will start from scratch and will be broken immediately.

We can say that such a function is not able to save and restore its state between subsequent invocations.

This also means that a function like this cannot be used as a generator.

Let’s replace exactly one word in the code:

def fun(n):
    for i in range(n):
        yield i

def fun(n):

for i in range(n):

yield i

There is one important limitation: such a function should not be invoked explicitly as – in fact – it isn’t a function anymore; it’s a generator object.

The invocation will return the object’s identifier, not the series we expect from the generator.

This is how we can use it:

def fun(n):
    for i in range(n):
        yield i

for v in fun(5):
    print(v)

def fun(n):

for i in range(n):

yield i

for v in fun(5):

print(v)

0
1
2
3
4

def powers_of_2(n):
    power = 1
    for i in range(n):
        yield power
        power *= 2

for v in powers_of_2(8):
    print(v)

def powers_of_2(n):

power = 1

for i in range(n):

yield power

power *= 2

for v in powers_of_2(8):

print(v)

1
2
4
8
16
32
64
128

Example.

s = '+' 
for i in range(2): 
    s += s 
    print("s=", s, end="\n")

s = '+'

for i in range(2):

s += s

print("s=", s, end="\n")

s= ++
s= ++++

def my_fun(n): 
    s = '+' 
    for i in range(n): 
        s += s 
        yield s
        
for x in my_fun(2): 
    print(x, end="")

def my_fun(n):

s = '+'

for i in range(n):

s += s

yield s

for x in my_fun(2):

print(x, end="")

++++++

3. A conditional expression is an expression built using the if-else operator. For example:

print(True if 0 >= 0 else False)

1	print(True if 0 >= 0 else False)

outputs True.

4. A list comprehension becomes a generator when used inside parentheses (used inside brackets, it produces a regular list). For example:

for x in (el * 2 for el in range(5)):
  print(x)

1 2	for x in (el * 2 for el in range(5)): print(x)

outputs 02468.

def powers_of_2(n):
    power = 1
    for i in range(n):
        yield power
        power *= 2


t = [x for x in powers_of_2(5)]
print(t)

def powers_of_2(n):

power = 1

for i in range(n):

yield power

power *= 2

t = [x for x in powers_of_2(5)]

print(t)

[1, 2, 4, 8, 16]

More examples list comprehensions:

list_1 = []

for ex in range(6):
    list_1.append(10 ** ex)

list_2 = [10 ** ex for ex in range(6)]

print(list_1)
print(list_2)

list_1 = []

for ex in range(6):

list_1.append(10 ** ex)

list_2 = [10 ** ex for ex in range(6)]

print(list_1)

print(list_2)

[1, 10, 100, 1000, 10000, 100000]

the_list = []
for x in range(10):
    the_list.append(1 if x % 2 == 0 else 0)

print(the_list)

the_list = []

for x in range(10):

the_list.append(1 if x % 2 == 0 else 0)

print(the_list)

[1, 0, 1, 0, 1, 0, 1, 0, 1, 0]

5. List comprehensions vs. generators

Just one change can turn any list comprehension into a generator. It’s the parentheses. The brackets make a comprehension, the parentheses make a generator.

the_list = [1 if x % 2 == 0 else 0 for x in range(10)]
the_generator = (1 if x % 2 == 0 else 0 for x in range(10))

for v in the_list:
    print(v, end=" ")
print()

for v in the_generator:
    print(v, end=" ")
print()

print(len(the_list))
print(len(the_generator))

the_list = [1 if x % 2 == 0 else 0 for x in range(10)]

the_generator = (1 if x % 2 == 0 else 0 for x in range(10))

for v in the_list:

print(v, end=" ")

print()

for v in the_generator:

print(v, end=" ")

print()

print(len(the_list))

print(len(the_generator))

1 0 1 0 1 0 1 0 1 0

10

Traceback (most recent call last):

File "main.py", line 15, in <module>

print (len(the_generator))

TypeError: object of type 'generator' has no len()

You can’t use the len() function on the generator.

6. The list() function can transform a series of subsequent generator invocations into a real list:

def powers_of_2(n):
    power = 1
    for i in range(n):
        yield power
        power *= 2

t = list(powers_of_2(3))
print(t)

def powers_of_2(n):

power = 1

for i in range(n):

yield power

power *= 2

t = list(powers_of_2(3))

print(t)

[1, 2, 4]

7. The in operator allows you to use a generator, too.

def powers_of_2(n):
    power = 1
    for i in range(n):
        yield power
        power *= 2


for i in range(20):
    if i in powers_of_2(4):
        print(i)

def powers_of_2(n):

power = 1

for i in range(n):

yield power

power *= 2

for i in range(20):

if i in powers_of_2(4):

print(i)

1
2
4
8

8. Fibonacci number generator looks much better than the objective version based on the direct iterator protocol implementation.

def fibonacci(n):
    p = pp = 1
    for i in range(n):
        if i in [0, 1]:
            yield 1
        else:
            n = p + pp
            pp, p = p, n
            yield n

fibs = list(fibonacci(10))
print(fibs)

def fibonacci(n):

p = pp = 1

for i in range(n):

if i in [0, 1]:

yield 1

else:

n = p + pp

pp, p = p, n

yield n

fibs = list(fibonacci(10))

print(fibs)

[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

9. A lambda is a function without a name (you can also call it an anonymous function). The declaration of the lambda function doesn’t resemble a normal function declaration in any way:

lambda parameters: expression

1	lambda parameters: expression

Example 1.

def foo(x, f):
  return f(x)

print(foo(9, lambda x: x ** 0.5))

def foo(x, f):

return f(x)

print(foo(9, lambda x: x ** 0.5))

outputs 3.0.

Example 2.

two = lambda: 2
sqr = lambda x: x * x
pwr = lambda x, y: x ** y

for a in range(-2, 3):
    print(sqr(a), end=" ")
    print(pwr(a, two()))

two = lambda: 2

sqr = lambda x: x * x

pwr = lambda x, y: x ** y

for a in range(-2, 3):

print(sqr(a), end=" ")

print(pwr(a, two()))

4 4
1 1
0 0
1 1
4 4

Example 3.

def print_function(args, fun):
    for x in args:
        print('f(', x,')=', fun(x), sep='')

def poly(x):
    return 2 * x**2 - 4 * x + 2

print_function([x for x in range(-2, 3)], poly)

def print_function(args, fun):

for x in args:

print('f(', x,')=', fun(x), sep='')

def poly(x):

return 2 * x**2 - 4 * x + 2

print_function([x for x in range(-2, 3)], poly)

f(-2)=18

f(-1)=8

f(0)=2

f(1)=0

f(2)=2

def print_function(args, fun):
    for x in args:
        print('f(', x,')=', fun(x), sep='')

print_function([x for x in range(-2, 3)], lambda x: 2 * x**2 - 4 * x + 2)

def print_function(args, fun):

for x in args:

print('f(', x,')=', fun(x), sep='')

print_function([x for x in range(-2, 3)], lambda x: 2 * x**2 - 4 * x + 2)

PEP 8, the Style Guide for Python Code, recommends that lambdas should not be assigned to variables, but rather they should be defined as functions.

This means that it is better to use a def statement, and avoid using an assignment statement that binds a lambda expression to an identifer. Analyze the code below:

# Not recommended:
f = lambda x: 3*x

# Recommended:
def f(x): return 3*x

# Not recommended:

f = lambda x: 3*x

# Recommended:

def f(x): return 3*x

10. The map(fun, list) function creates a copy of a list argument, and applies the fun function to all of its elements, returning a generator that provides the new list content element by element.

Example 1.

short_list = ['mython', 'python', 'fell', 'on', 'the', 'floor']
new_list = list(map(lambda s: s.title(), short_list))
print(new_list)

short_list = ['mython', 'python', 'fell', 'on', 'the', 'floor']

new_list = list(map(lambda s: s.title(), short_list))

print(new_list)

outputs

['Mython', 'Python', 'Fell', 'On', 'The', 'Floor'].

Example 2.

list_1 = [x for x in range(5)]
print(list_1)

list_2 = list(map(lambda x: 2 ** x, list_1))
print(list_2)

for x in map(lambda x: x * x, list_2):
    print(x, end=' ')
print()

list_1 = [x for x in range(5)]

print(list_1)

list_2 = list(map(lambda x: 2 ** x, list_1))

print(list_2)

for x in map(lambda x: x * x, list_2):

print(x, end=' ')

print()

[0, 1, 2, 3, 4]

[1, 2, 4, 8, 16]

1 4 16 64 256

11. The filter(fun, list) function creates a copy of those list elements, which cause the fun function to return True. The function’s result is a generator providing the new list content element by element.

Example 1.

The isinstance() function returns True if the specified object is of the specified type, otherwise False.

short_list = [1, "Python", -1, "Monty"]
new_list = list(filter(lambda s: isinstance(s, str), short_list))
print(new_list)

short_list = [1, "Python", -1, "Monty"]

new_list = list(filter(lambda s: isinstance(s, str), short_list))

print(new_list)

outputs

['Python', 'Monty'].

Example 2.

from random import seed, randint

seed()
data = [randint(-5,15) for x in range(5)]
filtered = list(filter(lambda x: x > 0 and x % 2 == 0, data))

print(data)
print(filtered)

from random import seed, randint

seed()

data = [randint(-5,15) for x in range(5)]

filtered = list(filter(lambda x: x > 0 and x % 2 == 0, data))

print(data)

print(filtered)

[1, 8, 2, 7, -2]

[8, 2]

12. A closure is a technique which allows the storing of values in spite of the fact that the context in which they have been created does not exist anymore.

Example 1:

def outer(par):
    loc = par

    def inner():
        return loc
    return inner


var = 1
fun = outer(var)
print(fun())

def outer(par):

loc = par

def inner():

return loc

return inner

var = 1

fun = outer(var)

print(fun())

1

Example 2:

def make_closure(par):
    loc = par

    def power(p):
        return p ** loc
    return power

fsqr = make_closure(2)   # loc = 2
fcub = make_closure(3)   # loc = 3

for i in range(5):
    print(i, fsqr(i), fcub(i))  # p = i

def make_closure(par):

loc = par

def power(p):

return p ** loc

return power

fsqr = make_closure(2) # loc = 2

fcub = make_closure(3) # loc = 3

for i in range(5):

print(i, fsqr(i), fcub(i)) # p = i

Note:

- the first closure obtained from make_closure() defines a tool squaring its argument (fsqr);
- the second one is designed to cube the argument (fcub).

This is why the code produces the following output:

0 0 0

1 1 1

2 4 8

3 9 27

4 16 64

Example 3:

def tag(tg):
  tg2 = tg
  tg2 = tg[0] + '/' + tg[1:]

  def inner(str):
    return tg + str + tg2
  return inner

b_tag = tag('<b>')
print(b_tag('Monty Python'))

def tag(tg):

tg2 = tg

tg2 = tg[0] + '/' + tg[1:]

def inner(str):

return tg + str + tg2

return inner

b_tag = tag('<b>')

print(b_tag('Monty Python'))

outputs

<b>Monty Python</b>

Exercise 1

What is the expected output of the following code?

class Vowels:
    def __init__(self):
        self.vow = "aeiouy "  # Yes, we know that y is not always considered a vowel.
        self.pos = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.pos == len(self.vow):
            raise StopIteration
        self.pos += 1
        return self.vow[self.pos - 1]

vowels = Vowels()
for v in vowels:
    print(v, end=' ')

class Vowels:

def __init__(self):

self.vow = "aeiouy " # Yes, we know that y is not always considered a vowel.

self.pos = 0

def __iter__(self):

return self

def __next__(self):

if self.pos == len(self.vow):

raise StopIteration

self.pos += 1

return self.vow[self.pos - 1]

vowels = Vowels()

for v in vowels:

print(v, end=' ')

a e i o u y

Exercise 2

Write a lambda function, setting the least significant bit of its integer argument, and apply it to the map() function to produce the string 1 3 3 5 on the console.

any_list = [1, 2, 3, 4]
even_list = # Complete the line here.
print(even_list)

any_list = [1, 2, 3, 4]

even_list = # Complete the line here.

print(even_list)

Lambda:

list(map(lambda n: n | 1, any_list))

1	list(map(lambda n: n \| 1, any_list))

Exercise 3

What is the expected output of the following code?

def replace_spaces(replacement='*'):
    def new_replacement(text):
        return text.replace(' ', replacement)
    return new_replacement


stars = replace_spaces()
print(stars("And Now for Something Completely Different"))

def replace_spaces(replacement='*'):

def new_replacement(text):

return text.replace(' ', replacement)

return new_replacement

stars = replace_spaces()

print(stars("And Now for Something Completely Different"))

And*Now*for*Something*Completely*Different