What the heck are Python Generator Expressions and List Comprehensions?

Have you ever thought on initializing a list with its values with a one liner expression? That's totally possible in Python using list comprehensions. But sometimes you wont use the list stored value for a while and you just wanted to initialize the values for a future use... That's a simple use case of or Generator Expressions.

PS: If you have read Fluent Python from @ramalhoorg , there is nothing new right here, but you can share this text to a friend, so she will be able to learn more about Python.

PS 2: Thanks @ramalhoorg for the examples on the book, they were very useful and some of them are used right here!

What are List Comprehensions?

List comprehensions are a list expression that creates a list with values already inside it, take a look at the example below:

>>> my_incredible_list = [x for x in range(5)]
>>> my_incredible_list
[0, 1, 2, 3, 4]

This list comprehension is the same as if you were doing a for loop appending values to a list.

>>> my_incredible_list = []
>>> for x in range(5):
>>>     my_incredible_list.append(x)
>>> my_incredible_list
[0, 1, 2, 3, 4]

With list comprehensions, you are able to generate lists with tuples inside iterating two other lists.

>>> foods = "apple banana sausages".split() # We declare a list here with foods
>>> beverages = "coca-cola pepsi guarana".split() # We declare a list of beverages
>>> [(food, beverage) for food in foods for beverage in beverages]
[('apple', 'coca-cola'), ('apple', 'pepsi'), ('apple', 'guarana'), ('banana', 'coca-cola'), ('banana', 'pepsi'), ('banana', 'guarana'), ('sausages', 'coca-cola'), ('sausages', 'pepsi'), ('sausages', 'guarana')]

What are Generator Expressions?

So generator expressions are just like a list comprehension, but it doesn't store it's values on the memory, it only generate the value you are iterating on when the generator is called. Take a look at the example below:

>>> my_incredible_list = (x for x in range(5))
>>> my_incredible_list
<generator object <genexpr> at 0x7f14149a3db0>

To check for a value on a generator you can either iterate over the generator using a for loop or you can check the values using the next() operator.

>>> # Using the next operator
>>> next(my_incredible_list)
0
>>> next(my_incredible_list)
1
>>> next(my_incredible_list)
2 
>>> next(my_incredible_list)
3
>>> next(my_incredible_list)
4
>>> next(my_incredible_list)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>
>>> # Using a for loop
>>> for item in my_incredible_list:
...     print(item)
... 
0
1
2
3
4

Checking out the Assembly of both of them to see the differences

Assembly instructions from Python disassambler

>>> import dis
>>> my_incredible_list = [x for x in range(5)]
>>> dis.dis(my_incredible_list)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/dis.py", line 67, in dis
    type(x).__name__)
TypeError: don't know how to disassemble list objects
>>> my_incredible_list = (x for x in range(5))
>>> dis.dis(my_incredible_list)
  1           0 LOAD_FAST                0 (.0)
        >>    2 FOR_ITER                10 (to 14)
              4 STORE_FAST               1 (x)
              6 LOAD_FAST                1 (x)
              8 YIELD_VALUE
             10 POP_TOP
             12 JUMP_ABSOLUTE            2
        >>   14 LOAD_CONST               0 (None)
             16 RETURN_VALUE
>>> 

So as we see right here, the disassemble of the list comprehension is not possible, because we cannot disassemble a data structure, but we can still disassemble the generator expression.

When we run dis.dis(<genexpr> right here) we are able to see that the expression is treated like a queue, we yield the value and then we pop it, so it's not called anymore.

If we call the dis.dis(<listcomp> here) we are able to see the running procedure to create a list using the listcomp expression:

>>> dis.dis("[x for x in range(5)]")
  1           0 LOAD_CONST               0 (<code object <listcomp> at 0x7f14149c84b0, file "<dis>", line 1>)
              2 LOAD_CONST               1 ('<listcomp>')
              4 MAKE_FUNCTION            0
              6 LOAD_NAME                0 (range)
              8 LOAD_CONST               2 (5)
             10 CALL_FUNCTION            1
             12 GET_ITER
             14 CALL_FUNCTION            1
             16 RETURN_VALUE

So as we see, the MAKE_FUNCTION piece is run generating the values for the list, we instantiate the function range, load a value to it (5), we call it and them we save it to the list instead of just yielding the value every time the generator is called.