Comprehensions in Python: Why are they faster? A deep dive.

I've asked this question to Python developers - beginners and advanced Python devs alike. Their answer, however, more often than not doesn't answer the fundamental question: are they the same as a simple for loop.

The internet does have a lot of articles around how you use them, but they do not cover the why - why do list comprehensions exist? What did the vanilla for loop miss that we had to create another syntax to do the same thing? I'm sharing a short summary of how you can use them and why are they faster, but if you want to deep dive, feel free to read more at this link.

What are comprehensions?

In short, comprehensions can help you shorten your code while giving a performance boost to your code. As an example, if you want to loop over a range of integers & add 1 to all, the for loop will be:

la_ints = []
for i in range(10**7):
    la_ints.append(i + 1)

However, a comprehension is much shorter and ~30% faster

la_ints = [i+1 for i in range(10**7)]

But.. why?

You might already know that Python itself is built on top of C - which is much faster than Python when it comes to handling raw data. When you're using a list comprehension, the evaluation logic of the loop is taken care of by the lower level implementation which is much faster. If you see the disassembly of these implementation using dis module, you'll see how append kills the performance of the for loop.

In my blog post Comprehensions in Python: The hard way, I deep dive into disassembling of the two implementations (including sweet sweet bytecodes), and how to manage memory with generator expressions when you're hitting memory limits.

Happy to hear any feedback about this post of the blog post! Thanks!