DEV Community

loading...

Python performance benefits of generator expressions

Ryan Westlund
I'm a programmer, writer, and philosopher. My Github account is yujiri8; all my content besides code is at yujiri.xyz.
・2 min read

I'm a huge fan of Python's generator expressions and list comprehensions, but today I thought I'd do a performance test. The following tests are done with Python 3.7.8 on FreeBSD.

>>> timeit.timeit('for n in filter(lambda n: n % 2 == 0, nums): n', setup = 'nums = [1,2,3,4,5,6,7,8,9]')
1.322555473074317
>>> timeit.timeit('for n in (n for n in nums if n % 2 == 0): n', setup = 'nums = [1,2,3,4,5,6,7,8,9]')
0.8492276258766651
Enter fullscreen mode Exit fullscreen mode

Damn! That's pretty significant.

The reason I'm iterating and evaluating the elements instead of just using list() around filter() and list comprehensions instead of generators I thought that would unfairly advantage the generator expression, since list is a function call.

And I'm right: that exacerbates the difference.

>>> timeit.timeit('list(filter(lambda n: n % 2 == 0, nums))', setup = 'nums = [1,2,3,4,5,6,7,8,9]')
1.4714625549968332
>>> timeit.timeit('[n for n in nums if n % 2 == 0]', setup = 'nums = [1,2,3,4,5,6,7,8,9]')
0.6537750540301204
Enter fullscreen mode Exit fullscreen mode

I'm surprised that replacing the for loop with building a list actually makes the generator expression faster.

Let's try map.

>>> timeit.timeit('for n in map(lambda n: n * 10, nums): n', setup = 'nums = [1,2,3,4,5,6,7,8,9]')
1.0912448461167514
>>> timeit.timeit('for n in (n * 10 for n in nums): n', setup = 'nums = [1,2,3,4,5,6,7,8,9]')
0.8008647549431771
Enter fullscreen mode Exit fullscreen mode

I guess map is faster than filter because it doesn't involve branching instructions.

Let's see what happens when we do both at once. My gut says this is the biggest win because the generator doesn't have to iterate twice (I promise I thought of that before testing it).

>>> timeit.timeit('for n in map(lambda n: n * 10, filter(lambda n: n % 2 == 0, nums)): n', setup = 'nums = [1,2,3,4,5,6,7,8,9]')
1.9710871148854494
>>> timeit.timeit('for n in (n * 10 for n in nums if n % 2 == 0): n', setup = 'nums = [1,2,3,4,5,6,7,8,9]')
0.9152004329953343
Enter fullscreen mode Exit fullscreen mode

It's over twice as fast! The generator implementation of both at once is barely any slower than just one of the operations.

So there's our answer: generators and comprehensions blow off map and filter on speed as well as they do on readability.

Discussion (0)