One Simple Trick to Reduce Your Memory Usage in Python

Connor — Thu, 20 May 2021 08:18:07 +0000

Intro
List Comprehension
Generator Comprehension

Intro

Python is ultimate "getting things done" language, where you can soo easily write code and not worry too much about performance and memory. However once your program becomes large, large memory usage can significantly slow down your program. One easy way to reduce memory usage and speed up your programs is to switch your list comprehensions into generator comprehensions.

Lets explore this with a simple example program to sum up a range of numbers.

List Comprehension

Code:

import sys
my_large_list = [i for i in range(100000)]
print(sum(my_large_list))
print(f"My list is {sys.getsizeof(my_large_list)} bytes")

Output:

4999950000
My list is 824456 bytes

Generator Comprehension

My code is often filled with a lot of list comprehensions, where instead we could use generators instead. Generators operate like lists, except they are evaluated "lazily", so the values are grabbed when needed.

All we need to do is use curly braces on all of our list comprehensions.

Code:

import sys
my_large_generator_list = (i for i in range(100000))
print(sum(my_large_list))
print(f"My generator is {sys.getsizeof(my_large_generator_list)} bytes")

Output:

4999950000
My list is 112 bytes

As we can see both give the same result, however the generator only uses a fraction of the memory (112 bytes instead of 824456). When you have hundreds of lists floating in your code, switching them to generators is an easy way to save on memory and increase your program's speed :).

DEV Community: Connor

One Simple Trick to Reduce Your Memory Usage in Python

Table of Contents

Intro

List Comprehension

Generator Comprehension