Choosing the Best Iteration Technique for Optimized Python Code

This article was originally created in Medium, check it here.

Python is a versatile language with multiple options to accomplish tasks. One of the most frequent tasks in programming is iterating through a collection of elements, and Python provides three popular ways to do this: using for loops, list comprehensions, and high-order functions. As Python developers, selecting the best method to guarantee best practices and optimal performance in our code is essential.

To compare the performance of these methods and discuss when to use each, we will explain the experiment in this article. So, let’s get started!
Experiment

To measure the performance of each process, we used the time library and the perf_counter() function. If you want to learn more about this function, you can check out the following link.

To ensure accurate results, we allocated four processor threads and 1GB of memory to the process using a container. Python will manage the jobs using the processor threads by default.

We chose a simple mathematical task of generating multiples to conduct our experiment. We ran the process 100 times, each time generating a different number of multiples of 2. For instance, if we generated the first 100 multiples of 2, the resulting sequence would be:

[0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138, 140, 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176, 178, 180, 182, 184, 186, 188, 190, 192, 194, 196, 198]

In the following results, we ran each process 10,000, 100,000, and 1,000,000 times, respectively. We executed each process separately to ensure accurate results and prevent memory overload that could interfere with the measurements.

We ran the process 102 times, removing the lowest and highest values to eliminate any results affected by system performance. Finally, we determined the minimum and maximum values for each step and calculated the mean of all values.

For loop

The for loop is a basic construct in Python for iterating over a sequence of elements. It has been around since the beginning of the language and is still widely used today. A for loop allows us to act on each element in a sequence one at a time. Here’s an example:

import time

def for_loop(rang: int, decimals: int) -> float:
    start_time_for_loop = time.perf_counter()
    my_list_for_loop = []
    n = 2
    for i in range(rang):
        n = i * n
        my_list_for_loop.append(n)
    return round(time.perf_counter() - start_time_for_loop, decimals)

For loops are easy to read and understand and great for simple iterations. However, for larger sequences, the performance of for loops can degrade due to the overhead of the loop construct.

List Comprehension

List comprehensions are a concise way of creating lists in Python. They are a syntactic shortcut for writing a for loop to create a list. Here’s an example:

import time


def comprehension_loop(rang: int, decimals: int) -> float:
    start_time_comprehension_loop = time.perf_counter()
    n = 2
    my_list_comprehension_loop = [n * i for i in range(rang)]
    return round(time.perf_counter() - start_time_comprehension_loop, decimals)

List comprehensions are generally faster than for loops for creating lists because they do not require the overhead of the loop construct. They are also more concise and easier to read than for loops, making them a great choice for simple iterations.

High-Order Functions

Python has several built-in high-order functions, including map, filter, and reduce. These functions take a function as an argument and apply it to every element in a sequence. Here’s an example using map:

import time


def high_order_func(rang: int, decimals: int) -> float:
    start_time_high_order_func = time.perf_counter()
    n = 2
    my_list_high_order_func = list(map(lambda i: i * n, range(rang)))

High-order functions can be faster than for loops and list comprehensions for complex operations because they are optimized for performance. They are also more concise and easier to read than for loops for complex operations, making them a great choice for complex iterations.

Results

Below are the results of the 100 tests in a range of 10,000 iterations. As seen in the graphs, we may encounter occasional peaks corresponding to longer processing times, which can be attributed to processor load caused by background processes. However, for the most part, the processing time remains relatively consistent with minimal variation.

CSV File	Mean	Min	Max
For loop	1.34	0.79	1.8
Comprehension list	0.91	0.7	1.3
Higher order functions	1.617	1.0	2.58

We can now begin to see how the tests will evolve based on the results of the following 100 tests, which were conducted over 100,000 iterations. As expected, the higher-order functions are starting to fall slightly behind the other two functions.

CSV File	Mean	Min	Max
For loop	12.66	10.25	14.23
Comprehension list	9.53	7.9	11.7
Higher order functions	15.59	13.26	17.87

However, there is still no clear difference between the three functions, and the execution times remain in the same range as the previous tests. This suggests that there may not be a significant performance advantage to using higher-order functions in this scenario.

Overall, these results provide valuable insights into the performance characteristics of the different functions. Further experimentation will be necessary to determine whether the observed trends hold up over larger datasets and longer execution times.

CSV File	Mean	Min	Max
For loop	103.61	98.1	130.59
Comprehension list	89.92	68.8	128.9
Higher order functions	141.2	133.86	172.17

The final test clearly shows a significant difference in performance between the three functions. Specifically, the higher-order functions perform much more slowly than the other two. This result provides us with a much clearer conclusion.

Discussion

When to use each

• for loops are great for simple iterations over small sequences. They are easy to read and understand and do not require specialized knowledge.

• List comprehensions are great for creating lists and performing simple operations on sequences. They are more concise than for loops and easier to read, making them an excellent choice for simple iterations.

• High-order functions are efficient and faster than for loops and list comprehensions in complex sequence operations. They are also more concise and easier to read, making them suitable for elegant and efficient code.

Conclusion

In conclusion, the test results show that high-order functions are not as efficient and fast in simple operations on sequences. However, loops and list comprehensions are better options in these cases and are easier to read and understand. Therefore, it is essential to consider the complexity of the task before choosing an iteration technique. In summary, loops are good for simple iterations and small sequences, list comprehensions are ideal for creating simple lists and operations, while high-order functions are efficient and better suited for complex operations on large sequences.

Comments

[hidden_dude]

You may very well be right, but I suspect that your testing methodology may have introduced some extra overhead. Mainly in the allocation of the list.

The higher order function calls list() at the end which makes a full copy of the entire result. The for-iteration is calling append repeatedly whereas the comprehension may have some trickery to allocate a list of the right size.

It would be useful to try to factor out the list allocation aspect of your test because those subtle differences are key.