For Loop VS List comprehension VS High order functions

Studying again the basic concepts of Python, a question that had always arisen came back to me, what is the difference in performance of each of the Loop options that we have within the language? (For Loop, List comprehension and High order functions), for this I had to understand how each one worked inside and find the best way to measure them, this is the reason why I began to investigate and test.

However, first I would like to explain what was the experiment that I carried out and how I came to think that this would be the best option.

Experiment

The experiment consisted of performing three functions in order to monitor the performance and not the process carried out by each of the iterables. At the beginning, a person from the community told me that the psutil module could help me by measuring the performance; however, this tool allows me to measure a complete process and it was very difficult and time-consuming to measure each of the functions separately, additionally we had a small detail and that is that these tests could not be done on large servers with a capacity of RAM and processors multicore, my machine was just a Macbook with 8 Gb of ram and a dual-core I5 core, a joke if we start to see the types of machines we have today.

With this in mind I decided to go for a basic mathematical iterative process that would start consuming enough resources that not too many processes would stall without affecting upcoming loops.

The solution was to iterate x number of times a number that was multiplied by the previous sequence and store everything in a list, something similar to fibonacci, but with much higher values provided that the list began to be much larger.

Hoping that the garbage collector would remove the variables in each iteration, I used the same names in all the calls and only changed the process, the code was as follows.

Code

def high_order_func(rang):
    start_time = time.time()
    n = 2
    my_list = map(lambda i: i * n, range(rang))
    my_list = list(my_list)
    end_time = time.time()

    how_much_time = end_time - start_time
    return how_much_time

def comprehension_loop(rang):
    start_time = time.time()
    n = 2
    my_list = [n * i for i in range(rang)]
    end_time = time.time()

    how_much_time = end_time - start_time
    return how_much_time

def loop(rang):
    start_time = time.time()
    my_list = []
    n = 2
    for i in range(rang):
        n = i * n
        my_list.append(n)
    end_time = time.time()

    how_much_time = end_time - start_time
    return how_much_time

if __name__ == '__main__':
    import time
    rang = 1000
    high_order_func = high_order_func(rang)
    comprehension_loop = comprehension_loop(rang)
    loop = loop(rang)

    print("loop ", str(loop), " sec")
    print("list comprehension ", str(comprehension_loop), " sec")
    print("Higher order funciton ", str(high_order_func), " sec")

Results

Each outcome is separated by its respective value of the rang variable and before you read on I would like you to try to guess which one is going to win and try to explain why you think this will happen.

Initially, it was tested with a small rang in order to see in small processes what the real difference was.

rang = 1000

Test 1

loop  0.009955167770385742  sec
list comprehension  0.02051258087158203  sec
Higher order funciton  8.821487426757812e-06  sec

Test 2

loop  8.0108642578125e-05  sec
list comprehension  5.507469177246094e-05  sec
Higher order funciton  0.0001220703125  sec

Test 3

loop  8.082389831542969e-05  sec
list comprehension  5.507469177246094e-05  sec
Higher order funciton  0.00012302398681640625  sec

At this point we can see that there is no clear difference between the three functions, let us remember that these values do not even take milliseconds yet, since at the end as you can see they have the scientific notation e-05 or e-06 which tries to say that they are They add that amount of zeros to the value at the beginning, so we can say that there is no winner since the values have a clear fluctuation of times but do not exceed e-05.

rang = 100000

Test 1

loop  0.008324146270751953  sec
list comprehension  0.007503986358642578  sec
Higher order funciton  0.012623071670532227  sec

Test 2

loop  0.009238958358764648  sec
list comprehension  0.007218837738037109  sec
Higher order funciton  0.015091180801391602  sec

Test 3

loop  0.00857686996459961  sec
list comprehension  0.007061004638671875  sec
Higher order funciton  0.013355016708374023  sec

This Output gives us an overview of how the tests are going to evolve from now on, since it shows us that the higher-order functions begin to slowly fall behind, however the other two remain quite similar in execution times.

rang = 100000000

Finally, the definitive test, since if I continued to increase the rang the machine began to throttle, which affected the tests beyond giving me a more realistic result than the one I was looking for.

Test 1

lloop  9.187693119049072  sec
list comprehension  9.024915933609009  sec
Higher order funciton  14.004309892654419  sec

Test 2

loop  9.79594111442566  sec
list comprehension  9.057484865188599  sec
Higher order funciton  15.923699855804443  sec

Test 3

loop  8.725109577178955  sec
list comprehension  8.681398868560791  sec
Higher order funciton  14.65591287612915  sec

In this last test we have a fairly clear difference and two winners that are close to being quite tied, since the higher order functions lagged far behind compared to the other two, but what happened here? What could have happened?

Conclusión

These results surprised me more than I expected, as at one point I thought I was doing double processing with these functions, however reading a bit of the [real python] article(https://realpython.com/python-map-function/) I understood that the map iterates over the first element creating an object with those values, but when passing it to the list it has to do a bigger process by operating all those bytes again, therefore, it makes them more expensive.

What do you think? Tell me in the comments your opinion about it, if this test seemed reliable or you think I had to take it from another point of view or something in my test is wrong.

I also invite you to read the following articles that allows you to understand some more things about loops.

List-comprehension: https://realpython.com/list-comprehension-python/
For Loops: https://realpython.com/python-for-loop/