DEV Community

Alin Climente
Alin Climente

Posted on

Loading a 600mb+ CSV file is faster in Python than in Go?

Ever wondered how much time it takes for Go to load a CSV file in comparison with the slower Python language?

I downloaded a relatively big csv file from the web (658 MB) to see which one wins.

Let's start with Python and the well known pandas package:
Pandas6sec
Pretty bad isn't it? 6+ seconds! Tried with engine="c" flag, but got same result. It's still good enough, now we have available a lot of pandas goodness.

Let's see in Go:
Go1.5
Nice! Finished in 1.5 seconds -> 21% faster! But, remember pandas offers a lot out of the box, with Go you need to write a lot of code to benefit from those extra goodies.

Is Python slow? Yes, but if the packages implementation you are mostly using are in C/Rust you won't feel it as much.

Now, let's switch pandas with a newer alternative polars:
polars700ms
It's even faster than Go!(yes, because of Rust). We now have a lot of functionalities out of the box (not as much as pandas, but will get there).

Time to learn Rust I guess...

Top comments (0)