DEV Community

loading...

Discussion on: Is Python’s asyncio Worth It?

Collapse
rhymes profile image
rhymes • Edited

Let's define concurrency and parallelism first because it's a subtle difference and sometimes they get confused one for the other.

Concurrency means the program can run different tasks in overlapping time periods. A standard multitasker on a single core CPU is concurrent, because it cycles between different tasks (your programs) but those tasks are not technically running at the same time.

Parallelism means the ability of running completely separate tasks at the same time.

A program can be concurrent but not parallel, parallel but not concurrent, both or neither if it's totally sequential.

There are many ways to achieve concurrency, probably the most well known is using an event loop that runs until termination and on which build your async program. Twisted, eventlet, asyncio all use event loops (usually through poll or kqueue system calls).

Multithreading can be concurrent or parallel. On a single core it will always be concurrent. On a multi core it might be parallel because parallelism means "running two separate tasks AT THE SAME TIME" which you might or not achieve using multithreading, depends on how you structure your program.

Let me repeat: given a unit of time concurrent means multiple tasks make progress, parallel means multiple tasks run at the same time.

Python's multithreading, because of GIL, can never be parallel but it is concurrent. That's why you can use ThreadPool to "advance your tasks in concurrency" but not to run those tasks at the same time on different CPU cores.

A way to use all the cores in Python is to use multiple processes. Having more than one process (the interpreter with your piece of code) running on multiple cores can help you achieve parallelism in Python.

Processes in Python have a limit though, because you are effectively running multiple interpreters for the duration of your parallel tasks so you have to be careful about how many (benchmarking help) and how much memory each process occupies.

You can also obviously combine multiple processes and threads.

My head is already hurting and we haven't even talked about parallel programming with multiple networked machines, synchronisation between different tasks or other concurrency models (like Go's, Erlang's)

So, to answer your questions: it depends on what you have to do :-)

asyncio (or uvloop) is a perfectly valid choice if you want to build a high performant concurrent application. I wouldn't introduce Twisted in an existing application. I know there's an asyncio clone for Python 2 but I have no idea how stable it is: trollius.

Keep in mind you need async libraries for IO: database access, HTTP calls and stuff, everything needs to support asyncio or be adapted to it.

Concurrency and parallelism always come with a price :-)