In this article, I will explain asynchronous programming in Python via AsyncIO library.
Before going forward let’s understand few terminologies
Event loops: It is used to run asynchronous tasks(which can be a coroutine, a future or any awaitable objects) concurrently, it can register the task which is going to be executed, execute them, delay or cancel them
Suppose we have 2 async tasks(task1 and task2) now I schedule both tasks on event loop now suppose event loop starts executing task1 and encounter an IO operation then it will take back the control from task 1 and give it to task2 to execute and when the task1 completes IO and when the control comes back to task1 it will resume from the state it was stopped thus two or more tasks can concurrently run together
Awaitable objects: We say that an object is an awaitable object if it can be used in an await or yield from expression.
There are three main types of awaitable objects in python: coroutines, Tasks, and Futures.
AsyncIO is a library which helps to run code concurrently using single thread or event loop, It is basically using async/await API for asynchronous programming.
AsyncIO was released in python 3.3 before that we use threads, greenlet and multiprocessing library to achieve asynchronous programming in python
As threads can be used for running several tasks concurrently but python threads are managed by Operating system and OS have to do more context switching b/w threads compared to green threads
Greenlet(green threads) takes away the scheduler from the OS and playing the scheduler itself but CPython doesn’t use green threads by default (this is what asyncio, gevent, PyPy, et al. are for)
By using Multiprocessing library we can create multiple processes and we can allow the program to make full use of all cores of the computer but processes are costly to spawn so for I/O operations, threads are chosen largely
In General, asyncIO is a more readable and cleaner approach towards asynchronous programming.
When asyncIO released it was using
@asyncio.coroutine decorator with
generator-based coroutines to achieve asynchronous programming
Asyncio generator coroutines use
yield from syntax to suspend coroutine.
In the below example
some_async_task() is a generator-based coroutine, to execute this coroutine, first, we need to get the event loop(at line 11) and then schedule this task to run in an event loop using(loop.run_until_complete)
Note: Directly calling
some_async_task()will not schedule this task to execute, it will only return a generator object
Here in line 8
asyncio.sleep()is a coroutine(we can use any coroutine or tasks/future here)and when the statement yield from executed it which will give up the control back to the event loop to let other coroutines execute and when coroutine
asyncio.sleep()completed and when the event loop gives back the control to
some_async_task()coroutine it will run further instruction(like line 9)
In Python 3.5 the language introduced native support for coroutines. Now we can use async/await syntax to define native coroutines
A method prefixed with
async defautomatically becomes a native coroutine.
awaitcan be used to obtain the result of awaitable objects(which can be coroutine, tasks or future)
Before python 3.7 we manually create/get event loop and then schedule our task like:
loop = asyncio.get_event_loop() #if there is no event loop then it will create new one. loop.run_until_complete(coroutine()) #run until coroutine is completed.
In python 3.7 and above, below is the preferred way to run the event loop
asyncio.run(coroutine()) # This function runs the passed coroutine, taking care of managing the asyncio event loop and finalizing asynchronous generators.
AsyncIO provides high-level API and low-level APIs, generally, application developer use high-level API and library or framework developer uses low-level API
It is a low-level awaitable object that is supposed to have a result in the future.
when a future object is awaited it means that the coroutine will wait until the Future is resolved in some other place.
This API exists to enable callback-based code to be used with async/await
Normally at the application level code, we don't deal with Future objects, it is usually exposed by asyncio API or libraries.
Task is a subclass of futures and it is used to run coroutines concurrently within an event loop.
There are many ways to create a task:
loop.create_task() → via low-level API and it only accepts coroutines.
asyncio.ensure_future() → via low-level API and it can accept any awaitable objects, this will work on all python version but it is less readable.
asyncio.create_task() → via high-level API and it is work in Python 3.7+ and it accepts coroutines and it will wrap them as tasks
When a coroutine is wrapped into a Task with functions like
asyncio.create_task() the coroutine is automatically scheduled to run soon
In Below example I am using
aiohttp library to fetch news articles from hacker-news public APIs, i created two task(task1 and task 2) to fetch two different news concurrently and showing the title for both news article.
It is similar to asyncio.create_task() but it can also accept the future as shown in the below example
It is responsible for gathering all the results, it will wait till all the awaitables objects are completed and return the results in order of given awaitable objects
If there is an exception in any of the awaitable object, it will not cancel the other awaitable objects
In the below example we are running two tasks concurrently and we can see that if there is an exception in
some_async_task2(), it won't cancel
False and if there is any exception raised in any of awaitable object then the await
asyncio.gather() returns immediately and show an error to the screen. so for demo purpose at line 17, we are awaiting another coroutine(which will resolve after 6 sec) to make sure that program doesn't exit as after 4 sec
And we can see the execution of line 6 in the output
If we want to gather all the result(along with exception) in an array then we can use
return_exceptions=True which will treat exception as a result and it will be aggregated in the result list.