Turn sync function to async - Python Tips

TL;DR

We can write a function wrap a sync function an async function:

import asyncio
from functools import wraps, partial

def async_wrap(func):
    @wraps(func)
    async def run(*args, loop=None, executor=None, **kwargs):
        if loop is None:
            loop = asyncio.get_event_loop()
        pfunc = partial(func, *args, **kwargs)
        return await loop.run_in_executor(executor, pfunc)
    return run 

To use it:

import time
import os

async_sleep = async_wrap(time.sleep)
async_remove = async_wrap(os.remove)

# or use decorator style
@async_wrap
def my_async_sleep(duration):
    time.sleep(duration)

Longer Description

If we use sync function time.sleep in a function and run it 3 times:

import time

def count():
    print("func start")
    time.sleep(1)
    print("func end")


def main():
    funcs = [count, count, count]
    for func in funcs:
        func()


if __name__ == "__main__":
    start = time.time()
    main()
    end = time.time()
    print(f"Time elapse: {end-start}")

Run it:

func start
func end
func start
func end
func start
func end
Time elapse: 3.00303053855896

The total time cost is around 3 seconds.

Now let's use the async version of time.sleep (We can use asyncio.sleep from Python's asyncio standard library, but here to demonstrate our async_wrap function works, we are going to use our own async_sleep function).

import asyncio
from functools import wraps, partial
import time


def async_wrap(func):
    @wraps(func)
    async def run(*args, loop=None, executor=None, **kwargs):
        if loop is None:
            loop = asyncio.get_event_loop()
        pfunc = partial(func, *args, **kwargs)
        return await loop.run_in_executor(executor, pfunc)
    return run 


async_sleep = async_wrap(time.sleep)


async def count():
    print("func start")
    await async_sleep(1)
    print("func end")


async def main():
    await asyncio.gather(count(), count(), count())


if __name__ == "__main__":
    start = time.time()
    asyncio.run(main())
    end = time.time()
    print(f"Time elapse: {end-start}")

Here we use asyncio.gather to run the count function three times. Now run it:

func start
func start
func start
func end
func end
func end
Time elapse: 1.007828950881958

We can see our async version only cost around 1 second! And our async_sleep function works!

Reference:

• https://stackoverflow.com/questions/43241221/how-can-i-wrap-a-synchronous-function-in-an-async-coroutine
• https://github.com/Tinche/aiofiles/blob/master/aiofiles/os.py
• https://github.com/yifeikong/aioify/blob/master/aioify/__init__.py
• https://realpython.com/async-io-python/

Comments

[Joao S. O. Bueno]

Pay attention that this code,as is, will only enable multi-threading of the sync functions.
To be able to run the function in a sub-process by using an explicit ProcessPoolExecutor, the original function, prior to been decorated, have to be preserved - and the decorated async-function needs to exist with a different name.

I just made it work for someone who has hit this: stackoverflow.com/questions/743598...