DEV Community: Matt

Writing a chainable CLI For Data Processing

Matt — Mon, 28 Dec 2020 19:27:42 +0000

Although some complex datasets or data exploratory require going to Jupyter notebooks, on the other hand, some datasets require simple processing like simple filtering, creating a pivot table.

So you probably end up opening it in a spreadsheet. However if spreadsheets are accommodating, they are difficult to automate and do not offer as many features as pandas.

Find out in this tutorial how to write a command-line interface to wrap some pandas features and automate simple tasks.

Serverless Machine Learning Engineering Project On AWS

Matt — Sat, 26 Dec 2020 22:51:47 +0000

Lots of tutorials on how to deploy a model in production directly integrates the serialized model into the API. This way of proceeding has the disadvantage of making the API coupled to the model. Another way to do this is to delegate the prediction load to workers using a queue. The schema below shows the solution architecture on the AWS environment.

The machine learning model is stored in an s3 bucket. It is loaded by workers, which are Lambda functions when a message containing prediction data is put in the SQS queue by the client through the API gateway/Lambda REST endpoint.

When the worker has finished the prediction job, it puts the result in a DynamoDb table in order to be accessed. Finally, the client requests the prediction result through an API endpoint that will read the DynamoDb table to fetch the result.

As you can see we are delegating the loading and prediction work to a worker and we do not integrate the model into the REST API. This is because a model can take a long time to load and predict. Therefore we manage them asynchronously thanks to the addition of an SQS queue and a DynamoDb table.

Complete article here including code.

Asynchronous HTTP requests in Python

Matt — Tue, 25 Feb 2020 09:49:22 +0000

In python, you can make HTTP request to API using the requests module
or native urllib3 module.

However, requests and urllib3 are synchronous. It means that only one HTTP call can be made at a time in a single thread. Sometimes you have to make multiples HTTP call and synchronous code will perform baldy. To avoid this, you can use multi-threading or since python 3.4 asyncio module.

Test case

In order to show the difference of time between sync and async code, i made a script that read a file with 500 cities names and perform HTTP call to an API to retrieve information about location, population and so on from the city name.

Sync code performance

Here is the sync code version with the requests module

@timeit
def fetch_all(cities):
    responses = []
    with requests.session() as session:
        for city in cities:
            resp = session.get(f"https://geo.api.gouv.fr/communes?nom={city}&fields=nom,region&format=json&geometry=centr")
            responses.append(resp.json())
    return responses

Finished 'fetch_all' in 38.7053 secs

Async code performance

I used aiohttp module to make the async code as the requests module doesn't support asyncio for now.

async def fetch(session, url):
    """Execute an http call async
    Args:
        session: contexte for making the http call
        url: URL to call
    Return:
        responses: A dict like object containing http response
    """
    async with session.get(url) as response:
        resp = await response.json()
        return resp

async def fetch_all(cities):
    """ Gather many HTTP call made async
    Args:
        cities: a list of string 
    Return:
        responses: A list of dict like object containing http response
    """
    async with aiohttp.ClientSession() as session:
        tasks = []
        for city in cities:
            tasks.append(
                fetch(
                    session,
                    f"https://geo.api.gouv.fr/communes?nom={city}&fields=nom,region&format=json&geometry=centr",
                )
            )
        responses = await asyncio.gather(*tasks, return_exceptions=True)
        return responses
@timeit
def run(cities):
    responses = asyncio.run(fetch_all(cities))
    return responses

Finished 'run' in 3.0706 secs

Conclusion

As you can see, the async version is a lot faster than the sync version so if you run into a situation where your code is performing multiple I/O calls then you should consider concurrency to improve performance. However asynchronous version requires more work as you can see.

If you want to see the threading version that works with the requests module and also see how to implement automatic retry and caching on your API call check out this tutorial.