I'm writing this post to report my first few hours of experimentation with asynchronous Python code. I have been coding in Python for the last few years but I had no previous experience with asynchronous code besides toy problems and have been flirting with the idea of learning it for a few months.
This weekend I was looking into a few project ideas and thought it would be a good time to give it a try. I remembered a few months or years ago I watched a talk in PyCon about how Trio was the best thing since sliced bread so appeared to be a good fit.
The initial documentation was very decent. It was approachable even for a beginner which usually isn't the case. I spent 1-2 hours reading the core concepts and following the getting started. The API is very well designed. There are context managers everywhere, exception handlers considering multiple edge cases, callback entry points, robust logging. Every signal that it would be an easy experience.
Then I went back to the project that has a very simple objective: given a few sites I wanted to crawl some of their contents. I plan to use this data to explore more into other software concepts. It didn't look hard to create a small parser for each site. The first site uses incremental IDs, so I generated a list of URLs to be GET.
Then comes the first problem, Requests, the most used HTTP lib, does not support async work by default. So I was pushed into more corner case libraries... no biggie, any libraries should handle this simple use case. There even seems to exist ports of Requests to support asynchronous calls.
Asks is one of the recommended HTTP clients recommended on Trio documentation. The library has very succinct documentation. I was able to replicate the code examples provided but trying to apply a rate limiter wasn't that trivial. I was able to find an issue on Github discussing it that had some code snippet that almost worked, fiddling with it a little I was able to get it running.
Altogether I had a working solution within a few hours. But it still bothers me that fundamental things like rate limiting are not present in the documentation, only to be found partially in GitHub issues and requiring a decent amount of experimentation. The examples were too simple to be useful and since this ecosystem still has very low adoption it is harder to find runnable examples from the community.
While frameworks like Scrapy are very robust and seem to have support for asynchronous code I felt that it was too much of a dependency for such small use case. I will be looking into other libraries on the ecosystem in the next few days to see if there are better solutions. Overall, this division between async/non-async libraries is the worst point of the experience. Being unable to run the most adopted tools makes it hard to migrate code to asynchronous. It leaves a taste in the mouth that probably Python is far behind other languages that have better co-routines support like Go.
Top comments (0)