Interested in Python FastAPI? Wondering how to execute long-running tasks in the background in Python? You came to the right place!
The use case
I learned about FastAPI and Celery when confronted with a simple yet interesting use case:
I had a Jupyter Notebook that connected to a database, ran some heavy processing on the data (using machine learning and everything), and saved aggregated data back to the database.
Since notebooks are great for developing, the requirement was to keep using notebooks for development but to be able to trigger the processing from an API call. The notebook should never be executed twice in parallel though. In other words, the API should return an error if the notebook were already being executed. Note that the notebook would be provided once at deployment time: it won't change during the lifecycle of the app.
The implementation
I was initially planning to use a simple Flask app, but soon got into trouble. How can I (1) run the notebook in a background thread and (2) restrict its execution to one at a time?
This is how I discovered FastAPI and Celery. I implemented an MLP (Minimum Lovable Product) based on those technologies available on GitHub.
⮕ ✨✨ github.com/derlin/fastapi-notebook-runner ✨✨
The tutorial
The use case was perfect for learning. This is why I cooked a complete tutorial based on it, along with schemas and explanations. The tutorial repository can be used as a base to follow along. Not only will you learn about FastAPI and Celery, but also Poetry, ruff, and other nice tips and tricks.
⮕ ✨✨ derlin.github.io/introduction-to-fastapi-and-celery ✨✨
Jump to the main sections:
I used the above website as a base for a talk at the GDG Fribourg and figured some of you could also benefit from it. Don't forget to leave a ⭐!
Top comments (6)
Hey, it was a nice article. Especially thanks for the pointer to Ruff. I will try that.
Glad you liked it! Yes, ruff is really nice :) I don't know if you saw the hidden section in the tutorial (about Poetry): derlin.github.io/introduction-to-f.... This is where I share my current setup. Let me know what you think!
Btw, (just asking) can you kindly collab on an example app created in a Python web framework I made? It is fully featured(cookies, redirects, dynamic URL routing, etc) and also provides SwaggerUI docs. It can be customised from top to bottom.
It is: github.com/re-masashi/willpyre. You have the docs and an
example_apifolder where you can find a base of how to use (apart from the docs). If you want any features to be added, just tell me, I'll be adding them.I'm all ears for any sorta feedback.
Also, Willpyre is faster than FastAPI, I ran the benchmarks locally on PyPy and CPython, they are yet to be published.
Cheers if you read this far.
I saw and I love black, but poetry, not much.
⭐
I was thinking of a star on the github repo, but your comment made my day 😂, thank you!