Working with Flask or the fabulous Quart in Python, you sometime want to "split" your routes (or views) in several packages. That's something very common in Go, but a bit more complex with Python. I've got something to propose.
The reason is how Python defined a package. With Go, a package is a set of files that are all injected once. But with Python, it's a set of module that are independent.
So, the time has come when you begin to use Blueprints
. A very nice way to write sub-routes that you can classify, prefix and activate.
That's ok, let's see the problem.
Note, I use Quart but you can change the entire examples to use Flask (remove
async
and change imports)
Classification of routes
Let's imagine we want to create this application structure:
app.py
api/
__init__.py
auth.py
content.py
The app.py
will load the api
package to register it as a Blueprint
. That's common, and we will not touch this file anymore. So, write the app.py
file like this:
""" The main application """
from quart import Quart
from api import api
app = Quart(__name__)
app.register_blueprint(api, url_prefix="/api")
if __name__ == "__main__":
app.run(debug=True)
Manage the routes in modules
Now, let's take a look at the routes
package. And see the problem happening.
First, declare a Blueprint in api/__init__.py
# api/__init__.py
from quart import Blueprint
api = Blueprint('api', __name__)
The api
variable is global and can be imported inside any modules.
That's not a bad idea.
# in api/auth.py
from . import api
@api.route("/login")
async def login():
return "Welcome"
@api.route("/logout")
async def logout():
return "Bye !"
And...
# in api/content.py
from . import api
@api.route("/")
async def index():
return "The content !"
Back to __init__.py
, how to import the modules?
There are many difficulties...
- You cannot import
auth
andcontent
modules before theapi
declaration, because the module is not yet initialized (keep in mind thatapi
should be initialized to use@api.route
decorator... - if you put imports after the
api
declaration, the linter will complain about the placement of imports - and moreover, if you import modules without any use of them, so the linter will complain (and it is right, it's bad, ugly)
So, let's make a "programmatic import"! A good idea, isn't it?
# in api/__init__.py
from quart import Blueprint
api = Blueprint('api', __name__)
for module in ["auth", "content"]:
__import__(f"{__name__}.{module}")
This is not that bad actually, everything is working as expected.
Yes, but... What if we forget a module or if we make a mistake in the name? And what if I move, rename or delete a module ?
Make it a few more automatic
We need to detect the modules inside the package. And, believe it or not, that's not so easy. We could, for example, use os.walk
or glob.glob
to find files. But there is a complication: a module can be "compiled" as a library (.so
file), and there are many other extensions to detect. By chance, we have a solution.
There is a __loader__
inside each package. A lot of (even good) Python developers ignore this, and that's a pity. Because this variable contains a very useful SourceFileLoader
that provides a FileReader
. This is a generator that gives a solution to retrieve the modules inside the package.
Ho !
So! It's time to make a nice and tricky loop!
# app/__init__.py
from importlib import import_module
from pathlib import Path
from quart import Blueprint
api = Blueprint('api', __name__)
for mod in __loader__.get_resource_reader().contents():
mod = Path(mod).stem
if mod == "__init__":
continue
import_module(__package__ + "." + mod, __package__)
But... one more time, a problem may happen.
What if another file than a Python source file (or precompiled file) exists in the module ? For example, a
README.md
file.
There are plenty of ways to check this, for example having a list of possible extensions.
But, are you sure to know the entire list of Python resource extensions?
I mean, you think about.py
and.pyc
- but do you remember that the.pyo
extention exists? And are they other forgotten names?
So, what I love to do is to guess the mimetype.
# app/__init__.py
import mimetypes
from importlib import import_module
from pathlib import Path
from quart import Blueprint
api = Blueprint('api', __name__)
for mod in __loader__.get_resource_reader().contents():
if "python" not in str(mimetypes.guess_type(mod)[0]):
continue
mod = Path(mod).stem
if mod == "__init__":
continue
import_module(__package__ + "." + mod, __package__)
And there we are!
Time to make it a "tool"
OK, copying this loop inside each package __init__
file is a waste of time, and it is not comfortable if we need to fix the behavior in case of bug.
Anyway, when you need to copy a portion of code, so it should be a function.
Note, the __loader__
is a SourceFileLoader
which is a bit tricky to find in modules, and not easy at all to use. So, I prefer to make a "representation" of the class I need.
It's time to create a "tool". Create a "utils" package at the root of the project, and make the __init__.py
file:
# utils/__init__.py
import mimetypes
from importlib import import_module
from pathlib import Path
from typing import Callable
class SourceFileLoader:
""" Represents a SouceFileLoader (__loader__)"""
name: str
get_resource_reader: Callable
def load_modules(loader: SourceFileLoader):
"""Load the entire modules from a SourceFileLoader (__loader__)"""
pkg = loader.name
for l in loader.get_resource_reader().contents():
if "python" not in str(mimetypes.guess_type(l)[0]):
continue
mod = Path(l).stem
if mod == "__init__":
continue
import_module(pkg + "." + mod, pkg)
In the api
and others packages, I can now do:
# api/__init__.py
from quart import Blueprint
from utils import load_modules
api = Blueprint("api", __name__)
load_modules(__loader__)
And now, that's fantastic! Our all modules are now injected from the package. I don't forget any file, everything is automated by a simple call to load_modules()
.
I know...
Yes, Python is an awesome language, easy to use and Flask or Quart are wonderful. But, as you can see, sometimes you need to make some tricky things to make your project more concise.
Here, to make this loader tool, we needed to know some deep knowledge of the language. But that's also something I really love when I use a technology: not to be happy of the basics. I like to use pdb
to observe and discover what's behind the scene. Using dir()
on a object, checking types...
So, this loader is a little example of what we can do to automate development and to make less code in our project.
I hope you enjoyed my article and that may help you 😄
Top comments (0)