Dependency Injection with Import Hooks in Python 3

sys.meta_path is one of the best-kept secrets in the Python standard library, despite being squarely located in a module every Python programmer is familiar with. Some might say the secrecy is well-warranted, though; it's not usually a good idea to go messing around with tools and functions that may subvert programmer expectations the way this oddly-named list potentially can. But isn't "do not" better than "know not?" So before a quick primer on dependency injection, let's examine some of the ins and outs of sys.meta_path.

Whenever you use an import statement, say, import os, that's functionally equivalent to the following code:

os = __import__("os")

It's a mild oversimplification, since stuff like submodules would require a little bit more work (as we'll have to deal with later, actually), but the main point to be made here is that the __import__ built-in does the heavy lifting under the hood for module imports in Python, and now we know that it exists, we can examine its control flow. First, it checks sys.modules, a dict containing already-loaded modules, just in case we've already imported the requested module before. Next, it checks sys.path - a list containing filesystem paths for Python to search through for potential modules; you probably know about this list already. However, if the sys.meta_path list is not empty, than before looking through sys.path, every meta importer hook in sys.meta_path will be queried before the filesystem is checked, giving us an easy way to intercept module imports and do all sorts of freakishly wonderful things, like dependency injection.

So what is dependency injection? It's roughly an applied version of the dependency inversion principle, often for module-sized stuff. Say your application has several different front-ends, maybe targeting different platforms or UI toolkits. As long as you have a common interface, dependency injection would allow you to do something like:

user_info = info_getter.get_user_info()
frontend = dependency_injector.get("frontend")
frontend.display_user_info(user_info)

"So what? I can do that already with a Facade or something, maybe a config switch," I can hear you say. For very simple use cases, that may actually be better - simple is better than complex, remember? However, with dependency injection, we can easily provide module-level reuse, as well as significantly cut down on boilerplate. What if I told you, by the end of this article, you could write something like this:

import myapp.dependency.frontend as frontend
popup = frontend.Popup(popup_style=frontend.ERROR)
popup.display()

One import that works regardless of whether we're using GTK+, Qt, ncurses, or whatever your program needs to function. You might be thinking some very dark thoughts at this point. It's true, all sorts of dark magic could be performed depending on the details of our import hooks. But readability counts, so it's important that we not stray too far off the beaten path. So let's set some requirements for our dependency injection framework before writing our implementation:

• All dependency-injected import paths begin with myapp.virtual
• Provided dependencies are known beforehand and registered with a provide method

All the magic - and there's not all that much - relies almost entirely on creating a meta path finder and adding it to sys.meta_path. A meta path finder is a very simple object with a single public method, find_spec:

import importlib.abc
import importlib.machinery

class DependencyInjectorFinder(importlib.abc.MetaPathFinder):
    def __init__(self, loader):
        # we'll write the loader in a minute, hang tight
        self._loader = loader
    def find_spec(self, fullname, path, target=None):
        """Attempt to locate the requested module
        fullname is the fully-qualified name of the module,
        path is set to __path__ for sub-modules/packages, or None otherwise.
        target can be a module object, but is unused in this example.
        """
        if self._loader.provides(fullname):
            return self._gen_spec(fullname)
    def _gen_spec(self, fullname):
        spec = importlib.machinery.ModuleSpec(fullname, self._loader)
        return spec
# we'll also add it to sys.meta_path later

If a meta path finder provides the requested module, then it should return an instance of the importlib.machinery.ModuleSpec class, which is a fairly simple affair with a small handful of attributes that lets Python's import machinery know what it needs to know to take the next steps in importing the module the user requested. For our purposes, we're interested in two attributes (the only required ones): ModuleSpec.name, which is the name of the requested module, and ModuleSpec.loader, which is the loader object that Python should use to actually load the module - you'll notice the self._loader lines above that reference a loader object, as well. A loader object is a very simple class with two required methods in modern Python (3.4 onwards): create_module, which takes a ModuleSpec as its sole argument and returns an object that Python will consider to be the new module, and exec_module, which takes the new module as its sole argument and executes it. So a no-op, barebones loader looks something like this:

class Loader(importlib.abc.Loader):
    def create_module(self, spec):
        raise NotImplementedError
    def exec_module(self, module):
        raise NotImplementedError

In addition to those two methods, we should implement the provide method we talked about earlier to signal that a certain dependency is provided, in addition to the provides method that our finder referenced earlier, that indicates whether the requested module is part of our dependency injection framework. Here's the implementation:

import importlib.abc
import sys
import types


class DependencyInjectorLoader(importlib.abc.Loader):
    _COMMON_PREFIX = "myapp.virtual."
    def __init__(self):
        self._services = {}
        # create a dummy module to return when Python attempts to import
        # myapp and myapp.virtual, the :-1 removes the last "." for
        # aesthetic reasons :) 
        self._dummy_module = types.ModuleType(self._COMMON_PREFIX[:-1])
        # set __path__ so Python believes our dummy module is a package
        # this is important, since otherwise Python will believe our
        # dummy module can have no submodules
        self._dummy_module.__path__ = []
    def provide(self, service_name, module):
        """Register a service as provided via the given module
        A service is any Python object in this context - an imported module,
        a class, etc."""
        self._services[service_name] = module
    def provides(self, fullname):
        if self._truncate_name(fullname) in self._services:
            return True
        else:
            # this checks if we should return the dummy module,
            # since this evaluates to True when importing myapp and
            # myapp.virtual
            return self._COMMON_PREFIX.startswith(fullname)
    def create_module(self, spec):
        """Create the given module from the supplied module spec
        Under the hood, this module returns a service or a dummy module,
        depending on whether Python is still importing one of the names listed
        in _COMMON_PREFIX.
        """
        service_name = self._truncate_name(spec.name)
        if service_name not in self._services:
            # return our dummy module since at this point we're loading
            # *something* along the lines of "myapp.virtual" that's not
            # a service
            return self._dummy_module
        module = self._services[service_name]
        return module
    def exec_module(self, module):
        """Execute the given module in its own namespace
        This method is required to be present by importlib.abc.Loader,
        but since we know our module object is already fully-formed,
        this method merely no-ops.
        """
        pass
    def _truncate_name(self, fullname):
        """Strip off _COMMON_PREFIX from the given module name
        Convenience method when checking if a service is provided.
        """
        return fullname[len(self._COMMON_PREFIX):]

There's seemingly a lot of code here at first glance (even though most of it is comments!), so let's walk through this class' methods. Up first are the provide and provides methods that we've talked about earlier; there's not a whole lot of magic present in either one. create_module returns a so-called dummy module if we're trying to import either myapp or myapp.virtual, and there's a good reason for that. Say we have the following line of code:

import myapp.virtual.frontend

Under the hood, this generates three distinct searches in Python's import machinery: One for myapp, another for myapp.virtual, and lastly one for myapp.virtual.frontend. Since obviously myapp and myapp.virtual don't actually exist anywhere on the system, but Python will complain if they aren't loaded, we claim we provide both - note how provides will return True if queried for both myapp and myapp.virtual and return an empty dummy module to placate Python's import machinery. Only when we encounter myapp.virtual.frontend do we check against the list of provided dependency-injection services.

At this point, we've actually written all the plumbing we need to, so let's wrap up and see how we'd use these classes in practice:

import sys


class DependencyInjector:
    """
    Convenience wrapper for DependencyInjectorLoader and DependencyInjectorFinder.
    """
    def __init__(self):
        self._loader = DependencyInjectorLoader()
        self._finder = DependencyInjectorFinder(self._loader)
    def install(self):
        sys.meta_path.append(self._finder)
    def provide(self, service_name, module):
        self._loader.provide(service_name, module)


class FrontendModule:
    class Popup:
        def __init__(self, message):
            self._message = message
        def display(self):
            print("Popup:", self._message)


if __name__ == "__main__":
    injector = DependencyInjector()
    # we could install() and then provide() if we wanted, order isn't
    # important for the below two calls
    injector.provide("frontend", FrontendModule())
    injector.install()
    # note that these last three lines could exist in any other source file,
    # as long as injector.install() was called somewhere first
    import myapp.virtual.frontend as frontend
    popup = frontend.Popup("Hello World!")
    popup.display()

At this point, it'd be worth it to take a step back and examine at a high level what is going on behind the scenes:

1. The call to injector.install appends our DependencyInjectorFinder onto the sys.meta_path list, so Python will use it for future imports.
2. The line import myapp.virtual.frontend as frontend triggers three module searches, as mentioned earlier - one for myapp, then myapp.virtual, then myapp.virtual.frontend:
   • Python combs through sys.meta_path, looking for meta path finders. If your system Python is anything like mine, our DependencyInjectorFinder will be the only one not defined by _frozen_importlib, which is a part of Python itself.
   • For each found meta path finder, Python queries the finder's find_spec method, seeing if the finder provides the given module.
   • Obviously myapp.virtual.frontend doesn't exist on the filesystem, so it falls to our meta path finder to handle it. In all three cases, we return a ModuleSpec instance with name set to the same name that Python asked if we could find, and loader set to our custom DependencyInjectorLoader.
   • Next, Python will call the create_module method on our loader for the ModuleSpec in question. For myapp and myapp.virtual, our loader recognizes that these are dummy modules and returns the same dummy module for both (tricking Python into believing the module was loaded), but returns the instance of FrontendModule we gave it with injector.provide() upon being asked to load myapp.virtual.frontend. Python allows any valid object to function as a module, so a plain old class is perfectly fine to return.
   • Python will finally call the exec_module method on our loader, passing it the object we returned from create_module. Python requires the exec_module method to be present but doesn't really care about its behavior; normally, the method would execute the module's code in a new namespace, but since we already have fully-formed modules ready to go - either our pre-made dummy module or our FrontendModule instance - we simply do nothing inside exec_module.
   • Wash, rinse, and repeat for each successive module search.
3. After all that, frontend.Popup is functionally the same thing as FrontendModule.Popup, and the rest, as they say, is history.

As I mentioned earlier in the article, it's important not to abuse language features like this - with great power comes great responsibility, after all - and this especially rings true when it comes to metaprogramming. Still, there's a time and place for everything, import hooks included; but please use them responsibly.

Finally, the complete code example is up as a gist on GitHub if you'd like something you can download and pore over. Happy hooking!

Comments

[rhymes]

Hi Kermit, super interesting. I didn't know about sys.meta_path - seems also very handy to develop plugin systems like you did! Great for a big customizable app

A small suggestion: if you re-edit your post you can add syntax highlighting with tags like this:

>>> import platform
>>> platform.python_version()
'3.7.1'

You just need to add python at the end of the three beginning ticks, like this:

[Kermit Alexander II]

Ah, appreciate the tip on syntax highlighting! I've updated the article accordingly.

[Florimond Manca]

Interesting post! Thanks.

Have you tried seeing how static type checking behaves with this meta_path trick? Or would IDEs complain that the myapp.virtual.foo module does not exist?