Super dependency injection in Python

Originally posted on my blog.

This isn't a blog post about some great dependency injection framework I'm fond of. Rather, it's about using the super() method to change the behaviour of our classes in different contexts. If you're familiar with how super works in single inheritance languages, this might sound a bit mad - in my experience elsewhere, super calls out to the parent class, so if you have class B extending class A, and B calls super anywhere, then class A's method is invoked. However, this is not the case in Python - for classes constructed using multiple inheritance, Python's Method Resolution Order, better known as the MRO, changes dynamically to determine how super is invoked. In effect - if you use super in your class, then one of your children invokes super, the child's parents(of which your class is only one!) determines which method is invoked.

Confused? Let's step through an example and hopefully that'll clear things up.

So, simple enough situation, we're building some kind of service that makes HTTP calls to other services. We're going to use the Requests library instead of writing all our own session handling and so on, but we do want to put some boilerplate stuff like custom headers, logging and error handling in one place, so we'll subclass Session:

import requests


class CustomSession(requests.Session):
    def __init__(self):
        super().__init__()
        self.headers.update({'User-agent': 'CustomSession', 'Accept': 'application/json'})

    def get(self, url, **kwargs):
        print('Performing GET on {}'.format(url))
        resp = super().get(url=url, **kwargs)
        resp.raise_for_status()
        print('{} returned status: {}'.format(url, resp.status_code))
        return resp.json()

With that out of the way, we'll use our CustomSession as a base for our client:

class MyAppClient(CustomSession):
    URL = 'https://myapp.corpdomain/'

    def get_stuff(self):
        resp = super().get(url='{}stuff'.format(self.URL))
        return resp

Hopefully your actual API wouldn't look so poorly, but that's not the point of this post. Anyway, with that done, MyAppClient().get_stuff() makes a GET request to https://myapp.corpdomain/stuff, throws an exception if the status is not 2xx/3xx, and returns the JSON response. Neat. However, we really do not want our tests making live HTTP calls constantly, so this might be a bit of a problem. Let's subclass CustomSession and make a TestSession that overrides our get() and just returns an empty dictionary for now:

class TestSession(CustomSession):
    def get(self, url, **kwargs):
        print('TEST: app tried to GET {}'.format(url))
        return {}

Then let's do the complicated task of creating a test class that contains all the methods in MyAppClient but replaces the get() calls inherited from CustomSession with calls to TestSession:

class TestMyAppClient(MyAppClient, TestSession):
    pass

Now when we run TestMyAppClient().get_stuff(), we see:

TEST: app tried to GET https://myapp.corpdomain/stuff

OK, so what's happening here? Basically the MRO calls the next method, not the parent method in our classes. If we check the help() for our TestMyAppClient, we can see the resolution order:

 |  Method resolution order:
 |      TestMyAppClient
 |      MyAppClient
 |      TestSession
 |      CustomSession
 |      requests.sessions.Session
 |      requests.sessions.SessionRedirectMixin
 |      builtins.object
 |  
 |  Methods inherited from MyAppClient:
 |  
 |  get_stuff(self)

So, we have a super().get() call inherited from MyAppClient, but TestSession is next in our resolution order, so that get() is invoked instead of the CustomSession's, despite MyAppClient having zero knowledge of TestSession.

Of course, this isn't just useful in testing - you could extend your CustomSession to fetch and parse application/xml responses instead of JSON, or have a DebugSession that adds an absurd amount of logging in place of our little print statements - once you're comfortable with determining how super is going to resolve, it opens up all kinds of possibilities.