Simply put, dependency injection is a collection of programming techniques that enables software components to have their dependencies replaced, thereby increasing re-usability, for example by allowing a database module to depend on a connection string in order that it can connect to multiple databases. Dependency injection also improves testability because complicated dependencies such as database connections can be easily replaced with mocks.
In this post we'll study dependency injection in Python. We'll see how it can be made completely type safe using functional programming, specifically with a modern functional programming library called pfun.
Dependency injection in Python is typically implemented using monkey patching. In fact, monkey patching as a form of providing dependencies is so common that the
unittest.mock.patch function was added to the standard library in Python 3.3.
While monkey patching is a simple technique, it often leads to rather complex patching scenarios, where it's tricky to figure out what and how to patch. Moreover, it can be tricky to achieve complete type safety with monkey patching on both the dependency consumer and provider side. Finally, there are no straight-forward ways of making sure that the dependency provider provides all required dependencies, often leading to statements such as
if dependency is not None.
An alternative (and potentially even simpler) method for dependency injection is the following: function arguments. In fact, this approach is so simple that using the term "dependency injection" to describe it seems almost pretentious. An attractive feature of implementing dependency injection with functions that take arguments is that it is completely type safe by default (provided that you use type annotations of course). Moreover, you can tell a function's dependencies simply by reading its signature, making it totally clear what needs to be provided, and in fact making it impossible to not provide all required dependencies.
The main drawback from using function arguments as your "injection" mechanism, is that functions that call functions that have "injected dependencies" now need to take those dependencies as arguments themselves.
For example, consider a function
connect that returns a connection to a database given a
connection_str as argument:
def connect(connection_str: str) -> Connection: ...
To achieve the promise of "dependency injection", every function that calls
connect must now take a
connection_str parameter in addition to its other parameters, in order that the calling function itself can be used against different databases:
def get_user(connection_str: str, user_id: int) -> User: connection = connect(connection_str) return connection.get_user(user_id)
For functions with many dependencies, this quickly becomes very tedious.
Thankfully, we can use functional programming to improve the situation by a margin: Notice how the only use
get_user has of
connection_str is to pass it to
connect. With functional programming we can abstract this pattern of functions taking arguments only to pass them to other functions into some general 'plumbing' functions. This will alow us to write
get_user without mentioning the
connection_str at all!
To keep things nice and readable (and save us some typing), lets define a type-alias that represents functions that require dependencies:
from typing import TypeVar, Callable R = TypeVar('R') A = TypeVar('A') Depends = Callable[[R], A]
Depends[R, A] is a function that depends on something of type
R to produce something of type
A. For example, our previous
connect function is a
Depends[str, Connection] value: it's a function that depends on a
str to produce a
Simple enough, but how do we use the
Depends represented by
get_user without explicitly taking
connection_str as a parameter and calling
connect? To do that, let's introduce our first plumbing function
B = TypeVar('B') def map_(d: Depends[R, A], f: Callable[[A], B]) -> Depends[R, B]: def depends(r: R) -> B: return f(d(r)) return depends
map_ precisely implements the "passing of parameters" plumbing we were talking about earlier: it creates a new
Depends value that calls
d, and passes the result to a function
f. This allows us to write
get_user as follows:
def get_user(user_id: int) -> Depends[str, User]: return map_(connect, lambda con: con.get_user(user_id))
And voila: we have type-safe dependency injection using only
Depends (which is just a type-alias for functions of 1 argument) and
map_ (which is just a function that composes
Depends values with functions). Realise that we could keep applying
Depends values to return a final
Depends all the way back to where we make desicions about what concrete connection strings to pass in (probably in the main section of our program). The pattern we've discovered here seems simple enough that it would be surprising if we were the firsts to think of it, and sure enough it's in fact a common pattern in functional programming called the reader effect.
That's all well and good, but what happens when we want to
map_ functions that return new
Depends values? For example, let's imagine that in addition to reading user data from databases, our application also needs to call an HTTP api with the user data. Doing so requires authentication credentials that we want to inject. The function that calls the api might look like this:
from typing import Tuple Auth = Tuple[str, str] def call_api(user: User) -> Depends[Auth, bytes]: def depends(auth: Auth) -> bytes: ... return depends
We might try to use
map_ to pass the result of the
Depends value returned by
call_api. But in doing so we would end up with a
Depends[str, Depends[Auth, bytes]]:
if __name__ == '__main__': user_1: Depends[str, User] = get_user(user_id=1) call_api_w_user_1: Depends[str, Depends[Auth, bytes]] = map_(user_1, call_api)
When we want to call
call_api_w_user_1, we need to first supply the connection string, and then the auth information:
result: bytes = call_api_w_user_1('user@prod')(('prod_user', 'pa$$word'))
This might not seem like big issue in this example, but notice that we might apply
map_ over several functions that produces
Depends values with the same dependency type, resulting in a situation where we have to pass in the same dependency many times. We might also have a varying number of dependencies if we use
map_ in a loop, leading to a situation where we don't know how many times we need to call the final
To fix this situation, let's introduce another plumbing function that can pass dependencies to both a
Depends value, and a
Depends value returned by a function that's being mapped. We'll call it
and_then since it chains together results of
Depends values with functions that return new
from typing import Any R1 = TypeVar('R1') def and_then(d: Depends[R, A], f: Callable[[A], Depends[R1, B]]) -> Depends[Any, B]: def depends(r: Any) -> B: return map_(d, f)(r) return depends
The observant reader will notice that there's a big problem with the typing of our
and_then function: it returns a
Depends[Any, B]! Here you might object: 'But wait a minute, I though you said this was "completely type safe"'! The reason for this epic fail is that the
r parameter passed to the
depends function must be passed to both
d, which means it must be of type
R, and the
Depends returned by
f, which means it must be of type
R1. In other words, the dependency must be of both type
R1 at the same time. In our example, using
and_then to combine
call_api should result in a type that is both a
str and a
call_api_w_user_1: Depends[?, bytes] = and_then(get_user(user_id=1), call_api)
Such a type is called an intersection type and unfortunately it's not supported by the Python
typing module (yet). So without introducing third party libraries, this is the best we can do.
This is where the library pfun comes into the picture. In
and_then are instance methods of a
Depends type, but the idea is exactly the same:
from pfun import Depends def connect() -> Depends[str, Connection]: ... def get_user(user_id: int) -> Depends[str, User]: connect().map(lambda con: con.get_user(user_id)) def call_api(user: User) -> Depends[Auth, bytes]: ... call_api_w_user_1 = get_user(user_id=1).and_then(call_api)
pfun provides mechanisms for combining
Depends values with different dependency types when using MyPy with one small requirement: the dependency type must be a
typing.Protocol. The reason is that intersections of protocol types are simply a new protocol that inherits from both:
from typing import Protocol class P1(Protocol): pass class P2(Protocol): pass class Intersection(Protocol, P1, P2): pass
This means that when using
Depends values that have protocol types as dependencies,
pfun can infer an intersection type automatically. Let's rewrite our example to take advantage of this feature:
from typing import Protocol, Tuple class HasConnectionStr(Protocol): connection_str: str class HasAuth(Protocol): auth: Tuple[str, str] def connect() -> Depends[HasConnectionStr, Connection]: ... def get_user(user_id: int) -> Depends[HasConnectionStr, Connection]: connect().map(lambda con: con.get_user(1)) def call_api(user: User) -> Depends[HasAuth, bytes]: ... call_api_w_user_1 = get_user(user_id=1).and_then(call_api)
The type of
call_api_w_user_1 in this example will be
Depends[pfun.Intersection[HasConnectionStr, HasAuth], bytes]. The dependency type
pfun.Intersection ensures that
call_api_w_user_1 must be
called with an argument that fulfills both the
HasConnectionStr protocol and the
HasAuth protocol. In other words, this would be a type error:
class Dependency: def __init__(self): connection_str = 'user@prod' call_api_w_user_1(Dependency()) # MyPy Error!
This compositional nature of
and_then means that complex dependency types are built from simple dependency types automatically, which means you can always figure out which dependencies a part of your application has, simply by inspecting it's return type. To learn more about functional programming with
pfun check out some of the other posts in the series, the documentation or the github repo.
Top comments (1)
Thanks Sune, this is a nicely written article which demonstrates how to compose functions in Python while taking care of the environment/dependencies in a type-safe manner.
The astute reader will notice that this ressembles mecanisms found in ZIO in Scala or effect-ts in TypeScript.