functools.partial

Intro

In this post, we will look at the partial function in the functools module.

The functools.partial function accepts a callable, with arguments (positional and keyword), and returns a partial object that has not yet executed the callable. When this partial object is called, it then passes the arguments to the callable and returns the output. The idea is to have an object that can store even some of the arguments required by the callabel, such that it can be executed in the future. Think of it like taking the callable, with any arguments that you can pass, and then freezing it for a later point.

Use cases

The first use case as mentioned above is when you do not have all the arguments required by a callable, and for some reason, you need to have an object ready with whatever partial arguments you have. One reason could be to pass this partial object to another function, which holds other arguments.

Another use case (which I encounter more) is when a callable with fixed values for one or more arguments is to be used by a set of ojbects, and with other values for another set of objects. Rather than creating a function or a lambda, I prefer partial and it feels more pythonic to me (see this example below).

In both the cases, it simplifies the signature of the callable, because now the code only needs to call the partial object with the remaining arguments.

Let's see some code for each of these use cases.

Usage

Consider a function that accepts multiple arguments, and we wish to create a partial object and pass around, populating arguments at different points in the code:

from functools import partial


def foo(x, y, *z, k=0):
    print(x, y, z, k)


##############################
# using `foo` from elsewhere #
##############################

# regular call, without `partial`
foo(1, 2, 3, 4, k=5)
# [Out]: 1 2 (3, 4) 5


# create a partial object with just the first argument
p = partial(foo, 1)

# add more arguments [1]
p = partial(p, 2)

# try executing the partial object [2]
p()
# [Out]: 1 2 () 0

# add more arguments
p = partial(p, 3, 4)    # or, partial(p, *(3, 4))
p()
# [Out]: 1 2 (3, 4) 0

# you can pass kwargs at any point, though
p = partial(p, k=5)
p()
# [Out]: 1 2 (3, 4) 5

A couple of points to note about the code:

• [1] Probably obvious, but in order to pass more arguments to the original callable (foo in this case), you need to pass it to the partial object.
• [2] The partial object can be called multiple times, it's not like a generator object that will lose the value once called.

Real-life scenarios

Here are some real-life examples where I was able to find partial helpful.

Reusing config object in different 'modes'

I once had to design a module for storing configuration values that were to be read from either the OS environment, or an env file, or from constants defined elsewhere, or default to some value if none of the above had it (more on this in another post). For this purpose I used the python-decouple library.

python-decouple in a nutshell

Using this library, you can define a config variable in your file:

##################
# test_config.py #
##################
from decouple import config

PG_PORT = config("PG_PORT", default=5432, cast=int)
print(f"PG_PORT is {PG_PORT}")

Then you can call this file in these three ways:

$ python test_config.py
PG_PORT is 5432

$ PG_PORT=5433 python test_config.py
PG_PORT is 5433

$ echo "PG_PORT=5434" >> .env
$ python test_config.py
PG_PORT is 5434

The use cases here was that some config variables were to be mandatory (meaning, the system should throw an exception if they didn't exist), and some optional. In python-decouple, setting default=None makes it optional.

Now, I didn't want to write default=None for all the variables that were to be optional, and this is where partial came in handy. I defined a partial object that already had default=None passed to it, and then just used that object to define optional config variables. Similarly, I also defined another partial object for config variables that should default to False:

from functools import partial

from decouple import config


opt_config = partial(config, default=None)
falsy_config = partial(config, default=False)

# mandatory vars
PG_HOST = config("PG_HOST")
PG_PORT = config("PG_PORT")
PG_PASSWORD = config("PG_PASSWORD")
...

# optional vars
SENTRY_DSN = opt_config("SENTRY_DSN")
HTTPS_PROXY = opt_config("HTTPS_PROXY")
...

# default False vars
DEBUG = falsy_config("DEBUG")
SEND_EMAIL = falsy_config("SEND_EMAIL")
...

Setting up FastAPI router

This one was when I had a FastAPI app with only some routes having the previx /v1. Routes like health-check, status, app-info need not be under version prefix, but routes with CRUD operations on resources had to be (see the FastAPI documentation on this topic).
Here I created a partial object from app.include_router that included the prefix. Then I could use this object to define routes that would have the prefix:

from functools import partial

from fastapi import Depends, FastAPI

from app.api import deps
from app.api.api_v1.routers import inventory, orders, probes, users
from app.api.probing.routers import probes

APP_VERSION = "v1"
ROUTE_PREFIX = f"/api/{APP_VERSION}"

app = FastAPI(
    title="My Web App",
    description="This is nice app.",
    version=APP_VERSION,
)

# probing-related routes without any prefix
app.include_router(probes.router, tags=["probes"])

# routes with prefix
route_include = partial(
    app.include_router,
    prefix=ROUTE_PREFIX,
    dependencies=[Depends(deps.authorize)],
)
route_include(users.router, tags=["users"])
route_include(orders.router, tags=["orders"])
route_include(inventory.router, tags=["inventory"])

Uploading to S3

There was a situation where I had to upload different files to different S3 buckets. I could use partial to have different objects for each bucket:

# function to upload files to S3
def upload_to_s3(client, bucket_name, path, file):
    ...

# from elsewhere

_upload_to_b = partial(upload_to_s3, env.client)
_upload_to_b1 = partial(_upload_to_b, env.bucket_1, env.path_1)
_upload_to_b2 = partial(_upload_to_b, env.bucket_2, env.path_2)
_upload_to_b3 = partial(_upload_to_b, env.bucket_3, env.path_3)

# few lines later...

# call relevant partial object for files
_upload_to_b1(file1)
_upload_to_b2(file2)
_upload_to_b3(file3)

Caveats

Different than functions

partial objects are slightly different than function objects, in that, you cannot access to the callable function's __name__ and __doc__ attributes. In order to access them you need to do it via the .func attribute of the partial:

def foo(x, y):
    """My fav func."""
    return x, y

p = partial(foo, 1)

p.func.__name__
# [Out]: 'foo'

p.func.__doc__
# [Out]: 'My fav func.'

See this for more information.

Incomplete or repetative arguments

If you create a partial without all the mandatory arguments, it will not throw error at that point, but when you call it. This is obvious because at the time of creation, the partial is not executing the callable, just freezing it in memory:

def foo(x, y):
    return x, y

# this will not throw error
p = partial(foo, 1)

# this will throw error
p()

If you create a partial with all the mandatory arguments, then you cannot pass any more arguments to it - else, it will throw error - again, not while defining the partial, but while executing it:

def foo(x, y):
    return x, y

p = partial(foo, 1, 2)

p()
# [Out]: (1, 2)

# this will not throw error yet
p2 = partial(p, 3)

# this will throw error now
p2()

# this will also throw error
p(3)

Behaviour with variable (*args) and keyword arguments (**kwargs)

If you are trying to create a partial out of a function that accepts *args and **kwargs, then there are few points that you need to take care of.

Consider the below code:

def foo(x, y, *args, **kwargs):
    return x, y, args, kwargs

p = partial(foo, 1, 2, 3, 4, p=5)

p()
# [Out]: (1, 2, (3, 4), {'p': 5})

p(6)
# [Out]: (1, 2, (3, 4, 6), {'p': 5})

p2 = partial(p, 6)
p2()
# same as above
# [Out]: (1, 2, (3, 4, 6), {'p': 5})

p(q=7)
# [Out]: (1, 2, (3, 4), {'p': 5, 'q': 7})

p(p=7)
# [Out]: (1, 2, (3, 4, 6), {'p': 7})

Notice how the behaviour is, with *args and **kwargs. Ignore the positional arguments x and y, they are here just for completion.

• If you pass an argument as positional to the partial, it gets appended to the *args. This can be seen when we execute p(6).
• If you pass a keyword argument, it gets updated in the kwargs dict. This means, if you pass value to an existing keyword, it will replace the previous one, but if you pass a new keyword it will get added to the kwargs dict.

Working with classes

A partial cannot be defined as a method inside a class:

class Foo:
    def __init__(self, x):
        self.x = x
        self.y = {}

    def add_to_y(self, **kwargs):
        self.y.update(kwargs)

    p = partial(add_to_y, foo="bar")

f = Foo(1)

# throws error
f.p()

To make this work, you need partialmethod. Simply replace partial with partialmethod:

class Foo:
    def __init__(self, x):
        self.x = x
        self.y = {}

    def add_to_y(self, **kwargs):
        self.y.update(kwargs)

    p = partialmethod(add_to_y, foo="bar")

f = Foo(1)

f.p()

f.y
# [Out]: {'foo': 'bar'}