Django leaky connection pool

Disclaimer

• In the case described below, we are NOT using traditional Django web server. We are just leverage django as a framework and start our own grpc server. That's why we hit this tricky scenario, that's used to be handled by django automatically

Background

Django web server will close old connection when web request starts / finishes using signals: https://github.com/django/django/blob/5.2.5/django/db/__init__.py#L62-L63

# Register an event to reset transaction state and close connections past
# their lifetime.
def close_old_connections(**kwargs):
    for conn in connections.all(initialized_only=True):
        conn.close_if_unusable_or_obsolete()


signals.request_started.connect(close_old_connections)
signals.request_finished.connect(close_old_connections)

Yet, request_started and request_finished will only be fired upon web requests. If we're not using django for web server, we need to bite the bullet and handle close connection on our own.

What we did

Interceptor on request

We have a grpc.ServerInterceptor, which closes the connection before request and after response.

class DatabaseInterceptor(grpc.ServerInterceptor):
    ...
    def before_request(...):
        django.db.close_old_connections()

    def after_response(...):
        django.db.close_old_connections()


server = grpc.server(
    ...
    interceptors=[
        DatabaseInterceptor(),
    ]
)

Define an internal ThreadPoolExecutor that helps us to close connection

We need this because the new threads spawned will get a connection if it runs a database query -> but will never return the connection back to the pool

class DatabaseThreadPoolExecutor(ThreadPoolExecutor):
    ...
    def submit(self, fn, /, *args, **kwargs):
        try:
            return fn(*args, **kwargs)
        finally:
            django.db.close_old_connections()

Other alternatives but not considered

We considered the on_commit callback like

transaction.on_commit(django.db.close_old_connections)

such that developers don't need to always remember to use the predefined DatabaseThreadPoolExecutor.

However Django doesn't provide rollback hook, that makes it not a good solution because we do want to release connection whenever there's an exception in an atomic block.

Thoughts

This is very tricky to discover but we cannot blame django for this. Django is built for handling web requests, it's on us using the framework to do what it's not supposed to do.

Yet, hope this helps anyone who face this situation and provide some insights. :D