Some insights into the low-level workflow of the Django framework

Running flow

You can use multiple methods to run a Django app, one of which is runserver which uses the web server of Django itself, and it is mostly used in the development and testing phases. Another is using protocols like fastcgi and uWSGI.

1. runserver

Usage: python3 manage.py runserver, this command is run under port 8080 by default. If you investigate into the source codes of manage.py, you will find that this line is actually executed through the internal runserver command in execute_from_command_line, and this command mostly does two things:

• Parse the parameters, and get the wsgi handler by django.core.servers.basehttp.get_internal_wsgi_application;
• Generate a WSGIServer object according to your IP address and port, and then accept the user's requests. The source codes of get_internal_wsgi_application is as followed:

def get_internal_wsgi_application():
    from django.conf import settings
    app_path = getattr(settings, 'WSGI_APPLICATION')
    if app_path is None:
        return get_wsgi_application()
    return import_by_path(
        app_path,
        error_prefix="WSGI application '%s' could not be loaded; " % app_path
    )

By the above we know that, Django will first obtain the handler via WSGI_APPLICATION in settings. While building the project, Django will create a wsgi.py by default.

2. uWSGI

Another approach is the most prevalent one while running Django in production, it uses Ngnix combined with uWSGI. To dive more deeply into this method, we would need to learn more about two protocols: WSGI and uWSGI.

• WSGI is short for Web Server Gateway Interface. It is a universally applicable interface that bridges python-based web servers and web apps/framworks, and is designed based on the current CGI standard. WSGI is actually a gateway, which helps conversion between multiple protocols.
• uWSGI is a web server which supports protocols like WSGI protocol, UWSGI protocol and HTTP Protocol, etc. It is known for rapid transmission, low memory usage and its excellence in multi-app management.

HTTP Request Process

Django's RESTful workflow is similar to other web frameworks: Receiving requests and returning responses. Let's see the details in the following:

When Django is creating the project via django-admin, it will automatically create some default files including settings.py and manage.py, and execute the following command before building the WSGIServer:

from django.conf import settings

During the execution of the above command, Django will read the DjANGO_SETTINGS_MODULE config info in os.environ and load the config files before creating the setting object. Therefore, inside the manage.py, it will first add the path of project settings to the os path before obtaining the WSGIServer.

1. Creating the WSGIServer

No matter you are running your Django project with runserver or uWSGI, Django will always call run() in django.core.servers.basehttp during the start-up, creating an instance of WSGIServer, and later call its serve_forever() to start the HTTP services. Below is the source codes of run():

 def run(addr, port, wsgi_handler, ipv6=False, threading=False):
     server_address = (addr, port)
     if threading:
        httpd_cls = type(str('WSGIServer'), (socketserver.ThreadingMixIn, WSGIServer), {})
    else:
        httpd_cls = WSGIServer
        httpd = httpd_cls(server_address, WSGIRequestHandler, ipv6=ipv6)
        # Sets the callable application as the WSGI application that will receive requests
        httpd.set_app(wsgi_handler)
        httpd.serve_forever()

From the above, we see that: While creating the WSGIServer instance, WSGIRequestHandler is used to designate the HTTP request handler. When the user's request arrives at the server, WSGIServerwill create a WSGIRequestHandler instance, using its handler function to process the http request, and actually in the end it calls run() in wsgiref.handlers.BaseHandler. With set_app(), WSGIServer set up a callable object as an application. The above mentioned handler function will call this application to process the request and return the response. Besides, WSGIServer inherits from wsgiref.simple_server.WSGIServer, but WSGIRequestHandler inherits from wsgiref.simple_server.WSGIRequestHandler, wsgiref is the reference implementation of WSGI from the Python standard library.

2. Process the request

In Django, the aforementioned application is in general an object of wsgi.WSGIHandler in django.core.handlers. WSGIHandler inherits from base.BaseHandler and it creates a WSGIRequest instance which further inherits from http.HttpRequest. These are the essenial logics on how Django processes the request.

3. Return the response

There is a get_response method in BaseHandler and it initially loads the ROOT_URLCONF in the Django project and then find out corresponding view methods or classes according to the url rules. The logics in the view functions/classes will create and return specific responses through the request instance.

After Django returns the results, the run() in wsgiref.handlers.BaseHandler will call finish_response to end the request, and return the contents to the user.

Seeing from the above image. 1-9 represent 9 steps, and the first steo is to send the request from a page, which reaches the Request middleware in the 2nd step. In the 3rd step, URLConf is querying for the corresponding View in url.py, and later the Views are called to answer the request with their functions invoked. During the 6th step, the view will access the model information and any interactions between model and databases are done by a built-in manager. Here Views can have contexts if needed, and if there is any context, it will be passed to Template for further rendering.

Processes a-e are done mostly in Templates. First filters and tags are used to render the output, which is later returned to Views. At the same time, Response will receive instances of HTTPResponse, and any response could add up new information or simply return a brand new body. In the end, user could receive the response from the browser.

Two most important components in these procedures are: Middleware and URLConf, and below we will briefly walk through them.

Middleware

Middleware does not exclusively belong to Django, but also exist in other frameworks. In Django, middleware could participate in 4 phases in the RESTful process: request, view, response, exception. Accordingly, every class of middleware has the following 4 methods in the pattern of: process_x, where x are request, view, response and exception You can define one or more of these methods depending on which stage you want the middleware to put effect on, and every method returns a response object.

URLConf

URLconf is the mapping of the websites supported by Django. Basically, it is a mapping table between the URL pattern and the view functions.

References

[1] https://hitesh.in/2009/django-flow/
[2] https://stackoverflow.com/questions/1836627/diagrams-explanations-of-django-request-processing
[3] https://docs.djangoproject.com/en/3.2/topics/http/urls/
[4] https://docs.djangoproject.com/en/3.2/topics/http/middleware/
[5] https://docs.djangoproject.com/en/3.2/topics/templates/
[6] https://medium.com/@jvdali966/understanding-djangos-work-flow-1ee521422092