DEV Community: Kracekumar

Profiling Django App

Kracekumar — Sun, 17 Oct 2021 04:07:52 +0000

TL:DR

Pyinstrument is a call stack sampling profiler with low overhead to find out time spent in your Django application.
QueryCount is a simplistic ORM query count middleware that counts the number of ORM queries, finds duplicate queries, and prints them in the terminal.
Django Silk is an extensive Django profiler that records the entire execution, SQL queries, source of origin, and persists the recordings. The complete Django profiler.

🔬 What's Profiling? 🔬

Profiling is a dynamic program analysis that measures a running program's time and(or) memory consumption. The profiler can instrument the entire running program or record samples for a fixed duration of time.

🚀 Why Profile Django code? 🚀

The Django application can consume more than expected memory or time. The blog post will focus on profiling performance. Some of the reasons why Django App can be slow

ORM could be making N+1 select queries. Example: The child model fetching parent models.
The custom python code can be making DB queries inside a for-loop.
The service or utility function may be making an HTTP or network call that consumes a lot of time.

Application Setup

Here is a sample book counter application that contains three models - Book, Author, and Shelf. A shelf contains a list of books. A book has many authors.

Models

# book/models.py

from django.contrib.auth.models import User
from django.db import models


class Author(models.Model):
    first_name = models.CharField(max_length=255, blank=False, null=False)
    middle_name = models.CharField(max_length=255, blank=True, null=True,
                                   default='')
    last_name = models.CharField(max_length=255, blank=True, null=True,
                                 default='')
    profile_photo = models.URLField(default='', blank=True, null=True)


class Book(models.Model):
    name = models.CharField(max_length=255, blank=False, null=False)
    cover_photo = models.URLField(default='', blank=True, null=True)
    language = models.CharField(max_length=255, blank=True, null=False)


#shelf/models.py
from django.db import models

# Create your models here.
from book.models import Book


class Shelf(models.Model):
    name = models.CharField(max_length=255, unique=True,
                            db_index=True, null=False, blank=False)
    books = models.ManyToManyField(Book, related_name="shelves")

Views

#shelf/views.py
class ShelfViewSet(viewsets.ViewSet):
    http_method_names = ["get"]
    queryset = Shelf.objects.all()

    def list(self, request):
        name = request.GET.get('name')
        qs = self.queryset

        if name:
            qs = qs.filter(name=name)

        data = ShelfSerializer(qs, many=True).data
        return Response(data)

Serializers

#book/serializers.py
from rest_framework import serializers
from book.models import Author, Book


class AuthorSerializer(serializers.ModelSerializer):
    class Meta:
        model = Author
        fields = ("first_name", "middle_name", "last_name", "profile_photo")


class BookSerializer(serializers.ModelSerializer):
    authors = AuthorSerializer(many=True)

    class Meta:
        model = Book
        fields = ("name", "cover_photo", "language", "authors")

#shelf/serializers.py
from rest_framework import serializers
from shelf.models import Shelf
from book.serializers import BookSerializer
import time

classic_books = ['War And Peace', 'Lolita', 'The Stranger']


class ShelfSerializer(serializers.ModelSerializer):
    books = BookSerializer(many=True)

    class Meta:
        model = Shelf
        fields = ("books", "name")

    def to_representation(self, obj):
        book = super().to_representation(obj)
        if book['name'] in classic_books:
            book['is_classic'] = True
        else:
            book['is_classic'] = False
            time.sleep(1)
        return book

The application has an endpoint /shelf/ that returns all the shelves along with books and authors using DRF model serializer.

🚆 Pyinstrument 🚆

Pyinstrument is a call stack Python profiler to find performance bottlenecks in your Python programs. It doesn't track every function call but samples the call stack every one millisecond. So it's a statistical profiler(not tracer), and it may lead to optimizing the wrong part of the program.

Since the profiler records the call stack, every fixed interval(can be configured to lower value too, say 1ns) has a low overhead.

Profiling time = program execution time + profiler recording time

If the program takes 0.33s to run, pyinstrument profiling takes 0.43s (30% extra) , cProfile takes 0.61s (84% extra).

Pyinstrument is a Python profiler that offers a Django middleware to record the profiling.

How to use it?

Pyinstrument middleware records the request-response time. The middleware class goes into MIDDLEWARE list variable. MIDDLEWARE+=[ 'pyinstrument.middleware.ProfilerMiddleware'].
The profiler stores the data in the configured directory mentioned in setting file. The profiler generates profile data for every request. The PYINSTRUMENT_PROFILE_DIR contains a directory that stores the profile data.

Output

By default, pyinstrument produces HTML profile information.
The right corner contains the metadata.
- SAMPLES - number of samples collected during the execution.
- DURATION - Total duration of the profiling.
The tree structure captures the flow of the code along with the time spent.
Each line is a frame that contains the duration, function/method name, file path, and line number.
data = ShelfSerializer(qs, many=True).data in the views, took 2.09 seconds.
The profiler auto hides any standard library and third-party calls to provide a brief overview of the program call graph.
From the profile, sleep function inside the serializer takes most of the time.

Upside

Fast and easy to use.
Requires no code modification other than configuring settings variables.
Less overhead to measure performance.

Downside

The library can only collect Python code and not SQL queries executed during the request-response cycle.
Since the library samples the call stack every one millisecond, there can be false positives. It's possible to configure the sample rate.

🧮 Django Query Count 🧮

Django QueryCount is a middleware that prints the number of database queries made during the request processing.

How to use it?

Add querycount.middleware.QueryCountMiddleware to the MIDDLEWARE settings variable.
The middleware supports printing duplicate queries. Following are some extra settings options.

  QUERYCOUNT = {
      'THRESHOLDS': {
          'MEDIUM': 50,
          'HIGH': 200,
          'MIN_TIME_TO_LOG':0,
          'MIN_QUERY_COUNT_TO_LOG':0
      },
      'IGNORE_REQUEST_PATTERNS': [],
      'IGNORE_SQL_PATTERNS': [],
      'DISPLAY_DUPLICATES': True,
      'RESPONSE_HEADER': 'X-DjangoQueryCount-Count'
  }

Ouptut

# output in the server console
[17/Oct/2021 11:05:43] "GET /shelf/ HTTP/1.1" 200 1031

http://localhost:8000/shelf/
|------|-----------|----------|----------|----------|------------|
| Type | Database  |   Reads  |  Writes  |  Totals  | Duplicates |
|------|-----------|----------|----------|----------|------------|
| RESP |  default  |    8     |    0     |    8     |     1      |
|------|-----------|----------|----------|----------|------------|
Total queries: 8 in 2.0803s


Repeated 1 times.
SELECT "book_book"."id", "book_book"."name",
"book_book"."cover_photo", "book_book"."language" FROM "book_book"
INNER JOIN "shelf_shelf_books" ON ("book_book"."id" =
"shelf_shelf_books"."book_id") WHERE "shelf_shelf_books"."shelf_id" =
1


Repeated 1 times.
SELECT "shelf_shelf"."id", "shelf_shelf"."name" FROM "shelf_shelf"

# output in the client
$http http://localhost:8000/shelf/ --header
HTTP/1.1 200 OK
Allow: GET
Content-Length: 1031
Content-Type: application/json
Date: Sun, 17 Oct 2021 11:05:43 GMT
Referrer-Policy: same-origin
Server: WSGIServer/0.2 CPython/3.9.7
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-DjangoQueryCount-Count: 8
X-Frame-Options: DENY

The middleware records the total number of SQL queries made and prints it in a tabular form in the terminal.
The table contains the total number of reads, total number of writes, total queries, and total duplicates. The number of duplicates is a good indicator of N + 1 select queries.
When DISPLAY_DUPLICATES is True, the middleware prints the duplicate query.

Upside

Simple to use.
Fast and accurate.
Low overhead to find out the SQL queries.

Downside

Simple, hence hard to find the origin of the SQL queries.
No option to print all the SQL queries.

🔥 Django Silk 🔥

Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection.

Installation

pip install git+https://github.com/jazzband/django-silk.git#egg=django-silk

The latest PyPI version doesn't support Django 3.2.

How to use it?

Add 'silk.middleware.SilkyMiddleware' to MIDDLEWARE settings variable.
Add silk to INSTALLED_APPS settings variable.
Add silk endpoints to url patterns. urlpatterns += [url(r'^silk/', include('silk.urls', namespace='silk'))].
Run migrate to create tables. python manage.py migrate.
Run collectstatic to collect static files, python manage.py collectstatic.

Output

/silk/ endpoint is the dashboard and lists all the recorded requests.
The quick view under Most Time Overall shows, the response status, total response time, total time spent in queries, and total number of queries.

Clicking on the request profile takes to the detail view. The detail view displays request headers details, response headers, and details on how to use the client to connect to the endpoint.

The SQL tab displays all the queries made during the request-response cycle.
Clicking on the single query displays the query plan and Python traceback to find the source of origin. SILKY_ANALYZE_QUERIES variable controls the feature to analyze the SQL query.

Silk also provides option to generate Python profile code for a view or any Python function using a decorator or a context manager. Below is the modified view.

from silk.profiling.profiler import silk_profile

class ShelfViewSet(viewsets.ViewSet):
    http_method_names = ["get"]
    queryset = Shelf.objects.all()


    @silk_profile(name='Shelf list')
    def list(self, request):
        name = request.GET.get('name')
        qs = self.queryset

        if name:
            qs = qs.filter(name=name)

        data = ShelfSerializer(qs, many=True).data
        return Response(data)

Profile tab displays the call graph and Python traceback. The page also shows the link to download ` profile file(.prof). snakeviz is an excellent tool to visualize the profile data.
Below is the profile visualized using snakeviz.

Snakeviz graph is visually pleasing and gives a breakdown of the function call based on time spent.

Other features

Dynamic Profiling - Profile specific function or class during runtime.

SILKY_DYNAMIC_PROFILING = [{ 'module': 'shelf.views', 'function': 'ShelfViewSet.list' }]

Silk can also display the time taken by the silk profiler. The setting variable SILKY_META controls it. When set to True, the dashboard displays the time taken by silk in red color in the display card.

Upside

Exhaustive profiling option and completeness.
UI dashboard to visualise and browse the profiling data.
Storing all the data in the database helps to analyze performance improvement on the same endpoint over time.
Highly customizable and extensible to profile.

Downside

The library is not well maintained, there are a lot of open issues, and the PyPI release doesn't support Django 3.2 version.
A bit of a learning curve and a lot of configuration.
It can be slow on slow endpoints since it stores all the details in the database.

Conclusion

Performance is a feature that lets the user of the application perform certain types of operations quicker. The profiling tools help developers to measure the application's performance and make decisions accordingly.

There are other Django/Python profilers, but these three profilers help measure and understand the Django application performance. Use these tools to make your Django application faster.

1. 🚀Profiling Django App 🚀- https://t.co/EifIAiW8ba. In this blog post, I discuss three tools. #Django #Python
— kracekumar || கிரேஸ்குமார் (@kracetheking) October 17, 2021

Image Credits

Foot Print Photo by Martin Widenka on Unsplash.
Pyinstrument cover image is from the source repository.
Query Count cover image is from the repository.
Django Silk cover image is from the repository.

Generate HTTP Request

Kracekumar — Sun, 10 Oct 2021 18:33:27 +0000

HTTPie is a command-line utility for making HTTP requests with more straightforward syntax(controversial, I agree). The interesting feature is --offline flag which prints HTTP raw request text. The client sends the HTTP request to the server, and the server responds to the request. It's an alternate to curl.

HTTP Syntax

HTTP Flow and syntrax from Wikipedia.

A client sends request messages to the server, which consist of

a request line, consisting of the case-sensitive request method, a space, the request target, another space, the protocol version, a carriage return, and a line feed (e.g. GET /images/logo.png HTTP/1.1)

zero or more request header fields, each consisting of the case-insensitive field name, a colon, optional leading whitespace, the field value, and optional trailing whitespace (e.g. Accept-Language: en), and ending with a carriage return and a line feed.

an empty line, consisting of a carriage return and a line feed;

an optional message body.

In the HTTP/1.1 protocol, all header fields except Host are optional.

A request line containing only the path name is accepted by servers to maintain compatibility with HTTP clients before the HTTP/1.0 specification in RFC 1945.

Throughout the post, I'll use --offline feature to understand how the HTTP request structure looks for educational purposes.

🉑 Accept only JSON response data 🉑

HTTPie uses : to separate header key and values in the terminal. Accept:application/json.
Accept header tells the server, the client accepts only specific MIME types, here JSON.

$http --offline httpbin.org/get Accept:application/json
GET /get HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: httpbin.org
User-Agent: HTTPie/2.5.0
# Sample request sent to the server
$ http httpbin.org/get Accept:application/json
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 310
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:21:40 GMT
Server: gunicorn/19.9.0

{
    "args": {},
    "headers": {
        "Accept": "application/json",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/2.5.0",
        "X-Amzn-Trace-Id": "Root=1-61629484-3b25a3631e2a89bf60f2600e"
    },
    "origin": "xxx.xxx.xxx.xxx",
    "url": "http://httpbin.org/get"
}

You can pass more than one value in the Accept header seperated by comma.
The response from the server contains Content-Type as JSON. The server can choose to ignore the Accept header.

📚 Request the Tamil language version of duckduckgo 📚

Accept-Language header instructs the web server to deliver particluar language version of the page.
Here Accept-Language is set to ta. ta is ISO 639-1 code.

$http --offline  https://duckduckgo.com Accept-Language:ta
GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: ta
Connection: keep-alive
Host: duckduckgo.com
User-Agent: HTTPie/2.5.0

$http  https://duckduckgo.com Accept-Language:ta
HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: keep-alive
Content-Encoding: gzip
Content-Security-Policy: default-src 'none' ; connect-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ https://duck.co ; manifest-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; media-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; script-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ 'unsafe-inline' 'unsafe-eval' ; font-src data:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; img-src data:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; style-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ 'unsafe-inline' ; object-src 'none' ; worker-src blob: ; child-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; frame-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; form-action  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ https://duck.co ; frame-ancestors 'self' ; base-uri 'self' ; block-all-mixed-content ;
Content-Type: text/html; charset=UTF-8
Date: Sat, 09 Oct 2021 21:44:33 GMT
ETag: W/"6161a338-16b8"
Expect-CT: max-age=0
Expires: Sat, 09 Oct 2021 21:44:32 GMT
Permissions-Policy: interest-cohort=()
Referrer-Policy: origin
Server: nginx
Strict-Transport-Security: max-age=31536000
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1;mode=block

<!DOCTYPE html>
...
<body id="pg-index" class="page-index body--home">
    <script type="text/javascript" src="/tl5.js"></script>
<script type="text/javascript" src="/lib/l123.js"></script>
<script type="text/javascript" src="/locale/ta_IN/duckduckgo50.js"></script>
<script type="text/javascript" src="/util/u588.js"></script>
<script type="text/javascript" src="/d3012.js"></script>
...
<noscript>
    <div class="tag-home">
        <div class="tag-home__wrapper">
            <div class="tag-home__item">
                Privacy, simplified&period;
                <span class="hide--screen-xs"><a href="/about" class="tag-home__link"> மேலும் கற்க</a></span>
            </div>
        </div>
    </div>
</noscript>
...
</html>

The header can carry more than one value separated by a comma.
When more than one value is present, an extra parameter q=0.5 represents the weightage among the values. Example: Accept-Language: ta,fr;q=0.50.
One of the link description is in Tamil, மேலும் கற்க.

🔑 Post Authorization token as part of login 🔑

Authorization header can be used to provide credentials to authenticate the user.
Example: Authorization:"Bearer my-dear-darkness".
Because of space between Bearer and the value quotes is mandatory.

$http --offline POST httpbin.org/auth Authorization:"Bearer my-dear-darkness"
POST /auth HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: Bearer my-dear-darkness
Connection: keep-alive
Content-Length: 0
Host: httpbin.org
User-Agent: HTTPie/2.5.0

🍪 Send Cookie as part of a request 🍪

The server can send HTTP Cookie as part of the response, and the client can send the Cookies in subsequent requests.
Cookie header carries the cookies previously sent by the user.
Semi-colon separates multiple cookie pairs.
Example: "name=not-unique;value=23"

$http --offline http://pie.dev/cookies Cookie:"name=not-unique;value=23"
GET /cookies HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cookie: name=not-unique;value=23
Host: dev.pie
User-Agent: HTTPie/2.5.0
$http http://pie.dev/cookies Cookie:"name=not-unique;value=23"
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be1b984c743c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:24:13 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2Zt9wQcKQYStY5dPukoEXFwn7K6pKNaVTiKljZ5h9oL3mcQZj0khYq8Kmzo8PmQinPncStZeHASetQqHzCODe0wbrljPEIJxCWGdRWMbry9rWOG%2FBheDvJs7"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "cookies": {
        "name": "not-unique",
        "value": "23"
    }
}

✉️ Send JSON Request ✉️

HTTPie supports sending form-encoded values or JSON values. = sign indiciates JSON key-value pair.
= is useful for primitive values like number, string, null, boolean.
Example: lang-py

$http --offline PUT http://pie.dev/put lang=py version=3.10
PUT /put HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 33
Content-Type: application/json
Host: pie.dev
User-Agent: HTTPie/2.5.0

{
    "lang": "py",
    "version": "3.10"
}
$HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be1e017fb53c12-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:25:52 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=GMW5PQHMwjxpVUeDG7uWo6M%2FiKLMzmQLd1e5BIG3AXljmuQgwCP9nzrFPdaidR2wL14eisfiViDhumDHpepgNB6yIrsVKXRybHa5tRqmH7lUDoQdRqCK1Ijg"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "{\"lang\": \"py\", \"version\": \"3.10\"}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "application/json, */*;q=0.5",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be1e017fb53c12-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "33",
        "Content-Type": "application/json",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": {
        "lang": "py",
        "version": "3.10"
    },
    "origin": "xxx.xxx.xxx.xxxx",
    "url": "http://pie.dev/put"
}

✉️ Send non-primitive JSON values ✉️

:= means the JSON value is non-primitve values like array/list and dictionary.
Single quote carries the value.
Example: os:='["GNU/Linux", "Mac OSX"]'.

$http --offline  PUT http://pie.dev/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]'
PUT /put HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 65
Content-Type: application/json
Host: pie.dev
User-Agent: HTTPie/2.5.0

{
    "lang": "py",
    "os": [
        "GNU/Linux",
        "Mac OSX"
    ],
    "version": "3.10"
}
$http  PUT http://pie.dev/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]'
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be224ce97c3c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:28:48 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=dgY0R%2FqomBbumKs0IVB7GjPR2chhHJ9wVKAzW33IOnvZ%2Fs4l2LYnvKKeXo6Xhd162AYKGzyIrK4pNrYdH8SEs1NvGmYfJqEDIPmfUOELqpC6HK9iP1zIENa0"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "{\"lang\": \"py\", \"version\": \"3.10\", \"os\": [\"GNU/Linux\", \"Mac OSX\"]}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "application/json, */*;q=0.5",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be224ce97c3c0c-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "65",
        "Content-Type": "application/json",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": {
        "lang": "py",
        "os": [
            "GNU/Linux",
            "Mac OSX"
        ],
        "version": "3.10"
    },
    "origin": "xx.xxx.xxx.xxx",
    "url": "http://pie.dev/put"
}

📤 Upload files 📤

-f flag indicates the data is form-encoded values.
@ symbol indicates the value is a file.
Example: cv@hello-world.txt. cv form field name.

$cat hello-world.txt
hello-world
$http --offline -f POST pie.dev/post name='Krace' cv@hello-world.txt
POST /post HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 277
Content-Type: multipart/form-data; boundary=183f70d3da41432d95bcd839e2cc20e2
Host: pie.dev
User-Agent: HTTPie/2.5.0

--183f70d3da41432d95bcd839e2cc20e2
Content-Disposition: form-data; name="name"

Krace
--183f70d3da41432d95bcd839e2cc20e2
Content-Disposition: form-data; name="cv"; filename="hello-world.txt"
Content-Type: text/plain

hello-world

--183f70d3da41432d95bcd839e2cc20e2--
$ http -f POST pie.dev/post name='Krace' cv@hello-world.txt
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be2564a8fc3c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:30:54 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=yNaTSZJ7ouYH%2FSJZw6LIR5vNl6KNTFeDKF1u8V60abb3ClKLzdOj0zkchcIAWqTvZDbxXm5MnffDkCLdqviMQuAo7DqFA2GH%2Bm%2FiZ6sH90oOr0HyFGAuS4Gp"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "",
    "files": {
        "cv": "hello-world\n"
    },
    "form": {
        "name": "Krace"
    },
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be2564a8fc3c0c-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "277",
        "Content-Type": "multipart/form-data; boundary=252c6fcd1dcc40e09de54958660d672d",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": null,
    "origin": "xxx.xxx.xxx.xxxx",
    "url": "http://pie.dev/post"
}

Explore more in HTTPie website.

🐈 How to verify the the generated request works? 🐈

Netcat is a utility for sending and receiving data in network connection using TCP or UDP. Netcat take hostname, port, and body as arguments and sends it to the server and displays the response.

$http --offline  PUT http://httpbin.org/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]' | nc httpbin.org 80
HTTP/1.1 200 OK
Date: Sun, 10 Oct 2021 08:27:58 GMT
Content-Type: application/json
Content-Length: 631
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {},
  "data": "{\"lang\": \"py\", \"version\": \"3.10\", \"os\": [\"GNU/Linux\", \"Mac OSX\"]}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "application/json, */*;q=0.5",
    "Accept-Encoding": "gzip, deflate",
    "Content-Length": "65",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "HTTPie/2.5.0",
    "X-Amzn-Trace-Id": "Root=1-6162a40e-34b9a83f40868b4a73e8fa09"
  },
  "json": {
    "lang": "py",
    "os": [
      "GNU/Linux",
      "Mac OSX"
    ],
    "version": "3.10"
  },
  "origin": "xxx.xxx.xxx.xxx",
  "url": "http://httpbin.org/put"
}

Netcat doesn't support encrypted network connections.

❓How to understand what HTTP request the client sends to the server?
💡Use HTTPie --offline feature to print the HTTP request.

Details on how to send various HTTPie options in the blog post - https://t.co/mIEG2FpBkP #TIL #Python #CLI pic.twitter.com/kBvfc9QQAt
— kracekumar || கிரேஸ்குமார் (@kracetheking) October 10, 2021

References

HTTPie - https://httpie.io/
HTTPBin - http://httpbin.org
NetCat - https://en.wikipedia.org/wiki/Netcat
HTTP MDN Docs: https://developer.mozilla.org/en-US/docs/Web/HTTP
HTTP Header Image: https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/HTTP_logo.svg/2880px-HTTP_logo.svg.png

Type Check Your Django Application

Kracekumar — Sat, 18 Sep 2021 12:33:27 +0000

Recently, I gave a talk, Type Check your Django app at two conferences - Euro Python 2021 and PyCon India 2021. The talk was about adding Python gradual typing to Django using third-party package Django-stubs focussed heavily around Django Models. The blog post is the write-up of the talk. Here is the unofficial link recorded video of the PyCon India talk.

Here is the link to PyCon India Slides. The slides to Euro Python Talk (both slides are similar).

Gradual Typing

Photo by John Lockwood on Unsplash

Python from the 3.5 version onwards started supported optional static typing or gradual typing. Some parts of the source code contain type annotation, and some parts may have no type annotation. The python interpreter doesn't complain about the lack of type hints. The third-party library mypy does the type check.

Throughout the post, the source code example follows Python 3.8+ syntax and Django version 3.2. By default, static type checker refers to mypy, even though there are other type checkers like pyre from Facebook and pylance/pyright from Microsoft.

Types

Photo by Alexander Schimmeck on Unsplash

Types at Run Time

>>> type(lambda x: x)
<class 'function'>
>>> type(type)
<class 'type'>
>>> type(23)
<class 'int'>
>>> type(("127.0.0.1", 8000))
<class 'tuple'>

Python's in-built function type returns the type of the argument. When the argument is ("127.0.0.1", 8000) the function returns type as tuple.

>>>from django.contrib.auth.models import User
>>>type(User.objects.filter(
    email='not-found@example.com'))
django.db.models.query.QuerySet

On a Django filter method result, type functions returns the type as django.db.models.query.QuerySet.

Types at Static Checker Time

addr = "127.0.0.1"
port = 8000

reveal_type((addr, port))

Similar to the type function, the static type checker provides reveal_type function returns the type of the argument during static type checker time. The function is not present during Python runtime but is part of mypy.

$mypy filename.py
note: Revealed type is 
  'Tuple[builtins.str, builtins.int]'

The reveal_type returns the type of the tuple as Tuple[builtins.str, builtins.int]. The reveal_type function also returns the type of tuple elements. In contrast, the type function returns the object type at the first level.

# filename.py
from django.contrib.auth.models import User

reveal_type(User.objects.filter(
  email='not-found@example.com'))

$ mypy filename.py
note: Revealed type is 
  'django.contrib.auth.models.UserManager
  [django.contrib.auth.models.User]'

Similarly, on the result of Django's User object's filter method, reveal_type returns the type as UserManager[User]. Mypy is interested in the type of objects at all levels.

Mypy config

# mypy.ini
exclude = "[a-zA-Z_]+.migrations.|[a-zA-Z_]+.tests.|[a-zA-Z_]+.testing."

allow_redefinition = false

plugins =
  mypy_django_plugin.main,

[mypy.plugins.django-stubs]
django_settings_module = "yourapp.settings"

The Django project does not contain type annotation in the source code and not in road map. Mypy needs information to infer the Django source code types. The mypy configuration needs to know the Django Stubs' entry point and the Django project's settings. Pass Django stub plugins to plugins variable and settings file location of the Django project to Django stubs plugin as django_settings_module variable in mypy.plugins.django-stubs.

Annotation Syntax

Photo by Lea Øchel on Unsplash

from datetime import date

# Example variable annotation
lang: str = "Python"
year: date = date(1989, 2, 1)

# Example annotation on input arguments 
# and return values
def sum(a: int, b: int) -> int:
  return a + b

class Person:
  # Class/instance method annotation
  def __init__(self, name: str, age: int, 
               is_alive: bool):
    self.name = name
    self.age = age
    self.is_alive = is_alive

Type annotation can happen in three places.

During variable declaration/definition. Example: lang: str = "Python". The grammar is name: <type> = <value>.
The function declaration with input arguments and return value types annotated. sum(a: int, b: int) -> int. The function sum input arguments annotation looks similar to variable annotation. The return value annotation syntax, -> arrow mark followed by return value type. In sum function definition, it's -> int.
The method declaration. The syntax is similar to annotating a function. The self or class argument needs no annotation since mypy understand the semantics of the declaration. Except __init__ method, when the function, method does return value, the explicit annotation should be -> None .

Annotating Django Code

Views

Django supports class-based views and function-based views. Since function and method annotations are similar, the example will focus on function-based views.

from django.http import (HttpRequest, HttpResponse, 
            HttpResponseNotFound)

def index(request: HttpRequest) -> HttpResponse:
    return HttpResponse("hello world!")

The view function takes in a HttpRequest and returns a HttpResponse. The annotating view function is straightforward after importing relevant classes from django.http module.

def view_404(request: 
               HttpRequest) -> HttpResponseNotFound:
    return HttpResponseNotFound(
      '<h1>Page not found</h1>')

def view_404(request: HttpRequest) -> HttpResponse:
    return HttpResponseNotFound(
      '<h1>Page not found</h1>')


# bad - not precise and not useful
def view_404(request: HttpRequest) -> object:
    return HttpResponseNotFound(
      '<h1>Page not found</h1>')

Here is another view function, view_404. The function returns HttpResponseFound- Http Status code 404. The return value annotation can take three possible values - HttpResponseNotFound, HttpResponse, object. The mypy accepts all three annotations as valid.

Why and How? MRO

Method Resolution Order(mro) is the linearization of multi-inheritance parent classes. To know the mro of a class, each class has a mro method.

>>>HttpResponse.mro()
[django.http.response.HttpResponse,
 django.http.response.HttpResponseBase,
 object]


>>>HttpResponseNotFound.mro()
[django.http.response.HttpResponseNotFound,
 django.http.response.HttpResponse,
 django.http.response.HttpResponseBase,
 object]

HTTPResponseNotFound inherits HttpResponse, HTTPResponse inherits HttpResponseBase, HttpResponseBase inherits objects.

LSP - Liskov substitution principle

Liskov substitution principle states that in an object-oriented program, substituting a superclass object reference with an object of any of its subclasses, the program should not break.

HTTPResponseNotFound is a special class of HTTPResponse and object; hence mypy doesn't complain about the type mismatch.

Django Models

Photo by Jan Antonin Kolar on Unsplash

Create

from django.db import models
from django.utils import timezone


class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField("date published")


def create_question(question_text: str) -> Question:
    qs = Question(question_text=question_text, 
                  pub_date=timezone.now())
    qs.save()
    return qs

Question is a Django model with two explicit fields: question_text of CharField and pub_date of DateTimeField . create_question is a simple function that takes in question_text as an argument and returns Question instance.

When the function returns an object, the return annotation should be the class's reference or the class's name as a string.

Read


def get_question(question_text: str) -> Question:
    return Question.objects.filter(
      question_text=question_text).first()

get_question takes a string as an argument and filters the Question model, and returns the first instance.

error: Incompatible return value type 
(got "Optional[Any]", expected "Question")

Mypy is unhappy about the return type annotation. The type checker says the return value can be None or Question instance. But the annotation is Question.

Two solutions

from typing import Optional

def get_question(question_text: str) -> Optional[Question]:
    return Question.objects.filter(
      question_text=question_text).first()

Annotate the return type to specify None value.
Typing module contains an Optional type, which means None. The return value Optional[Question] means None type or Question type.

# mypy.ini
strict_optional = False

def get_question(question_text: str) -> Question:
    return Question.objects.filter(
        question_text=question_text).first()

By default, mypy runs in strict mode. strict_optional variable instructs mypy to ignore None type in the annotations(in the return value, in the variable assignment, ...). There are a lot of such config variables mypy to run in the lenient mode.

The lenient configs values can help to get type coverage quicker.

Filter method

In [8]: Question.objects.all()
Out[8]: <QuerySet [<Question: Question object (1)>, 
                   <Question: Question object (2)>]>

In [9]: Question.objects.filter()
Out[9]: <QuerySet [<Question: Question object (1)>, 
                   <Question: Question object (2)>]>

Django object manager filter method returns a QuerySet and is iterable. All bulk read, and filter operations return Queryset. QuerySet carries the same model instances. It's a box type.

Photo by Alexandra Kusper on Unsplash


def filter_question(text: str) -> QuerySet[Question]:
    return Question.objects.filter(
      text__startswith=text)

def exclude_question(text: str) -> QuerySet[Question]:
    return Question.objects.exclude(
      text__startswith=text)

Other object manager methods that return queryset are all, reverse, order_by, distinct, select_for_update, prefetch_related, ...

Aggregate


class Publisher(models.Model):
    name = models.CharField(max_length=300)

class Book(models.Model):
    name = models.CharField(max_length=300)
    pages = models.IntegerField()
    # use integer field in production
    price = models.DecimalField(max_digits=10, 
                                decimal_places=2)
    rating = models.FloatField()
    publisher = models.ForeignKey(Publisher)
    pubdate = models.DateField()

The aggregate query is a way of summarizing the data to get a high-level understanding of the data. Publisher model stores the data of the book publisher with name as an explicit character field.

The Book model contains six explicit model fields.

name - Character Field of maximum length 300
pages - Integer Field
price - Decimal Field
rating - Decimal Field of maximum 10 digits and minimum 2 decimal digits
publisher - Foreign Key to Publisher Field
Pubdate - Date Field

>>>def get_avg_price():
      return Book.objects.all().aggregate(
        avg_price=Avg("price"))

>>>print(get_avg_price())
{'avg_price': Decimal('276.666666666667')}

The function get_avg_price returns the average price of all the books. avg_price is a Django query expression in the aggregate method. From the get_avg_price function output, the output value is a dictionary.

from decimal import Decimal


def get_avg_price() -> dict[str, Decimal]:
    return Book.objects.all().aggregate(
      avg_price=Avg("price"))

Type annotation is simple here. The return value is a dictionary. dict[str, Decimal] is the return type annotation. The first type of argument(str) in the dict specification is the dictionary's key's type. The second type of argument(Decimal) is the value of the key, Decimal.

Annotate Method

Photo by Joseph Pearson on Unsplash

From Django doc's on annotate queryset method

Annotates each object in the QuerySet with the provided list of query expressions. An expression may be a simple value, a reference to a field on the model (or any related models), or an aggregate expression (averages, sums, etc.) that has been computed over the objects that are related to the objects in the QuerySet.

def count_by_publisher():
    return Publisher.objects.annotate(
      num_books=Count("book"))


def print_pub(num_books=0):
    if num_books > 0:
        res = count_by_publisher().filter(
          num_books__gt=num_books)
    else:
        res = count_by_publisher()
    for item in res:
        print(item.name, item.num_books)

The count_by_publisher function counts the books published by the publisher. The print_pub function filters the publisher count based on the num_book function argument and prints the result.

>>># after importing the function
>>>print_pub()
Penguin 2
vintage 1

print_pub prints publication house name and their books count. Next is adding an annotation to both the function.

from typing import TypedDict
from collections.abc import Iterable

class PublishedBookCount(TypedDict):
    name: str
    num_books: int

def count_by_publisher() -> 
    Iterable[PublishedBookCount]:
  ...

count_by_publisher returns more than one value, and the result is iterable. TypedDict is useful when the dictionary contents keys are known in advance. The attribute names of the class are the key names(should be a string), and the value type is an annotation to the key. count_by_publisher 's annotation is Iterable[PublishedBookCount].

$# mypy output
scratch.py:46: error: Incompatible return value 
    type (got "QuerySet[Any]", expected
"Iterable[PublishedBookCount]")
        return Publisher.objects.annotate(
          num_books=Count("book"))
               ^
scratch.py:51: error: 
      "Iterable[PublishedBookCount]" has no attribute "filter"
       res = count_by_publisher().filter(
         num_books__gt=num_books)

The mypy found out two errors.

error: Incompatible return value type (got "QuerySet[Any]", expected "Iterable[PublishedBookCount]")

Mypy says the .annotate method returns QuerySet[Any] whereas annotation says return type as Iterable[PublishedBookCount].

"Iterable[PublishedBookCount]" has no attribute "filter"

print_pub uses return value from count_by_publisher to filter the values. Since the return value is iterable and the filter method is missing, mypy complains.

How to fix these two errors?

def count_by_publisher() -> QuerySet[Publisher]:
   ...

def print_pub(num_books: int=0) -> None:
    ...
    for item in res:
        print(item.name, item.num_books)

Modify the return value annotation for count_by_publisher to QuerySet[Publisher] as suggested by mypy. Now the first error is fixed, but some other error.

# mypy output
$mypy scratch.py
scratch.py:55: error: "Publisher" has 
      no attribute "num_books"
      print(item.name, item.num_books)

Django dynamically adds the num_books attribute to the return QuerySet. The publisher model has one explicitly declared attribute name, and num_books is nowhere declared, and mypy is complaining.

This was a bug in Django Stubs project and got fixed recently. The newer version of Django stubs provides a nice way to annotate the function.

Option 1 - Recommended

from django_stubs_ext import WithAnnotations

class TypedPublisher(TypedDict):
    num_books: int

def count_by_publisher() -> WithAnnotations[Publisher, TypedPublisher]:
        ...

WithAnnotation takes two argument the model and TypedDict with on the fly fields.

Option 2 - Good solution

Another solution is to create a new model TypedPublisher inside TYPE_CHECKING block, which is only visible to mypy during static type-checking time. The TypedPublisher inherits Publisher model and declares the num_books attribute as Django field, Then mypy will not complain about the missing attribute.

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    class TypedPublisher(Publisher):
        num_books = models.IntegerField()

        class meta:
            abstract = True


def count_by_publisher() -> QuerySet[TypedPublisher]:
    return Publisher.objects.annotate(
      num_books=Count("book"))


def print_pub(num_books: int=0) -> None:
    if num_books > 0:
        res = count_by_publisher().filter(
          num_books__gt=num_books)
    else:
        res = count_by_publisher()
    for item in res:
        print(item.name, item.num_books)

The earlier solution is elegant and works with simple data-types, which group by/annotate returns.

Tools

Board Photo by Nina Mercado on Unsplash

It's hard to start annotation when the project has a significant amount of code because of the surface area and topics to learn. Except for Django ORM, most of the custom code in the project will be Python-specific data flow.

Pyannotate

Pyannotate is a tool to auto-generate type hints for a given Python project. Pyannotate captures the static types during execution code and writes to an annotation file. Pytest-annotate is a pytest plugin to infer types during test time. In the anecdotal micro-benchmark, because of pytest-annotate, the tests take 2X time to complete.

Phase 0 - Preparation

from django.http import (HttpResponse, 
        HttpResponseNotFound)

# Create your views here.
# annotate the return value
def index(request):
    return HttpResponse("hello world!")

def view_404_0(request):
    return HttpResponseNotFound(
      '<h1>Page not found</h1>')

Here is a simple python file with no type annotations.

from polls.views import *
from django.test import RequestFactory

def test_index():
    request_factory = RequestFactory()
    request = request_factory.post('/index')
    index(request)


def test_view_404_0():
    request_factory = RequestFactory()
    request = request_factory.post('/404')
    view_404_0(request)

Then add relevant test cases for the files.

Phase 1 - Invoking Pyannotate

$DJANGO_SETTINGS_MODULE="mysite.settings" PYTHONPATH='.' poetry run pytest -sv polls/tests.py --annotate-output=./annotations.json

While running the pytest pass extra option, --annotate-ouput to store the inferred annotations.

Phase 2 - Apply the annotations

$cat annotations.json
[...
    {
        "path": "polls/views.py",
        "line": 7,
        "func_name": "index",
        "type_comments": [
            "(django.core.handlers.wsgi.WSGIRequest) -> 
          django.http.response.HttpResponse"
        ],
        "samples": 1
    },
    {
        "path": "polls/views.py",
        "line": 10,
        "func_name": "view_404_0",
        "type_comments": [
            "(django.core.handlers.wsgi.WSGIRequest) -> 
          django.http.response.HttpResponseNotFound"
        ],
        "samples": 1
    }
]

After running the test, annotations.json file contains the inferred annotations.

$poetry run pyannotate --type-info ./annotations.json -w polls/views.py --py3

Now, apply the annotations from the annotations.json to the source code in pools/views.py. --py3 flag indicates, the type-annotations should follow Python 3 syntax.

from django.http import HttpResponse, HttpResponseNotFound
from django.core.handlers.wsgi import WSGIRequest
from django.http.response import HttpResponse
from django.http.response import HttpResponseNotFound

def index(request: WSGIRequest) -> HttpResponse:
    return HttpResponse("hello world!")

def view_404_0(request: WSGIRequest) -> HttpResponseNotFound:
    return HttpResponseNotFound('<h1>Page not found</h1>')

After applying the annotations, the file contains the available annotations and required imports.

One major shortcoming of pyannotate is types at test time, and runtime can be different. Example: Dummy Email Provider. That's what happened in the current case. Django tests don't use HTTPRequest, and the tests use WSGIRequest the request argument type annotation is WSGIRequest.

For edge cases like these, pyannotate is better(run Django server as part of pyannotate) and infers the type correctly.

Python Typing Koans

Photo by John Lockwood on Unsplash

Demo

Python Typing Koans repository contains the standalone python programs to learn gradual typing in Python. The programs contain partial or no type hints. The learner will understand how type-checkers evaluate the types by adding type hints and fixing the existing type hints error.

The project contains koans for Python, Django, and Django Rest Framework. By removing the errors in each file, the learner will understand the typing concepts.

The detailed write up about the project in the blog post.

Conclusion

Disclaimer: Gradual Typing is evolving and not complete yet. For example, it's still hard to annotate decorators(python 3.10 release should make it easier), so it's hard to annotate all dynamic behaviors. Adding type-hints to a project comes with its own cost, and not all projects would need it.

I hope you learned about Python Django, and if you're using type-hints, I'd like to hear about it.

If you're struggling with type-hints in your projects or need some advice I'll be happy to offer. Shoot me an email!

Found a bug or typo, and have spare time to fix, send a PR and the file is here!

References

Euro PyCon - https://ep2021.europython.eu/talks/BsaKGk4-type-check-your-django-app/
PyCon India - https://in.pycon.org/cfp/2021/proposals/type-check-your-django-app~ejRql/
Mypy - http://mypy-lang.org/
Django Stub - https://github.com/TypedDjango/django-stubs
Django Models - https://docs.djangoproject.com/en/3.2/topics/db/models/
Video recording - https://youtu.be/efs3RXaLJ4I
PyCon India Slides - https://slides.com/kracekumar/type-hints-in-django/fullscreen
LSP - https://en.wikipedia.org/wiki/Liskov_substitution_principle
Method Resolution Order - https://www.python.org/download/releases/2.3/mro/
Mypy Config variables - https://mypy.readthedocs.io/en/stable/config_file.html#none-and-optional-handling
Django Stubs Annotate fix - https://github.com/typeddjango/django-stubs/pull/398
Pyannotate - https://github.com/dropbox/pyannotate
Pytest-annotate - https://pypi.org/project/pytest-annotate/
Python Typing Koans - https://github.com/kracekumar/python-typing-koans
Python Typing Koans blog post - https://kracekumar.com/post/python-typing-koans

Images References

Fruits Photo by Alexander Schimmeck on Unsplash
Highway Photo by John Lockwood on Unsplash
Leather Jacket Photo by Lea Øchel on Unsplash
Cup cake Photo by Alexandra Kusper on Unsplash
Tool Board Photo by Nina Mercado on Unsplash
Store Photo by Jan Antonin Kolar on Unsplash
Knot Photo by John Lockwood on Unsplash
Records Photo by Joseph Pearson on Unsplash

Discussions

Lobste.rs - https://lobste.rs/s/exvuuc/type_check_your_django_application
Hacker News - https://news.ycombinator.com/item?id=28640033
Reddit - r/python and r/Django
Twitter Thread - https://twitter.com/kracetheking/status/1441329460754595846

1/3. Blog Post of @pyconindia and @europython talk, Type Check your Django App is out. https://t.co/hAWhBljSYD #Python #Django
— kracekumar || கிரேஸ்குமார் (@kracetheking) September 24, 2021

Notes:

Some the django stubs bugs mentioned were captured during prepartion of the talk, while you're reading the blog post bug might be fixed.

Markdown Backticks: TIL

Kracekumar — Sun, 22 Aug 2021 18:07:27 +0000

In markdown spec, content between triple back ticks "

```" is a code fence/code block.


python
var = "this is a python variable"

According to GFM and this blog post, it's possible to have more than three backpacks as long followed by code and equal number of backticks. It's valid to have four backticks and five backticks. What's the use case?


`markdown

### Markdown content

```python

print("Hell of a boring life")

```
### Closing thoughts


`

Note: dev.to doesn't render four backticks properly(mostly they may not follow GFM).

From GFM Spec

The closing code fence must be at least as long as the opening fence:

Source

The outer block can be one type and inner codeblock can be another type. Say, outer block can be markdown and inner block can be Python.

Preview from https://dillinger.io/

Preview from Github

The post initially appeared in my website https://til.kracekumar.com/posts/markdown_backtick/

References

Python Typing Koans

Kracekumar — Sun, 27 Jun 2021 10:30:00 +0000

Python 3 introduced type annotation syntax. PEP 484 introduced a provisional module to provide these standard definitions and tools, along with some conventions for situations where annotations are not available.

Python is a dynamic language and follows gradual typing. When a static type checker runs a python code, the type checker considers code without type hints as Any.

def print_name(name):
       print(name)

planet: str = "earth"

In the above example, the name argument type hint will be Any since the type hint is missing while the type hint for the planet variable is a string.

Gradual typing is still an emerging topic in Python, and there is a gap in resources to educate the Python developers about the utility and Python typing concepts.

On the surface, it looks easy to annotate the code. But the dynamic nature makes a certain part of the code harder to annotate. I have been using type-hints over the past three years and find it hard sometimes. A lot of new developers also face the same problem.

Koans

To make the learning easier, simpler, I have created a GitHub repository.

The repository contains the standalone python programs. The programs contain partial or no type hints. By adding new type hints and fixing the existing type hints, the learner will understand how type-checkers evaluate the types and what's a difference in types at run-time.

Here is a simple demo to use of the command line.

https://asciinema.org/a/419119

Steps

Clone the repository. git clone git@github.com:kracekumar/python-typing-koans.git
Install all the dependencies(advised to use Python Poetry, virtual env should also work.). poetry install. It requires Python 3.9.4
List all the koans using the command line program. poetry run python cli.py list
Pick up a file to learn.
Run the file with the command line program. poetry run python cli.py one koans/py/100-easy-variable-wrong-type.py
Repeat the process till there are no type errors.

One central missing part is how the learner will know to fix the type errors?

The comments in the files carry the links to relevant concepts, which aids the learner in understanding the ideas to use.

Screenshots of a few koans

Topics

Python topics covered are

Primitive Types
dictionaries - dict/typedict
Callables
Design pattern - factory pattern, the builder pattern
Decorators
Type Alias
Protocol, classes, objects

20 Python programs(koans) help the learner to understand gradual typing. The filenames indicate the learning level like easy, medium, and hard.

The repository also contains Django and Django Rest Framework examples.

The Django koans teach the annotating views, models, model methods, queryset methods like filter, all, annotate, aggregate, Q object etc..

The DRF koans teach how to annotate DRF serializers and DRF Views.

If you face any issues while solving the koans, please open an issue in the Github repository; I'd happy to answer and explain the relevant concepts.

Notes

The blog post first appeared in my blog
Photo by Unsplash

Model Field - Django ORM Working - Part 2

Kracekumar — Sun, 27 Jun 2021 08:27:12 +0000

The last post covered the structure of Django Model. This post covers how the model field works, what are the some important methods and functionality and properties of the field.

Object-Relational Mapper is a technique of declaring, querying the database tables using Object relationship in the programming language. Here is a sample model declaration in Django.


class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')

Each class inherits from models.Model becomes a table inside the SQL database unless explicitly marked as abstract. The Question model becomes <app_name>_question table in the database. question_text and pub_date become columns in the table. The properties of the each field are declared by instantiating the respective class. Below is the method resolution order for CharField.

In [5]: models.CharField.mro()
Out[5]:
[django.db.models.fields.CharField,
 django.db.models.fields.Field,
 django.db.models.query_utils.RegisterLookupMixin,
 object]

CharField inherits Field and Field inherits RegisterLookUpMixin.

High-level role of Field class

The role of field class is to map type of the field to SQL database type.
Serialization - to convert the Python object into relevant SQL database value.
DeSerialization - to convert the SQL database value into Python object.
Check declared validations at the field level and built-in checks before serializing the data. For example, in a PositiveIntegerField the value should be greater than zero - built-in constraint.

Structure of Field Class

# Find out all the classes inheriting the Field

In [7]: models.Field.__subclasses__()
Out[7]:
[django.db.models.fields.BooleanField,
 django.db.models.fields.CharField,
 django.db.models.fields.DateField,
 django.db.models.fields.DecimalField,
 django.db.models.fields.DurationField,
 django.db.models.fields.FilePathField,
 django.db.models.fields.FloatField,
 django.db.models.fields.IntegerField,
 django.db.models.fields.IPAddressField,
 django.db.models.fields.GenericIPAddressField,
 django.db.models.fields.TextField,
 django.db.models.fields.TimeField,
 django.db.models.fields.BinaryField,
 django.db.models.fields.UUIDField,
 django.db.models.fields.json.JSONField,
 django.db.models.fields.files.FileField,
 django.db.models.fields.related.RelatedField,
 django.contrib.postgres.search.SearchVectorField,
 django.contrib.postgres.search.SearchQueryField,
 fernet_fields.fields.EncryptedField,
 enumchoicefield.fields.EnumChoiceField,
 django.contrib.postgres.fields.array.ArrayField,
 django.contrib.postgres.fields.hstore.HStoreField,
 django.contrib.postgres.fields.ranges.RangeField]

Here fernet_fields is a third-party library which implements the EncryptedField by inheriting the Field class. Also these are high level fields. For example, Django implements other high-level fields which inherit the above fields.

For example, EmailField inherits the CharField.

In [10]: models.CharField.__subclasses__()
Out[10]:
[django.db.models.fields.CommaSeparatedIntegerField,
 django.db.models.fields.EmailField,
 django.db.models.fields.SlugField,
 django.db.models.fields.URLField,
 django.contrib.postgres.fields.citext.CICharField,
 django_extensions.db.fields.RandomCharField,
 django_extensions.db.fields.ShortUUIDField]

Here is the Field class initializer signature

In [11]: models.Field?
Init signature:
models.Field(
    verbose_name=None,
    name=None,
    primary_key=False,
    max_length=None,
    unique=False,
    blank=False,
    null=False,
    db_index=False,
    rel=None,
    default=<class 'django.db.models.fields.NOT_PROVIDED'>,
    editable=True,
    serialize=True,
    unique_for_date=None,
    unique_for_month=None,
    unique_for_year=None,
    choices=None,
    help_text='',
    db_column=None,
    db_tablespace=None,
    auto_created=False,
    validators=(),
    error_messages=None,
)

The Field initializer contains 22 arguments. Most of the arguments are related to SQL database column properties and rest of the arguments are for Django admin and model forms.

For example, Django provides Admin interface to browse the database records and allows you to edit. blank parameter determines whether the field is required while filling up data in the admin interface and custom form. help_text field is used while display the form.

The most commonly used fields are max_length, unique, blank, null, db_index, validators, default, auto_created. null attribute is a boolean type when set to True, the allows null value while saving to the database. db_index=True created a B-Tree index on the column. default attribute stores the default value passed on to the database, when the value for the field is missing.

validators attribute contains list of validators passed on by the user and Django's internal validators. The function of the validator is to determine the value is valid or not. For example, in our question_text field declaration max_length is set to 200. When the field value is greater than 200, Django raises ValidationError. max_length attribute is useful only for text field and MaxLengthValidator will be missing in non-text fields.

In [29]: from django.core.exceptions import ValidationError

In [30]: def allow_odd_validator(value):
    ...:     if value % 2 == 0:
    ...:         raise ValidationError(f'{value} is not odd number')
    ...:

In [31]: int_field = models.IntegerField(validators=[allow_odd_validator])

In [32]: int_field.validators
Out[32]:
[<function __main__.allow_odd_validator(value)>,
 <django.core.validators.MinValueValidator at 0x1305fdac0>,
 <django.core.validators.MaxValueValidator at 0x1305fda30>]

In [33]: # let's look into question_text field validators

In [38]: question_text.validators
Out[38]: [<django.core.validators.MaxLengthValidator at 0x12e767fa0>]

As long as the validator function or custom class doesn't raise exception, the value is considered as valid.

The details of each field can found in the Django model field reference

Field Methods

In [41]: import inspect

In [44]: len(inspect.getmembers(models.Field, predicate=inspect.isfunction))
Out[44]: 59

In [45]: len(inspect.getmembers(models.Field, predicate=inspect.ismethod))
Out[45]: 6

The Field class consists of (along with inherited ones) 65 methods. Let's look at some of the important ones.

to_python

to_python method is responsible to convert the value passed on to the model during intialization. For example, to_python for IntegerField will convert the value to Python integer. The original value could be string or float. Every field will override to_python method. Here is an example of to_python method invocation on an IntegerField.

In [46]: int_field.to_python
Out[46]: <bound method IntegerField.to_python of <django.db.models.fields.IntegerField>>

In [47]: int_field.to_python('23')
Out[47]: 23

In [48]: int_field.to_python(23)
Out[48]: 23

In [49]: int_field.to_python(23.56)
Out[49]: 23

get_db_prep_value

get_db_prep_value method is responsible to convert Python value to SQL database specific value. Each field may have a different implementation depending on field type. For example, Postgres has a native UUID type, whereas in SQLite and MySQL Django uses varchar(32). Here is the implementation for get_db_prep_value from UUIDField.

def get_db_prep_value(self, value, connection, prepared=False):
    if value is None:
    return None
  if not isinstance(value, uuid.UUID):
    value = self.to_python(value)

  if connection.features.has_native_uuid_field:
    return value
  return value.hex

connection is a Database Connection or Wrapper object of underlying database. Below is an example output from a Postgres Connection and SQLite Connection for uuid field check.

In [50]: from django.db import connection
    ...:

In [51]: connection
Out[51]: <django.utils.connection.ConnectionProxy at 0x10e3c8970>

In [52]: connection.features
Out[52]: <django.db.backends.postgresql.features.DatabaseFeatures at 0x1236a6a00>

In [53]: connection.features.has_native_uuid_field
Out[53]: True

In [1]: from django.db import connection

In [2]: connection
Out[2]: <django.utils.connection.ConnectionProxy at 0x10fe3b4f0>

In [3]: connection.features
Out[3]: <django.db.backends.sqlite3.features.DatabaseFeatures at 0x110ba5d90>

In [4]: connection.features.has_native_uuid_field
Out[4]: False

One thing to note, Django uses psycopg2 driver for Postgres and it will take care of handling UUID specific to Postgres because UUID Python object needs to be converted to string or bytes before sending to the Postgres server.

Similar to get_db_prep_value, get_prep_value which converts Python value to query value.

formfield

Django supports ModelForm which is one to one mapping of HTML form to Django model. The Django admin uses ModelForm . The form consists of several fields. Each field in the form maps to field in the model. So Django can automatically construct the form with a list of validators from the model field.

Here is the implementation for the UUIDField.

def formfield(self, **kwargs):
    return super().formfield(**{
    'form_class': forms.UUIDField,
    **kwargs,
   })

When you create a custom database field, you need to create a custom form field to work with Django admin and pass it as an argument to super class method.

deconstruct

deconstruct method returns value for creating an exact copy of the field. The method returns a tuple with 4 values.

The first value is the name of the field passed during initialisation. The default value is None.
The import path of the field.
The list of positonal arguments passed during the field creation.
The dictionary of keyword arguments passed during the field creation.

In [62]: # Let's see the question_text deconstruct method return value

In [63]: question_text.deconstruct()
Out[63]: (None, 'django.db.models.CharField', 
[], {'max_length': 200})

In [65]: # let's create a new integer field with a name

In [66]: int_field = models.IntegerField(name='int_field', validators=[allow_odd_validator])

In [67]: int_field.deconstruct()
Out[67]:
('int_field',
 'django.db.models.IntegerField',
 [],
 {'validators': [<function __main__.allow_odd_validator(value)>]})

In [68]: models.IntegerField(**int_field.deconstruct()[-1])
Out[68]: <django.db.models.fields.IntegerField>

In [69]: int_2_field = models.IntegerField(default=2)

In [70]: int_2_field.deconstruct()
Out[70]: (None, 'django.db.models.IntegerField', 
[], {'default': 2})

Also when you implement a custom field, you can override the deconstruct method. Here is the deconstruct implementation for UUIDField.

def deconstruct(self):
    name, path, args, kwargs = super().deconstruct()
    del kwargs['max_length']
    return name, path, args, kwargs

init

__init__ method is a good place to override some of the default values. For example, UUIDField max_length should always be 32 irrespective of the value passed on. In the decimal field, max_digits can be modified during initialization.

Here is the UUIDField initializer method implementation.

def __init__(self, verbose_name=None, **kwargs):
    kwargs['max_length'] = 32
    super().__init__(verbose_name, **kwargs)

db_type

db_type method takes Django connection as an argument and returns the database specific implementation type for this field. The method takes connection as an argument. Here is the output of db_type for Postgres and SQLite.

In [72]: # Postgres

In [73]: uuid_field = models.UUIDField()

In [74]: uuid_field.db_type(connection)
Out[74]: 'uuid'

In [8]: # Sqlite

In [9]: uuid_field = models.UUIDField()

In [10]: uuid_field.rel_db_type(connection)
Out[10]: 'char(32)'

get_internal_type method returns internal Python type which is companion to the db_type method. In practice, Django fields type and database field mapping is maintained as class variable in DatabaseWrapper. You can find, Django fields and Postgres fields mapping in backends module. Below is the mapping taken from source code.

class DatabaseWrapper(BaseDatabaseWrapper):
    vendor = 'postgresql'
    display_name = 'PostgreSQL'
    # This dictionary maps Field objects to their associated PostgreSQL column
    # types, as strings. Column-type strings can contain format strings; they'll
    # be interpolated against the values of Field.__dict__ before being output.
    # If a column type is set to None, it won't be included in the output.
    data_types = {
        'AutoField': 'serial',
        'BigAutoField': 'bigserial',
        'BinaryField': 'bytea',
        'BooleanField': 'boolean',
        'CharField': 'varchar(%(max_length)s)',
        'DateField': 'date',
        'DateTimeField': 'timestamp with time zone',
        'DecimalField': 'numeric(%(max_digits)s, %(decimal_places)s)',
        'DurationField': 'interval',
        'FileField': 'varchar(%(max_length)s)',
        'FilePathField': 'varchar(%(max_length)s)',
        'FloatField': 'double precision',
        'IntegerField': 'integer',
        'BigIntegerField': 'bigint',
        'IPAddressField': 'inet',
        'GenericIPAddressField': 'inet',
        'JSONField': 'jsonb',
        'OneToOneField': 'integer',
        'PositiveBigIntegerField': 'bigint',
        'PositiveIntegerField': 'integer',
        'PositiveSmallIntegerField': 'smallint',
        'SlugField': 'varchar(%(max_length)s)',
        'SmallAutoField': 'smallserial',
        'SmallIntegerField': 'smallint',
        'TextField': 'text',
        'TimeField': 'time',
        'UUIDField': 'uuid',
    }

get_internal_type values are keys and values are Postgres field names.

I have skipped the implementation of the rest of the methods like __reduce__ and check . You can go through the source code of Django fields in GitHub and also you will find class variables and private methods usages.

Django documentation has an excellent page on how-to write custom model field.

Summary

models.Field is the root of all the model fields.
Field initializer takes configuration details like name, default, db_index, null for the database columns and blank, help_text for non-column features like Django model form and Django admin.
__init__ method in the child class can override the user passed value and(or) set custom default value.
validators attribute in the field contains the user-defined validators and default validators specific to the field.
Every field needs to implement a few methods to work with the specific databases. Some of the useful methods are to_python, get_db_prep_value, get_prep_value, deconstruct, formfield, db_type.
Django Connection object or wrapper contains details and features of the underlying the database.

Notes

Cover Photo by Unsplash
The post first appeared in my blog.

Structure - Django ORM Working - Part 1

Kracekumar — Sun, 27 Jun 2021 07:39:59 +0000

Django ORM hides a lot of complexity while developing the web application. The data model declaration and querying pattern are simplified, whereas it's structured differently behind the scenes. The series of blog posts will explain Django ORM working(not just converting Python code to SQL), model declaration, querying (manager, queryset), supporting multiple drivers, writing custom queries, migrations etc...

Consider a model definition from the Django tutorial.

from django.db import models


class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')


class Choice(models.Model):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    choice_text = models.CharField(max_length=200)
    votes = models.IntegerField(default=0)

The Question and Choice model class derives from models.Model. Inheriting Model signals Django at run-time the class is a database model. Question model(later converted to a table) contains two extra class variables, question_text and pub_date, which will be two columns in the table. Their type is indicated by creating an instance of the respective type of fields, here models.CharField, models.DateTimeField. A similar work applies to the Choice model.

Querying

Let's display all the instances of Question (all the rows in the table - polls_question).

>>> Question.objects.all()
<QuerySet []>

As expected, the table is empty, and the method call returns empty result. The two things to notice are objects.all() and QuerySet. All the data entry in and out of the database through Django ORM happens through the object's interface. All the results are wrapped inside the QuerySet, even empty ones.

From where does the `objects` instance come?

>>> class Foo:
...     pass
...
>>> class Bar(Foo):
...     pass
...
>>> Bar.mro()
[<class 'Bar'>, <class 'Foo'>, <class 'object'>]

Every Python class has a Method Resolution Order, which determines the behavior of the class/method invocation. In the above example, the Bar class inherits the Foo class. The mro method returns the order from child to patent class, Bar inherits Foo, Foo inherits object. As you can see, the Foo class has no inherited class in the code, but the mro method says Foo inherits object. All the Python class inherits from the object; hence mro method returns object as the last parent in the list.

>>> Question.mro()
[<class 'polls.models.Question'>, 
<class 'django.db.models.base.Model'>, 
<class 'object'>]

MRO for Question is clear and returns result as expected.

>>> Bar
<class 'Bar'>
>>> Question
<class 'polls.models.Question'>
>>> Question.objects
<django.db.models.manager.Manager object at 0x10bd7f1c0>
>>> Question.objects.mro()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
AttributeError: 'Manager' object has no attribute 'mro'
>>> Question.objects.__class__.mro()
[<class 'django.db.models.manager.Manager'>, <class 'django.db.models.manager.BaseManagerFromQuerySet'>, <class 'django.db.models.manager.BaseManager'>, <class 'object'>]

The representation of Question.objects is different from the representation of Bar and Question classes. As the name indicates, objects is an instance of the Manager. The objects, the instance of Manager class, inherits BaseManagerFromQuerySet and BaseManager.

>>> Choice
<class 'polls.models.Choice'>
>>> Choice.objects
<django.db.models.manager.Manager object at 0x10bd7f0a0>
>>> Question.objects
<django.db.models.manager.Manager object at 0x10bd7f1c0>
>>> Choice.objects is Question.objects
False
>>> Choice.objects == Question.objects
True

What? Even though the instance uses different id but equality test returns True.

# It's decalared in manager class

def __eq__(self, other):
        return (
            isinstance(other, self.__class__) and
            self._constructor_args == other._constructor_args
        )

The logic for checking the equality has two pieces - both operands should be of the same type and their constructor args are same. In this case(behind the scenes), both the managers were called with empty arguments.

>>> Question.objects._constructor_args
((), {})
>>> Choice.objects._constructor_args
((), {})

In the next post, I'll cover how models.*Field works.

Summary

The Django model inherits model.Model and all the class variables initialized with model.*Field automatically behaves like a column in the table.
The interactions to the database happen through ModelManager via objects attribute.

Notes:

This post initially appeared in my blog.
Cover Photo by Unsplash

DEV Community: Kracekumar

Profiling Django App

TL:DR

🔬 What's Profiling? 🔬

🚀 Why Profile Django code? 🚀

Application Setup

Models

Views

Serializers

🚆 Pyinstrument 🚆

How to use it?

Output

Upside

Downside

🧮 Django Query Count 🧮

How to use it?

Ouptut

Upside

Downside

🔥 Django Silk 🔥

Installation

How to use it?

Output

Other features

Upside

Downside

Conclusion

Image Credits

Generate HTTP Request

HTTP Syntax

🉑 Accept only JSON response data 🉑

📚 Request the Tamil language version of duckduckgo 📚

🔑 Post Authorization token as part of login 🔑

🍪 Send Cookie as part of a request 🍪

✉️ Send JSON Request ✉️

✉️ Send non-primitive JSON values ✉️

📤 Upload files 📤

🐈 How to verify the the generated request works? 🐈

References

Type Check Your Django Application

Gradual Typing

Types

Types at Run Time

Types at Static Checker Time

Mypy config

Annotation Syntax

Annotating Django Code

Views

Why and How? MRO

LSP - Liskov substitution principle

Django Models

Create

Read

Two solutions

Filter method

Aggregate

Annotate Method

How to fix these two errors?

Option 1 - Recommended

Option 2 - Good solution

Tools

Pyannotate

Phase 0 - Preparation

Phase 1 - Invoking Pyannotate

Phase 2 - Apply the annotations

Python Typing Koans

Conclusion

References

Images References

Discussions

Markdown Backticks: TIL

References

Python Typing Koans

Koans

Steps

Topics

Links

Notes

Model Field - Django ORM Working - Part 2

High-level role of Field class

From where does the `objects` instance come?