Type Annotate an existing Python Django Codebase with MonkeyType

Why Add Types?

There has been quite a buzz around type annotations in the recent past, so you may ask yourself if type checking is something you should do. This article is not a general introduction to type annotations and type checks, I recommend Hypermodern Python: Typing to get an overview of the background.

In a nutshell citing the above article:
"Type annotations are a way to annotate functions and variables with types. Combined with tooling that understands them, they can make your programs easier to understand, debug, and maintain. A static type checker can use type annotations and type inference to verify the type correctness of your program without executing it, helping you discover many bugs that would otherwise go unnoticed." or Dropbox puts it like this: "If you aren’t using type checking in your large-scale Python project, now is a good time to get started — nobody who has made the jump I’ve talked to has regretted it. It really makes Python a much better language for large projects."

In addition to the above the tooling around type annotations is growing continuously and helps to keep your code DRY.
There are libraries like Desert or Typical to help with data validation, Typeguard or strongtyping to check types at runtime, sphinx-autodoc-typehints to help build documentation, Typer to build CLIs, FastAPI a web framework for building APIs, and even transpilers to compile your python code with Mypy to Python C or Rust.

For a more complete list please refer to Awesome Python Typing

Do Not Let Legacy Code Become Technical Debt

As Dropbox describes in Our journey to type checking 4 million lines of Python it is quite daunting to type annotate an existing codebase. The article describes in depth how you can approach gradual improvements with type annotations to your codebase.

Enter MonkeyType

But why do I have to write my annotations manually? After all Python knows what type a variable is at runtime. The folks at Instagram thought the same and created MonkeyType. A similar projects is PyAnnotate created by Dropbox. Pytype generates files of inferred type information, located by default in .pytype/pyi. Pyre has a powerful feature for migrating codebases to a typed format. The infer command-line option ingests a file or directory, makes educated guesses about the types used, and applies the annotations to the files.

MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type annotations directly to your Python code based on the types collected at runtime.

Have a look at the Introduction to MonkeyType for an overview of its functionality or listen to the MonkeyType podcast.

Setup MonkeyType For a Django Project With PyTest

While you can run MonkeyType in a production like environment and get real world data on how you code is called, the most common use case is probably to get this data out of test runs. If you do not use pytest with django Testing Your Django App With Pytest should get you started.

Install the required dependencies by creating a file monkeytype.txt with the contents

mypy
mypy-extensions
MonkeyType
pytest-monkeytype
django-stubs
djangorestframework-stubs

and install the dependencies with pip install -r monkeytype.txt As of writing this pytest-monkeytype needs MonkeyType==19.11.2, this is fixed in this Pull Request

Configure mypy with mypy.ini e.g.

[mypy]
disallow_any_generics = True
disallow_untyped_calls = True
disallow_untyped_defs = True
disallow_untyped_decorators = True
ignore_errors = False
ignore_missing_imports = True
implicit_reexport = False
strict_optional = True
strict_equality = True
no_implicit_optional = True
warn_unused_ignores = True
warn_redundant_casts = True
warn_unused_configs = True
warn_unreachable = True
warn_no_return = True
warn_return_any = True

plugins =
    mypy_django_plugin.main,
    mypy_drf_plugin.main

[mypy.plugins.django-stubs]
django_settings_module = "test_settings"

[mypy-drfdapc.test_permissions]
ignore_errors = True

Create a monkeytype_config.py file:

# Standard Library
import os
from contextlib import contextmanager
from typing import Iterator

# 3rd-party
from monkeytype.config import DefaultConfig


class MonkeyConfig(DefaultConfig):
    @contextmanager
    def cli_context(self, command: str) -> Iterator[None]:
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_settings")
        import django

        django.setup()
        yield


CONFIG = MonkeyConfig()

Change the DJANGO_SETTINGS_MODULE to the one that is used in tests.
Now you can run your tests with pytest --monkeytype-output=./monkeytype.sqlite3 and export MT_DB_PATH=./monkeytype.sqlite3 on Bash, or setenv MT_DB_PATH ./monkeytype.sqlite3 on C Shell.

Check which modules MonkeyType has collected infomation for with monkeytype list-modules.

drfdapc$ monkeytype list-modules
drfdapc.permissions
drfdapc.test_permissions

You can then use the monkeytype stub some.module command to generate a stub file for a module, or apply the type annotations directly to your code with monkeytype apply some.module.

drfdapc$ monkeytype stub drfdapc.permissions
from django.core.handlers.wsgi import WSGIRequest
from unittest.mock import Mock


def allow_all(*args, **kwargs) -> bool: ...


def allow_authenticated(request: WSGIRequest, *args, **kwargs) -> bool: ...


def allow_authorized_key(
    request: WSGIRequest,
    view: Mock,
    *args,
    **kwargs
) -> bool: ...


def allow_staff(request: WSGIRequest, *args, **kwargs) -> bool: ...


def allow_superuser(request: WSGIRequest, *args, **kwargs) -> bool: ...


def deny_all(*args, **kwargs) -> bool: ...


class DABasePermission:
    def has_object_permission(
        self,
        request: WSGIRequest,
        view: None,
        obj: Mock
    ) -> bool: ...
    def has_permission(self, request: WSGIRequest, view: None) -> bool: ...


class DACrudBasePermission:
    def has_object_permission(
        self,
        request: WSGIRequest,
        view: None,
        obj: Mock
    ) -> bool: ...
    def has_permission(self, request: WSGIRequest, view: None) -> bool: ...


class DARWBasePermission:
    def has_object_permission(
        self,
        request: WSGIRequest,
        view: None,
        obj: Mock
    ) -> bool: ...
    def has_permission(self, request: WSGIRequest, view: None) -> bool: ...

Checking the annotations with mypy shows that there is still some work to do and it demonstrates nicely which errors mypy catches.

drfdapc$ mypy drfdapc/permissions.py
...
drfdapc/permissions.py:71: error: Function is missing a type annotation for one or more arguments
drfdapc/permissions.py:86: error: Untyped decorator makes function "allow_superuser" untyped
...
drfdapc/permissions.py:162: error: Argument 2 of "has_permission" is incompatible with supertype "BasePermission"; supertype defines the argument type as "View"
drfdapc/permissions.py:162: note: This violates the Liskov substitution principle
drfdapc/permissions.py:162: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
...
Found 17 errors in 1 file (checked 1 source file)

Remember that MonkeyType’s annotations are an informative first draft, to be checked and corrected by a developer.
In the above example we never pass a view, all objects are mocked and WSGIRequest is too specific, so we need to adjust this. Also run black and isort on this file to have a nicely formatted code. You can see the workflow in this pull request

Start with small modules that have few (or even better no) dependencies on other parts of your code, apply the types, check the correctness of the annotations with mypy some/module.py fix the typing information, and move on to the next module.

Little Helpers

You will want to keep your type information clean, readable and consistent. To achieve this there are some plugins for flake8.

flake8-annotations-complexity reports on too complex, hard to read type annotations. Complex type annotations often means bad annotation usage, wrong code decomposition or improper data structure choice.

flake8-annotations-coverage reports on files with a lot of code without type annotations. This is mostly useful when you add type annotations to existing large codebase and want to know if new code in annotated modules is annotated.

flake8-type-annotations is used to validate type annotations syntax as it was originally proposed

flake8-annotations detects the absence of function annotations.
What this won't do: Check variable annotations respect stub files, or replace mypy.

Comments

[Christian Ledermann]

This looks also interesting
Typilus: A GitHub action that suggests type annotations for Python using machine learning.

[Christian Ledermann]

Type Check Your Django Application - An article based on two recent talks on adding type checks to Django. This also discusses adding type annotations with Pyannotate