Why Add Types?
There has been quite a buzz around type annotations in the recent past, so you may ask yourself if type checking is something you should do. This article is not a general introduction to type annotations and type checks, I recommend Hypermodern Python: Typing to get an overview of the background.
In a nutshell citing the above article:
"Type annotations are a way to annotate functions and variables with types. Combined with tooling that understands them, they can make your programs easier to understand, debug, and maintain. A static type checker can use type annotations and type inference to verify the type correctness of your program without executing it, helping you discover many bugs that would otherwise go unnoticed." or Dropbox puts it like this: "If you aren’t using type checking in your large-scale Python project, now is a good time to get started — nobody who has made the jump I’ve talked to has regretted it. It really makes Python a much better language for large projects."
In addition to the above the tooling around type annotations is growing continuously and helps to keep your code DRY.
There are libraries like Desert or Typical to help with data validation, Typeguard or strongtyping to check types at runtime, sphinx-autodoc-typehints to help build documentation, Typer to build CLIs, FastAPI a web framework for building APIs, and even transpilers to compile your python code with Mypy to Python C or Rust.
For a more complete list please refer to Awesome Python Typing
Do Not Let Legacy Code Become Technical Debt
As Dropbox describes in Our journey to type checking 4 million lines of Python it is quite daunting to type annotate an existing codebase. The article describes in depth how you can approach gradual improvements with type annotations to your codebase.
But why do I have to write my annotations manually? After all Python knows what type a variable is at runtime. The folks at Instagram thought the same and created MonkeyType. A similar projects is PyAnnotate created by Dropbox. Pytype generates files of inferred type information, located by default in
.pytype/pyi. Pyre has a powerful feature for migrating codebases to a typed format. The infer command-line option ingests a file or directory, makes educated guesses about the types used, and applies the annotations to the files.
MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type annotations directly to your Python code based on the types collected at runtime.
Have a look at the Introduction to MonkeyType for an overview of its functionality or listen to the MonkeyType podcast.
Setup MonkeyType For a Django Project With PyTest
While you can run MonkeyType in a production like environment and get real world data on how you code is called, the most common use case is probably to get this data out of test runs. If you do not use pytest with django Testing Your Django App With Pytest should get you started.
Install the required dependencies by creating a file
monkeytype.txt with the contents
mypy mypy-extensions MonkeyType pytest-monkeytype django-stubs djangorestframework-stubs
and install the dependencies with
pip install -r monkeytype.txt As of writing this
MonkeyType==19.11.2, this is fixed in this Pull Request
Configure mypy with
[mypy] disallow_any_generics = True disallow_untyped_calls = True disallow_untyped_defs = True disallow_untyped_decorators = True ignore_errors = False ignore_missing_imports = True implicit_reexport = False strict_optional = True strict_equality = True no_implicit_optional = True warn_unused_ignores = True warn_redundant_casts = True warn_unused_configs = True warn_unreachable = True warn_no_return = True warn_return_any = True plugins = mypy_django_plugin.main, mypy_drf_plugin.main [mypy.plugins.django-stubs] django_settings_module = "test_settings" [mypy-drfdapc.test_permissions] ignore_errors = True
# Standard Library import os from contextlib import contextmanager from typing import Iterator # 3rd-party from monkeytype.config import DefaultConfig class MonkeyConfig(DefaultConfig): @contextmanager def cli_context(self, command: str) -> Iterator[None]: os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_settings") import django django.setup() yield CONFIG = MonkeyConfig()
DJANGO_SETTINGS_MODULE to the one that is used in tests.
Now you can run your tests with
pytest --monkeytype-output=./monkeytype.sqlite3 and
export MT_DB_PATH=./monkeytype.sqlite3 on Bash, or
setenv MT_DB_PATH ./monkeytype.sqlite3 on C Shell.
Check which modules MonkeyType has collected infomation for with
drfdapc$ monkeytype list-modules drfdapc.permissions drfdapc.test_permissions
You can then use the
monkeytype stub some.module command to generate a stub file for a module, or apply the type annotations directly to your code with
monkeytype apply some.module.
drfdapc$ monkeytype stub drfdapc.permissions from django.core.handlers.wsgi import WSGIRequest from unittest.mock import Mock def allow_all(*args, **kwargs) -> bool: ... def allow_authenticated(request: WSGIRequest, *args, **kwargs) -> bool: ... def allow_authorized_key( request: WSGIRequest, view: Mock, *args, **kwargs ) -> bool: ... def allow_staff(request: WSGIRequest, *args, **kwargs) -> bool: ... def allow_superuser(request: WSGIRequest, *args, **kwargs) -> bool: ... def deny_all(*args, **kwargs) -> bool: ... class DABasePermission: def has_object_permission( self, request: WSGIRequest, view: None, obj: Mock ) -> bool: ... def has_permission(self, request: WSGIRequest, view: None) -> bool: ... class DACrudBasePermission: def has_object_permission( self, request: WSGIRequest, view: None, obj: Mock ) -> bool: ... def has_permission(self, request: WSGIRequest, view: None) -> bool: ... class DARWBasePermission: def has_object_permission( self, request: WSGIRequest, view: None, obj: Mock ) -> bool: ... def has_permission(self, request: WSGIRequest, view: None) -> bool: ...
Checking the annotations with mypy shows that there is still some work to do and it demonstrates nicely which errors mypy catches.
drfdapc$ mypy drfdapc/permissions.py ... drfdapc/permissions.py:71: error: Function is missing a type annotation for one or more arguments drfdapc/permissions.py:86: error: Untyped decorator makes function "allow_superuser" untyped ... drfdapc/permissions.py:162: error: Argument 2 of "has_permission" is incompatible with supertype "BasePermission"; supertype defines the argument type as "View" drfdapc/permissions.py:162: note: This violates the Liskov substitution principle drfdapc/permissions.py:162: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides ... Found 17 errors in 1 file (checked 1 source file)
Remember that MonkeyType’s annotations are an informative first draft, to be checked and corrected by a developer.
In the above example we never pass a view, all objects are mocked and
WSGIRequest is too specific, so we need to adjust this. Also run
isort on this file to have a nicely formatted code. You can see the workflow in this pull request
Start with small modules that have few (or even better no) dependencies on other parts of your code, apply the types, check the correctness of the annotations with
mypy some/module.py fix the typing information, and move on to the next module.
You will want to keep your type information clean, readable and consistent. To achieve this there are some plugins for flake8.
flake8-annotations-complexity reports on too complex, hard to read type annotations. Complex type annotations often means bad annotation usage, wrong code decomposition or improper data structure choice.
flake8-annotations-coverage reports on files with a lot of code without type annotations. This is mostly useful when you add type annotations to existing large codebase and want to know if new code in annotated modules is annotated.
flake8-type-annotations is used to validate type annotations syntax as it was originally proposed
flake8-annotations detects the absence of function annotations.
What this won't do: Check variable annotations respect stub files, or replace mypy.
Top comments (2)
Type Check Your Django Application - An article based on two recent talks on adding type checks to Django. This also discusses adding type annotations with Pyannotate
This looks also interesting
Typilus: A GitHub action that suggests type annotations for Python using machine learning.