Modern Python setup for quality development

In this article, i'll show you what is my common python setup.

Goals are:

• High quality code standard
• Respect of the python standard
• Dev experience quality
• Security
• repeatability

We'll see:

• The IDE
• Python versions
• Package and virtual environments
• Code linter/formatter
• Tests tools
• Documentation
• Pre-commit
• Security

This is entirely subjective :)

IDE: VSCode

VSCode is a great IDE for python programming.

One of his best feature is his extensibility though extensions.

Here are some extensions i used:

For the general development environment

• Remote-Container: Open any folder or repository inside a Docker container and take advantage of Visual Studio Code's full feature set. Great to ensure every dev environment through a team are identical.

• Python: Python IntelliSense (Pylance), Linting, Debugging (multi-threaded, remote), Jupyter Notebooks, code formatting, refactoring, unit tests, and more. I don't use Jupyter so i cant talk about it. However Pylance is a great tool. Under the hood it used PyRights which is a super fast code analyzer (way faster than mypy!). Don't forget add this to your settings:

• vscode-icons: Each folder type have his own well design icons. Big project with a lots of files are way more easy to understand with that.

• Window Color: Automatically adds a unique color to each window's activityBar and titleBar. A project will always have the same color. Great when a teams works on multiple project, everybody see the same color.

• GitLens: Now you can see who wrote that directly into the code

For data parsing/formatting/transformation...

There's lots of extensions for that depending on your use case (docker, helm, etc...)

I think these one are required by pretty much every project:

• Better TOML
• change-case
• Code Spell checker
• JSON Tools
• XML Tools
• YAML
• vscode-base64
• Sort JSON objects

Python versions: Pyenv

Pyenv is so far the best python versions manager.

Python package and venv: Poetry

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.

Basically, it's pip under steroids. It works with the latest package file format pyproject.toml . His dependencies and build system are very reliable. You can still export requirements.txt if necessary. Virtual environments works like a charms. Multiple python version too. It solve a lot of pain point. This is a must have.

Code linter/formatter

• flake8: Flake8 is a wrapper around these tools: PyFlakes pycodestyle Ned Batchelder's McCabe script

Flake8 runs all the tools by launching the single flake8 command. It displays the warnings in a per-file, merged output.

• black: Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters.

Blackened code looks the same regardless of the project you're reading. Formatting becomes transparent after a while and you can focus on the content instead.

Black makes code review faster by producing the smallest diffs possible.

• isort: isort your imports, so you don't have to.

For the tests

• pytest: Python allow to write way less code than using directly unittest. Lots of good library handle it very well. Some library extend it to be even more efficient and user friendly (pytest-mock, pytest-freezegun, pytest-sugar...)

• nox: The first point of nox is to run your tests into multiple environment (multiple python version). In my opinion way better than tox

For the documentation: Mkdocs

Mkdocs allow you to write documentation using markdown format. Really easy to use and integrate. You can easily integrate your documentation to your CI/CD pipeline for automatically test it then deploy it.

Pre-commit

This is a must have. pre-commit allow you to run multiple scripts (a.k.a pre-commit) before your commit validation. There's a big catalogue of builtin pre-commit. Some will modify files. Some will just return warning or errors. You can automatically perform a TONS of tests/validation before committing anything.

This is basically a local CI pipeline run before the real one.

Here is an example of one .pre-commit-config.yaml config file i use, among others it will:

• sort your import automatically
• black will format your code
• flake8 will warn you about style issue
• Validate .gitlab-ci.yaml against my enterprise gitlab server.
• check style for bash script
• and so much more...

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.0.1
    hooks:
      - id: check-added-large-files
      - id: check-ast
      - id: check-builtin-literals
      - id: check-case-conflict
      - id: check-docstring-first
      - id: check-executables-have-shebangs
      - id: check-json
      - id: check-merge-conflict
      - id: check-symlinks
      - id: check-toml
      - id: check-vcs-permalinks
      - id: check-xml
      - id: check-yaml
        args: [--allow-multiple-documents]
      - id: debug-statements
      - id: detect-aws-credentials
        args: [--allow-missing-credentials]
      - id: destroyed-symlinks
      - id: end-of-file-fixer
      - id: fix-byte-order-marker
      - id: fix-encoding-pragma
        args: [--remove]
      - id: forbid-new-submodules
      - id: mixed-line-ending
        args: [--fix=auto]
      - id: name-tests-test
        args: [--django]
      - id: requirements-txt-fixer
      - id: trailing-whitespace
  - repo: local
    hooks:
      - id: black
        name: black
        entry: poetry run black
        language: system
        types: [python]
      - id: flake8
        name: flake8
        entry: poetry run flake8
        language: system
        types: [python]
  - repo: https://github.com/pycqa/isort
    rev: "5.9.1"
    hooks:
      - id: isort
        args:
          - --profile
          - black
          - --filter-files
  - repo: https://github.com/adrienverge/yamllint.git
    rev: v1.26.1
    hooks:
      - id: yamllint
        args: [-c=.yamllint.yaml]
  - repo: https://gitlab.com/devopshq/gitlab-ci-linter
    rev: v1.0.2
    hooks:
      - id: gitlab-ci-linter
        args:
          - "--server"
          - "https://your.gitlab.server" # Need env var GITLAB_PRIVATE_TOKEN with gitlab api read token
  - repo: https://github.com/commitizen-tools/commitizen
    rev: v2.17.11
    hooks:
      - id: commitizen
        stages: [commit-msg]
  - repo: https://github.com/jumanjihouse/pre-commit-hooks
    rev: 2.1.5 # or specific git tag
    hooks:
      - id: forbid-binary
      - id: shellcheck
      - id: shfmt

Security

• bandit: Bandit is a tool designed to find common security issues in Python code.

To do this Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files it generates a report.

• safety: Safety checks your installed dependencies for known security vulnerabilities.

By default it uses the open Python vulnerability database Safety DB, but can be upgraded to use pyup.io's Safety API using the --key option.

• gitleaks: Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.

Job is done but if someone have better alternative, i would try them.

Conclusion

I hope you leaned a thing or two in this article.

Using these tools can help you falling in some common pitfall.

Pylance warning & errors are great best practices teachers. Your code understanding and therefore quality will greatly progress understanding and correcting them.

Comments

[slidenerd]

Unpopular opinion: python should get rid of all virtual environment stuff from their codebase and let Docker take care of it

[Kalob Taulien]

Love this article! pyenv is the best for changing Python versions - until you need to change your Python paths for any reason (like if homebrew updates on a mac, for instance) then it becomes one of the biggest Python nightmares.

If anyone has ever run into that issue with Pyenv, try using Docker. You can use a slim Python image to keep it small, too.

[Richard Devers]

Thanks!