In this article, i'll show you what is my common python setup.
Goals are:
- High quality code standard
- Respect of the python standard
- Dev experience quality
- Security
- repeatability
We'll see:
- The IDE
- Python versions
- Package and virtual environments
- Code linter/formatter
- Tests tools
- Documentation
- Pre-commit
- Security
This is entirely subjective :)
IDE: VSCode
VSCode is a great IDE for python programming.
One of his best feature is his extensibility though extensions.
Here are some extensions i used:
For the general development environment
Remote-Container: Open any folder or repository inside a Docker container and take advantage of Visual Studio Code's full feature set. Great to ensure every dev environment through a team are identical.
Python: Python IntelliSense (Pylance), Linting, Debugging (multi-threaded, remote), Jupyter Notebooks, code formatting, refactoring, unit tests, and more. I don't use Jupyter so i cant talk about it. However Pylance is a great tool. Under the hood it used PyRights which is a super fast code analyzer (way faster than mypy!). Don't forget add this to your settings:
vscode-icons: Each folder type have his own well design icons. Big project with a lots of files are way more easy to understand with that.
Window Color: Automatically adds a unique color to each window's activityBar and titleBar. A project will always have the same color. Great when a teams works on multiple project, everybody see the same color.
GitLens: Now you can see who wrote that directly into the code
For data parsing/formatting/transformation...
There's lots of extensions for that depending on your use case (docker, helm, etc...)
I think these one are required by pretty much every project:
- Better TOML
- change-case
- Code Spell checker
- JSON Tools
- XML Tools
- YAML
- vscode-base64
- Sort JSON objects
Python versions: Pyenv
Pyenv is so far the best python versions manager.
Python package and venv: Poetry
Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.
Basically, it's pip under steroids. It works with the latest package file format pyproject.toml . His dependencies and build system are very reliable. You can still export requirements.txt if necessary. Virtual environments works like a charms. Multiple python version too. It solve a lot of pain point. This is a must have.
Code linter/formatter
- flake8: Flake8 is a wrapper around these tools: PyFlakes pycodestyle Ned Batchelder's McCabe script
Flake8 runs all the tools by launching the single flake8 command. It displays the warnings in a per-file, merged output.
- black: Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters.
Blackened code looks the same regardless of the project you're reading. Formatting becomes transparent after a while and you can focus on the content instead.
Black makes code review faster by producing the smallest diffs possible.
- isort: isort your imports, so you don't have to.
For the tests
pytest: Python allow to write way less code than using directly unittest. Lots of good library handle it very well. Some library extend it to be even more efficient and user friendly (pytest-mock, pytest-freezegun, pytest-sugar...)
nox: The first point of nox is to run your tests into multiple environment (multiple python version). In my opinion way better than tox
For the documentation: Mkdocs
Mkdocs allow you to write documentation using markdown format. Really easy to use and integrate. You can easily integrate your documentation to your CI/CD pipeline for automatically test it then deploy it.
Pre-commit
This is a must have. pre-commit allow you to run multiple scripts (a.k.a pre-commit) before your commit validation. There's a big catalogue of builtin pre-commit. Some will modify files. Some will just return warning or errors. You can automatically perform a TONS of tests/validation before committing anything.
This is basically a local CI pipeline run before the real one.
Here is an example of one .pre-commit-config.yaml config file i use, among others it will:
- sort your import automatically
- black will format your code
- flake8 will warn you about style issue
- Validate .gitlab-ci.yaml against my enterprise gitlab server.
- check style for bash script
- and so much more...
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.0.1
hooks:
- id: check-added-large-files
- id: check-ast
- id: check-builtin-literals
- id: check-case-conflict
- id: check-docstring-first
- id: check-executables-have-shebangs
- id: check-json
- id: check-merge-conflict
- id: check-symlinks
- id: check-toml
- id: check-vcs-permalinks
- id: check-xml
- id: check-yaml
args: [--allow-multiple-documents]
- id: debug-statements
- id: detect-aws-credentials
args: [--allow-missing-credentials]
- id: destroyed-symlinks
- id: end-of-file-fixer
- id: fix-byte-order-marker
- id: fix-encoding-pragma
args: [--remove]
- id: forbid-new-submodules
- id: mixed-line-ending
args: [--fix=auto]
- id: name-tests-test
args: [--django]
- id: requirements-txt-fixer
- id: trailing-whitespace
- repo: local
hooks:
- id: black
name: black
entry: poetry run black
language: system
types: [python]
- id: flake8
name: flake8
entry: poetry run flake8
language: system
types: [python]
- repo: https://github.com/pycqa/isort
rev: "5.9.1"
hooks:
- id: isort
args:
- --profile
- black
- --filter-files
- repo: https://github.com/adrienverge/yamllint.git
rev: v1.26.1
hooks:
- id: yamllint
args: [-c=.yamllint.yaml]
- repo: https://gitlab.com/devopshq/gitlab-ci-linter
rev: v1.0.2
hooks:
- id: gitlab-ci-linter
args:
- "--server"
- "https://your.gitlab.server" # Need env var GITLAB_PRIVATE_TOKEN with gitlab api read token
- repo: https://github.com/commitizen-tools/commitizen
rev: v2.17.11
hooks:
- id: commitizen
stages: [commit-msg]
- repo: https://github.com/jumanjihouse/pre-commit-hooks
rev: 2.1.5 # or specific git tag
hooks:
- id: forbid-binary
- id: shellcheck
- id: shfmt
Security
- bandit: Bandit is a tool designed to find common security issues in Python code.
To do this Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files it generates a report.
- safety: Safety checks your installed dependencies for known security vulnerabilities.
By default it uses the open Python vulnerability database Safety DB, but can be upgraded to use pyup.io's Safety API using the --key option.
- gitleaks: Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.
Job is done but if someone have better alternative, i would try them.
Conclusion
I hope you leaned a thing or two in this article.
Using these tools can help you falling in some common pitfall.
Pylance warning & errors are great best practices teachers. Your code understanding and therefore quality will greatly progress understanding and correcting them.
Top comments (3)
Love this article!
pyenvis the best for changing Python versions - until you need to change your Python paths for any reason (like if homebrew updates on a mac, for instance) then it becomes one of the biggest Python nightmares.If anyone has ever run into that issue with Pyenv, try using Docker. You can use a slim Python image to keep it small, too.
Thanks!
Unpopular opinion: python should get rid of all virtual environment stuff from their codebase and let Docker take care of it