In this article, i'll show you what is my common python setup.
- High quality code standard
- Respect of the python standard
- Dev experience quality
- The IDE
- Python versions
- Package and virtual environments
- Code linter/formatter
- Tests tools
This is entirely subjective :)
VSCode is a great IDE for python programming.
One of his best feature is his extensibility though extensions.
Here are some extensions i used:
For the general development environment
Remote-Container: Open any folder or repository inside a Docker container and take advantage of Visual Studio Code's full feature set. Great to ensure every dev environment through a team are identical.
Python: Python IntelliSense (Pylance), Linting, Debugging (multi-threaded, remote), Jupyter Notebooks, code formatting, refactoring, unit tests, and more. I don't use Jupyter so i cant talk about it. However Pylance is a great tool. Under the hood it used PyRights which is a super fast code analyzer (way faster than mypy!). Don't forget add this to your settings:
vscode-icons: Each folder type have his own well design icons. Big project with a lots of files are way more easy to understand with that.
Window Color: Automatically adds a unique color to each window's activityBar and titleBar. A project will always have the same color. Great when a teams works on multiple project, everybody see the same color.
GitLens: Now you can see who wrote that directly into the code
For data parsing/formatting/transformation...
There's lots of extensions for that depending on your use case (docker, helm, etc...)
I think these one are required by pretty much every project:
- Better TOML
- Code Spell checker
- JSON Tools
- XML Tools
- Sort JSON objects
Pyenv is so far the best python versions manager.
Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.
Basically, it's pip under steroids. It works with the latest package file format pyproject.toml . His dependencies and build system are very reliable. You can still export requirements.txt if necessary. Virtual environments works like a charms. Multiple python version too. It solve a lot of pain point. This is a must have.
- flake8: Flake8 is a wrapper around these tools: PyFlakes pycodestyle Ned Batchelder's McCabe script
Flake8 runs all the tools by launching the single flake8 command. It displays the warnings in a per-file, merged output.
- black: Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters.
Blackened code looks the same regardless of the project you're reading. Formatting becomes transparent after a while and you can focus on the content instead.
Black makes code review faster by producing the smallest diffs possible.
- isort: isort your imports, so you don't have to.
pytest: Python allow to write way less code than using directly unittest. Lots of good library handle it very well. Some library extend it to be even more efficient and user friendly (pytest-mock, pytest-freezegun, pytest-sugar...)
nox: The first point of nox is to run your tests into multiple environment (multiple python version). In my opinion way better than tox
Mkdocs allow you to write documentation using markdown format. Really easy to use and integrate. You can easily integrate your documentation to your CI/CD pipeline for automatically test it then deploy it.
This is a must have. pre-commit allow you to run multiple scripts (a.k.a pre-commit) before your commit validation. There's a big catalogue of builtin pre-commit. Some will modify files. Some will just return warning or errors. You can automatically perform a TONS of tests/validation before committing anything.
This is basically a local CI pipeline run before the real one.
Here is an example of one .pre-commit-config.yaml config file i use, among others it will:
- sort your import automatically
- black will format your code
- flake8 will warn you about style issue
- Validate .gitlab-ci.yaml against my enterprise gitlab server.
- check style for bash script
- and so much more...
repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.0.1 hooks: - id: check-added-large-files - id: check-ast - id: check-builtin-literals - id: check-case-conflict - id: check-docstring-first - id: check-executables-have-shebangs - id: check-json - id: check-merge-conflict - id: check-symlinks - id: check-toml - id: check-vcs-permalinks - id: check-xml - id: check-yaml args: [--allow-multiple-documents] - id: debug-statements - id: detect-aws-credentials args: [--allow-missing-credentials] - id: destroyed-symlinks - id: end-of-file-fixer - id: fix-byte-order-marker - id: fix-encoding-pragma args: [--remove] - id: forbid-new-submodules - id: mixed-line-ending args: [--fix=auto] - id: name-tests-test args: [--django] - id: requirements-txt-fixer - id: trailing-whitespace - repo: local hooks: - id: black name: black entry: poetry run black language: system types: [python] - id: flake8 name: flake8 entry: poetry run flake8 language: system types: [python] - repo: https://github.com/pycqa/isort rev: "5.9.1" hooks: - id: isort args: - --profile - black - --filter-files - repo: https://github.com/adrienverge/yamllint.git rev: v1.26.1 hooks: - id: yamllint args: [-c=.yamllint.yaml] - repo: https://gitlab.com/devopshq/gitlab-ci-linter rev: v1.0.2 hooks: - id: gitlab-ci-linter args: - "--server" - "https://your.gitlab.server" # Need env var GITLAB_PRIVATE_TOKEN with gitlab api read token - repo: https://github.com/commitizen-tools/commitizen rev: v2.17.11 hooks: - id: commitizen stages: [commit-msg] - repo: https://github.com/jumanjihouse/pre-commit-hooks rev: 2.1.5 # or specific git tag hooks: - id: forbid-binary - id: shellcheck - id: shfmt
- bandit: Bandit is a tool designed to find common security issues in Python code.
To do this Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files it generates a report.
- safety: Safety checks your installed dependencies for known security vulnerabilities.
By default it uses the open Python vulnerability database Safety DB, but can be upgraded to use pyup.io's Safety API using the --key option.
- gitleaks: Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.
Job is done but if someone have better alternative, i would try them.
I hope you leaned a thing or two in this article.
Using these tools can help you falling in some common pitfall.
Pylance warning & errors are great best practices teachers. Your code understanding and therefore quality will greatly progress understanding and correcting them.