Dependency management in Python is very painful. Everyone that uses Python in multiple projects someday had a problem with package versions across the machine and even between team members. To solve these problems a lot of solutions appeared: venv, conda environments, Pipenv and many other tools that aim to create an isolated environment for you projects.
The objective of this quick tutorial is to help you easily setup a reproducible and isolated virtual environment. using Pyenv to manage your python versions and Poetry to manage your packages.
Managing your Python Versions: PyEnv
The objective of PyEnv is to help you install and switch between multiple Python versions in your system without worrying about breaking things.
Installing PyEnv
To install PyEnv you can follow their Installation tutorial. Personally I like to use their official automatic installer found here. For any problem I also recommend reading the Common build problems specially the prerequisites section.
Installing the Python Version
With PyEnv installed you can run the command below to obtain the list of possible Python versions that you can install:
pyenv install --list
The command above shows a huge list of python versions, you can choose whichever you need. For example, you can install Python 3.11.3 by running:
pyenv install 3.11.3
To know if everything went well you can check all the installed versions running:
pyenv versions
You should see an output similar to this:
system
* 3.11.3 (set by /usr/local/pyenv/version)
This means that are 2 python versions installed into your machine. The system is the one that comes with your OS. The other is the versions that we installed.
Choosing the versions
The * symbol means that this is the default version used whenever you call python (i.e. the global version). You can change the global version by running the command pyenv global
. For example, to set 3.11.3 as the global version you can run:
pyenv global 3.11.3
You can also set the Python version in a folder, this is useful when you need a different versions of python for many projects. To do this use the pyenv local
command. Following our example:
pyenv local 3.11.3
Managing your packages: Poetry
With Python installed in the correct version now you need a way to easily manage your packages in an isolated environment. Poetry is a tool to help manage your packages and its dependencies. It provides packaging and dependency resolution features (we are going to focus on this one) for you not to worry more about conflicts in your dependencies.
Installing Poetry
First you need to install the tool by following the installation guide.
Setup your project
Now you need to go to your project folder and run the poetry init
command. This command will prompt you for some information including name of the project, authors, packages that you want to install from the beginning and other information. Note that this command will use the Python version setup by PyEnv giving preference to the local version.
poetry init
You will notice that the command will create 2 new files. The pyproject.toml
is a human-friendly file introduced on PEP518 and expanded in some other PEPs (PEP 517, PEP 621 and PEP 660). This file store many information about your project that can be used by many tools of the Python ecosystem.
The poetry.lock
files contains the installed packages (including their dependencies) and whatever is necessary for them to work. You should not edit this file manually.
If your project already have a
pyproject.toml
inside with[tool.poetry]
sections it means that someone already setup the tool. You can skip thepoetry init
and go directly to runpoetry install
to install all the required packages in your machine.
Adding Packages
To add packages to the environment you just need to run the poetry add
command. The command accepts a flag --group
that is a way to organize the dependencies into groups. A common example is if you want to differ between production packages and development ones (e.g. test packages, code style and so on). In this case you can create a group called dev
for the non-production environment. See an example below:
poetry add pandas==2.1.1 # Add pandas 2.1.1 on prod environment
poetry add pandas==2.1.1 --group dev # Add pandas 2.1.1 on dev environment
Using your environment
With your environment configured, you need to activate it to use every time you call python. This is as simple as running:
poetry shell
If you don't want to activate the shell you can also run any python command using the environment with poetry run
inside the project folder. For example, to run an example python script using the environment you can do:
poetry run python some-script.py
For more Poetry commands and usage, see Poetry Docs
Final Thoughts
Since I started my python journey I had used many tools to manage packages: venv, Pipenv, requirements, conda environments. This setup was the one that I had less headaches with the many projects and versions that I worked on. This tutorial should help you achieve that.
Comment below if you use any other setup, if this tutorial helped you or any other comment that you want. Thanks for reading!
Top comments (0)