Setup your Python Environment with Pyenv and Poetry

Dependency management in Python is very painful. Everyone that uses Python in multiple projects someday had a problem with package versions across the machine and even between team members. To solve these problems a lot of solutions appeared: venv, conda environments, Pipenv and many other tools that aim to create an isolated environment for you projects.

The objective of this quick tutorial is to help you easily setup a reproducible and isolated virtual environment. using Pyenv to manage your python versions and Poetry to manage your packages.

Managing your Python Versions: PyEnv

The objective of PyEnv is to help you install and switch between multiple Python versions in your system without worrying about breaking things.

Installing PyEnv

To install PyEnv you can follow their Installation tutorial. Personally I like to use their official automatic installer found here. For any problem I also recommend reading the Common build problems specially the prerequisites section.

Installing the Python Version

With PyEnv installed you can run the command below to obtain the list of possible Python versions that you can install:

pyenv install --list

The command above shows a huge list of python versions, you can choose whichever you need. For example, you can install Python 3.11.3 by running:

pyenv install 3.11.3

To know if everything went well you can check all the installed versions running:

pyenv versions

You should see an output similar to this:

system
* 3.11.3 (set by /usr/local/pyenv/version)

This means that are 2 python versions installed into your machine. The system is the one that comes with your OS. The other is the versions that we installed.

Choosing the versions

The * symbol means that this is the default version used whenever you call python (i.e. the global version). You can change the global version by running the command pyenv global. For example, to set 3.11.3 as the global version you can run:

pyenv global 3.11.3

You can also set the Python version in a folder, this is useful when you need a different versions of python for many projects. To do this use the pyenv local command. Following our example:

pyenv local 3.11.3

Managing your packages: Poetry

With Python installed in the correct version now you need a way to easily manage your packages in an isolated environment. Poetry is a tool to help manage your packages and its dependencies. It provides packaging and dependency resolution features (we are going to focus on this one) for you not to worry more about conflicts in your dependencies.

Installing Poetry

First you need to install the tool by following the installation guide.

Setup your project

Now you need to go to your project folder and run the poetry init command. This command will prompt you for some information including name of the project, authors, packages that you want to install from the beginning and other information. Note that this command will use the Python version setup by PyEnv giving preference to the local version.

poetry init

You will notice that the command will create 2 new files. The pyproject.toml is a human-friendly file introduced on PEP518 and expanded in some other PEPs (PEP 517, PEP 621 and PEP 660). This file store many information about your project that can be used by many tools of the Python ecosystem.

The poetry.lock files contains the installed packages (including their dependencies) and whatever is necessary for them to work. You should not edit this file manually.

If your project already have a pyproject.toml inside with [tool.poetry] sections it means that someone already setup the tool. You can skip the poetry init and go directly to run poetry install to install all the required packages in your machine.

Adding Packages

To add packages to the environment you just need to run the poetry add command. The command accepts a flag --group that is a way to organize the dependencies into groups. A common example is if you want to differ between production packages and development ones (e.g. test packages, code style and so on). In this case you can create a group called dev for the non-production environment. See an example below:

poetry add pandas==2.1.1             # Add pandas 2.1.1 on prod environment
poetry add pandas==2.1.1 --group dev # Add pandas 2.1.1 on dev environment

Using your environment

With your environment configured, you need to activate it to use every time you call python. This is as simple as running:

poetry shell

If you don't want to activate the shell you can also run any python command using the environment with poetry run inside the project folder. For example, to run an example python script using the environment you can do:

poetry run python some-script.py

For more Poetry commands and usage, see Poetry Docs

Final Thoughts

Since I started my python journey I had used many tools to manage packages: venv, Pipenv, requirements, conda environments. This setup was the one that I had less headaches with the many projects and versions that I worked on. This tutorial should help you achieve that.

Comment below if you use any other setup, if this tutorial helped you or any other comment that you want. Thanks for reading!

Sources

• pip: pyproject.toml
• Poetry Docs
• Pyenv