Managing python projects like a pro!

Sudarsaan — Mon, 18 Mar 2024 18:20:44 +0000

Hello folks! When I first started with python and was working on some hobby projects I used to struggle managing dependencies and sharing the whole code base to my friends to test it out. Another pain was to resolve python versions for some packages or projects I used. When I first heard about poetry and pyenv I was like, "holy sheeet which island I was on". Here we will see one among many professional ways to set up python using pyenvand installing poetry for dependency management. This blog gonna be bit long, but trust me it will be save lots of time in the future.

Note:

I am only going to cover installation and setup for windows as I am on one. I am basically writing this for myself which will help me get back to it whenever I need.

Installing `pyenv` on Windows using pyenv-win

Installing using powershell

Run the below command in the powershell, if you find this blog post to be too old, please follow the steps in the official repository of pyenv-win.

Invoke-WebRequest -UseBasicParsing -Uri "https://raw.githubusercontent.com/pyenv-win/pyenv-win/master/pyenv-win/install-pyenv-win.ps1" -OutFile "./install-pyenv-win.ps1"; &"./install-pyenv-win.ps1"

If the command fail due to issues regarding permissions, try adding -Scope Currentuser _(Only if you are in an admin account and using powershell) or simply run the powershell as administrator.

Once done with the installation, verify it running the following command

pyenv --version

You should see something like

pyenv 3.x.x

Now let's quickly install python. But before installing it, let's make use of pyenv for which it is installed for, Yes you got it right! choosing the version of the python to be installed easily and switching it back and forth to the versions we need.

You can see the python versions available to install using pyenv by running following command.

pyenv install -l

Then choose a version you need for your project and install it using.

pyenv install <version>

Once installation is successful, make the installed python as the global version using the following command.

pyenv global <version>

What this basically does is, it updates/sets the path of the python interpreter in the environment variable for us to use. This saves us so much time and makes our life easier by smoothly moving from one version from another easily.

Hurrayy! We successfully installed python with the version we need.

Hold on a sec though, let's verify all the wheels are moving appropriately.

Let's test is using following commands

pyenv version
python -c "import sys; print(sys.executable)"

First one's output should give you something like

<version> (set by \path\to\.pyenv\pyenv-win\.python-version)

Second one's output should be something like

\path\to\.pyenv\pyenv-win\versions\<version>\python.exe

If these are working, then yes we managed to setup python. Next we will go ahead and install poetry.

Poetry Installation

Before installing poetry let's understand what is scoop for windows, pipx and why is recommended. If you are in a hurry please skip to the installation part of the blog below.

Scoop

Scoop is a command-line installer for Windows, aimed at making it easier for users to manage software installations and maintain a clean system. It's designed with developers and power users in mind but can be beneficial for any Windows user looking for an efficient way to manage software. Basically it makes our life easier when it comes to software installation of any sort. Scoop support installation for large number of software. Check it out here Scoop.

pipx

pipx is nothing but a tool for installing and running Python applications in isolated environments. It ensures that each application and its dependencies are contained in their own virtual environment, preventing conflicts with other applications or the system-wide Python environment. pipx allows users to easily install, run, and manage Python CLI tools globally while maintaining the benefits of environment isolation. This approach simplifies the management of Python tools and enhances security and stability by isolating dependencies.

Next question you might get is, why do we need pipx for poetry?

Poetry recommends installation using pipx and here are the main reasons.

Environment Isolation: pipx installs Python applications in isolated virtual environments. This isolation ensures that Poetry and its dependencies do not interfere with the system-wide Python environment or other project-specific environments. It helps prevent dependency conflicts and ensures that the Python ecosystem on your system remains clean and manageable.
Global Accessibility with Isolation: While pipx installs Poetry in an isolated environment, it also makes Poetry globally accessible from the command line. This means you can run Poetry commands in any directory to manage project dependencies, without needing to activate a specific virtual environment. This setup combines the convenience of global access with the benefits of environment isolation.

Let's install Poetry now!

Make sure you installed pipx on your system. if you have not done it follow this official pipx installation documentation
Then using pipx install poetry with the below command. If you wish to install a specific version of poetry for some reason follow this poetry documentation.

pipx install poetry

Hurray! you are finally set. Now you can use poetry by navigating inside a directory where your project is and run

poetry init

It will ask you series of questions like package name, version and dependencies it needs to initialize etc. Follow the instructions and finish the procedure and you will have a project.toml and poetry.lock files.

To install any package to your project all you have to do is

poetry add <package-name>

Here by default PyPi will be the default repository for installing the packages but it supports other sources too. You can read this documentationto understand more about the procedure.

Other important note on poetry installation

Adding development packages

Development dependencies are packages that are needed for development purposes, such as testing, linting, or building documentation, but not for running the actual project code. Here's how you add a development package. Distinguishing between development and production dependencies keeps your project's environment clean and minimal. Only the packages necessary for running the project are installed in production, reducing the deployment size and potential attack surface for security vulnerabilities. You can install dev packages using the following command.

poetry add <package-name> --group dev

Understanding pyproject.toml file

The pyproject.toml file is divided into several sections, each serving a specific purpose:

[tool.poetry]: This section contains the project metadata like name, version, description, authors, and more. It's essential for identifying the project and its versioning.

[tool.poetry.dependencies]: Lists the project's dependencies. These are packages your project needs to run. Poetry automatically manages these dependencies and their versions to avoid conflicts.

[tool.poetry.dev-dependencies]: Specifies the development dependencies. These are packages required for development tasks, like testing, linting, or documentation, but not needed for running the project.

[build-system]: Defines the build system requirements, specifying which tool (like Poetry) should be used to build the project and its dependencies.

In summary, the pyproject.toml file is a powerful tool for Python project management, offering a standardized way to specify project settings and dependencies. By making full use of pyproject.toml, you can enhance project portability, simplify configuration management, and streamline development and deployment processes.

What is the `poetry.lock` File?

When you specify dependencies in the pyproject.toml file using Poetry, you often do so without specifying exact versions (or using very broad constraints). This approach keeps your project flexible but can lead to inconsistencies across installations if dependencies are updated.

The poetry.lock file is automatically generated or updated by Poetry when you add, update, or install dependencies. It contains a detailed list of your project's direct and indirect dependencies, including the exact version and checksum of each package. This specificity ensures that no matter where or when you install your project's dependencies, Poetry will install the exact same versions, as specified in the poetry.lock file.

That's it, now you have a good idea of how to setup a project environment professionally. you are good to go!

Summary

we dive into the essential tools and practices for effective Python project management. Highlighting the initial challenges of dependency management and Python version conflicts, this post introduces pyenv, poetry, and pipxas game-changing tools for any Python developer's toolkit.

Starting with pyenv for Python version management, we explore how to install and utilize this tool on Windows to seamlessly switch between Python versions.
Next, we delve into poetry for dependency management, explaining its advantages, such as environment isolation and simplification of dependency handling. We also discuss the importance of using pipx to install Python CLI tools globally while keeping them isolated. Through step-by-step instructions, the post guides on setting up these tools to create a more efficient and streamlined development workflow. By embracing these tools, we can save time, avoid common pitfalls, and focus on building great projects with confidence.

Best wishes with your projects and let me know if there is anything in the comment section.

See ya!

DEV Community: Sudarsaan