Friction-less dependency management with Python

Now... hear me out this combination might seem a bit weird to many of you but just have a read through to see if I can convince you that it's actually useful :)

So I got into an issue with managing dependencies a few weeks ago and thought of writing this quick little article on what I've been using and think is the best way to manage dependencies for your Python projects.

If you have no idea of what environments are or don't have familiarity with them, then the next section will give you an overview. Else you can just skip ahead

Environments and their reason to exist

When working in a project with python, you have to install many packages in order to complete what you are making, but if you are not careful then those packages can pile up in your python installation; which is not necessarily bad but when talking in the context of deploying to production, you will have to generate a requirements file so your code can run there. It's mostly done with a similar command:

pip freeze > requirements.txt

which dumps all your installed packages in the requirements.txt file. But if you take a look in that file, it will include not only the packages you installed while working but every single one which you have installed in the past. And it's basically a pain to deal with.

To solve this, environments are used in Python. Which simply putting it are:

An Isolated container/place for having a separate dependency tree for each project that you are working on.

The benefit of using such a tool is that you don't have to work with all those unnecessary dependencies which can build up overtime if you keep working with python in a single environment, you can already imagine the difficulty in the management.

Hope that clears up whatever question you had. If you still have question about environments in python & in general, then Google and YouTube are your best friends :)

Now with that aside, let's dive into what we actually are here for

Understanding

Now, both conda & poetry have a very similar reason to exist, to help manage environments. But they both work a bit differently. Now the technique can be used without conda completely, but the reason I include it is because of conda's capability to allow you to use whichever python's version you want to in an environment. And it's done with a simple command like:

conda create -n envname python=3.10

With this I can have like 100 environments, each with a different python version and it's own set of dependencies. And my system installed python can stay clean and simple like a fresh installation.

Now, conda is in of itself is good enough for a lot of people as it has it's own package manager and quite a few handy tools (inclined more towards Data Scientists), but the reason I wrote this article is because I use python a lot in development, and I want to keep my production environment as minimal as possible and it's simply not possible to have separate dev & prod dependency sets ( like in NodeJS for example, if you've worked with it ) with a simple requirements.txt without some nifty hacks

Which is where poetry comes in. Poetry is responsible for managing the packages. It's somewhat new (v1.1 as of writing this article) and is being updated relatively frequently. Poetry has a really cool management system which is quite similar to NodeJS, where you can manage normal and dev. dependencies separately.

The upside of this is when you let's say: package your application with poetry for use in a production environment; it won't matter even if you had like a 1000 packages installed with pip in your conda environment, you'll only be passing ahead the packages that are marked in the pyproject.toml file manged by poetry, which already reduced most of the pain with having a clean final requirements file. On top of that you can also exclude out the kinds of dependencies like linters and formatters (black I'm looking at you) which are not used in a production environment. And that's basically the gist of it, if you're sold on the idea and want to give it a try then keep reading on other wise this is all the wisdom I wanted to share for today and hopefully you got something out of that ~

A Quick Setup

If you haven't already setup conda and poetry, then for conda you should first go and install it from here

After downloading, if on Windows just run the executable, if on mac it's pretty similar but as for you linux chads out there, you should know what you're doing, but either it's just 2 commands to set it up after the install:

1. chmod +x <your anaconda installation filename here>.sh
2. /bin/bash <your anaconda installation filename here>.sh

and just follow their steps and it'll set it up for you.

Now restart/open your terminal and typing in conda will show a similar output

usage: conda [-h] [-V] command ...

conda is a tool for managing and deploying applications, environments and packages.

Options:

positional arguments:
  command

... more output

by default conda puts you in a base environment which you can optionally disable it if you don't like it by typing in:

conda config --set auto_activate_base false

Now setup an environment with the command

conda create -n my_awesome_project python=3.10

Then activate it with conda activate my_awesome_project

Now there are many ways to setup poetry, but the quickest way is to just run

pip install poetry

And just like that you're ready to start working on whatever you're wanting to with minimal hassle :^]

For a simple FastAPI server setup, create a folder and run poetry init and follow the prompt. It'll generate the pyproject.toml file. Now type in

poetry add fastapi

following right after with everyone's faviourite formatter black

poetry add black -D ~ you can skip this if you want to, I'm just showing how you're gonna separate your prod/dev dependencies. and you'll endup with

[tool.poetry]
name = "my awesome project"
version = "0.1.0"
description = ""
authors = ["username <[e-mail]>"]

[tool.poetry.dependencies]
python = "^3.10"
fastapi = "^0.75.0"
uvicorn = "^0.17.6"

[tool.poetry.dev-dependencies]
black = "^22.1.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Now you're pretty much set and ready to go 👍

Conclusion

So, hopefully that convinced you that why this setup is useful over the environment management setups.

Hope you got something out of reading the article, and if so then do leave some feedback as this is my first article and I'm not really used to doing this :^]

Comments

[drmikecrowe]

Pair this with direnv and you will love it. I achieve mostly the same thing with pyenv on a per folder basis now, though I may go back and look at anaconda again

[Ahmad]

Will sure do