Let's Set Up Poetry - Clean Python DevEnv for 2020

unsplash-logoScott Umstattd

I have used a couple of dev env on python.

1. System-provided Python (Mac)
   As many of you know, the version is too outdated right now. When I used it, it was fine but at that time I wanted to use python3 so I didn't use it for a long time.

2. Pyenv
   Pyenv allows us to install multiple versions of python which is very useful, but when I used it, I didn't use Virtualenv, so my dev envs were very messy.

3. Anaconda
   Then, I started learning about machine learning and one ITP resident recommended us to use Anaconda since it's very easy to create a virtual environment and install some specific modules/packages.
   found out this article(https://jacobian.org/2019/nov/11/python-environment-2020/) that is from Jacob Kaplan-Moss, co-creator of Django.

4. Pyenv + Virtualenv
   When I used 3, I also used Pyenv, but sometimes that made me confused and also Anaconda had some issues when I run flask. Eventually, I uninstalled Anaconda and started using Pyenv and Virtualenv. Actually, I'm still using this. But, the issue is that I heavily use a specific virtual env and it's very messy so it's about to ruin my python env lol

5. Pipenv
   The team I joined used python and the lead engineer recommended to use pipenv But, pipenv is slow and has caused some issues when I update
   modules/packages. I end up using 4 for other things/projects.

Recently, I read the article on python dev env that is written in Japanese lol. The article mentioned 3 three things, Pipenv, Poetry, and Pyflow. It says that Poetry could be a better choice. After reading that, I researched about Poetry a little bit. Then, I found out this article(https://jacobian.org/2019/nov/11/python-environment-2020/) that is from Jacob Kaplan-Moss, co-creator of Django.

Poetry
https://python-poetry.org/

Why do I need to use Poetry?

Before introducing the install steps, I would like to mention the reason why I want to switch Pyenv + Virtualenv to Poetry. The biggest reason is that I can share a project easily.
When I was an ITP resident and led a python workshop, I spent so much time to create an environment for sample codes since most students didn't follow guides I shared before the workshop lol.
Also, there are a few reasons.

What Poetry can do is to allow me to manage python projects like js/nodejs projects which are managed by yarn/npm. This feature enables me to discard a project easily(I will be able to delete the project.) when I don't need it. I can use virtual env with Poetry.
In addition, Poetry has commands that build a package and publish it to PyPI. This is also good for me since I have published a couple of packages to PyPI. Plus Poetry has an export command for requirements.txt.

https://python-poetry.org/docs/cli/#export

I decided to set up Poetry on my Ubuntu since Mac's environment will take some time to switch and I noticed that I haven't used Pyenv on my Ubuntu. (I don't know why I haven't used it.)

In this article, I will share steps to set up Poetry on my Ubuntu.
The env will be Pyenv + Poetry + Pipx.
This setup is very similar to my js dev env. (nvm + yarn)

Step1. Install Pyenv
Step2. Install Pipx for python CLI tool (especially, youtube-dl)
Step3. Install Poetry
Step4. Create a sample project with Poetry
Step5. Allow making venv in a project

Step1 Install Pyenv

There are a couple of ways to install Pyenv on Ubuntu. In this article, I use brew since I'm using brew on Mac and currently we can use brew on Linux too.

https://docs.brew.sh/Homebrew-on-Linux
https://github.com/pyenv/pyenv

$ sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"

$ brew install pyenv

$ pyenv --version
pyenv 1.2.15

system pythons

$ python --version
Python 2.7.17

$ python3 --version
Python 3.7.5

Let's install python 3.6 and 3.8

$ pyenv install 3.6.5
$ pyenv install 3.8.0
$ pyenv global 3.8.0

Ugh, got BUILD FAILED (Ubuntu 18.04 using python-build 20180424) with 3.6.5...

The solution is the following

sudo apt-get remove libssl-dev
sudo apt-get update
sudo apt-get install libssl1.0-dev

https://github.com/pyenv/pyenv/wiki/Common-build-problems

Now the global Python version is changed by Pyenv

$ python --version
Python 3.8.0

Step2 install pipx

Of course, you can install CLI tools via brew, but this time I'm using pipx to try a new thing lol.

$ pip list
Package    Version
---------- -------
pip        19.2.3 
setuptools 41.2.0 


$ python -m pip install pipx
$ pipx install youtube-dl
$ pipx list
   package youtube-dl 2019.12.25, Python 3.8.0
    - youtube-dl
# Actually the command is a little bit longer, but you can use alias to shorten the command.
$ pipx run youtube-dl url

Step3 install Poetry

Eventually reached the most important part of this post.
The official page is using curl to install Poetry, but this post uses pipx because I need to understand pipx.

I think following the official site is better.

Poetry - Official
https://python-poetry.org/docs/

$ pipx install poetry 

Step4 Create a sample project

$ mkdir hoge && cd hoge
$ pipx run poetry init
or
$ pipx run poetry new hoge

# add package
$ pipx run poetry add ipython
$ pipx run poetry run ipython
Python 3.8.0 (default, Dec 25 2019, 21:53:01) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.10.2 -- An enhanced Interactive Python. Type '?' for help.

In [1]: print('a')                                                              
a

In [2]:      

pyproject.toml

[tool.poetry]
name = "hoge"
version = "0.1.0"
description = ""
authors = ["koji <[e-mail]>"]

[tool.poetry.dependencies]
python = "^3.8"
ipython = "^7.10.2"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

export

$ pipx run poetry export -f requirements.txt > requirements.txt

Step5 Allow making venv in a project

The default settings create a virtual environment in a specific place, so this is the same as pyenv + virtualenv that I'd like to avoid. So I need to modify the setting.

$ pipx run poetry config --list
cache-dir = "/home/koji/.cache/pypoetry"
virtualenvs.create = true
virtualenvs.in-project = false <-- chage this
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/koji/.cache/pypoetry/virtualenvs

$ pipx run poetry config virtualenvs.in-project true
$ pipx run poetry config --list                                          
cache-dir = "/home/koji/.cache/pypoetry"
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/koji/.cache/pypoetry/virtualenvs

Caution

I just started using Poetry before 2020, so there might be some mistakes.
If you find out the mistake, please let me know!

Comments

[Pacharapol Withayasakpunt]

I use macOS, then use pyenv to globalize Python 3.8.0; then install Poetry via cURL; then run poetry directly.

But yeah, after that I set poetry config virtualenvs.in-project true

The rest is, to prevent running commands with global Python; I can always use poetry run python ./script.py or poetry run python -m http.server.

I also use poetry init to create virtualenv regularly, before running the project in PyCharm.

I have never grown to like Pipenv.

[0xkoji]

Sounds good. I will use Poetry on macOS soon.
Thank you for sharing the info!

[Guilherme Marthe]

Ok, this is a great post. I never saw the point for using pipx and this post clarified it to me.

How ever, I have a problem where I use multiple python version on different projects.

Should I just install pipx + poetry for each of the pyenv versions I work with?

And this gets messed up some times, since my choice of REPL is ipython and sometimes pyenv just points to the wrong ipython, while everything is working well with the python REPL,
I guess your tutorial taught me that I can just let the venv be in the root of the project and I can just alias everything to the binaries contained in those.

:/

[0xkoji]

Hi @gui42 ,

Thank you for reading this.
In terms of ipython, installing pipx makes sense to me.

I can just let the venv be in the root of the project and I can just alias everything to the binaries contained in those.

I think that is the way to keep your dev env clean.