DEV Community

Cover image for How I Set up My Python Workspace
Miguel Brito
Miguel Brito

Posted on • Edited on • Originally published at miguendes.me

How I Set up My Python Workspace

#python #tutorial #beginners

TL;DR: In this tutorial, I’ll teach you what I do to have multiple Python versions and tools installed without conflicts.

I’ve been working with Python for the last 5 years. In the beginning academically and as a hobby, then professionally. One thing that has always been very common is having multiple projects in different Python versions. Regardless of the language you use, having projects targeting at distinct versions can be a nightmare. When it comes to Python, it’s no different. As a matter of fact, it’s worse. For instance, there are libraries that only work with Python 2, despite not being supported anymore. Fortunately, there are a couple of tools that can help us have seamless experience handling different Python versions.

This guide will focus on Linux, but you can easily adapt to macOS. In fact, I have a MacBook at work and follow the same steps to configure my workspace with little modifications. Also, I use zsh as my default shell, but this guide applies to bash as well.

So, without further ado, here's how I setup my Python workspace:

Step 1. Define your requirements

Before getting into the actual configuration, you must take some time to define what your optimal setup looks like. For me, it looks like this:

  1. It must have Python 2.7+, Python 3.6, Python 3.7, and Python 3.8 installed.
  2. Python 3.8 must be the default version.
  3. I must be able to switch between versions easily.
  4. I must be able to fire up a ipython using the default python version I specified.
  5. Dependency tools I use - pipenv and poetry - must work with all Python 3+ versions.

Step 2. Installing and configuring multiple Python versions

To install multiple versions and being able to switch between them, I use pyenv.
pyenv has several benefits such as:

  • Lets you change the global Python version on a per-user basis.
  • Provides support for per-project Python versions.
  • Everything is installed in the $HOME directory. This means no risk of messing up the default Python installation.
  • Supports pypy, anaconda, CPython, Stackless-Python and others!

In addition to pyenv, I also use the pyenv-virtualenv plugin to manage my virtual environments.

Installing pyenv and its plugins

Since last year, I've been using Homebrew both on macOS and Linux. So, the following steps are OS agnostic for me.

brew install pyenv
brew install pyenv-virtualenv
Enter fullscreen mode Exit fullscreen mode

Creating a directory for the virtualenvs

Each project should have each own virtual environment associated with. This way we can work on more than one project at a time without introducing conflicts in their dependencies. I keep all my virtualenvs under the $HOME/.ve directory.

mkdir -p $HOME/.ve
Enter fullscreen mode Exit fullscreen mode

Once the folder is created I add the path to my ~/.zshrc (for bash users, add it to ~/.bashrc):

cat <<"EOT" >> ~/.zshrc
# pyenv config
# Set virtualenv dir
export WORKON_HOME=~/.ve
# Initialize pyenv
if command -v pyenv 1>/dev/null 2>&1; then
  eval "$(pyenv init -)"
fi
# Initialize pyenv-virtualenv
eval "$(pyenv virtualenv-init -)"
EOT
Enter fullscreen mode Exit fullscreen mode

The variable WORKON_HOME tells pipenv where to place your virtual environments.

Once that's done, you can restart your shell by either closing and opening a new window or running:

exec $SHELL
Enter fullscreen mode Exit fullscreen mode

Installing all Python versions I need. 

PY_DEFAULT=3.8.5
PY_VERSIONS=( $PY_DEFAULT 3.7.8 3.6.11 2.7.18 )

for py_version in "${PY_VERSIONS[@]}"
do
    echo -e "Installing Python $py_version...\n\n"
    # Install specific Python version
    pyenv install $py_version
done
Enter fullscreen mode Exit fullscreen mode

Step 3. Installing the tools

For dependency management I use two different tools, pipenv, and more recently poetry. Eventually I will probably settle on poetry by at the moment I need both.

Also, I rely a lot on jupyter notebooks, for quick data analysis tasks; and ipython as a fancy Python interpreter.

Again, I don't want to pollute the global installation. To avoid that we can use pyenv-virtualenv to create virtualenvs for the tools.

Let's install jupyter, pipenv, and poetry.

# Creating tools3 venv
pyenv virtualenv $PY_DEFAULT tools3

# Activating
pyenv activate tools3

# Upgrade pip
pip install --upgrade pip

# Install Jupyter
pip install jupyter

# Install Jupyter extensions
pip install jupyter_nbextensions_configurator rise
jupyter nbextensions_configurator enable --user

# pipenv
pip install pipenv

# poetry (using preview version to fix a bug in the 1.0.10 version)
pip install --pre poetry -U
Enter fullscreen mode Exit fullscreen mode

If you use pipenv, make sure to install its completion script. You can add it to your shell like so:

echo 'eval "$(pipenv --completion)"' >> ~/.zshrc
Enter fullscreen mode Exit fullscreen mode

And for poetry:

# Oh-My-Zsh
mkdir -p $ZSH/plugins/poetry
poetry completions zsh > $ZSH/plugins/poetry/_poetry
Enter fullscreen mode Exit fullscreen mode

Since the completion is a oh-my-zsh plugin, I need to add poetry to the plugins list in my ~/.zshrc file.

plugins(
    poetry
    ...
)
Enter fullscreen mode Exit fullscreen mode

This can be accomplished using sed:

sed -i.bak 's/^plugins=(\(.*\)/plugins=(poetry\n        \1/' ~/.zshrc
Enter fullscreen mode Exit fullscreen mode

Now, we need to configure poetry to create virtualenvs inside ~/.ve, just like pipenv.

echo 'export POETRY_VIRTUALENVS_PATH=$WORKON_HOME' >> ~/.zshrc
Enter fullscreen mode Exit fullscreen mode

More poetry configurations

One more thing, pipenv's default behaviour is to load any environment variables defined in the .env in the root of the project. It does that on two occasions, when running pipenv shell and pipenv run. poetry, on the other hand, does not support that. As a workaround, I need to load it manually.

To automate the process, I created a shell function that loads it whenever I run poetry shell or poetry run. If I want to disable it, I can set the POETRY_DONT_LOAD_ENV variable. This is, again, similar to how PIPENV_DONT_LOAD_ENV works.

cat <<"EOT" >> ~/.zshrc
# Get the poetry's full path
POETRY_CMD=$(which poetry)
# Allow poetry to load .env files
function poetry() {
    # Define the full command. i.e. poetry [run|shell|version]
    POETRY_FULL_CMD=($POETRY_CMD "$@")

    # if POETRY_DONT_LOAD_ENV is *not* set, then load .env if it exists
    # also, only loads when for "run" and "shell" commands.
    if [[ -z "$POETRY_DONT_LOAD_ENV" && -f .env && ("$1" = "run" || "$1" = "shell") ]]; then
        echo 'Loading .env environment variables…'
        env $(grep -v '^#' .env | tr -d ' ' | xargs) $POETRY_FULL_CMD
    else
        $POETRY_FULL_CMD
    fi
}
EOT
Enter fullscreen mode Exit fullscreen mode

Now deactivate the virtualenv tools3.

# Deactivating the venv
pyenv deactivate
Enter fullscreen mode Exit fullscreen mode

Step 4. Setting interpreters priority

Now that we have all the versions we need installed we must establish some sort of priority. Basically, I want to use poetry, or any other tool, without activating the virtualenv where I installed them. We can do that using pyenv global command.

PY_DEFAULT=3.8.5
PY_VERSIONS=( $PY_DEFAULT 3.7.8 3.6.11 2.7.18 )
pyenv global $PY_VERSIONS tools3 system
Enter fullscreen mode Exit fullscreen mode

Let's check:

$ pyenv versions
* system (set by /home/miguel/.pyenv/version)
* 2.7.18 (set by /home/miguel/.pyenv/version)
  2.7.18/envs/tools2
* 3.6.11 (set by /home/miguel/.pyenv/version)
* 3.7.8 (set by /home/miguel/.pyenv/version)
* 3.8.5 (set by /home/miguel/.pyenv/version)
  3.8.5/envs/tools3
* tools3 (set by /home/miguel/.pyenv/version)
Enter fullscreen mode Exit fullscreen mode

Everything looks good, it's time to restart the shell and that's it.

Preventing accidental installation of packages

One thing that happened a lot to me was installing packages inside one of the global interpreters using pip. As I mentioned earlier, it's a good idea to keep each Python interpreter intact. If we need to install anything, I prefer to create a new virtualenv, like I did for tools3.

Now, how can we lock each Python installation?

That's actually pretty straightforward. We can set pip to only install packages if there's a virtualenv active.

echo 'export PIP_REQUIRE_VIRTUALENV=true' >> ~/.zshrc
Enter fullscreen mode Exit fullscreen mode

What about pipenv and poetry?

Now, if I have a poetry project that requires Python 3.6, I tell poetry to use the 3.6 version and it will automatically create a virtualenv using the Python 3.6.11 I installed.

Example:

# pyproject.toml 
...
[tool.poetry.dependencies]
python = "~3.6"
...
Enter fullscreen mode Exit fullscreen mode

Output:

$ poetry env use 3.6   
Creating virtualenv sandbox-aBhl6cgV-py3.6 in /home/miguel/.ve
Using virtualenv: /home/miguel/.ve/sandbox-aBhl6cgV-py3.6
$ poetry shell        
Loading .env environment variables…
Spawning shell within /home/miguel/.ve/sandbox-aBhl6cgV-py3.6
$ . /home/miguel/.ve/sandbox-aBhl6cgV-py3.6/bin/activate
(sandbox-aBhl6cgV-py3.6) $
Enter fullscreen mode Exit fullscreen mode

Conclusion

That’s pretty much it! I hope this tutorial is useful for you, just as it’s for me. Whenever I need to reconfigure my workspace, I follow those steps. Also, I’ll probably create a shell script to automate it instead of copying and pasting from this guide. This tutorial was initially based on https://medium.com/@henriquebastos/the-definitive-guide-to-setup-my-python-workspace-628d68552e14, which served as inspiration on how to setup my own workspace.

How I Set up My Python Workspace first appeared on miguendes's blog.

Top comments (0)

Read next

chipm0nk profile image

From Concept to Code: Building a Simple Endless Runner Game

Chipm0nk -

hyperkai profile image

eq and ne in PyTorch

Super Kai (Kazuya Ito) -

robertobutti profile image

Building a website using Markdown content with Next.js App Router and Fusionable

Roberto B. -

alexmercedcoder profile image

Leveraging Python's Pattern Matching and Comprehensions for Data Analytics

Alex Merced -