Use pyenv and virtual environments to manage Python complexity

In my earlier post, I wrote about how pyenv is a great tool for running multiple versions of Python on the same host. It makes it simple to install multiple versions of Python on your workstation or server and control which version executes in a shell. But as a Python developer, the Python version is only one part of the environment. Most Python developers will work on more than one project at a time and want to install a number of Python packages for use in each project. Installing modules globally is rarely a good idea, especially if you are likely to use that Python version for more than one project. What happens when one projects wants a specific version of a package that won’t work with another project? Instead, using virtualenv or anaconda is the way to go. Luckily, both work well with pyenv. In this post I’ll look at using basic virtualenv, the pyenv-virtualenv plugin, and anaconda to build an isolated virtual environment that has a package installed in it that will be isolated to that environment.

Before discussing these details, I’ll mention that this post does not talk about the complexity of maintaining package compatibility within virtual environments. That is a topic for another post (or set of posts).

virtualenv

Assuming you followed the installation steps in thefirst post on pyenv, you should already know how to setup your shell to use a specific version of Python. As in that post, I’ll go ahead and install a unique Python version. Then, using that version I’ll

• install virtualenv (globally)
• use the virtualenv command to make a virtualenv
• activate the virtualenv
• use pip to install packages into that virtualenv

Note that all of these examples were run on a Mac running macOS Catalina and using zsh. This can all be run using the shell of your choice on Mac, Linux, or Windows using WSL.

❯ pyenv install --list | grep 3.8 # look for the latest 3.8 version
  3.8.0
  3.8-dev
  3.8.1
  3.8.2
  3.8.3
  3.8.4
  3.8.5
  3.8.6
  miniconda-3.8.3
  miniconda3-3.8.3
❮ pyenv install 3.8.6
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Installing Python-3.8.6...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.6 to /Users/mcw/.pyenv/versions/3.8.6

❯ pyenv shell 3.8.6 # sets the version just for this shell
❯ pyenv which pip # show which executable is running for pip, it's the newly installed one
/Users/mcw/.pyenv/versions/3.8.6/bin/pip
❯ pip install virtualenv
Collecting virtualenv
  Using cached virtualenv-20.1.0-py2.py3-none-any.whl (4.9 MB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting distlib<1,>=0.3.1
  Using cached distlib-0.3.1-py2.py3-none-any.whl (335 kB)
Collecting six<2,>=1.9.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: appdirs, filelock, distlib, six, virtualenv
Successfully installed appdirs-1.4.4 distlib-0.3.1 filelock-3.0.12 six-1.15.0 virtualenv-20.1.0

Now that virtualenv is installed in the Python environment, I can setup a virtualenv for our test project. For this example, I’m putting the virtualenv in a projects directory, but you can put it anywhere you want, including in a hidden directory in our source tree like .env.

❯ cd projects
❯ virtualenv myenv
created virtual environment CPython3.8.6.final.0-64 in 407ms
  creator CPython3Posix(dest=/Users/mcw/projects/myenv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/mcw/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
❯ . myenv/bin/activate # activates the 3.8.6 virtualenv to isolate our pip installs
❯ pip install requests
Collecting requests
  Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Installing collected packages: idna, chardet, urllib3, certifi, requests
Successfully installed certifi-2020.6.20 chardet-3.0.4 idna-2.10 requests-2.24.0 urllib3-1.25.11
❯ python
Python 3.8.6 (default, Nov 1 2020, 17:41:10)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests. __file__'/Users/mcw/projects/myenv/lib/python3.8/site-packages/requests/ __init__.py'
>>> requests. __version__'2.24.0'

Now just to show you quickly why virtualenvs are great, we’ll install a different version of requests in a different virtualenv. It doesn’t take much work at all.

❯ deactivate # deactivate our old environment
❯ virtualenv myenv2. # same output as above, but for 2nd environment
❯ . myenv2/bin/activate # activate our new one
❯ pip install requests==2.23.0. # pick a different version than last time
❯ python
Python 3.8.6 (default, Nov 1 2020, 17:41:10)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests. __version__'2.23.0'

So now there are two isolated environments, each with a different version of the requests module. We can easily switch between the two as needed.

pyenv-virtualenv plugin

It turns out that pyenv supports plugins, and the pyenv-virtualenv plugin helps you use pyenv with virtualenv (or conda, which we’ll talk about next). Take a look at their docs for the installation process. It’s very similar to the installation of pyenv itself (I used brew to install it in my environment). Once installed, you get some new commands available in pyenv.

   activate Activate virtual environment
   deactivate Deactivate virtual environment
   virtualenv Create a Python virtualenv using the pyenv-virtualenv plugin
   virtualenv-delete Uninstall a specific Python virtualenv
   virtualenv-init Configure the shell environment for pyenv-virtualenv
   virtualenv-prefix Display real_prefix for a Python virtualenv version
   virtualenvs List all Python virtualenvs found in `$PYENV_ROOT/versions/*'.

If we use these commands to replicate what we did above, it would look like this.

❯ pyenv virtualenv 3.8.6 myenv3 # makes a 3rd virtualenv using the 3.8.6 version
# output as before
❯ pyenv virtualenvs
  3.8.6/envs/myenv3 (created from /Users/mcw/.pyenv/versions/3.8.6)
  myenv3 (created from /Users/mcw/.pyenv/versions/3.8.6)
❯ pyenv deactivate # if you still have another virtualenv activated from earlier, you can deactivate this way
❯ pyenv activate myenv3
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
❯ pip install requests
# output as before
❯ python
Python 3.8.6 (default, Nov 1 2020, 17:41:10)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests. __file__'/Users/mcw/.pyenv/versions/myenv3/lib/python3.8/site-packages/requests/ __init__.py'

conda

Another nice thing about pyenv is that it makes working with Anaconda pretty straightforward. I won’t get into Anaconda in detail here, but if you plan on working on data science projects that have a large number of dependencies (like numpy, pandas, scikit-learn, tensorflow, etc.), it is usually going to be much easier to just use anaconda or miniconda to get all of your dependencies installed. Anaconda is the full install, miniconda will just install the bare necessities for you to pick and choose the packages you want quicker installs and need to save disk space.

Pyenv makes it easy to search for all versions of anaconda and miniconda without having to wade through the web site to search for an installer. Once you install a version of Anaconda using pyenv, using conda (the command for managing environments and dependencies) fits in pretty well with the environment so you don’t have to use conda commands for the basic environment creation.

❯ pyenv install --list | grep conda # to see what's available
❯ pyenv install miniconda3-4.7.12
Downloading Miniconda3-4.7.12-MacOSX-x86_64.sh...
-> https://repo.anaconda.com/miniconda/Miniconda3-4.7.12-MacOSX-x86_64.sh
Installing Miniconda3-4.7.12-MacOSX-x86_64...

# <lots of other output snipped>

❯ pyenv versions
  system
  3.6.10
  3.8.6
  3.8.6/envs/myenv3
  3.9.0
  miniconda3-4.7.12
* myenv3 (set by PYENV_VERSION environment variable)
❯ pyenv deactivate # if we had an environment activated already (myenv3 from above)
❯ pyenv virtualenv myconda
# this will generate lots of output as conda builds the environment

❯ pyenv activate myconda
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.

Now the conda tools are available in the shell and can be used instead of pyenv to create environments. Conda is also used to install packages. I won’t get into details here, but to be complete, let’s replicate the earlier task that was done using virtualenv, but this time with conda.

❯ conda env list # see that we have our new environment activated
# conda environments:
#
base /Users/mcw/.pyenv/versions/miniconda3-4.7.12
myconda * /Users/mcw/.pyenv/versions/miniconda3-4.7.12/envs/myconda
❯ conda install requests # instead of pip
# this generates a lot of output showing all the dependencies being installed
❯ python
Python 3.8.5 (default, Sep 4 2020, 02:22:02)
[Clang 10.0.0] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests. __file__'/Users/mcw/.pyenv/versions/myconda/lib/python3.8/site-packages/requests/ __init__.py'

Pyenv is a useful tool for not only installing and isolating multiple versions of Python, but can easily be used to manage virtual environments, both in Anaconda and using virtualenv. Once a Python developer needs more than one version of Python on a workstation or server, it’s a great way to manage that complexity.