You can easily and sensibly run multiple versions of Python with pyenv

Python 3.9 just came out recently, and I thought it would make sense to check out some of the new features (dict union operators,string remove prefix and suffix, etc.). Of course, doing this requires a Python 3.9 environment. Since new versions of Python may break existing code, I don’t want to update my entire system to try a new feature, rather I’d like to be able to control which version of Python I use, ideally keeping older versions around as needed. There are many ways to maintain Python environments, but one very useful tool is pyenv.

From the pyenv docs, we learn that pyenv is a set of pure shell functions that allow you to change your global Python version on a per-user basis, have per-project Python versions, or override the Python version used with an environment variable. Since pyenv depends on a shell, it’s only supported on Mac/Linux (or Windows using WSL). In this post, I’ll walk through the installation process and a few common usage scenarios.

Before we do that, it’s important to note that pyenv doesn’t handle virtualenv creation and maintenance by itself (but you can use the pyenv-virtualenv plugin for that), or just use virtualenv itself. I will plan on covering those details in a future post.

If you’re using a Mac, consider using homebrew to install pyenv. With this method, you only need one more step to finish the install (skip to step #3 below)

brew update
brew install pyenv

Step #1

As an alternative, you can install using git. It’s recommended to just place this in $HOME\.pyenv, but it could be installed anywhere.

 git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Step #2

At this point in a git install, you need to add two variables to your environment. PYENV_ROOT needs to to point to the root of the install, and your PATH needs to include $PYENV_ROOT/bin at the front. Check out the docs for different shells, but here’s what you’d need to do for zsh, the current Mac default shell as of 10.15 Catalina.

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc

Now if you installed with either homebrew or the git install, you need to add a pyenv init - to your environment. This is step #3 and is required for both install types (this example is for zsh, check the docs for other shells).

echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.zshrc

At this point, you’ll need to restart your shell. If you’ve never built python from scratch before, you’ll need to install the Python build dependencies. This is really just a few simple commands, and once you’ve done this you should be able to build any version of Python. You’ll want to make sure your dependencies are up to date because otherwise you may build a version of Python that will give you strange errors at startup, or it may not build at all.

OK, now that the install is over, you should be able to run pyenv (or pyenv -h) in your shell and see the usage help listing.

❯ pyenv
pyenv 1.2.21
Usage: pyenv <command> [<args>]

Some useful pyenv commands are:
   --version Display the version of pyenv
   commands List all available pyenv commands
   exec Run an executable with the selected Python version
   global Set or show the global Python version(s)
   help Display help for a command
   hooks List hook scripts for a given pyenv command
   init Configure the shell environment for pyenv
   install Install a Python version using python-build
   local Set or show the local application-specific Python version(s)
   prefix Display prefix for a Python version
   rehash Rehash pyenv shims (run this after installing executables)
   root Display the root directory where versions and shims are kept
   shell Set or show the shell-specific Python version
   shims List existing pyenv shims
   uninstall Uninstall a specific Python version
   version Show the current Python version(s) and its origin
   version-file Detect the file that sets the current pyenv version
   version-name Show the current Python version
   version-origin Explain how the current Python version is set
   versions List all Python versions available to pyenv
   whence List all Python versions that contain the given executable
   which Display the full path to an executable

See `pyenv help <command>' for information on a specific command.
For full documentation, see: https://github.com/pyenv/pyenv#readme

Now in getting back to my original motivation, I’m ready to start checking out a new version of Python, and I need to install it. To see how install works, check out the help. You can run help for any of the commands for detailed options.

❯ pyenv help install
Usage: pyenv install [-f] [-kvp] <version>
       pyenv install [-f] [-kvp] <definition-file>
       pyenv install -l|--list
       pyenv install --version

  -l/--list List all available versions
  -f/--force Install even if the version appears to be installed already
  -s/--skip-existing Skip if the version appears to be installed already

  python-build options:

  -k/--keep Keep source tree in $PYENV_BUILD_ROOT after installation
                     (defaults to $PYENV_ROOT/sources)
  -p/--patch Apply a patch from stdin before building
  -v/--verbose Verbose mode: print compilation status to stdout
  --version Show version of python-build
  -g/--debug Build a debug version

For detailed information on installing Python versions with
python-build, including a list of environment variables for adjusting
compilation, see: https://github.com/pyenv/pyenv#readme

If we look for the available versions (using -l or --list), we’ll see a huge list (over 400 versions as of today).

❮ pyenv install -l
Available versions:
  2.1.3
  2.2.3
  2.3.7
  2.4.0
  2.4.1
...
  stackless-3.5.4
  stackless-3.7.5

I just want to check out the latest 3.9 version (3.9.0 as of this writing), so I’ll install it. This will take a little while.

> pyenv install 3.9.0
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.0.tar.xz...
-> https://www.python.org/ftp/python/3.9.0/Python-3.9.0.tar.xz
Installing Python-3.9.0...
python-build: use readline from homebrew
python-build: use zlib from Xcode sdk
Installed Python-3.9.0 to /Users/mcw/.pyenv/versions/3.9.0

Now that it is installed, we can see it in our list. (Note I had earlier installed version 3.6.10).

❯ pyenv versions
  system
  3.6.10
  3.9.0

So now, we need to figure out how to use the different versions of Python we have installed. First, there is a global version. You can set this to just one version, or a chain of versions if you want the version specific shims to find specific versions. For example, based on the Python versions I have installed, I could set my global pyenv versions like this:

pyenv global system 3.6.10 3.9.0

This will cause the system Python to be found first, but the 3.6 and 3.9 versions can be picked up by their specific shims. Having multiple global versions can be helpful for tools that need to be able to run multiple versions of python in the same shell by invoking the specific versions (e.g. running python3.6 or python3 instead of python).

A quick side note: you may be tempted when using pyenv to use the shell builtin function which to determine which python version is in your path. That will always disappoint you, however, since it will just tell you the location of the shim. Use the pyenv which command instead. Use pyenv whence to find all the installed versions that have given Python binary commands installed.

❯ which python
/Users/mcw/.pyenv/shims/python
❯ pyenv which python
/usr/bin/python
❯ pyenv which python3.6
/Users/mcw/.pyenv/versions/3.6.10/bin/python3.6
❯ pyenv which python3.9
/Users/mcw/.pyenv/versions/3.9.0/bin/python3.9
❯ pyenv which python3.7
pyenv: python3.7: command not found
❯ pyenv whence pip
3.6.10
3.9.0

Now a global version is of some use but where pyenv is really helpful is in using local versions. You can force a specific version of Python just for a local directory. So for my motivating example of trying out Python 3.9 features, I can isolate usage of this version to a single directory. pyenv accomplishes this through a .python-version file placed in that directory, so to stop using a local version you can remove that file or use pyenv local --unset to revert to the global version.

❯ mkdir -p ~/projects/python3.9
❯ cd ~/projects/python3.9
❯ pyenv local 3.9.0
❯ python --version
Python 3.9.0
❯ pyenv version
3.9.0 (set by /Users/mcw/projects/python3.9/.python-version)
❯ cat .python-version
3.9.0

Another way to set your Python version is to use a shell specific version. When this option is used, that instance of your shell will use the specified version and ignore the local and global options. The underlying implementation just sets the environment variable PYENV_VERSION, so you can just set this without using a command if you like. You can use the pyenv version command to see which version you are currently using, or pyenv versions to see all versions that are available to you.

❯ pyenv version
system (set by /Users/mcw/.pyenv/version)
3.6.10 (set by /Users/mcw/.pyenv/version)
3.9.0 (set by /Users/mcw/.pyenv/version)
❯ pyenv shell 3.9.0
❯ pyenv version
3.9.0 (set by PYENV_VERSION environment variable)
❯ export PYENV_VERSION=3.6.10
❯ pyenv version
3.6.10 (set by PYENV_VERSION environment variable)
❯ pyenv shell --unset
❯ pyenv version
system (set by /Users/mcw/.pyenv/version)
3.6.10 (set by /Users/mcw/.pyenv/version)
3.9.0 (set by /Users/mcw/.pyenv/version)

And now, let me look at one of those new Python 3.9 features:

❯ python
Python 3.9.0 (default, Oct 25 2020, 16:22:53)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> s = "a test example"
>>> s.removeprefix("a ")
'test example'

And it worked!

Hopefully this has been a useful overview of how pyenv works and why you might want to use it to support multiple Python version in your environment. I find it very helpful for allowing me to have different Python versions for different projects and utilities and still play around with a new version quickly. It is especially useful for installing shell utilities that are generally useful and written in Python (think of things like linters, testing tools, or third party tools like aws tools). In the future, I’ll look at how you can also use virtual environments with pyenv to further isolate your projects.

Comments

[Waylon Walker]

I've used conda for several years now as nothing more than an environment manager. I haven't ran a conda install since moving completely to wsl in 2017. I should take another look at pyenv.