There are two hard problems to solve in a Python project: installing packages, and installing packages in other projects.
This post was originally published on my website. Check it out for more awesome content!
Over the years, the challenges I’ve come across in Python projects has narrowed down to two topics 1 :
- Installing packages for just my current project – not for all projects using that version of Python
- Using different Python versions for different projects
Problem (1) can be solved with Python’s built in virtual environments, but they don’t automatically activate when you enter a project folder. As a result, I often forget to run
source venv/activate and then accidentally install a bunch of packages to my system Python.
I highly recommend Pyenv to solve both problems. Pyenv is a fantastic tool for managing Python environments simply by changing your current folder. Pyenv also integrates with
virtualenv so that you can create virtual environments for any Python version you’d like to use.
By the end of this post you will know how to:
pyenvon your computer
- Build a version of Python with
- Change your global and local folder settings to run the new version of Python
virtualenvto create a new virtual environment
First, install Xcode tools and a few development libraries that the Python build step will need:
# Note: These are macOS specific commands
# Update XCode and Homebrew
# Install libraries you'll need to build Python from source
brew install openssl readline sqlite3 xz zlib
# Install pyenv
brew install pyenv
Note: The installation instructions for
pyenv change frequently. These are the steps for
pyenv==2.3.1 on macOS. If you are using a different version or OS, see the
pyenv installation guide.
If you’re using a zsh shell, run these commands to add
pyenv to your dot files. If you’re using another shell, follow the installation steps for your shell under the Shell environment section of the pyenv README.
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc
Restart your terminal for the changes to take effect, and verify your installation by entering the command
which python3 and confirming that
pyenv/.shims is somewhere in the output:
$ which python3
Next let’s install Python 3.8.2. I chose this particular version of Python because I use it in a few of my side projects, so you can use any other version of Python if you’d like.
Recently, I came across a bug in macOS Big Sur that prevents libraries like
pandas to use the
lzma package. Credit to the thread for pyenv Issue #1737 for the below instructions to correct the 3.8.2 build.
To install Python 3.8.2, first reinstall the
bzip2 libraries via Homebrew:
brew reinstall zlib bzip2
~/.bashrc if you’re using bash), add the below flags for the Python compiler to read:
export LDFLAGS="-L/usr/local/opt/zlib/lib -L/usr/local/opt/bzip2/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include -I/usr/local/opt/bzip2/include"
Then install Python 3.8.2 using the below command. This may take some time to compile.
CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" \
LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.2 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
Verify that you built and installed Python 3.8.2 by checking what versions of Python
$ pyenv versions
Pyenv has two types of versions:
local– the Python version for your current directory
global– the default Python version to use if no
localversion is set.
When you switch directories in your terminal, Pyenv checks for a
.python-version file in the root of that directory. If there’s no
.python-version, Pyenv will search each parent folder until one is found.2
You can set what version of Python
pyenv will use in your current folder with
$ pyenv local 3.8.2
You can check
.python-version to see what version is used:
$ cat .python-version
pyenv version will also tell you what version of Python you’re currently using as well as what config file set it:
$ pyenv version
3.8.2 (set by /Users/nicky/Developer/my-project/.python-version)
If you move to a different folder without a
.python-version, Pyenv will use your
global setting, which is
system by default:
$ cd /path/to/other/folder
$ pyenv version
system (set by /Users/nicky/.python-version)
pyenv superpowers by installing pyenv-virtualenv. With this plugin,
pyenv can manage virtual environments like
venv and Conda environments.
Install the plugin with Homebrew:
brew install pyenv-virtualenv
And add the following to your shell’s
.rc file (such as
eval "$(pyenv virtualenv-init -)"
To create a virtual environment for the Python version you’re using with
pyenv virtualenv [version] [new environment name]. For example, to create a new
venv for a sample project using 3.8.2:
pyenv virtualenv 3.8.2 my-project-3.8.2
While not required, I recommend adding the Python version to your environment names to manage them easier.
You can also create a new virtual environment using the current
pyenv Python version:
pyenv virtualenv my-default-venv
pyenv, your virtual environment will automatically activate whenever you move into that folder!
Below is a list of commands for
pyenv that I use often.
To show all of your Python versions and all virtual environments:
To set a different version for the
python3 commands for your
# (This works the same with `local` as well)
pyenv global [python version] [python3 version]
To create a new virtual environment and use it locally:
pyenv virtualenv [python version] [new environment name]
pyenv local [new environment name]