DEV Community

Cover image for Ultimate Solution to Python Virtual Environments: pyenv + virtualenvwrapper
Aly Sivji
Aly Sivji

Posted on • Updated on • Originally published at alysivji.github.io

Ultimate Solution to Python Virtual Environments: pyenv + virtualenvwrapper

#python #tutorial #productivity #terminal

(This post was originally published on Siv Scripts)

I love using the Command Line; it's my favourite part of programming. It wasn't always this way. I used to worry that the commands I entered would destory the delicate equilibrium where everything just worked. Would this be the pip install that ended it all?

Enter virtual environments.

A virtual environment (venv) is a tool that allows us to keep project dependencies isolated. If Project A requires pandas version 0.17 and Project B requires pandas version 0.21, we can create a venv for each project and keep workflows independent. More info in the Python documentation.

It's been around a year since I started using virtual environments in my projects, but I had yet to find the perfect dev workflow.

Tried Anaconda; it was too bloated for non-Data Science work.

Next came virtualenv + virtualevnwrapper. Did everything I wanted, but I had to manually manage Python installations when setting up environments for different versions.

Like everyone who learns Docker, I went thru a containerize all the things phase. The non-existent battery life on my Mac told me this wasn't the most feasible solution.

Then I found pyenv...

Just Right

In this post, I will describe how to install pyenv and pyenv-virtualenvwrapper. I will also walk through the steps required to setup and use a new virtual environment.

pyenv

pyenv lets you easily switch between multiple versions of Python

Background

Anytime we run a command, the OS goes looking for the executible within the list of directories in the $PATH environment variable.

$ echo $PATH
/Users/alysivji/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/Users/alysivji/bin
$ which python
/usr/bin/python
$ which python3
/usr/local/bin/python3
Enter fullscreen mode Exit fullscreen mode

We see both the default Python and Python3.6 installations are located in the $PATH var.

When we install pyenv, we add an additional directory of shims to the front of our $PATH. Each shim intercepts commands (python, pip, etc) and redirects them to the appropriate location.

Installation

Check the project's Github for detailed installation instructions. These instructions assume you have macOS + Homebrew installed.

brew update
brew install pyenv
Enter fullscreen mode Exit fullscreen mode

Add the following line to ~/.bash_profile:

eval "$(pyenv init -)"
Enter fullscreen mode Exit fullscreen mode

Let's make sure our installation worked:

$ source ~/.bash_profile
$ echo $PATH
/Users/alysivji/.pyenv/shims:/Users/alysivji/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/Users/alysivji/bin
Enter fullscreen mode Exit fullscreen mode

Looks good! The shims at the front of our $PATH variable will intercept and redirect all python-related commands.

Commands

We use pyenv install --list to get a list of all Python versions available for installation. Let's install a few different versions of Python so we can see how pyenv works.

pyenv install 2.7.13
pyenv install pypy-5.7.1
pyenv install 3.6.2
pyenv install 3.7-dev
Enter fullscreen mode Exit fullscreen mode

Set the Python we want as our default version:

$ pyenv global 3.6.2
$ pyenv versions
  system
  2.7.13
* 3.6.2 (set by /Users/alysivji/.pyenv/version)
  3.7-dev
  pypy-5.7.1
Enter fullscreen mode Exit fullscreen mode

How does the shim know which version of Python to use? It goes down the hierarchy until it finds what it's looking for.

Activate different versions of Python using the pyenv shell [version] command.

$ which python
/Users/alysivji/.pyenv/shims/python
$ python
Python 3.6.2 (default, Aug 24 2017, 00:00:01)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
$ pyenv shell 2.7.13
$ which python
/Users/alysivji/.pyenv/shims/python
$ python
Python 2.7.13 (default, Aug 24 2017, 00:14:24)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Enter fullscreen mode Exit fullscreen mode

Notice how the pyenv shims take care of everything for us. Next, we'll learn how to create a venv.

pyenv-virtualenvwrapper

pyenv-virtualenvwrapper is a pyenv plugin ... to manage your virtualenvs with virtualenvwrapper

Installation

Detailed instructions are on Github. macOS + Homebrew users can follow along:

brew install pyenv-virtualenvwrapper
Enter fullscreen mode Exit fullscreen mode

Append to ~/.bash_profile:

export PYENV_VIRTUALENVWRAPPER_PREFER_PYVENV="true"
export WORKON_HOME=$HOME/.virtualenvs
pyenv virtualenvwrapper_lazy
Enter fullscreen mode Exit fullscreen mode

Reload the shell:

source ~/.bash_profile
Enter fullscreen mode Exit fullscreen mode

Commands

Anytime we install a new version of Python, we will need to install virtualenvwrapper. This is done with either the pyenv virtualenvwrapper or pyenv virtualenvwrapper_lazy (preferred) command:

$ pyenv shell 3.6.2
$ pyenv virtualenvwrapper_lazy
Collecting virtualenvwrapper
Collecting stevedore (from virtualenvwrapper)
  Using cached stevedore-1.25.0-py2.py3-none-any.whl
Collecting virtualenv (from virtualenvwrapper)
  Using cached virtualenv-15.1.0-py2.py3-none-any.whl
Collecting virtualenv-clone (from virtualenvwrapper)
  Using cached virtualenv-clone-0.2.6.tar.gz
Collecting pbr!=2.1.0,>=2.0.0 (from stevedore->virtualenvwrapper)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting six>=1.9.0 (from stevedore->virtualenvwrapper)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: pbr, six, stevedore, virtualenv, virtualenv-clone, virtualenvwrapper
  Running setup.py install for virtualenv-clone ... done
Successfully installed pbr-3.1.1 six-1.10.0 stevedore-1.25.0 virtualenv-15.1.0 virtualenv-clone-0.2.6 virtualenvwrapper-4.7.2
Enter fullscreen mode Exit fullscreen mode

Now we can use the virtualenvwrapper shell commands to manage our environment:

$ mkvirtualenv venv_test
Using base prefix '/Users/alysivji/.pyenv/versions/3.6.2'
New python executable in /Users/alysivji/.virtualenvs/venv_test/bin/python3.6
Also creating executable in /Users/alysivji/.virtualenvs/venv_test/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /Users/alysivji/.virtualenvs/venv_test/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/alysivji/.virtualenvs/venv_test/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/alysivji/.virtualenvs/venv_test/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/alysivji/.virtualenvs/venv_test/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/alysivji/.virtualenvs/venv_test/bin/get_env_details
(venv_test) $ which python
/Users/alysivji/.virtualenvs/venv_test/bin/python
(venv_test) $ python
Python 3.6.2 (default, Aug 24 2017, 00:00:01)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Enter fullscreen mode Exit fullscreen mode

We can also switch between virtual environments using the deactivate and workon commands:

(venv_test) $ deactivate
$ workon venv_test
(venv_test) $ python
Python 3.6.2 (default, Aug 24 2017, 00:00:01)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Enter fullscreen mode Exit fullscreen mode

That's pretty much it. We can create a virtual environment for any version of Python in two commands.

Notes

Conclusion

We should always look for ways to optimize our development workflows. There are tools out there to help us manage complexity and make our life easy. Find these tools. Use them. It'll make you more productive in the long run.

Additional Resources

Top comments (5)

Collapse
 
taylor profile image
Taylor D. Edmiston •

Great post! I've been describing my setup of pyenv + virtualenvwrapper via homebrew for a while now as I think it's the best of all words when one needs to develop on multiple subversions of Python.

When I mention it especially to beginners to Python they can be a little overwhelmed by what it takes to get all of this setup the first time. For that case, I'm going to send them to this guide next time. Thanks for putting all of it together in one place!

Collapse
 
icicleta profile image
Iciar Novo •

Nice, I've been searching for a while the best way to install python and virtual environments from scratch and nothing was convincing me. Pyenv seems to be what I was looking for. Your tutorial took me easily with the installation. After I found out about pipenv, the article said that it was the last and best way to setup python and env's. But I find this one easier.
One question please, do I need the python version installed with brew?, since now is pyenv the one installing python, I thought I could uninstall the version on Cellar.
Thanks for the article.

Collapse
 
alysivji profile image
Aly Sivji •

You should be able to uninstall the brew Python. pyenv controls everything via the shims.

Collapse
 
icicleta profile image
Iciar Novo •

Great! thanks!

Collapse
 
ybryan profile image
Bryan Yu •

THANK YOU! This read like my entire history with python

  1. Homebrew + Conda. Messed up all my paths, no idea what linked to what, reformat.
  2. Tried just conda. Not enough community
  3. Docker everything. Too much containers
  4. Back to virutalenvs and pyenv

Read next

hystou profile image

Solutions to Fix WHEA_UNCORRECTABLE_ERROR Blue Screen Error in Win11（Fix 1-3）

Hystou -

intersystemsdev profile image

Step by step guide to create customized chatbot by using spaCy (Python NLP library)

InterSystems Developer -

alakkadshaw profile image

How to use TURN server with PeerJs

alakkadshaw -

danielox profile image

Python Data Engineering: Comprehensive Workflow for Data Modeling, Analytics with DuckDB

danial.shabbir -