Dead Simple Python (12 Part Series)
Virtual environments. If you've ever done any meaningful work in Python, you've almost certainly heard of these. Chances are, you've even been told they're non-negotiable. Trouble is, you have no idea what they are, much less how to make them.
For my first several dozen attempts at using virtual environments, I managed to get something horribly wrong. They never worked. I hate to admit it, but I don't even know what I did anymore! Ever since I've learned how virtual environments worked, I haven't had a single problem with them.
A virtual environment, or virtualenv as it is sometimes called, is a sandbox where you can install only the Python packages you need.
"Yeah, and what's a package?"
Well, Python is rather famous for being "batteries included." Most things "just work" with a simple
But what if you want something more than the built-in packages? For example, you might want to create a snazzy user interface, and you decide to use PySide2. PySide2, like thousands of other third-party libraries, aren't already built into Python - you have to install them.
Thankfully, installing most third-party libraries is easy! The library authors already bundled the whole library up into a package, which can be installed using a handy little Python tool called
pip. (We'll get to that later.)
But this is where it gets tricky. Some packages require other packages to be installed first. Certain packages don't like certain other packages. Meanwhile, you can actually install specific versions of a package, depending on what exactly you need.
Did I mention that some of the applications and operating system components on your system rely on those Python packages?
If you're not careful, you'll end up with this mess...
This is why we have virtual environments! We can create a different little sandbox for each project, install only the packages we want in it, and keep everything neatly organized. Bonus, we never actually change what Python packages are installed on our system, so we avoid breaking important things that have nothing to do with our project.
pip. Just for the sake of example, we'll also be installing Python 3. If you already have it, reinstalling doesn't hurt; otherwise, you can just skip that part.
On Linux, your distribution's package repository almost certainly has what you need.
sudo apt install virtualenv python3-virtualenv python3-pip
sudo dnf python3-virtualenv python3-pip
On Mac, you can use either Macports or Homebrew to install.
Here's Macports, for Python 3.7. If you want 3.6, change all the instances of
sudo port install python37 py37-pip py37-virtualenv sudo port select --set python python37 sudo port select --set pip py37-pip sudo port select --set virtualenv py37-virtualenv
brew install python pip install virtualenv
On Windows, you'll need to first download and install Python. This should also automatically install
However, if you try to run
pip in the command line, you should download get-pip.py onto your Desktop, navigate to that directory on your command line, and run it via
pip install virtualenv
(Thank you to this guide for the Windows instructions.)
Whew! With that out of the way, let's get to the fun part...CREATING virtual environments!
Once again, a virtual environment is like a sandbox, containing only the packages you choose, and ignoring (by default) all the Python packages installed elsewhere on your system. Each virtual environment resides in a dedicated directory. Conventionally, we name this folder
For each project, I typically like to create a dedicated virtual environment inside the project folder. (If you're using Git or another VCS, there's an additional setup step we'll get to in a moment.)
To create the virtualenv, we first change our working directory in our command line to our project folder. (Remember, that's the
cd command.) Then, we create the virtualenv and its directory in one step.
virtualenv -p python3 venv
The last part of that command,
venv, is the name of the directory you're creating for the virtual environment. Technically, you can call it whatever you want, but like I mentioned before,
venv is the convention.
Note, we're explicitly specifying we want to use
python3 here, although we can give the path to whatever Python executable we want to use (such as
virtualenv -p /usr/bin/python3.6 venv)
If you only have one version of Python installed on your system, you can actually skip the
-p python3 section altogether, and just say
virtualenv venv. Being primarily an Ubuntu user, however, I usually have multiple versions of Python installed, including Python 2.7. Thus, it's easiest just to specify exactly what I want.
If you look at your working directory, you'll notice that the directory
venv/ has been created.
Great, so how do we use this thing?
Actually, it's deliciously simple.
On UNIX-like systems (Mac, Linux, etc.), just run...
On Windows, run...
Like magic, you're now using your virtual environment! On UNIX systems, you'll probably see
(venv) at the start of your command line prompt, to indicate that you're using a virtual environment named
Naturally, if you named your virtual environment something else, like
bob, you'd need to change the activation command accordingly (
One of the wonderful things about virtual environments on a system with multiple versions of Python is that you no longer have to specify the path in your commands. While the virtual environment is activated,
python whatever_your_command_is.py will use the version of Python you selected when you created the
venv. Every time.
Most of us have great expectations for Python's package system. (See what I did there? No? Sigh.)
pip is pretty easy to use, and much easier than it used to be in the bad-old-days. It used to be so clunky, in fact, that someone felt they needed to create something called
pip is now very much painless to use.
To install a package - say,
pyside2, just run...
pip install PySide2
If you want to install a specific version of something, that's easy too.
pip install PySide2==5.11.1
Bonus, you can even use operators like
>= ("at least this version, or greater"), and so forth. These are called requirement specifiers. So this...
pip install PySide2>=5.11.1
Would install the latest version of PySide2 that is at least version 5.11.1, or else later. This is really helpful if you want to make sure someone actually has access to a minimum version of a package (they might not).
You can actually save even more time for yourself and others by writing a
requirements.txt file for your project. On each line, list the name of the
pip package, exactly as you would type it into the
For example, if you had a
requirements.txt file like this...
...you could install all those packages in one shot with...
pip install -r requirements.txt
You can update an already installed package using the
pip install command and the
--upgrade flag. For example, to install the latest version of PySide2, just run...
pip install --upgrade PySide2
You can also upgrade all your required packages at once with...
pip install --upgrade -r requirements.txt
Removing things is just as easy.
pip uninstall PySide2
Great, so we can install, upgrade, and remove things. But how do we know what packages
pip even has to offer?
There are two ways. The first is to use
pip itself to run a search. Say you want a package for web scraping.
pip search web scraping
That will give you a whole ton of results to sift through, but it's helpful if you simply forgot the name of a package.
If you want something a little more browsable and informative, PyPI.org is the official Python Package Index.
Once you've installed the packages you need for your virtual environment, you're good to go! The next time you start the virtual environment, those packages will still be there, exactly as you left them, waiting for you.
No matter who tells you to, never, ever ever, EVER use
sudo pip on a UNIX system. It will do so many bad things to your system installation that your system package manager cannot correct, you will be regretting the decision for the lifetime of your system.
All the problems that
sudo pip appears to fix can be solved with virtual environments.
Friends don't let friends
Great, so how do you get out of the virtual environment, and back to reality...er, ahem, the system.
You ready for this, UNIX users?
I know! Simple, right?
Of course, things are just slightly more complicated on Windows...
Eh, still pretty painless. (Remember, like with activation, if you named your virtual environment something else, you'd have to change that line accordingly.)
So, one last little detail. You may have noticed that most Python files start with something like...
First, this is called a "she-bang" (short for haSH-BANG, or
#!), and it allows the script to be run without
python being tacked onto the beginning of the terminal command.
Second, the above line is very, very wrong. It forces the computer to use a particular system-wide copy of Python, which more or less throws the whole virtual environment thing out the window.
Instead, you should always use the following she-bang for Python3 scripts:
If you happen to have a script which runs on both Python2 and Python3, use:
(By the way, the rules about
python3 officially come from PEP 394.)
Remember that warning earlier, about
venv if you're using a VCS like Git?
Within a virtual environment's directory are the actual packages you installed with
pip. Those would clutter up your repository with big, unnecessary files, and you can't necessarily copy a virtual environment folder from one computer to another and expect it to work anyway.
Thus, we don't want to track these files in our VCS. In Git, you'll have a file called
.gitignore in the root directory of your repository. Create or edit that file, and add this line somewhere in it...
Naturally, if you used a different name for your virtual environment, you'd change that line to match.
Conventionally, every developer who clones your repository will build their own virtual environment, probably using that
requirements.txt file you created.
If you're using a different VCS, like Subversion or Mercurial, check the documentation to see how to ignore a directory like
A lot of Python developers will probably frown deeply at you for putting your virtual environment in the repository folder at all. The main disadvantage to my method above is, if you name your virtual environment anything but
venv (or whatever you put in your
.gitignore, it's going to get committed, and that's bad.
The best habit is actually to keep your virtual environment out of the repository directory altogether. But, if we're honest, most of us actually don't. It just feels more convenient to do it the "wrong" way.
That's why we add
venv to the
.gitignore. You, or someone else, is probably going to stick the virtual environment in your repository directory, so this just helps prevent some accidental commits.
Some of my Python developer friends on Freenode IRC, as well as folks in the comments, pointed out a few extra tricks that will be helpful to virtual environment users.
If you're using Python 3.3 or later, you can actually use
python -m venv venv to create a virtual environment with the name
venv. The first
venv is a command, and the second
venv is a name. Thus, you could also say
python -m venv myvirtualenv to create a virtual environment called
Of course, if you're on a system with both Python 2 and Python 3, be sure you use
python3 -m venv or whatever is appropriate. This trick doesn't work on any version of Python before 3.6.
You can also use the binaries that are part of the virtual environment without actually activating it. For example, you can execute
venv/bin/python to run the virtualenv's own Python instance, or
venv/bin/pip to run its instance of
pip. It'll actually work the same as if you had activated the virtual environment!
For example, I could do this (assuming my virtual environment is
venv/bin/pip install pylint venv/bin/python >>> import pylint
...and it works! Yet,
import pylint will NOT work on the system-wide Python shell still...unless, of course, you installed it on the system. ;)
I've heard a lot of suggestions for using
pipenv, including in the comments section. I won't be covering it in this article, but it has a neat looking workflow, and many vocal fans. You can find out more here: pipenv on PyPI
I hope this guide has completely demystified virtual environments and
pip for you. Naturally, I recommend you keep the documentation under your pillow:
Comics courtesy XKCD
Thank you to
deniska (Freenode IRC
#python) for suggested revisions.