Python: DIY virtualenv

virtualenv in Python is a very well known tools. Almost all tutorial will recommend you to use it. In the past, I'd also wrote about why you should not use the system python.

Being essential it is, starting with python 3.4 it was bundled within python itself as venv module.

There's nothing magic about virtualenv actually. It just a copy (or symlink) to the python interpreter that you already have in your system, plus a couple of other files.

As we have learned when we kids, the best way to learn and understand stuff is by breaking it, or try to build it from scratch. So let's try to build venv/virtualenv without using the built-in module.

As mentioned before, virtualenv is just a collection of directories and files.

mkdir myenv
mkdir myenv/bin

Then we need to find out where is the system python interpreter.

which python3
/usr/local/bin/python3

Let's copy it into our "virtualenv":-

cp /usr/local/bin/python3 myenv/bin/

Now let's try to invoke our "new" python interpreter:-

myenv/bin/python3

We should see the usual prompt such as:-

Python 3.6.0 (default, Jan 24 2017, 16:44:16)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Now our "virtualenv" got it own interpreter, we also must make sure that it has it own site-packages directory, where all the packages we're going to install will live. This is the main reason we use virtualenv, so that packages we're installing do not mess up with the system python or other project virtualenv. Run myenv/bin/python3 and run this code:-

>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', '/Users/kamal/Library/Python/3.6/lib/python/site-packages', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages']
>>> sys.prefix
'/Library/Frameworks/Python.framework/Versions/3.6'

Notice that our myenv directory is none in the output. The docs on sys.prefix says this:-

If a virtual environment is in effect, this value will be changed in site.py to point to the virtual environment. The value for the Python installation will still be available, via base_prefix.

And following the link on virtual environment it says this:-

A virtual environment is a directory tree which contains Python executable files and other files which indicate that it is a virtual environment.

But what indicate a virtual environment? I cheated a bit here and created a new virtualenv using python -mvenv tmp_env and in the newly created virtualenv, I noticed a file called pyenv.cfg which contain this:-

home = /usr/local/bin
include-system-site-packages = false
version = 3.6.0

So let's try adding this file in our virtualenv as myenv/pyenv.cfg. After adding this file, our interpreter will give the following output:-

>>> import sys
>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload']
>>> sys.prefix
'/Users/kamal/myenv'

Notice the sys.prefix value. Now we're onto something. It correctly pointed to our virtualenv directory. However our virtualenv still not in sys.path. I forgot something! We still haven't created the lib directory. So let's do it now:-

mkdir -p myenv/lib/python3.6/site-packages

And check our sys.path:-

>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', '/Users/kamal/myenv/lib/python3.6/site-packages']

Now our virtualenv is added to sys.path! Time to install some packages:-

myenv/bin/python3 -mpip install requests

But we will get the following error:-

/Users/kamal/myenv/bin/python3: No module named pip

Obviously our site-packages directory still empty, so we don't have pip yet. Fortunately getting pip is not that hard.

wget https://bootstrap.pypa.io/get-pip.py
myenv/bin/python3 get-pip.py
Collecting pip
  Using cached https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/ff/47/1dfa4795e24fd6f93d5d58602dd716c3f101cfd5a77cd9acbe519b44a0a9/wheel-0.32.3-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
Successfully installed pip-18.1 setuptools-40.6.3 wheel-0.32.3

Now we have pip installed, let's try again installing requests:-

myenv/bin/python3 -mpip install requests

This time, requests will be installed without problem. Let's check it get installed into the correct location:-

myenv/bin/python3
>>> import requests
>>> requests
<module 'requests' from '/Users/kamal/myenv/lib/python3.6/site-packages/requests/__init__.py'>

It's correct, we have a fully functioning virtualenv now!

Astute reader might noticed that all this far, we have been invoking our python as myenv/bin/python3. What if we want to invoke it simply as python3? In virtualenv, there's a concept of activate that you do after you created the new env. This basically just adding the newly created env directory to the PATH environment variable, which is a list of search path used by the OS shell to find where the particular program exists. The activate script basically look like this:-

OLD_PATH=$PATH
PATH=`pwd`/myenv/bin:$PATH

To deactivate, we replace PATH back to OLD_PATH.

Alternatively, we can also start a subshell with PATH containing the new PATH:-

PATH=`pwd`/myenv/bin:$PATH sh

And to deactivate, we simply exit from the subshell. Personally I don't recommend this "activate" thing. I prefer to invoke the virtualenv interpreter directly, using it full path.