ViP-IPykernel: Start Jupyter Kernels in the Closest Virtual Environment

Venvs are essential for robust analysis, but juggling them when using Jupyter can be overwhelming, ViP-IPykernel aims to help

Logo is a modified version of the IPykernel logo, created by Project Jupyter. This project has no affiliation with Project Jupyter or IPython

Reproducibility has been a hot topic for years in STEM subjects, more and more people are realising how important it is for your work to be reproducible, both from a purely scientific point of view but also as a way to facilitate and improve collaboration between colleagues.

When using Python, an essential element of having reproducible work is using well defined isolated environments so that you can be sure of what packages are available and required for your code to execute correctly.

A common way to do this is to use Python venv's, however using Jupyter Notebooks with virtual environments can be a bit awkward for a few reasons, this post will go through these problems and explain how ViP-IPykernel tries to help avoid them. If you're already thinking that notebook automatically using the virtual environment they're in would be helpful then feel free to skip the explanation and check out the project here: RobertRosca/vip-ipykernel

Many people will install Jupyter under their users home directory (i.e. with the pip install --user flag), or use a system provided Jupyter installation. If you do this, and then use virtual environments to isolate your projects you have a few options:

1. Install Jupyter in every venv, and then always activate and run Jupyter out of the environment you want to work with. This way any notebooks you open in Jupyter, when using the default python3 kernel, will use the python environment set up for the project you started Jupyter in.
2. Install just IPykernel in every venv, and then run python3 -m ipykernel install --user --name PROJECT-NAME, now when you start your user or site-level installation of Jupyter you can open your notebooks and select the relevant kernel for your project.

The first approach has the downside of being a bit time consuming, Jupyter has a lot of dependencies, and it also adds in the additional step of always having to activate the correct environment before starting Jupyter. Mild inconveniences like these add up over time and create a barrier which may discourage people from using best practices, even if they know they should create an environment for each separate project. Plus if you're using a remote or centralised Jupyter-Hub server this isn't really an option.

The second approach has a similar downside of the mild inconvenience of having to install IPykernel in every environment, and then having to create a new kernel for each project; if you have a lot of projects you can quickly end up with dozens of kernels, and if you don't have a good naming system you may accidentally end up overwriting them or forgetting what does what.

Additionally, setting a custom kernel for each notebook can impact collaboration negatively in a few ways:

• if you share a notebook or commit it to a common git repo then collaborators will have to set the kernel back to the default python3 one if they don't use virtual environments, or use them and start Jupyter from within the environment (as in option 1)
• alternatively they'll have to change it to their own kernel if they do use virtual environments to provide kernels (as in option 2), and once they save and commit their changes then everybody else will end up modifying the kernel again each time
• committing notebooks saved with non-standard kernels may affect some automated pipelines for testing or documentation which may execute the notebooks

These seem like minor problems, but if you work with notebooks daily, or if you're trying out Jupyter notebooks for the first time, these small annoyances can be enough to make you think "Eh, I can't be bothered to make another kernel, using a venv isn't that important anyway".

This project has no affiliation with Project Jupyter or IPythonWhich brings me to ViP-IPykernel: an IPython kernel which automatically runs python out of a Virtualenv in a Parent directory (hence ViP).

The project is pretty simple, all it does is look for a virtual environment (currently only checks for .venv or venv directories but this will be expanded later) and if one is found in the directory the notebook is in, or in one of the parent directories, it runs IPykernel out of that environment automatically. If no venv is found then it falls back to the python environment vip-ipykernel was installed with.

If you want the kernel to fall back to the standard Python kernel behaviour when no environment is found then this kernel should be installed under your user space, not inside a virtual environment, alternatively, it can be installed inside a venv if you'd like the fallback to be to a different environment to your users one.

After installation, you can then either override the default python3 kernel with vip-ipykernel or install it as a separate kernel:

pip install --user vip-ipykernel
#  To override the default kernel:
python3 -m vip_ipykernel.kernelspec --user
#  To install as a new kernel:
python3 -m vip_ipykernel.kernelspec --user --name vip-ipykernel

Overriding the default kernel is a useful approach as that way you never have to change the kernel a notebook is using: leave it as the default python3 kernel and as long as the notebook is in a directory or subdirectory where a .venv exists it will always use that environment, avoiding both the issues of creating a kernel or installing Jupyter for every environment.

This way you get the best of both worlds, you can use virtual environments to manage your dependencies, and you don't have to juggle multiple kernels or run Jupyter out of a specific environment each time.

Small quality of life improvements like this are pretty trivial, but when trying to convince others (or yourself) to change workflows and muscle memory built up and reinforced by years of experience, any small perceived downside can be seen as 'enough' of a reason to stick with what you've always done.

If this sounds useful to you, the project is available on PyPI and can be installed via pip, or you can view the code here: RobertRosca/vip-ipykernel