DEV Community: Mario Kostelac

Github Actions: using python version from .python-version file (pyenv)

Mario Kostelac — Fri, 09 Apr 2021 08:41:56 +0000

I was recently creating a CI pipeline for a toy ML project. It was to make sure that our training accuracies stay within some thresholds as we change the code. We were hosting the project on GitHub so GitHub Actions seemed like a great fit to run our CI.

I’d found GitHub’s actions/setup-python@v2 to set up the specific python version, but it wasn’t clear how to set a version used pyenv (a great tool for managing multiple python versions btw), one written in the .python-version file.

Turns out combining action contexts and environment variables does the trick.

One step reads the .python-version file and writes that to environment variable
actions/setup-python@v2 installs the version from the environment variable.

Here's the snippet defining the job steps

 steps:
    - uses: actions/checkout@v2
    - name: Get python version
      run: |
        python_version=$(cat .python-version)
        echo "python_version=${python_version}" >> $GITHUB_ENV
    - name: Set up Python ${{ env.python_version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ env.python_version }}

Importing packages in Jupyter notebooks

Mario Kostelac — Thu, 07 Feb 2019 00:00:00 +0000

Seeing “ImportError: No module named tensorflow” but you know you installed it? Sometimes you can import packages from the console, but not from the Jupyter notebook? !pip install tensorflow sometimes just does not work?

It’s not about you. It’s not about python being flaky.It’s actually about your machine running multiple python installations (environments).Let’s build a basic understanding of what’s happening there and what causes your packages missing even after being installed. Once you understand that, you’ll have fewer problems installing and importing packages you need.

You are running multiple environments of python. That is fine!

Not fine like this! It is actually fine and not your fault. Because not all python2 tools are ported to python3, it is possible that your operating system runs both versions without you touching anything.

For example, I have both python 2 and python 3 installed. When I call python in my console – python 2.7.10 gets invoked; when I call python3, python 3.7.1 is invoked. It gets even better. I have multiple environments of python 3.6.3 (yes, all the same version) and they load dynamically depending on the project I am currently working on (thanks to conda). I create and destroy python environments daily.

So how do I know what environment is currently running? Easy! By running import sys; sys.prefix in interactive console. That reveals what environment of python you are currently in.

Different python environment means different sets of packages

Unless you’ve modified $PYTHONPATH variable (it’s ok if you haven’t heard of it), each of these environments will use a completely separate set of packages. That is the reason you can’t import the package and you know you installed it. You installed it in the wrong environment!

But you’re running Jupyter and not the console and how does it all fit together? We’re getting there just now.

Jupyter kernel you use is not using the same python environment as your console.

Jupyter does not run python the same way your console does. It has a concept of a kernel (if you are not familiar with that concept, think about it as python environment registered with Jupyter). The kernel running your notebook likely uses a different python environment and definitely does not have all the environment variables set as your console does. That is fine, too. We know how to figure out which environment is running our code so we can do exactly the same in Jupyter notebook.

Aha!!! You know which environment Jupyter uses. Now you just have to:

make sure your console (temporarily) uses the same python environment as your Jupyter notebook.
install the package with conda install or pip install (if you don’t know what is the difference, quickly go to this guide).

Cool, cool, cool. But you don’t know how to make your console use the same environment? All good, we’ll do it from your Jupyter notebook, but not like the last time you did it.

You are installing packages straight from Jupyter notebook and you are doing it wrong.

Browsing through StackOverflow about similar issues made me realise people are suggesting the thing that won’t work most of the time. If you are installing packages by running

!conda install tensorflow
# or if you want to use pip
!pip install tensorflow

you are using very fragile commands (if run in notebook) and that’s the reason packages you installed can’t be imported. It is not fine this time. But we will fix it 🙏. If you are interested in low-level details about why it does not work, read this great blog post from Jake Vanderplas.

Instead, use these commands:

import sys
!conda install --yes --prefix {sys.prefix} tensorflow
# or if you want to use pip
!{sys.executable} -m pip install tensorflow

and you will not have problems with damn ImportError again.

Understanding these concepts helped me and my teammates to understand what’s happening and why we were getting these errors. If you think I made a mistake, I missed something or you just want to say thanks, please do that in comments 🗣.

What is the difference between conda and pip and how not to break your environment again?

Mario Kostelac — Sat, 02 Feb 2019 00:00:00 +0000

One of the first things I’ve noticed while coming into Machine Learning Engineer role was package management mess.People often ask me “Should I use conda or pip to install packages?”. “Is conda just python with preinstalled packages?”. “I am getting some compiler errors. I thought we’re using just python?!”.

And I did not know what to answer. Everybody is mix and matching words pip, conda and install until it starts working! And it usually works, until it breaks your environment completely. And you do not know why it broke so you remove and install all packages from scratch. I was there. I can guarantee it does not have to be like that!

After you read what pip and conda do and how they work, you won’t be breaking your environment again.

pip

pip (recursive “pip Installs Packages”) is a Python package installer. It downloads and installs packages you want to use. Conda does that as well. The difference is that pip does just that and nothing else!

Pip is very simple:

it does not support multiple versions of same package installed - that’s why you can’t have to projects using different versions of same package!
for many packages, it uses your compiler, which could be incompatible - this is where your compiler errors are coming from
it does not know anything about python versions - python 2, python 3? Python 3.5 or 3.6? It can’t help with that.

conda

conda does far more than pip! It’s made for people like you, like me, people doing machine learning and data science.

It allows you to use different versions of the same package.
It gives you the power to use any python version you want!
It compiles packages before publishing to the package repository so you don’t get compiler errors.

(It actually does far more, but we’ll stop here 🤓)

Conda can do that because is a package manager, but also an environment manager. It isolates different python and package versions so they do not interact with each other.

Use one conda per project to stay away from the trouble!

If you don’t have it installed, go install anaconda!

System that helps me stay away from the trouble is to create one conda environment per project. Whenever you start working on a new project, just run conda env create -n .

If you have a specific version of python you want to use, run conda env create -n <project_name> python=3.5 (for version 3.5).

It’s going to install some basic packages and give instructions for activating created environment. This is important to remember. You have to activate the environment explicitly every time you want to use it in new console session. But it’s easy, just type conda activate <project_name>.

Installing packages

Once you activate the conda environment, you can install any package you want - just type conda install <package_name>. Same as you would do with pip.

But I often have these once-off notebooks that don’t fit into any project! 🔧

No problem, just create a onceoff environment! (hint: conda env create -n onceoff). It’s possible you will break this one, but you remove it with conda env remove onceoff. It is much faster than uninstalling all pip packages or all of conda 🍻.

I’ve forgotten the environment name 🤦🏼‍♂️

conda env list will list all existing environments.

This setup saved me hours of debugging issues, figuring out it’s the wrong package version and then reinstalling everything from scratch. I’m interested to hear how it works for you! Or if it does not, what does? 🗣