DEV Community

Cover image for Python Virtual Environments tutorial using Virtualenv and Poetry
Dmitriy Zub ☀️
Dmitriy Zub ☀️

Posted on • Updated on • Originally published at serpapi.com

Python Virtual Environments tutorial using Virtualenv and Poetry

#python #poetry #virtualenv #programming

A mini-guided tutorial showing how to use virtual environment and why it's matters on virtualenv and poetry illustrated examples.

What this blog post is about

This blog post is mostly aimed at people who didn't work with it. Here you will find that this is not a complete Python virtual environment reference, it is rather a mini-guided tutorial about:

  • what Python virtual environment is.
  • why bother using it.
  • how to use it.

... with example process of two popular modules: virtualenv and poetry, and software such as IntelliJ IDE via Python plugin, PyCharm Community Edition, VSCode, Windows system, and Git Bash terminal.

📌Note: If you're using JetBrains products you also need to index installed site-packages from the virtual environment, which is a core JetBrains features: code completion, inspections, finding usages, navigation, syntax highlighting, refactoring, and more.

What is virtual environment?

A thing that isolates other things

Python virtual environment is basically a separate folder that creates an independent set of installed packages, Python binaries in its own directory, that isolates any other installation of Python on your computer.

A thing that prevents conflicts

Python virtual environment is used to prevent interfering with the behavior of other applications. Therefore it will prevent packages or Python version conflicts when working with different projects that are running on the same system.

what is venv_03

Why to use virtual environment?

To avoid Python version conflicts

Python virtual environment allows multiple versions of Python to coexist with each other. It will let you work with the old version of Python after installing a newer version all on the same system.

If you try to do it without separated virtual environment things will break pretty quickly:

illustration by xkcd
Python environment - xkcd.com

To avoid package version conflicts

Say you're on two projects, two of them are using serpapi library which is installed globally (system-wide) with a 1.15 version.

Project_1 depends on the 1.05 version and Project_2 depends on the 1.08 version. What will happen if each project tries to import a serpapi library...

Example without the virtual environment

Since Python doesn’t distinguish between different versions of the same library in the /site-packages directory, this leads to the problem when you have two projects that require different versions of the same library and globally installed library have a completely different version.

why to use venv_1_04

Example with the virtual environment

When using a Python virtual environment you can use different versions of the same library or different versions of the Python separated by different virtual environments - folders.

why to use venv_2_04

📌Note: You can install globally different versions of site-packages and use them but as stated before it would become a mess pretty quickly and could break system tools or other projects.

How to use virtual environment?

Let's look at examples of how to use Python virtual environment from the initial install, creating and activating environment, adding dependencies using virtualenv and poetry modules, and deactivating virtual environment when done.

how to use venv_1_07

An example process of using Virtualenv

Install virtualenv:

$ pip install virtualenv
Enter fullscreen mode Exit fullscreen mode

Create environment folder inside the current package/project directory:

$ python -m venv env
Enter fullscreen mode Exit fullscreen mode
Code Explanation
-m executes module venv
env is a folder name created by venv module. The name of the virtual environment may be different, for example kitty or yolo.

Activate environment:

# On Windows
source env/Scripts/activate

(env)
Enter fullscreen mode Exit fullscreen mode 
# On Linux
$ source env/bin/activate

(env) $
Enter fullscreen mode Exit fullscreen mode
  • (env) indicates that you're in the virtual environment.

Add site-packages (third-party libraries) to the activated environment based on the folder you've created.

Add the latest version:

(env) $ pip install google-search-results
Enter fullscreen mode Exit fullscreen mode

Add specific version using equals == sign:

(env) $ pip install 'google-search-results==1.3.0'
Enter fullscreen mode Exit fullscreen mode

📌Note: if you're on Windows and using Command Line Prompt, use double quotes " when specifying versions:

pip install 'google-search-results==1.3.0'
ERROR: Invalid requirement: "'google-search-results==1.3.0'"
Enter fullscreen mode Exit fullscreen mode

Add specific version without overwriting lower version(s):

(env) $ pip install -I 'google-search-results==1.3.0'
Enter fullscreen mode Exit fullscreen mode

A quick look at how you can install site-package (virtualenv) and create a virtual environment for a specific Python version:

# For Windows:
# install package for specific Python version (https://bit.ly/3pXtHng)
$ py -3.6 -m pip install virtualenv

# create venv for specific Python version (https://bit.ly/3oQ008v)
$ py -3.6 -m venv my_test_env
Enter fullscreen mode Exit fullscreen mode 
# For Linux:
# install package for specific Python version
$ python3.6 -m pip install virtualenv

# create venv for specific Python version
$ python3.6 -m venv my_test_env
Enter fullscreen mode Exit fullscreen mode

Use and index added site-packages inside JetBrains IDE

Refer to activate and index installed packages section with the illustrated process using poetry examples for PyCharm, IntelliJ, and VSCode.

Everything is almost the same except you don't need to find a poetry cache folder via command line to find a path to python.exe file because the env folder is already in your project directory that was created earlier above.

Deactivate virtual environment when done:

(env) $ deactivate
$
Enter fullscreen mode Exit fullscreen mode

An example process of using Poetry

Install poetry:

$ pip install poetry
Enter fullscreen mode Exit fullscreen mode

A quick look at how you can install site-package (poetry) for a specific Python version:

# For Windows:
$ py -3.6 -m pip install poetry

# For Linux:
$ python3.6 -m pip install poetry
Enter fullscreen mode Exit fullscreen mode

Create (initialize) poetry inside current package/project directory:

$ poetry init
Enter fullscreen mode Exit fullscreen mode

The init command will ‘initialize’ an existing directory and create a pyproject.toml which will manage your project and its dependencies:

# pyproject.toml file

[tool.poetry]
name = "virtual environments"
version = "0.1.0"
description = ""
authors = ["Dimitry Zub <dimitryzub@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.9"
google-search-results = "^2.4.0"
# other site-packages will appear here..

[tool.poetry.dev-dependencies]
# development dependencies will appear here..

[build-system]
requires = ["poetry-core=1.0.0"]
build-backend = "poetry.core.masonry.api"
Enter fullscreen mode Exit fullscreen mode

What the heck is pyproject.toml?

In short, pyproject.toml is the new unified Python project settings file that contains build system requirements and information, which are used by pip to build the package/project, and it is almost a replacement for setup.py.

Before pyproject.toml creation, $ poetry init will interactively ask you to fill the fields about your package/project:

Argument Explanation
--name Name of the package/package.
--description Description of the package.
--author Author of the package.
--python Compatible Python versions.
--dependency Package to require with a version constraint. Should be in format package:1.0.0(version).
--dev-dependency Development requirements, see --require

Add dependencies to your package/project:

$ poetry add google-search-results
Enter fullscreen mode Exit fullscreen mode 
...
Creating virtualenv PROJECT-9SrbZw5z-py3.9 in C:\Users\USER\AppData\Local\pypoetry\Cache\virtualenvs

Using version ^2.4.0 for google-search-results

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing google-search-results (2.26.0)
Enter fullscreen mode Exit fullscreen mode

The add command adds dependencies to pyproject.toml and poetry.lock, and installs them.
Creating virtualenv will create a virtual environment with the showed path. Environment creation will be done once.
Writing lock file will write dependencies to poetry.lock file.

poetry.lock prevents from automatically getting the latest versions of your dependencies. You can explicitly write lock command to lock dependencies listed in the pyproject.toml

Add specific version:

# multiple ways
# double quotes ("foo") for Windows CMD 

$ poetry add google-search-results@^2.1.0

$ poetry add 'google-search-results>=1.8.5'

$ poetry add 'google-search-results==1.8.5'

$ poetry add google-search-results@latest
Enter fullscreen mode Exit fullscreen mode

If you specify a constraint (@ or >=), the dependency will be updated by using the specified constraint. Otherwise, if you try to add a package that is already present, you will get an error.

(optional) Install from existing project/package dependencies.

If you're using an already created project that has either poetry.lock or pyproject.toml files, you can install those dependencies to the virtual environment:

$ poetry install
Enter fullscreen mode Exit fullscreen mode

The install command read pyproject.toml or poetry.lock file and installs all listed dependencies.

If there's a poetry.lock file:

  • Poetry uses the exact versions listed in poetry.lock.

If there is no poetry.lock file:

  • Poetry will resolves all dependencies from the pyproject.toml file and downloads the latest version of their files.

(optional) To not install development dependencies, use --no-dev argument:

$ poetry install --no-dev
Enter fullscreen mode Exit fullscreen mode

Use added site packages inside IDE

If using poetry, find a location of the initialized environment first via config --list command. Look for virtualenvs.path in the output:

$ poetry config --list

cache-dir = "C:\\Users\\USER\\AppData\\Local\\pypoetry\\Cache"
experimental.new-installer = true
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = null
👉virtualenvs.path = "{cache-dir}\\virtualenvs"  👉👉👉# C:\Users\USER\AppData\Local\pypoetry\Cache\virtualenvs
Enter fullscreen mode Exit fullscreen mode

A few more steps to do:

  • Go to the virtualenvs.path folder and open created environment folder (in my case its: PROJECT-9SrbZw5z-py3.9).

  • Go to Scripts (Windows) or bin (Linux) folder, copy the full path and add python.exe at the end of the path:

C:\Users\USER\AppData\Local\pypoetry\Cache\virtualenvs\PROJECT-9SrbZw5z-py3.9\Scripts\python.exe
Enter fullscreen mode Exit fullscreen mode
  • virtualenvs.path is needed to find a path to python.exe inside created virtual environment which will let JetBrains or VSCode to index installed site-packages.

If using virtualenv, go to env\Scripts\python.exe folder in your project and copy the full path to the python.exe file and enter it as a System Interpreter inside IDE.

Activate and index installed packages

Currently, if you run the script inside IDE, it will look at the globally installed package (serpapi, for example) and will throw an error because globally there's no such library installed (it won't throw an error if it's installed):

Traceback (most recent call last):
  File "C:\Users\USER\PyCharmProjects\PROJECT\environment.py", line 1, in <module>
    from serpapi import GoogleSearch
ModuleNotFoundError: No module named 'serpapi'
Enter fullscreen mode Exit fullscreen mode

VSCode version

To fix this in VSCode we need to select a virtual environment Python Interpreter and set it as a System Interpreter.

Open command palette CTRL+SHIFT+P and type: Python: System Interpreter (Python extension should be installed).

Vscode_1

Both for virtualenv and poetry, VSCode should automatically detect a proper python.exe file from the virtual environment.

If you don't see a proper path to python.exe from your virtual environment then you need to locate and enter it.

Vscode_2

Now you can run your Python scripts from the virtual environment either by the command line or using VSCode Code Runner extension.

PyCharm version

To fix this in PyCharm we need to add the path to python.exe from the virtualenv folder and set it as a PyCharm System Interpreter which will index all site-packages from the virtual environment:

pycharm_env_1

pycharm_env_2

pycharm_env_3

pycharm_env_4

IntelliJ IDEA version

To fix this in IntelliJ IDEA we need to add the path to python.exe from the virtualenv folder as well and set it as a PyCharm System Interpreter with a few additional tweaks which will index all site-packages from the virtual environment:

intellij_env_1

intellij_env_2

intellij_env_3

intellij_env_4

intellij_env_5

Deactivate virtual environment when done

To deactivate virtual environment in order to use system Python both in PyCharm, IntelliJ IDEA and VSCode you need to set Python System Interpreter back to the default one without virtualenv prefix for example: "Python 3.9 virtualenv.." ==> "Python 3.9", a reverse process of what's being shown above.

Using Gitpod

Gitpod fixes this issue pretty much completely. You don't have to install Python, create environments, it's all there from a click of a button.

Well, of course you need to install something specific for a specific case. Plus, you can code in the directly browser if you really want to.

For example, with Gitpod Chrome extension you can open specifc:

  • Issue
  • PR
  • Branch
  • File
  • Something more I might forget

image

The great thing that you will be immediately on the different branch if you opened issue for example. Its similar to a CI server and will continuously prepare prebuilds for all your branches and pull requests or other types. You can also setup a workflow which let's you do prebuilds (installing site-packages) or start a dev server or something else.

You can install VSCode extensions on the prebuild via Open VSX registry ✨

Conclusion

Different project - different environment

In short, it is better to use a virtual environment if you need to work with several projects at the same time which:

  • use different library versions.
  • use different Python versions.

Installing globally will become a mess

Installing globally different versions of the same library for different projects will quickly turn into a mess, there will be no order, or if there will be a need to install different versions of Python it will turn into a mess of all messes:

image

Acknowledgments

A big thanks to these guys for helping out with the feedback about illustrations:

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

Oldest comments (2)

Collapse
 
ariasbitner profile image
AriasBitner •

I found this book to be really helpful. It does a good job explaining some of the quirks of JavaScript, as well as some easy project walk-throughs to practice some of those ideas. Spell to break someone's relationship

Collapse
 
chris_nelis profile image
Chris Nelis •

There are several Web3 clouds I know about. Data security is an obvious concern with such platforms -- you have to give away your information to third parties you cannot trust. The problem can be solved with confidential computing (protecting data-in-use), but it's a relatively new tech and not the easiest to implement.
I don't know many teams going in this direction. If you're interested in Web3 or cloud computing, you should probably check Super Protocol out. The project is still in an early stage of development though, but they recently launched a public testnet with several premade Python solutions (face recognition, speech recognition, and image classification). As far as I know, they are going to allow (and incentivize) uploading your own.
Anyway, here's a Typeform (you can google their website yourself Wink): superprotocol.typeform.com/testnet
They send out invitations a few times a week.

Read next

imashwani profile image

JavaScript Primitive Data Types

Ashwani Singh -

mcharytoniuk profile image

How to Write a PHP Extension with Zig?

Mateusz Charytoniuk -

jonathanbrizio profile image

Avoiding Code Chaos: The Broken Windows Principle in Software Development

Jonathan Brizio -

sainsec profile image

Information Disclosure

Hussain -