If you have been developing in Python, you must have had a tough time with dependency conflicts when working on multiple projects, and it's kind of inevitable you have to face this issue at least once in your python development career before you look for solutions and come across the term virtual-environments or virtualenv.
If you have faced such issues then this post is exactly for you to give an understanding of how python run time works and why virtual-environments are needed.
Pip's Default Choice
Pip is the default package manager for python just like npm is for Node, and by default, pip will install any package globally and this is where things start to get a little out of hands. Now imagine you are working on two projects both of these projects have a common dependency, and for the sake of simplicity, we will call it dependency A. Project 1 requires version 1.0.0 of dependency A and Project 2 requires version 2.0.0 of dependency A and since at any given time the python interpreter can only maintain one version of the dependency A we won't be able to run these two projects without having to manually install the correct version every time we switch between projects which will soon become a hindrance in our development flow. This was a very basic example now imagine a specific version of dependency A is dependent on a specific version of dependency B and you can kind of get an idea on how this could potentially break other projects.
Virtual-Environments to Rescue
This is where Virtual Environments come to play and facilitate us in creating isolated environments for each project. Virtual Environment is basically a fancy term for a directory within your project that encapsulates executables and packages that are required to run the project. Virtual Environment contains links to the python interpreters that are installed on the disk and this essentially means that by tweaking this property you can define which python-runtime you want to run your project with, sounds cool right let's see how we can set up a virtual environment in our project.
Steps
- pip3 install virtualenv.
- cd < project-path >.
- virtualenv < name-of-your-env >. (Generally a good practice to name it venv)
- source ./< name-of-your-env >/bin/activate. (To activate your environment)
- deactivate (To deactivate)
- virtualenv -p < path-to-python-interpreter > < name-of-your-env > (To switch to python interpreter of your choice)
And that's it once the virtual environment is created you can install any package and that will be installed locally within that project without affecting your global packages and creating dependency conflicts.
Now if you use a modern IDE like PyCharm you would have noticed that every time you create a new project PyCharm creates a virtual environment for you and configures it, and this is exactly what's going on behind the scenes. Now that you know what virtualenv is why its a handy tool when developing in python you should always try to use virtual environments, this would greatly increase your productivity and save you a lot of time that is spent figuring out dependency conflicts.
Food for thought
If you have had the chance of working in Node or Javascript you must have noticed that npm deals with this very nicely by default installation in the local space rather than global-space plus the package.json file helps maintain the dependencies and differentiates between dev-dependencies and production dependencies in an elegant manner. Managing dependencies for a python project with a requirements.txt file can become very tricky when your dependencies have sub-dependencies etc. Although both have their pro's and con's I have always wondered why pip installs packages globally and why there isn't any project configuration file when working in Python. Let me know in the comments if you know why thing's happen like this in python.
If you like this post do share it with your friends.
Top comments (4)
This will become less necessary once PEP-582 lands. They are proposing a local packages directory per project more like node_modules etc.
python.org/dev/peps/pep-0582/
Also I would highly recommend a dependency manager like poetry instead of using pip directly. It will take care of everything from creating a virtualenv and managing all of your dependencies and their versions. It is much more like npm in this regard, with a lock file that can give you reproducible builds.
This sounds really nice, I didn’t know about poetry I have seen pipenv in action I’ll give poetry a try thanks for the recommendation
Poetry is very similar to pipenv in both the cli args and purpose. I have used both and poetry is much better in my opinion. It is also more actively maintained.
Will definitely give it a try but definitely looking forward to a local packages directory 🙏