DEV Community

Cover image for Dead Simple Python: Virtual Environments and pip

Dead Simple Python: Virtual Environments and pip

Jason C. McDonald on January 13, 2019

Like the articles? Buy the book! Dead Simple Python by Jason C. McDonald is available from No Starch Press. Virtual environments. If you've eve...
Collapse
 
cwrinn profile image
Christopher Wrinn • Edited

Check out pipenv, it so very nicely married pip and venv/virtualenv. I've quite enjoyed that it tracks the requirements for me as I go.

Collapse
 
kip13 profile image
kip • Edited

If you're gonna start with Python 3.6+, venv module (built in) is my recommendation, of course only if you will work with 3.6+ version and you're not looking for previous versions.

Good start for this good serie !

Collapse
 
pencillr profile image
Richard Lenkovits

Nice one! Could add pip freeze step for requirements.txt creation, it results in better dependency management, as it handles transitive cases too.

Collapse
 
ardunster profile image
Anna R Dunster

What are your thoughts on using Anaconda? I've done a couple online courses and so far they've all started with installing python via anaconda. It has its own package management and virtual environment stuff (or at least its own commands for them).

Collapse
 
codemouse92 profile image
Jason C. McDonald • Edited

Honestly, I've never used it. I know Anaconda is a Python distribution platform geared specifically towards data science. If you like it, by all means, enjoy it. :3 But you should know how regular Python works, and then find out what makes Anaconda different. I don't know much about it at all.

Collapse
 
ardunster profile image
Anna R Dunster

Makes sense. Everywhere I've seen commands they look pretty similar. I wouldn't be surprised if conda is using a lot of the same stuff behind the scenes, but I don't actually know. (it is definitely recommended to NOT use pip to install anything if you're using conda, though...or if you absolutely must, do it last.)

I'm not sure how I like it, comparatively, having not done anything with Python without it. Sometimes it seems like more overhead than necessary, though.

Collapse
 
ravin309 profile image
Ravinder Kumar

Hello Jason,
I found this article very informative and point-to-point coverage on topics.
Though I still have doubts on she-bang. As you said above if we write !#/usr/bin/env as first line in python script we don't need to call it with 'python' command on terminal. So I tried this by writing 'test.py' on Linux but this script is not getting executed without typing 'python' onto the beginning of the terminal command and it gives error : 'test.py' is not a command..
So I didn't understand the use of she-bang properly.
Are there any exceptions to these?

Collapse
 
codemouse92 profile image
Jason C. McDonald

That error has nothing to do with Python at all. Running just...

test.py
Enter fullscreen mode Exit fullscreen mode

...is going to search your system environment paths for an executable file called test.py, and I can practically guarantee your project folder isn't part of that environment path.

Thus, you need to run...

./test.py
Enter fullscreen mode Exit fullscreen mode

The . means "current working directory", and then you build the relative path from there.

Collapse
 
tomekbuszewski profile image
Tomek Buszewski

Hi, nice article, but honestly I've abandoned virtualenv in favor of Docker. What are your thoughts about this tool?

Collapse
 
codemouse92 profile image
Jason C. McDonald • Edited

Docker has a lot of uses, but merely as a replacement for virtual environments? It's overkill. If a virtualenv is a sandbox, Docker is a clean room. You can use it, naturally, but if the only thing you were needing was a controlled set of Python packages, it's a lot of wasted time and disk space.

Now, if you actually need to control your environment beyond what virtualenv allows you to control, Docker is fine. (It makes sense for web development, for example.)

Just not as a straight virtualenv replacement.

Collapse
 
thomasjunkos profile image
Thomas Junk惄 • Edited

I do not share the point of docker being "overkill" because if one is fluent with it, there would be no "time wasted". And disk space is cheap - as long as you aren't on a Mac (SCNR - But admittedly one of my box is a Mac and I am too guilty of suffering from the lack of a bunch of GBs).

I put it more mildly: It has no advantages over virtual envs if you do not use docker for deployment.

Thread Thread
 
codemouse92 profile image
Jason C. McDonald • Edited

It has no advantages over virtual envs if you do not use docker for deployment.

Well, and see, that's why I said that if it was merely a replacement for virtualenvs, it was overkill. A virtual environment only replicates the bare minimum necessary, in terms of the Python interpreter and libraries you're using.

If you're actually deploying with Docker, that's another story altogether.

First, not everyone has Docker configured on their machine. Second, resources like disk space are not always "cheap" on any system. (Macs aren't the only machines to suffer from low disk space.) Internet may be limited, slow, or inexpensive, making a Docker build prohibitive. Time may be of the essence, RAM might be limited, or CPU might be in demand. There are so many scenarios in which Docker is just not going to be work.

Virtual environments are the one canonical "works everywhere" solution for sandboxing Python. Anyone and everyone who can run Python 3 (and most versions of 2) can create a virtual environment. They require very little overhead, generally minimal network time, and nothing in the way of extra processing power. Ironically, they're one of the fastest and most reliable ways to deploy a Python project in a Docker!

To this aim, your project should be configured properly so that it can be run in a virtual environment. (It doesn't take that much work; certainly less work than a Dockerfile does.)

The third problem is, if the Python project in question has anything to do with GUI or the like, you're going to be beyond the "just start a Docker container" situation. You'll have to configure and work with VNC, which for testing a local application is generally impractical. The same could be said of working with local filesystems (you have to plan what to access and mount it), system integration features, and the list goes on.

Docker certainly has a purpose in some Python development situations, but it should be considered a separate tool altogether, and never as a "replacement" for virtual environments.

Thread Thread
 
tomekbuszewski profile image
Tomek Buszewski

Hey Jason, thanks for the great write-up! It certainly solves a lot of questions for me. But one is still there ā€“Ā what to do when I have databases and I don't want to have pq, mysql and mongo installed on my system in various versions?

Thread Thread
 
codemouse92 profile image
Jason C. McDonald • Edited

Assuming SQLite is not an option, this is exactly the sort of scenario where Docker is a good solution: you aren't just needing the sandbox the Python environment itself (virtual environment), but the actual system environment.

Virtual environments really become extra helpful here, too, because it simplifies your Dockerfiles. Assuming your Python project has a requirements.txt file, you'd only need the following:

RUN python3 -m venv venv && \
    venv/bin/pip install -r requirements.txt

You'll note, I had no need to "activate" the virtual environment. You can just use the venv's binaries directly; they know to use the virtual environment.

Collapse
 
tomekbuszewski profile image
Tomek Buszewski

Thanks for your reply, Jason.

I enjoy using Docker, mostly because I always have a database or something similar standing next to the actual application, plus deploying Docker is super easy, but perhaps I should try using venv more. Thank you :)

Collapse
 
thomasjunkos profile image
Thomas Junk惄

Both docker and virtualenvs try to solve the same problem: separating the system side of things from the user side of things. In case of virtual envs you are getting basically a bit "shell magic" where paths are tweaked in such a way that things work as you were interfering with the system's python but you aren't. That makes it impossible to have interfering installations of python packages and versions.

Docker does the same on a more fundamental level:
Docker divides the kernel-land from the user-land. So it behaves more or less like a virtual environment; but instead of relying on "shell magic" it relies on "kernel magic".

From that point of view there is no real difference or advantage of using one over the other. It totally depends on your deployment model. If your deployment involves docker, it would make more sense to use docker from the first line of code written - and it would be a minimal smoketest for what you are doing anyways. If your deployment model doesn't involve docker, I see no advantage of using docker over a virtual environment.

Collapse
 
imronlearning profile image
Michael Learns

Awesome! Quick question tho on sudo pip. I've had encounters before with it.

Why is it wrong,
and what happens if you do do it,
and how do you "undo" such an action?

Collapse
 
jcsvveiga profile image
JoĆ£o Veiga

It's not exactly wrong, it might not do what you are expecting.
People might run pip install requests and fail because of permissions. They might try sudo pip instead of pip install --user.

Sudo pip will install things in the system's python, rather than the user's python.

Collapse
 
codemouse92 profile image
Jason C. McDonald

Good explanation, @jcsvveiga .

The reason I say you should "never" use sudo pip is, on any system maintained by a dedicated package manager, it can easily conflict with the system-maintained packages, which can make a real mess, especially where dependencies are concerned.

Collapse
 
imronlearning profile image
Michael Learns

Ah! That makes sense. So it would be much better then to do pip install --user than sudo?

Thread Thread
 
jcsvveiga profile image
JoĆ£o Veiga

Yes, or if you want a ditectory/project dependencies, use a virtual environment

Collapse
 
tacoder profile image
Jaime Lopez A.K.A. Tacoder

Great article Jason, i love how easy is to understand virtualenvs using this.
Recently i came across a problem trying to teach a colleague how to set up his own virtualenv to simulate what i had done with mine since we were trying to mimic the entire environment of mine we faced a struggle and ended up failing :(

A day later we came across pipenv wich reproduced the exact ambient for us and problem solved... now i have to ask whats your stance in this virtualenv vs pipenv that seems to be happening around the python world?

and also are you planning to add it to your dead simple series? i will love to use it to teach my colleagues at college, i love the style of your writing (fan girl screaming on the background).

greetings and lots of tacos for you.

Collapse
 
codemouse92 profile image
Jason C. McDonald

Honestly, I never planned on adding pipenv to this, but I may have to cave and add it anyway! Everyone keeps asking. :P

So glad you enjoyed it!

Collapse
 
leob profile image
leob

Just started with Python a week ago and the first advice I came across: pipenv! Combine venv, pip and whatnot in one simple command. It seems to be "the new standard" and until now seems to work flawlessly.

Collapse
 
clozach profile image
Chris Lozac'h

This is excellent! I did something wrong awhile back and ended up in a situation that makes that second xkcd look very familiar. šŸ˜¬

(Speaking of xkcd, I don't see any attribution for the comics. While I'm not exactly sure what legal attribution compliance with the license looks like, a link to the source material seems like an appropriate minimum.)

This has me excited to try python again, this time following your clear and concise instructions, and taking advantage of venv! Thank you!

Collapse
 
codemouse92 profile image
Jason C. McDonald • Edited

So glad you liked the article!

I went ahead and added an attribution at the bottom, thanks. XKCD is so widespread at this point, they get embedded rather commonly anymore (so much so, that Randall just hands out embed links, which I used). That's why I overlooked attribution. It's polite, anyhow. ;)

Collapse
 
ffreemanjr profile image
ffreemanjr

Thank you for this introductory post to begin the series. I look forward to going through the rest as they are released.

Collapse
 
openpencil profile image
JS

Thanks for this clear post! What is the difference between:

virtualenv -p python3 venv
and
python3 -m venv myvirtualenv

Collapse
 
codemouse92 profile image
Jason C. McDonald

There basically isn't one. The latter is preferred now, though, because it feels a little more natural feeling to create a virtual environment using a specific version of Python:

python3.8 -m venv myvirtualenv
Collapse
 
richardeby profile image
Richard Eby

This has some quality content. Thanks for explaining this. Starts to put things together a little more for me

Collapse
 
teckseeker profile image
Teckseeker

This article is a lifesaver for Python enthusiasts struggling with virtual environments. The author's candid admission of past struggles adds a relatable touch. The analogy of virtual environments as sandboxes brilliantly simplifies their purpose. Explaining the intricacies of package installations with humor (cue the XKCD comics) makes it an enjoyable read.

The step-by-step guide on creating, activating, and leaving virtual environments is like having a tech-savvy friend guiding you. The inclusion of commands for various operating systems ensures inclusivity. Kudos to the author for demystifying pip and making it sound "deliciously simple." Thanks a ton for this gem, Jason C. McDonald! Your efforts in breaking down complexities are truly appreciated.

ā€¢ For similar topics, read this article:

Collapse
 
heitrix profile image
Crocodile Forest

On what directory should I run the "venv\Scripts\activate.bat" command?

Collapse
 
heitrix profile image
Crocodile Forest

Ah i got it. I should run it in the same directory as venv/ and if i want to run the python i should use the one inside the 'venv\script\python' instead of only typing 'python'.

Collapse
 
tomisavolainen profile image
Tomi Savolainen

So I already have python2 and python3 on my Mac. How to handle that? Should I remove python2?

Collapse
 
codemouse92 profile image
Jason C. McDonald • Edited

You can absolutely leave them both installed. That's part of the beauty of a virtual environment! Just specify which one you want to use when you create the virtualenv. For example...

virtualenv -p python3 venv
Collapse
 
bhupesh profile image
Bhupesh Varshney šŸ‘¾

After countless searching of blogs/websites
Finally found the best and easiest tutorial of virtualenv šŸ˜„

Collapse
 
jimyuill profile image
jimyuill

Very helpful tutorial--thanks!
I've coded in Python for a while, but didn't know about virtual-environments.

Collapse
 
rpalo profile image
Ryan Palo

Iā€™m very excited for this series! Amazing so far! šŸ˜

Collapse
 
pedroalvesfi profile image
Pedro Alves Filho

Great recommendations!