Stop using virtualenv, pyenv, nvm, goenv and Use Docker images

Notes

• Requires Docker and at least 4 GB RAM (see comment section bellow for further details)

• Not applicable for single app

Why I use Docker images

Using version managers like pyenv and nvm are obsolete.

For example, virtualenv/pyenv only encapsulates Python dependencies.

Docker containers encapsulate an entire OS with several dependencies.

Docker already allows to pull several one-click images like python, nodejs, golang, etc ...

docker pull python:3.9.0

# Dockerfile
FROM python:3.9.0

Docker images and containers

A Docker image includes the elements needed to run an application as a container -- such as code, config files, environment variables, libraries and run time.

A Docker container can be seen as a replica or as a printout of that image.

For example, you can have one python image and multiple containers related to this image.

Isolation with the OS (Clean Install)

You may not have python installed on your OS and you can pull the python 3.9 docker image.

So you can only use python inside a docker container.

docker pull python:3.9.0

You may have the python 2.7 installed on your OS and pull the python 3.9 docker image.

They will not conflict together, but you can only use python 3.9 inside a docker container.

If you don't want the python 3.9 anymore , you just have to delete the docker container related to the python 3.9 image and the python 3.9 image itself.

Multiple Versions

You can pull the python 2.7 docker image and the python 3.9 docker image.

They will not conflict together.

Multiple Layers

You can pull the python 2.9 docker image and install additional dependencies.

# Dockerfile

FROM python:3.9.0

...

COPY requirements.txt /app/requirements.txt

RUN pip install -r requirements.txt

Upgrade - Switch versions

You can pull the python 3.8 docker image and then switch to the 3.9 image.

# Dockerfile

FROM python:3.8.0 -> FROM python:3.9.0

Example : python

# Dockerfile

FROM python:3.9.0

# Dockerfile

FROM python:3.9.0-alpine

Example : node

# Dockerfile

FROM node:14.16.0

# Dockerfile

FROM node:14.16.0-alpine

Hub

• https://hub.docker.com/

• https://artifacthub.io/

Links

• https://www.docker.com/resources/what-container

• https://searchitoperations.techtarget.com/definition/Docker-image

• https://towardsdatascience.com/building-a-data-science-development-environment-with-docker-compose-1407aa5147b9

• https://stackoverflow.com/questions/50974960/whats-the-difference-between-docker-and-python-virtualenv

Comments

[james]

I'd disagree that version managers are obsolete. I have great success with nvm and brew, with no complications. When I use Docker, a lot of RAM is used. I have a 32GB Mac and yet it still slows things down.

I do use Docker for some things successfully, but I can't imagine running all these different containers to do my day-to-day work. Wherever possible, I run things locally on my machine as close to the metal as possible.

Thanks for the post, I'm curious what your day-to-day looks like with all of these different images running?

[Javid Mougamadou]

I have also a 32 GB RAM Mac and I assigned 12-14 GB RAM for Docker (+2 GB for Swap).

I agree that the Docker Virtual Machine (Mac/Windows) could take a lot RAM. In my case 12 RAM is used whether there is any container or not.

For Linux, It takes all of the available Memory. So if for some reason there is a High Memory or Out of Memory, the System crash. For Mac/Windows, only Docker crash and then restart.

Everyday, I run a fullstack containers (api, app, db, broker, etc ) with at least 10 containers.

I know that there is a limitation with the number of containers on the machine (depend on the memory and cpu) but If you want, you can control the memory and the cpu usage of your containers.

If these containers launch dev-servers (like API or APP) so it is the same to launch manually servers on the OS because it is not costly.

I noticed that containers take a lot of RAM when building the image. In my case, I build the image only once and then bind it with a volume.

If you have a single app with few dependencies, so docker could be useless.

[Junxiao Shi]

My servers are mostly 1GB or 2GB, so I can't fit many containers there.

Laptop is 8GB and no Docker. I have everything on Windows or in WSL2.

I wish I can afford 32GB.

[Nasim Uddin]

For Linux, It takes all of the available Memory. So if for some reason there is a High Memory or Out of Memory, the System crash

Are you sure about that? I have used docker in Ubuntu. It almost used around 500mb memory. Though I don't always use docker. But docker is very close to linux. Correct me if I am wrong. And linux has a way to kill the process if it's using too much ram.

[Dave]

You're indeed correct.

Sincerely,
Another Ubuntu Docker user

[Sattwik Sahu]

32 GB RAM!!! I have begun questioning my mere existence... I got a 3GB RAM junkbox 😒😒

[james]

Haha yeah. This was provided by work, I've never had so much RAM before!

[Javid Mougamadou]

Same for me

[Bashar Abdullah]

This! Docker is resource hungry and I constantly need to go back and cleanup storage.

I use Docker, but not for these cases.

[Muas]

agree

[Junxiao Shi]

Docker is great for production deployment but painful for development.

• There's no incremental builds (rebuild changed files only). Docker has to rebuild the whole "layer".
• There's no GDB debugger.
• If you need a database, it has to run as another container.
• IPv6 doesn't work well.
• Docker cannot access certain hardware devices such as the serial port and CSI camera.

[Javid Mougamadou]

You don't have to necessary need another container for your database.
You could also have it directly installed on your OS.

In my case, I prefer to containerised all of my stack (db, api, app, broker, etc,)

[Nucu Labs]

To quote: You don't have to necessary need another container for your database.
You could also have it directly installed on your OS..

In my opinion that's the WORST WAY to use Docker. I rather run the 10 containers Confluent provides for Kafka and have everything working and setup than waste time configuring Kafka for my OS (which most probably won't work if it's Windows).

[Javid Mougamadou]

For development, you can do theses things :
1) Keep the container awake with a non-stop command ( ex: CMD tail -f /dev/null)

FROM python:3.9.0

...

COPY requirements.txt /data/requirements.txt
RUN pip install -r requirements.txt
COPY . .

# Keep the container awake
CMD tail -f /dev/null

2) Bind a volume on your container (with docker-compose.yml)

version: "3"
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.local
    volumes:
      - .:/data/
    environment:
      - TZ=Europe/Paris
    ports:
      - "8000:8000"
    networks:
      - default

3) Use a development server with hot-reloader (runserver for Django, run for flask, react-scripts start for Reactjs, etc)

[Confidence Yobo]

About running a database on another container I see that as an advantage as it helps isolate parts of your application. But if you want to run your db and app server on the same container, that can also be achieved.

[Vincent Milum Jr]

I'd suggest checking out FreeBSD with Jails then. They are the containerization system that Linux modeled after (but IMO didn't do it anywhere near as well). Jails are functionally containers, but from a UX perspective are closer to full blown virtual machines. You can SSH into them, local console access into them, they have the full FreeBSD package manager, can install and run virtually anything that could run on the host. They also have their own independent network stack via VNET, which fully supports DHCP, IPv6, the whole array of networking protocols, or optionally share the host's networking stack instead. Additionally, you get native ZFS support accessible on the host, and optionally accessible inside the jail for each snapshotting/backup/replication/deployment.

[Dave]

Our entire development environment runs in Docker. To address your points.

• That's Dockerfile design, better organisation & forethought into image content might help you.
• Yes there is, run your containers with SYS_PTRACE etc.
• That's not a negative, but also, nothing about Docker prevents you running a DB somewhere else, or indeed, within the application container at the same time.
• We use IPv6 without issue.
• Yep, it can, mostly what you're seeing is an OS problem. Run Linux, install docker, and mount /dev (with the proper kernel options) into the container.

[Sunny Singh]

I like the idea of Docker, but it does cause pain for local development. Others in the comments have already expressed the main issues, but for me it's mainly just the slowness (e.g. having to rebuild) and extra complexity for simple tasks (e.g. setting up git pre-commit hooks).

For simple Node.js-only projects (no dependency on Postgres or Redis), Node.js version managers like NVM work great and keeps setups very simple. It's not a complete replication of the production environment, but you should have that in a deployed development environment.

Conceptually, we should use containers for everything. In practice (at least in my own experience), it causes unnecessary problems for local development.

[Javid Mougamadou]

Ok I will mention it that version managers could be used for simple project

[Jason C. McDonald]

Or...perhaps just use what works best for your workflow, no matter how many articles say you should "Stop using that one thing that literally does the job for you".

Docker is a regular part of my workflow, but no, I'm not going to spin up a Docker container every single time I want to run a Python file with particular packages. I'm quite proficient with virtualenv, and it works very well.

Meanwhile, nvm saved my hide when testing a PR in an Electron app in an actual user environment, which Docker is not (without a lot of wasted time building it, at least.)

To say "only use Docker images for building" is like saying "always rent a food truck when you need to cook supper, rather than wasting your time cleaning and setting up your home kitchen." It's overkill. There's a place for both, and one cannot just replace the other.

But then, "don't use X, use Y" is virtually always fad-chasing, rather than fact-based.

[Whaison]

I use nvm and pyenv with huge success most of the times, but I also know that it can be a huge struggle when these tools fail what they promise.

That's why I don't use Docker for these tasks but I often use it for starting a local test database with one command and throwing it away when I don't need it anymore.

But I also use Docker when I don't want to install a certain tool in development. Then I quickly write a docker-compose.yml and mount the development folder into the container. Then there is no need to rebuild the container for every change and I can run the tool with shell inside the container.

[Javid Mougamadou]

I respect all developer point of views and theirs laptop configuration (memory, cpu)

It is a pleasure to have this kind of debate in dev.to (I couldn't do the same at the office).

I will notice your feedbacks at the top of my post.

Thanks for all replies.

[Francisco Quintero 🇨🇴]

You should add a disclaimer:

When you have tons of RAM and disk space.

Docker images consume a lot of disk space and running containers use a lot of RAM due to the need of a virtualization tool in the OS (in case of macos) or because of the containers themselves.

[Ægir Æxx]

I really need to get comfortable with Docker and by extension Kubernetes.

[Teo Selenius]

I recommend Mumshad's course on Udemy. I used it when I prepared for the CKA exam and it was the best online course I have taken in a long time. I'm not affiliated with the man in any way so this is coming from my heart.. (:

[Javid Mougamadou]

Thanks for the recommandation.

[Teo Selenius]

I would definitely add security to the benefits of using Docker vs. virtualenvs.

PyPI repositories are a wild west and you could end up running all sorts of crap on your host OS if you don't contain it.

Of course you have to audit your dependencies nevertheless because eventually the code will be deployed somewhere. But still, it at least makes me sleep my nights better.

[Javid Mougamadou]

I totally agree with you.

There is the security layer.

And I would also add the famous quote of : "it doesn't work on my machine".

Now sharing images will help to build app on several similar environment (local, development, staging, production) and look like the same environement (almost)

[John Teague]

I've used Docker on GKE for federated (multiregion) delivery, but I found LXD containers to be easy to manage, secure, and far less resource intensive for our dev env. It's really simple to spin up a cloned container using your favorite distro in seconds and work in a familiar linux server env. Each solution has it's upside and critters to deal with, but that's what works for us. Thanks for the good write up.