DEV Community

loading...
Cover image for Stop using virtualenv, pyenv, nvm, goenv and Use Docker images

Stop using virtualenv, pyenv, nvm, goenv and Use Docker images

Javid Mougamadou
Simple is better than complex (PEP20)
・2 min read

Notes

  • Requires Docker and at least 4 GB RAM (see comment section bellow for further details)

  • Not applicable for single app

Why I use Docker images

Image

Using version managers like pyenv and nvm are obsolete.

For example, virtualenv/pyenv only encapsulates Python dependencies.

Docker containers encapsulate an entire OS with several dependencies.

Docker already allows to pull several one-click images like python, nodejs, golang, etc ...

docker pull python:3.9.0
Enter fullscreen mode Exit fullscreen mode
# Dockerfile
FROM python:3.9.0
Enter fullscreen mode Exit fullscreen mode

Docker images and containers

image

A Docker image includes the elements needed to run an application as a container -- such as code, config files, environment variables, libraries and run time.

A Docker container can be seen as a replica or as a printout of that image.

For example, you can have one python image and multiple containers related to this image.

image

Isolation with the OS (Clean Install)

You may not have python installed on your OS and you can pull the python 3.9 docker image.

So you can only use python inside a docker container.

docker pull python:3.9.0
Enter fullscreen mode Exit fullscreen mode

You may have the python 2.7 installed on your OS and pull the python 3.9 docker image.

They will not conflict together, but you can only use python 3.9 inside a docker container.

If you don't want the python 3.9 anymore , you just have to delete the docker container related to the python 3.9 image and the python 3.9 image itself.

Multiple Versions

You can pull the python 2.7 docker image and the python 3.9 docker image.

They will not conflict together.

Multiple Layers

You can pull the python 2.9 docker image and install additional dependencies.

# Dockerfile
FROM python:3.9.0

...

COPY requirements.txt /app/requirements.txt
RUN pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Upgrade - Switch versions

You can pull the python 3.8 docker image and then switch to the 3.9 image.

# Dockerfile
FROM python:3.8.0 -> FROM python:3.9.0
Enter fullscreen mode Exit fullscreen mode

Example : python

# Dockerfile
FROM python:3.9.0
Enter fullscreen mode Exit fullscreen mode
# Dockerfile
FROM python:3.9.0-alpine
Enter fullscreen mode Exit fullscreen mode

Example : node

# Dockerfile
FROM node:14.16.0
Enter fullscreen mode Exit fullscreen mode
# Dockerfile
FROM node:14.16.0-alpine
Enter fullscreen mode Exit fullscreen mode

Hub

Links

Discussion (41)

Collapse
varjmes profile image
james

I'd disagree that version managers are obsolete. I have great success with nvm and brew, with no complications. When I use Docker, a lot of RAM is used. I have a 32GB Mac and yet it still slows things down.

I do use Docker for some things successfully, but I can't imagine running all these different containers to do my day-to-day work. Wherever possible, I run things locally on my machine as close to the metal as possible.

Thanks for the post, I'm curious what your day-to-day looks like with all of these different images running?

Collapse
javidjms profile image
Javid Mougamadou Author

I have also a 32 GB RAM Mac and I assigned 12-14 GB RAM for Docker (+2 GB for Swap).

I agree that the Docker Virtual Machine (Mac/Windows) could take a lot RAM. In my case 12 RAM is used whether there is any container or not.

For Linux, It takes all of the available Memory. So if for some reason there is a High Memory or Out of Memory, the System crash. For Mac/Windows, only Docker crash and then restart.

Everyday, I run a fullstack containers (api, app, db, broker, etc ) with at least 10 containers.

I know that there is a limitation with the number of containers on the machine (depend on the memory and cpu) but If you want, you can control the memory and the cpu usage of your containers.

If these containers launch dev-servers (like API or APP) so it is the same to launch manually servers on the OS because it is not costly.

I noticed that containers take a lot of RAM when building the image. In my case, I build the image only once and then bind it with a volume.

If you have a single app with few dependencies, so docker could be useless.

Collapse
yoursunny profile image
Junxiao Shi

My servers are mostly 1GB or 2GB, so I can't fit many containers there.

Laptop is 8GB and no Docker. I have everything on Windows or in WSL2.

I wish I can afford 32GB.

Collapse
nasimuddin profile image
Nasim Uddin

For Linux, It takes all of the available Memory. So if for some reason there is a High Memory or Out of Memory, the System crash

Are you sure about that? I have used docker in Ubuntu. It almost used around 500mb memory. Though I don't always use docker. But docker is very close to linux. Correct me if I am wrong. And linux has a way to kill the process if it's using too much ram.

Thread Thread
190245 profile image
Dave

You're indeed correct.

Sincerely,
Another Ubuntu Docker user

Collapse
swagwik profile image
Sattwik Sahu

32 GB RAM!!! I have begun questioning my mere existence... I got a 3GB RAM junkbox 😒😒

Thread Thread
varjmes profile image
james

Haha yeah. This was provided by work, I've never had so much RAM before!

Thread Thread
javidjms profile image
Collapse
bashar3a profile image
Bashar Abdullah

This! Docker is resource hungry and I constantly need to go back and cleanup storage.

I use Docker, but not for these cases.

Collapse
muasx88 profile image
Muas

agree

Collapse
yoursunny profile image
Junxiao Shi

Docker is great for production deployment but painful for development.

  • There's no incremental builds (rebuild changed files only). Docker has to rebuild the whole "layer".
  • There's no GDB debugger.
  • If you need a database, it has to run as another container.
  • IPv6 doesn't work well.
  • Docker cannot access certain hardware devices such as the serial port and CSI camera.
Collapse
javidjms profile image
Javid Mougamadou Author • Edited

For development, you can do theses things :
1) Keep the container awake with a non-stop command ( ex: CMD tail -f /dev/null)

FROM python:3.9.0

...

COPY requirements.txt /data/requirements.txt
RUN pip install -r requirements.txt
COPY . .

# Keep the container awake
CMD tail -f /dev/null
Enter fullscreen mode Exit fullscreen mode

2) Bind a volume on your container (with docker-compose.yml)

version: "3"
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.local
    volumes:
      - .:/data/
    environment:
      - TZ=Europe/Paris
    ports:
      - "8000:8000"
    networks:
      - default
Enter fullscreen mode Exit fullscreen mode

3) Use a development server with hot-reloader (runserver for Django, run for flask, react-scripts start for Reactjs, etc)

Collapse
javidjms profile image
Javid Mougamadou Author

You don't have to necessary need another container for your database.
You could also have it directly installed on your OS.

In my case, I prefer to containerised all of my stack (db, api, app, broker, etc,)

Collapse
nuculabs_dev profile image
Nucu Labs

To quote: You don't have to necessary need another container for your database.
You could also have it directly installed on your OS.
.

In my opinion that's the WORST WAY to use Docker. I rather run the 10 containers Confluent provides for Kafka and have everything working and setup than waste time configuring Kafka for my OS (which most probably won't work if it's Windows).

Collapse
confidenceyobo profile image
Confidence Yobo

About running a database on another container I see that as an advantage as it helps isolate parts of your application. But if you want to run your db and app server on the same container, that can also be achieved.

Collapse
darkain profile image
Vincent Milum Jr

I'd suggest checking out FreeBSD with Jails then. They are the containerization system that Linux modeled after (but IMO didn't do it anywhere near as well). Jails are functionally containers, but from a UX perspective are closer to full blown virtual machines. You can SSH into them, local console access into them, they have the full FreeBSD package manager, can install and run virtually anything that could run on the host. They also have their own independent network stack via VNET, which fully supports DHCP, IPv6, the whole array of networking protocols, or optionally share the host's networking stack instead. Additionally, you get native ZFS support accessible on the host, and optionally accessible inside the jail for each snapshotting/backup/replication/deployment.

Collapse
190245 profile image
Dave

Our entire development environment runs in Docker. To address your points.

  • That's Dockerfile design, better organisation & forethought into image content might help you.
  • Yes there is, run your containers with SYS_PTRACE etc.
  • That's not a negative, but also, nothing about Docker prevents you running a DB somewhere else, or indeed, within the application container at the same time.
  • We use IPv6 without issue.
  • Yep, it can, mostly what you're seeing is an OS problem. Run Linux, install docker, and mount /dev (with the proper kernel options) into the container.
Collapse
sunnysingh profile image
Sunny Singh

I like the idea of Docker, but it does cause pain for local development. Others in the comments have already expressed the main issues, but for me it's mainly just the slowness (e.g. having to rebuild) and extra complexity for simple tasks (e.g. setting up git pre-commit hooks).

For simple Node.js-only projects (no dependency on Postgres or Redis), Node.js version managers like NVM work great and keeps setups very simple. It's not a complete replication of the production environment, but you should have that in a deployed development environment.

Conceptually, we should use containers for everything. In practice (at least in my own experience), it causes unnecessary problems for local development.

Collapse
javidjms profile image
Javid Mougamadou Author

Ok I will mention it that version managers could be used for simple project

Collapse
codemouse92 profile image
Jason C. McDonald • Edited

Or...perhaps just use what works best for your workflow, no matter how many articles say you should "Stop using that one thing that literally does the job for you".

Docker is a regular part of my workflow, but no, I'm not going to spin up a Docker container every single time I want to run a Python file with particular packages. I'm quite proficient with virtualenv, and it works very well.

Meanwhile, nvm saved my hide when testing a PR in an Electron app in an actual user environment, which Docker is not (without a lot of wasted time building it, at least.)

To say "only use Docker images for building" is like saying "always rent a food truck when you need to cook supper, rather than wasting your time cleaning and setting up your home kitchen." It's overkill. There's a place for both, and one cannot just replace the other.

But then, "don't use X, use Y" is virtually always fad-chasing, rather than fact-based.

Collapse
whaison profile image
Whaison

I use nvm and pyenv with huge success most of the times, but I also know that it can be a huge struggle when these tools fail what they promise.

That's why I don't use Docker for these tasks but I often use it for starting a local test database with one command and throwing it away when I don't need it anymore.

But I also use Docker when I don't want to install a certain tool in development. Then I quickly write a docker-compose.yml and mount the development folder into the container. Then there is no need to rebuild the container for every change and I can run the tool with shell inside the container.

Collapse
javidjms profile image
Javid Mougamadou Author

I respect all developer point of views and theirs laptop configuration (memory, cpu)

It is a pleasure to have this kind of debate in dev.to (I couldn't do the same at the office).

I will notice your feedbacks at the top of my post.

Thanks for all replies.

Collapse
seriouslee13 profile image
SeriousLee13 • Edited

Man, I think I must be misunderstanding this docker craze. Like I once tried to implement a Vue.js development environment on docker, and it was such a mission to get it to run locally on docker. It took me half a day to put together from several different articles, and then it still required all sorts of hacks and tricks to run locally vs building for production, for instance. In the end I gave up, and installed node, npm and vue to my pc and carried on working like I always have, with zero headaches.

So I suppose, I'd like to know what it really means to replace dev environments with docker, if it doesn't mean that you're setting up a fresh project to run locally on a docker container, and then ultimately building to a docker container?

Collapse
190245 profile image
Dave

Nothing in our workflow requires docker for dev, but it is handy.

For example, we have a separate repository with branches that are environment names. Throw a war (or jar) into the right directory, and then docker-compose up ....

That way, I can checkout a different branch, and I get the config set for a different environment, spin up my container and my version of the application, is connected to a different environment, so I can triage issues or test that my new build doesn't break something in that environment.

Sure, I could manage that in other ways, but some of our stuff runs in Tomcat, and I'd rather not have to worry about configuring my machine to match the config of some other environment, just so I can test stuff, when it's less than 10% of my time to test stuff like that anyway.

I did know a guy, that ran his whole IDE in a container, and used it to produce images. I think he was a little bonkers though.

Collapse
seriouslee13 profile image
SeriousLee13

Coz it also seems to me that if you're successfully able to set up a containerized dev environment, you're willingly adding another vector for bugs and problems. Like if you run into complicated bug on your containerized dev project, how do you know it's a coding bug and not a issue with your docker setup? And how do you explain your setup to the kind folks over on stackoverflow? I don't want to first have to type a 6 page essay on the configuration of my project, every time I'm over there, asking a question.

Collapse
cescquintero profile image
Francisco Quintero 🇨🇴

You should add a disclaimer:

When you have tons of RAM and disk space.

Docker images consume a lot of disk space and running containers use a lot of RAM due to the need of a virtualization tool in the OS (in case of macos) or because of the containers themselves.

Collapse
hyperpress profile image
John Teague • Edited

I've used Docker on GKE for federated (multiregion) delivery, but I found LXD containers to be easy to manage, secure, and far less resource intensive for our dev env. It's really simple to spin up a cloned container using your favorite distro in seconds and work in a familiar linux server env. Each solution has it's upside and critters to deal with, but that's what works for us. Thanks for the good write up.

Collapse
aegiraexx profile image
Ægir Æxx

I really need to get comfortable with Docker and by extension Kubernetes.

Collapse
appsecmonkey profile image
Teo Selenius

I recommend Mumshad's course on Udemy. I used it when I prepared for the CKA exam and it was the best online course I have taken in a long time. I'm not affiliated with the man in any way so this is coming from my heart.. (:

Collapse
javidjms profile image
Javid Mougamadou Author

Thanks for the recommandation.

Collapse
appsecmonkey profile image
Teo Selenius

I would definitely add security to the benefits of using Docker vs. virtualenvs.

PyPI repositories are a wild west and you could end up running all sorts of crap on your host OS if you don't contain it.

Of course you have to audit your dependencies nevertheless because eventually the code will be deployed somewhere. But still, it at least makes me sleep my nights better.

Collapse
javidjms profile image
Javid Mougamadou Author

I totally agree with you.

There is the security layer.

And I would also add the famous quote of : "it doesn't work on my machine".

Now sharing images will help to build app on several similar environment (local, development, staging, production) and look like the same environement (almost)

Collapse
cyprx_ profile image
cyprx • Edited

I'd prefer mixing between Docker images and package manager. For daily use languages/ applications such as golang, python3.8 (create venv when needed), node, etc. I install them directly in the OS, for all other temporary components, which are used for specific purposes, I use docker and expose their ports to OS for comumication with my developing app.
Benefits from this way are it's easy for me to quickly run some command, python scripting, etc. no need to remember different syntax with docker, or turn docker on. Temporary docker containers and data could be easily destroyed as well after being used, which keeps my machine clean.

Collapse
zarszz profile image
Ganjar Gingin Tahyudin

i just still not comfortable using Docker for Development purpose, especially when i coding and debugging, yes, because remote debugging not right choice for me.

BTW i always use Docker image to install things like database or service broker.

Collapse
michaelsuggs profile image
Michael Suggs

Full disclosure: I’ve never used docker much outside of when it’s been required.

I kinda fail to see what Docker does better than tools like py/jenv. I can already install and set up different versions with their own separate packages with these, and I don’t have to worry about another layer of config/setup. Since I’ve already got my system set up pretty much the way I want, it seems like an unnecessary hassle.

I can see a big use for these in my data science work though. Having entire separate system environments could really be useful for things pulling env variables and whatnot, I suppose. I’d love to see a reason to use docker more in my personal workflow, but I guess I just haven’t found that magic feature that makes it a must-have yet.

Collapse
nuculabs_dev profile image
Nucu Labs

I don't agree with your post. I use the Python from my system when developing and I use images for testing (sometimes) and for external services and software that I need (ElasticSearch, Mongo, PostgresSQL).

Docker is painful, slow and remote debugging is tricky.

Collapse
digitallyinduced profile image
digitallyinduced

Another alternative is nix with nix-shell, which can result in just-as-reliable reproducible development environments (especially when combined with direnv and lorri).

Collapse
xowap profile image
Rémy 🤖

I'd be curious to know how is your IDE integration experience?

Collapse
javidjms profile image
Javid Mougamadou Author

Hi, I just used docker volumes, so it is easy to integrate with IDE (ex: node_modules)

Collapse
jolares profile image
Jo (Joaquin) Lares

Stop using Docker images and Use Linux Containers, lol jk.

Collapse
jascha030 profile image
Jascha van Aalst

I couldn't imagine running a container for everything going on, on a day to day basis. I think my mac would actually explode and burn my house down.

Forem Open with the Forem app