Say Goodbye to Docker Volumes 👋

Ever tried to use Docker volumes for hot-reloading in your web app? If you had the same horrible experience as me, you will enjoy the newest feature that Docker just released: docker-compose watch! Let me show you how to upgrade your existing project to have a wonderful Docker dev setup that your team will actually enjoy using 🤩

TL;DR: Check out this docker-compose file and the official documentation

Let's get started!

Introduction

Docker just released Docker Compose Watch with Docker Compose Version 2.22. With this new feature, you can use docker-compose watch instead of docker-compose up and automatically synchronize your local source code with the code in your Docker container without needing to use volumes!

Let us take a look at how this works in a real-word project by using a project that I previously wrote about.

In this project, I have a monorepo with a frontend, backend, and some additional libraries for the UI and database.

├── apps
│   ├── api
│   └── web
└── packages
    ├── database
    ├── eslint-config-custom
    ├── tsconfig
    └── ui

Both apps (api and web) are already dockerized and the Dockerfiles are in the root of the project (1, 2)

The docker-compose.yml file would look like this:

services:
  web:
    build:
      dockerfile: web.Dockerfile
    ports:
      - "3000:3000"
    depends_on:
      - api
  api:
    build:
      dockerfile: api.Dockerfile
    ports:
      - "3001:3000"from within the Docker network

That's already pretty good, but as you already know it's a PITA to work with this during development. You will have to rebuild your Docker images whenever you change your code, even though your apps will probably support hot-reloading out of the box (or with something like Nodemon if not).

To improve this, Docker Compose Watch introduces a new attribute called watch. The watch attribute contains a list of so-called rules that each contain a path that they are watching and an action that gets executed once a file in the path changes.

Sync

If you would want to have a folder synchronized between your host and your container, you would add:

services:
  web: # shortened for clarity
    build:
      dockerfile: web.Dockerfile
    develop:
      watch:
        - action: sync
          path: ./apps/web
          target: /app/apps/web

Whenever a file on your host in the path ./apps/web/ changes, it will get synchronized (copied) to your container to /app/apps/web. The additional app in the target path is required because this is our WORKDIR defined in the Dockerfile. This is the main thing you will probably use if you have hot-reloadable apps.

Rebuild

If you have apps that need to be compiled or dependencies that you need to re-install, there is also an action called rebuild. Instead of simply copying the files between the host and the container, it will rebuild and restart the container. This is super helpful for your npm dependencies! Let's add that:

services:
  web: # shortened for clarity
    build:
      dockerfile: web.Dockerfile
    develop:
      watch:
        - action: sync
          path: ./apps/web
          target: /app/apps/web
        - action: rebuild
          path: ./package.json
          target: /app/package.json

Whenever our package.json changes we will now rebuild our entire Dockerfile to install the new dependencies.

Sync+Restart

Besides just synchronizing and rebuilding there is also something in between called sync+restart. This action will first synchronize the directories and then immediately restart your container without rebuilding. Most frameworks usually have config files (such as next.config.js) that can't be hot-reloaded (just sync isn't enough) but also don't require a slow rebuild.

This would change your compose-file to this:

services:
  web: # shortened for clarity
    build:
      dockerfile: web.Dockerfile
    develop:
      watch:
        - action: sync
          path: ./apps/web
          target: /app/apps/web
        - action: rebuild
          path: ./package.json
          target: /app/package.json
        - action: sync+restart
          path: ./apps/web/next.config.js
          target: /app/apps/web/next.config.js

Caveats

As always, there is no free lunch and a few caveats 😬

The biggest problem with the new watch attribute is that the paths are still very basic. The documentation states that Glob patterns are not supported yet which can result in a huge amount of rules if you want to be specific.

Here are some examples of what works and what does not:

✅ apps/web
This will match all the files in ./apps/web (e.g. ./apps/web/README.md, but also ./apps/web/src/index.tsx)

❌ build/**/!(*.spec|*.bundle|*.min).js
Globs are sadly (not yet?) supported

❌ ~/Downloads
All paths are relative to the project root!

Next Steps

If you are still not happy with your Docker setup, there are still many ways to improve it!

Collaboration is a big part of software development and working in silos can be seriously damaging to your team. Slow Docker builds and complicated setups don't help! To counteract this and promote a culture of collaboration you can use Docker extensions such as Livecycle to instantly share your local docker-compose apps with your teammates. Since you are already using Docker and docker-compose, all you need to do is install the Docker Desktop Extension and click on the share toggle. Your apps will then be tunneled to the internet and you can share your unique URL with your team to get feedback! I've written more about that in this post if you want to check out more use cases of Livecycle :)

As always, make sure that your Dockerfile is following best practices, especially around multi-stage builds and caching. While this might make writing the initial Dockerfile harder, it will make your Docker apps a lot more pleasant to use during development.

Creating a basic .dockerignore file and separating dependency installation from code building goes a long way!

Conclusion

As always, I hope you learned something new today! Let me know if you need any help setting up your Docker project, or if you have any other feedback

Cheers,
Jonas Co-Founder sliplane.io

Comments

[Risal Hidayat]

I just discovered this docker compose watch things. However, I have two services that share the same code base (web and workers). Using docker compose watch makes infinite build and sync issues. because bot services are updating each other. I think I will stick to the volumes until I can figure out that issue :(

[Jonas Scholz]

Oh interesting, didnt even consider that this might be the case. Thanks for sharing!

[Nicol Acosta]

I personally work on personal and professional stuff with docker and this its gold to me, thanks

[cabrel92]

Thanks for your Post.
Nowadays, Developer shall think of working with docker, I personally try to dockerise every application I work on. I have stopped working with vagrant for a while, because of ressources consumption, and many other things.
So for me docker is the most for Dev.
Nos that I can use watch ( sync/rebuild/ sync+), now that I have opportunity ti share my Container with colleagues, I feel more comfortable.

[anhvandev]

Hopefully there will be a feature to dispatch multiple commands after changing package settings :v it will look like a CD tool in the local environment.

[Grigor Khachatryan]

This news is quite impressive; Thanks for sharing it!

[Jonas Scholz]

Glad that you enjoyed it!:)

[Nadir Hamid]

Great read. This is definitely better than using volumes. They were a pain to manage and annoying to integrate with app workflows. Indeed, each framework has a unique set of development challenges and there is no one size fits all, but this new feature helps. It would be good to use some of these rebuild actions in case a basic watch procedure doesn't work.

I hope other dev teams find this soon as it can save them large amounts of time.

[Victor Vincent]

Let's store node_modules twice on our machines!

Jokes aside, it's nice to have, but I'll probably still stick to volumes, especially on large projects. I can't see how any watcher can be sufficient enough on large codebases. There's a reason all tools exclude 3rd party lib folders, like node_modules from their watchers.

[Jonas Scholz]

node_modules are excluded from the watch if they are in .gitignore :)

[Victor Vincent]

So that's even worse, because I need to install both places anyway to make the IDE work.

[Jonas Scholz]

Well sure, but with a fast internet connection and enough disk space that isn't really a concern for me luckily:)

[Victor Vincent]

Well, enough is quite relative. I have 4TB+ SSD RAID and soon I need to extend again. 60% of space are node_modules :D

[Jonas Scholz]

Lmao that is impressive

[namdevgg]

Goood

[Eli Vargas]

I'm guessing volumes are a problem for non-Linux users. I have no issues with Docker volumes. But non-linux Docker actually runs on a VM. So I can see that could cause problems with hot reloads. 🤔 Good luck guys. 😅

[Julien Dephix]

Same here.
After reading the title I was wondering what was wrong with volumes but I'm running Ubuntu so that could be why I have no problems: yarn watch and code! ^_^

[Paul Keen]

Why do you need to add your code to the container? When would you need that instead of mount? I have not see dev containers with adding code to the image.

[Paul Keen]

You can find robust version to prevent hacks with watch here jtway.co/running-tests-in-containe...