How to persist data with docker compose

One problem I see when people start to work with docker and docker compose is the problem of understanding what docker volumes are and what's their use.
And in many cases you don't really need them, for example if you have a container that runs in isolation and has no dependencies, and you are not going to store any data, or maybe it connects to an external database that it's on the cloud.
But in other cases you do need to persist data on a container, thus the need for docker volumes.
But let's try to explain how they work with a simple example.

You have a project which has two services, one is the application you are building, like some node.js or Python APIs, which connects to a the other service, a database, either Mongodb or MySQL (doesn't really matter for our example).
So perhaps your docker-compose.yml looks like this:

version: '3'
services:
  api:
    build: .
    ports:
      - 9000:9000
    command: npm run dev
    networks:
      - my-network
  mongo:
    image: mongo:4.4
    environment:
      - MONGO_INITDB_ROOT_USERNAME=root
      - MONGO_INITDB_ROOT_PASSWORD=pass
    ports:
      - '27017:27017'
    networks:
      - my-network
networks:
  my-network:
    driver: bridge

As it is, once you start the containers, everything will work fine, you can save data on the database, you can read from it, etc...
But, the very moment you do docker-compose down, the containers will be removed by docker, and all the data will be gone!.
And that's probably fine with you as you don't need anything to persist, but in the other case, what you need to do is to use volumes.

Bind mounts vs Volumes

There are two ways where you can create a volume, bind mounts and volumes.
Whichever you choose, once you have set up a volume to the folder where the data is stored in the container, if you do a docker-compose down, and then a docker-compose up, your data will not be erased and it will become persistent.

Bind mounts

Bind Mounts are volumes that mount to a host path, and they can be modified by other processes outside docker.
Essentially it shares a folder with your host computer and the container, and it's defined with the syntax <host-folder>:<container-folder>.
With a bind mount our project will look like this:

mongo:
    image: mongo:4.4
    ...other settings
    volumes:
      - ./db:/data/db

So what you will see that from your host machine, you will see a .db folder, which is going to contain all the data stored in the database.

Note: /data/db is where the MongoDB image store the databases data, this will be different on other databases like Redis, MariaDB, MySQL, etc...

The only problem with bind mounts, is that the folder is managed by the host computer, so if you change file permissions, or you delete a file for example, you will corrupt the data in the container, which could lead to errors or the container not working anymore.

Volumes

Volumes on the other hand live under the docker engine, it behaves more or less like a virtual hard drive, managed by docker and the files inside it are not accessible from the host computer.

Add a volume to docker-compose is rather simple, it's similar to bind mounts, but with the difference that you need to add the volume under the volumes array in the main docker-compose.yml, and in the service configuration you use the syntax <volume-name>:<container-path>, for example if we want to store the data in a container called mongodata, our docker-compose.yml is going to look something like this:

version: '3'
services:
  api:
    ...api configuration
  mongo:
    ...mongo configuration
    volumes:
      - mongodata:/data/db
networks:
  my-network:
    driver: bridge
volumes:
  mongodata:

Note, the colon after mongodata is not an error.

Same as bind mounts, now when you stop the containers and restart them, the data will be there.

How I do remove a volume?

In case you need to remove a volume, for the bind mounts is just a matter to stop the containers, and delete the folder.
In the case of a volume you need to use a docker command to delete it, but it's rather easy:

docker volume rm mongodata

That's it!

Bonus

A couple of common commands you can use to manage volumes with docker:

# create a volume
docker volume create myvolume

# list volums
docker volume ls

# delete a volume
docker volume rm myvolume

Comments

[Ralf Longwitz]

For both volumes and binds I find that changes to files/folders and added files/folders persist. But removed files and folders reappear after a compose down/up. Why is that? How can deletions be made to persist?

[Alessio Michelini]

Maybe you have some process that it recreates them on startup?

[Ralf Longwitz]

I believe I found the reason why this happens:
stackoverflow.com/questions/393176...
When running docker-compose, a completely new container is created, with all original files. Then the files that are in my bound volume are added to those files.
I still have not figured out how to do it right. Something about running an image with lots of parameters. Super confusing. I cannot believe that such a common need does not have a simple solution ...

[Ralf Longwitz]

Thanks, Alessio. So I assume this is not normal behviour? If there is such a process, how could I find out?

[Alessio Michelini]

it could be either an init script in the docker-compose, or it could be something in your Dockerfile, or maybe one of the libraries you are using might do that on startup, it's hard to say unfortunately, but as far I know, it's not a normal behaviour

[Ralf Longwitz]

Thank you! I'll go check my files.