Docker copies your data when the image has a VOLUME

Why CI jobs failed with 4 MB of disk left, before the first step even ran.

In a project I work on, the integration tests started to fail a few times per day. The error was always the same:

##[warning]You are running out of disk space. The runner will stop working when
the machine runs out of disk space. Free space left: 4 MB
Error response from daemon: failed to copy files: write
/var/lib/docker/volumes/<id>/_data/ibdata1: no space left on device

My first guesses were all wrong. The runner image did not grow. The test database image did not grow either. Its compressed size was flat for a month. There was no leak, and the tests did not change.

The real reason was different. Docker was making a full second copy of a 6.6 GiB MySQL data directory every time a job started. This was new to me. Docker does it by design, and it is easy to miss.

The setup

This pattern is common for integration tests. You bake a ready database into an image, so each CI job gets real data in seconds. No need to run all migrations from an empty database.

FROM mysql:8.0
COPY --chown=mysql:mysql ./mysql /var/lib/mysql   # a full database, already migrated

Then you use it as a service container:

services:
  mysql:
    image: my-registry/test-db:latest
    ports: ["3307:3306"]

It works well. The container starts, the data is there, the tests run. But it also hides a 2x disk cost.

What Docker actually does

The official mysql image has this line inside:

VOLUME /var/lib/mysql

When you create a container from an image that declares a VOLUME, Docker makes a new empty volume for that path. If the image already has files in that path, Docker copies all of them into the new volume.

This is documented, but it is easy to miss. Normally that path is empty in the image, so there is nothing to copy and you never notice. If you bake your data into the same path, every container copies the whole thing again.

In CI this is even harder to see. In GitHub Actions, service containers start before step 1 of your job, so the copy is already done before your own code runs. A "free up disk space" step cannot protect you. The job in this project had such a step at the top for months. It looked safe, but it always ran too late.

Here is the disk math on a standard GitHub-hosted Linux runner:

Item	Size
Unpacked image layers	about 7.7 GB
The copy Docker makes at create	about 6.6 GB
Peak before step 1 runs	about 14.3 GB
Free space on the runner root disk	about 14 GB

The jobs were only about 300 MB over the limit. This is why they failed a few times per day and not every time. Each runner image version leaves a slightly different amount of free space. The failure looked random, but it was simple math.

Try it yourself in two minutes

You do not need a 7 GB image to see this. A small one is enough:

# 1. Create about 200 MB of MySQL data
docker run -d --name seed -e MYSQL_ROOT_PASSWORD=example mysql:8.0 --skip-log-bin
sleep 30
docker exec seed mysql -uroot -pexample -e 'create database probe;
  create table probe.t(i int); insert into probe.t values (42);'
docker exec seed mysqladmin -uroot -pexample shutdown; sleep 5
docker cp seed:/var/lib/mysql ./mysql

# 2. Bake the data into the VOLUME path (the usual way)
cat > Dockerfile.volume <<'EOF'
FROM mysql:8.0
COPY --chown=mysql:mysql ./mysql /var/lib/mysql
CMD ["--skip-log-bin"]
EOF
docker build -f Dockerfile.volume -t probe:volume .

# 3. Bake the data somewhere else, and tell mysqld where it is
cat > Dockerfile.nonvolume <<'EOF'
FROM mysql:8.0
COPY --chown=mysql:mysql ./mysql /var/lib/mysql-snap
CMD ["--datadir=/var/lib/mysql-snap","--skip-log-bin"]
EOF
docker build -f Dockerfile.nonvolume -t probe:nonvolume .

# 4. Compare only the create step
time docker create --name c1 probe:volume       # copies the data
time docker create --name c2 probe:nonvolume    # copies nothing

In my earlier test on my laptop, with about 200 MB of data, the create step took 1.539s for the image with data in the VOLUME path and 0.070s for the other one. Those numbers came from my own test with slightly different commands, not from the script above, so treat them as the shape of the result and not as an exact benchmark. Your own numbers will depend on your disk, but the pattern is the same: one command copies your data, the other does not.

You can also check that the volume stays empty:

docker run -d --name c3 probe:nonvolume && sleep 25
docker exec c3 sh -c 'du -sh /var/lib/mysql /var/lib/mysql-snap'
# 4.0K  /var/lib/mysql      <- the new volume, created but empty
# 199M  /var/lib/mysql-snap <- the server reads from here
docker exec c3 sh -c 'mysql -uroot -pexample -N -B -e "select @@datadir, (select i from probe.t)"'
# /var/lib/mysql-snap/  42  <- data is there, no new database was created

Now scale this to a 6.6 GiB database. That is where the 49 seconds and 6.6 GB per job came from, on every pull request.

The fix

Do not put your data in the VOLUME path of the image. Put it in another path and tell the server to use it. Docker still creates the volume, but it is empty, so there is nothing to copy.

Two details are important here.

You cannot remove a VOLUME from a base image. There is no UNVOLUME in Dockerfiles. If the base image declares one, your image has it too. So moving your data to another path is the only simple option.

Write the path into a config file, not only into CMD. Some users of the image override the container command:

services:
  db:
    image: my-registry/test-db:latest
    command: --some-other-flag        # your --datadir is gone now

If your --datadir only lives in CMD, this removes it. The server then uses the default path, finds it empty, and creates a fresh empty database. The tests fail later, and the reason is not obvious at all. So put the path in a config file too:

RUN printf '[mysqld]\ndatadir=/var/lib/mysql-snap\n' > /etc/mysql/conf.d/zz-datadir.cnf

The zz prefix is needed. Config files are read in order, and the base image already sets datadir, so your file must come last. Keep the CMD flag as well, as a second protection.

What I learned

The error message sent me to the wrong place. It said the disk was full, so I looked for something that grew. Nothing grew. The real hint was in the path: /var/lib/docker/volumes/.... A volume was being filled, and nobody had asked for a volume.

The second lesson is about base images. You inherit everything they declare, and VOLUME is one line that changes how every container starts. It is easy to read a Dockerfile and miss it.

Result on the same runners: create went from about 49 seconds to 0.36 seconds, peak disk before step 1 went from about 14.3 GB to about 7.7 GB, and the disk warnings are gone.

Quick check for your own project

If you bake data into images for tests, run these three commands:

# 1. Does your image declare a VOLUME on your data path?
docker image inspect <your-image> --format '{{json .Config.Volumes}}'

# 2. How long does create alone take? Slow means it is copying.
time docker create --name check <your-image>

# 3. How big is the volume it just created?
docker inspect check --format '{{range .Mounts}}{{.Name}}{{end}}' \
  | xargs -I{} docker run --rm -v {}:/v alpine du -sh /v
docker rm -f check

If the first command shows the same path where your data lives, you pay for that data twice on every container.