Docker Volume Debugging: Finding Where Your Data Actually Lives

Originally published at harshit.cloud on 2024-12-14.

---

TIL: Docker Volume Debugging: Finding Where Your Data Actually Lives

Spent 2 hours debugging why data wasn't persisting. Turns out, understanding Docker volumes is the actual job.

the problem

I had a container with a volume mount, but couldn't figure out where the data was actually stored on the host:

docker run -v mydata:/data myapp

Where is mydata? What's inside it?

the solution

find volume location

# List all volumes
docker volume ls

# Inspect a specific volume
docker volume inspect mydata

Output:

[
    {
        "CreatedAt": "2024-12-14T10:30:00Z",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/mydata/_data",
        "Name": "mydata",
        "Options": {},
        "Scope": "local"
    }
]

The Mountpoint tells you exactly where the data lives.

view volume contents (the trick)

You can't just cd to that path (permission denied). Instead, use a temporary container:

docker run --rm -v mydata:/data alpine ls -la /data

Or for interactive browsing:

docker run --rm -it -v mydata:/data alpine sh
cd /data
ls -la

a better named-volume workflow

Create a simple alias:

# Add to ~/.bashrc or ~/.zshrc
alias dvol='docker run --rm -it -v'

Usage:

# Browse any volume interactively
dvol mydata:/data alpine sh

# Quick listing
dvol mydata:/data alpine ls -la /data

# Check file contents
dvol mydata:/data alpine cat /data/config.json

# Copy file out of volume
docker run --rm -v mydata:/data -v $(pwd):/backup alpine cp /data/important.txt /backup/

debugging bind mounts

For bind mounts (host path to container):

docker run -v /host/path:/container/path myapp

To see what the container actually sees:

docker exec -it container_name ls -la /container/path

common volume issues

issue 1, volume not mounting

# Check if volume exists
docker volume ls | grep mydata

# Create it if missing
docker volume create mydata

issue 2, wrong permissions

# Check ownership in volume
docker run --rm -v mydata:/data alpine ls -ln /data

# Fix permissions (if needed)
docker run --rm -v mydata:/data alpine chown -R 1000:1000 /data

issue 3, dangling volumes

# List dangling volumes (not used by any container)
docker volume ls -f dangling=true

# Remove them
docker volume prune

# Be careful! This deletes data!

issue 4, volume vs bind mount confusion

# Named volume (managed by Docker)
-v mydata:/data

# Bind mount (you manage the host path)
-v /host/path:/data
-v $(pwd):/data

# Anonymous volume (Docker creates and manages)
-v /data

pro tips

1. backup a volume

# Backup volume to tar file
docker run --rm \
  -v mydata:/data \
  -v $(pwd):/backup \
  alpine tar czf /backup/mydata-backup.tar.gz -C /data .

2. restore a volume

# Restore from backup
docker run --rm \
  -v mydata:/data \
  -v $(pwd):/backup \
  alpine tar xzf /backup/mydata-backup.tar.gz -C /data

3. copy volume to another

# Copy all data from vol1 to vol2
docker run --rm \
  -v vol1:/source:ro \
  -v vol2:/dest \
  alpine sh -c "cp -av /source/. /dest/"

4. clone a volume

# Create new volume as copy of existing
docker volume create vol2
docker run --rm \
  -v vol1:/source:ro \
  -v vol2:/dest \
  alpine cp -av /source/. /dest/

real-world example

I was debugging a database that wasn't persisting data:

# Check if volume exists
docker volume inspect postgres_data
# Error: No such volume

# Ah! The docker-compose.yml had a typo
# It said: postgress_data (3 s's)
# Should be: postgres_data (2 s's)

# Found all volumes
docker volume ls
# Found: postgress_data (the typo!)

# Renamed it
docker volume create postgres_data
docker run --rm \
  -v postgress_data:/source \
  -v postgres_data:/dest \
  alpine cp -av /source/. /dest/

# Removed the typo volume
docker volume rm postgress_data

useful one-liners

# Find which containers use a volume
docker ps -a --filter volume=mydata

# Remove all stopped containers' volumes
docker container prune -f && docker volume prune -f

# List volumes with size (requires Docker 20.10+)
docker system df -v

# Find volumes larger than 1GB
docker system df -v | awk '$4 > 1000'

the gotcha I learned

Docker Compose creates volumes with prefixes:

# docker-compose.yml
volumes:
  mydata:

Creates volume named: projectname_mydata

To use a specific name:

volumes:
  mydata:
    name: mydata

This one trick would have saved me those 2 hours.