DEV Community

Christoph Dieck
Christoph Dieck

Posted on Originally published at slashgordon.link

The 35-Watt Roommate: Docker Cleanup and Surgical Data Recovery

#homelab #docker #proxmox #gitea

Sometimes you just want to clean up a bit and accidentally tear down half the house. That's what happened to me last week with my home lab. I wanted to free up some space on the Lenovo M920q and ran a routine Docker cleanup: delete a few unused images, remove orphaned volumes.

But when I opened my Gitea instance afterward, the page was blank. No repositories, no users, everything reset to factory defaults. All my code was gone.

VM Recovery from Proxmox Backup

The fatal error in the compose file

The answer was in my docker-compose.yml, where I had made a bad architectural mistake during setup. My volume mapping looked like this:

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    volumes:
      - /var/lib/docker/volumes/infra_volume/_data/gitea_data:/data
Enter fullscreen mode Exit fullscreen mode

At first glance it looks like a normal path. The problem is that /var/lib/docker/volumes/ is Docker's internal storage. I had reached around Docker and pointed a bind mount straight at the raw data of another volume (infra_volume). Docker didn't see that dependency, so the cleanup job classified the original volume as unused and deleted it.

Surgical recovery from the Proxmox backup

Restoring the entire VM from backup would have taken forever and overwritten the rest of my setup. I only needed one directory out of the backup, so I had to extract the data directly from the Proxmox backup without ever booting the VM.

The backup lay as a compressed file on my storage:

vzdump-qemu-100-2026_02_23-01_00_02.vma.zst
Enter fullscreen mode Exit fullscreen mode

First, I had to remove the compression:

zstd -d vzdump-qemu-100-2026_02_23-01_00_02.vma.zst
Enter fullscreen mode Exit fullscreen mode

Then I extracted the raw VMA archive using Proxmox's own tool into a temporary folder:

vma extract vzdump-qemu-100-2026_02_23-01_00_02.vma /mnt/restore
Enter fullscreen mode Exit fullscreen mode

Now I had a raw image file called tmp-disk-drive-scsi0.raw in front of me. You can't just mount such an image. It contains its own partitions and, in Ubuntu's case, LVM structures. I had to mount the image as a loop device:

losetup -f
losetup /dev/loop0 /mnt/restore/tmp-disk-drive-scsi0.raw
kpartx -av /dev/loop0
Enter fullscreen mode Exit fullscreen mode

With the kpartx command, I made the partitions in the image visible to the host system. Since Ubuntu uses LVM, I had to wake up the volume group inside the backup:

vgscan
vgchange -ay
Enter fullscreen mode Exit fullscreen mode

The system immediately found the volume group ubuntu-vg. Now I could finally mount the partition like a normal disk and navigate to my lost data:

mount -t ext4 /dev/ubuntu-vg/ubuntu-lv /mnt/vm
cd /mnt/vm/var/lib/docker/volumes/infra_volume/_data/gitea_data
Enter fullscreen mode Exit fullscreen mode

There they were again. My repositories and the Gitea database were intact.

The NFS trap and the SQLite repair

Copying the data back brought the next hurdle. My new destination was an NFS share on my Synology NAS. NFS uses root_squash by default, so the root user of the Proxmox server can't push files with full permissions to the share, and a normal copy failed.

The solution was rsync without taking over permissions:

rsync -rlt --no-perms --no-owner --no-group /mnt/vm/var/... /mnt/gitea/
Enter fullscreen mode Exit fullscreen mode

The relief was short-lived. The file gitea.db was corrupted, probably because the backup ran at the exact moment the database was being written to. SQLite has a repair mode for this:

sqlite3 gitea.db ".recover" | sqlite3 gitea_fixed.db
Enter fullscreen mode Exit fullscreen mode

If that aborts, you can force SQLite to ignore errors during import:

sqlite3 gitea.db ".recover" | sqlite3 -init <(echo ".bail off") gitea_fixed.db
Enter fullscreen mode Exit fullscreen mode

The new setup

After replacing the database and cleanly unmounting everything (umount, vgchange -an, losetup -d), I rebuilt the setup to avoid the mistake.

The new compose file keeps the data on the NAS via a direct mount point instead of a path inside Docker's internal storage. Whatever Docker does on the Lenovo, the data stays on the NAS:

version: "3.9"

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    restart: always
    networks: [cloudflare]
    ports:
      - "3000:3000"
      - "2222:2222"
    volumes:
      - /mnt/gitea:/data

  runner:
    image: gitea/act_runner:latest
    container_name: gitea-runner
    restart: always
    depends_on: [gitea]
    networks: [cloudflare]
    environment:
      GITEA_RUNNER_NAME: CI-Runner
      CONFIG_FILE: /data/config.yaml
      GITEA_INSTANCE_URL: http://gitea:3000
      GITEA_RUNNER_REGISTRATION_TOKEN_FILE: /run/secrets/gitea_runner_token
    volumes:
      - runner_data:/data
      - /var/run/docker.sock:/var/run/docker.sock

networks:
  cloudflare:
    external: true

volumes:
  runner_data:
    external: true

secrets:
  gitea_runner_token:
    file: /data/compose/secrets/gitea/gitea_runner_token
Enter fullscreen mode Exit fullscreen mode

What I took away from it

Never use /var/lib/docker/volumes/ or any of Docker's internal paths as the source for a bind mount. Use --no-perms --no-owner --no-group when rsyncing onto an NFS mount. Learn SQLite's .recover mode before you need it. And a backup is only worth something once you have actually pulled a single directory out of one and put it back.

This post was originally published on www.slashgordon.link.

Top comments (1)

Collapse
 
p_o_26e854a54d851cd606f08 profile image
P O •

the NFS and SQLite trap is a good reminder that a backup is only real after a restore test. before repairing, i'd snapshot the VM, keep the app stopped, run PRAGMA integrity_check, and copy the database locally instead of editing it over NFS. explicit volume paths plus a healthcheck in compose should make the next reboot much less exciting.