DEV Community: William Mbotta

Why i love docker

William Mbotta — Thu, 12 Sep 2024 00:24:43 +0000

1. The problem

This morning, I had a problem: my server, a Raspberry Pi 2 bought in 2016, no longer starts. I used this old machine to host many services (WireGuard, Nextcloud, Bitwarden, ODPS server, etc.).

After a few unsuccessful attempts, I gave up on the idea of repairing it and decided to use another one of my servers instead. I unplugged the external hard drive from my Raspberry Pi and plugged it into my other server.

And within 10 minutes, all the services on the new machine were running just like they did on the Raspberry Pi, with all the data intact. How did I do that?

Simply:

$ docker compose up -d

2. How I Started

I’ve been an almost exclusive Linux user for about fifteen years now on my laptop, starting with Ubuntu/Debian and then Arch, just to show off a bit. But I'm not an advanced user—I mostly do pretty basic stuff. I only use the shell when I have an issue or for my work as a JavaScript developer.

Of course, like many, I bought a few Raspberry Pis back in the day, but it was mostly a toy. Even though I did manage to install a private Git server and stream my music library using MPD.

But it's only recently that I started systematically installing a lot of services: Nextcloud, PhotoPrism, Bitwarden, and many more...

Installing each of these services can be long and tedious, for example, installing Nextcloud. Even if everything goes well, it won't take just 10 minutes :)

And after that, you’ll still need to retrieve the old data for Nextcloud, for example:

$ cp -r /mnt/storage/nextcloud/var/www/html /new_nextcloud_dir

Then you’ll need to redo the post-installation configuration, create accounts, and repeat this process for every service, each with its own way of working.

Docker and Docker Compose greatly simplify this process.

3. How It Works

On my old Raspberry Pi, I had many directories on my external drive like these:

old@server:/mnt/seagate$ ls nextcloud/ photoprism/ bitwarden/ wireguard/ odps/

All these directories are structured more or less the same way. For example, Nextcloud:

old@server:/mnt/seagate/nextcloud$ ls
data  db  docker-compose.yml  nextcloud.sql  redis

The file we’re going to modify here is docker-compose.yml. The files or directories are generated by the containers and contain the data we generated while using the service.

In general, I don't write the docker-compose.yml files myself. Most projects have one, or if not, you can usually find someone who has made one.

# docker-compose.yml file
services:
  nc:
    image: nextcloud:apache
    environment:
      - POSTGRES_HOST=db
      - POSTGRES_PASSWORD=nextcloud
      - POSTGRES_DB=nextcloud
      - POSTGRES_USER=nextcloud
      - REDIS_HOST=redis
    ports:
      - 4080:80
    restart: always
    volumes:
      - ./data:/var/www/html # I only modify these lines
    depends_on:
      - redis
      - db
  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_PASSWORD=nextcloud
      - POSTGRES_DB=nextcloud
      - POSTGRES_USER=nextcloud
    restart: always
    volumes:
      - ./db:/var/lib/postgresql/data # I only modify these lines
    expose:
      - 5432
  redis:
    image: redis:alpine
    restart: always
    volumes:
      - ./redis:/data # I only modify these lines
    expose:
      - 6379

Docker containers are isolated processes that share the host operating system’s kernel but run in compartmentalized environments (called "namespaces") that separate them from the rest of the system and from other containers. In terms of communication with the outside world, containers can be configured to interact via networks, but by default, they are isolated.

Regarding data persistence, by default, a Docker container does not retain data once stopped or deleted because anything written to its internal filesystem is ephemeral. To persist data beyond the lifecycle of a container, you must explicitly mount volumes or directories. This allows you to save data in the host’s filesystem or on external storage.

When defining a volume or directory mount, the left-hand path (in Docker syntax) specifies the location of files on the host machine, and the right-hand path indicates where these files will be accessible inside the container.

The modification I make simply tells Docker that the data will always be in the current directory, which is itself located on my external drive.

This allows all the necessary components to keep our services running to be located in the same places on my external drive: configuration and data.

This clear separation between data and application is what I truly appreciate about Docker.

It’s what allows me to unplug this hard drive and mount it elsewhere. If Docker Compose is already installed on the machine, I just do:

new@server:$ cd /mnt/seagate/nextcloud
new@server:/mnt/seagate/nextcloud$ docker compose up -d
new@server:$ cd /mnt/seagate/bitwarden
new@server:/mnt/seagate/bitwarden$ docker compose up -d
new@server:$ cd /mnt/seagate/wireguard
new@server:/mnt/seagate/wireguard$ docker compose up -d

And that’s it! You find your setup exactly as you left it. I really mean everything: the database connections, users, the latest changes—it all feels like you never switched machines! Even the Redis cache is restored to the state it was in the last time the service ran on my Pi.

I find this really amazing. The hype around Docker was definitely not exaggerated. I even dream that all the software I use could work this way, even on my laptop.

For the record, since I’m too lazy to go into each directory manually, I wrote a little script to do it for me:

# Find all directories that are exactly one level deep and contain a docker-compose.yml file
for dir in $(find . -mindepth 2 -maxdepth 2 -type f -name "docker-compose.yml" -exec dirname {} \;); do
  echo "Entering directory: $dir"
  cd "$dir"
  # Start docker compose in the current directory
  docker compose up -d
done

However, it’s worth noting that this isn’t always necessary at startup if you’ve selected the restart: always option in your Docker Compose configuration. The Docker daemon itself takes care of reviving all services when the server starts.

4. Updating is Even Simpler

I didn’t mention it earlier, but when you run docker compose up -d for the first time, three commands are actually executed:

$ docker compose pull
$ docker compose build
$ docker compose start

In subsequent runs, it’s equivalent to just a start.

If you want to update your container, you first need to change the version number, or if you like living dangerously, you can leave the latest tag next to your image name, so it always fetches the latest image.

To update:

$ docker compose pull
$ docker compose up -d  # It restarts the container only if `docker pull` found a newer image.

5. Conclusion

Honestly, I don’t know what you think, but I find this approach simple, elegant, and ultra-convenient.

There are thousands of Docker images available, and the principle is always the same.

WordPress? Docker Compose. A CMS? Docker Compose. Video or audio streaming? Docker Compose.

So, next time you think you might need a SaaS solution, just try this small reflex: search for my problem self-hosted in your favorite search engine.

If you find something that fits your needs, look for the docker-compose.yml, make the necessary modifications for the volumes, and the world is yours!

I recommend this site, which lists an incredible number of services that can be installed simply with docker compose.

We’ll discuss later how to access these services; most of the time, I prefer using a VPN, as it avoids exposing my services on the internet. Of course, WireGuard installs in a snap with its own docker compose (though you’ll need to forward UDP ports from your router to your instance). You even get a nice web interface with authentication and the ability to generate a QR code for each client as a bonus.

Or, if you’re using a VPS, I can show you how to associate each service with a domain. It’s really simple, although it wasn’t for me until late 2022.

Until then, see you soon, and thanks for reading this far.

How I Back Up My Data

William Mbotta — Thu, 22 Aug 2024 14:00:00 +0000

In my last article, I explained how Docker saved me when my Raspberry Pi, which hosted all my services, suddenly failed. Indeed, Docker allows for good compartmentalization of configurations and data, and lets you choose where to store them, which in my case was an external hard drive. In case of a failure, you just need to take the hard drive and connect it to another machine, and you're done.

These kinds of accidents happen quite often unfortunately, and the method I just described is quite effective in solving this problem.
However, what happens if it's the external hard drive that fails or worse, if I have a fire? In that case, do I lose everything?

This type of incident is rarer. I've never experienced a hard drive failure. All the hard drives I bought in the previous decade are still working. So it was more difficult for me to build an infrastructure resilient to this kind of problem. I managed to do it anyway, and I'm going to present how I proceed. I'll introduce you to deduplication with Borg Backup and Amazon S3 Glacier.

1. 3-2-1 Rule

For setting up a backup plan, I follow this principle that seems to be a consensus of having three different copies of your data:

Use at least 3 copies
Use two different types of media
And have one off-site copy

The off-site copy is what allows you to survive a fire, for example.

That's good for the overview and guiding principle, let's see how I apply it.

2. Borg Backup and Deduplication

1. Install Borg Backup

The main problem when making backups is the redundancy of data between different backups. Fortunately, tools like Borg exist, allowing us to make incremental backups. This means, the first time it backs up all your files, but the other times it only records the changes, which saves bandwidth and also space on the backup disk.

If Borg Backup is not yet installed:

$ sudo apt install borgbackup

2. Configure AWS CLI

Configure AWS CLI with your AWS access information:

aws configure

You will be prompted to enter the following information:

AWS Access Key ID: Your AWS access key.
AWS Secret Access Key: Your AWS secret key.
Default region name: The AWS region in which your S3 bucket is located (e.g. us-west-2).
Default output format: You can leave this blank or choose json.

Now you need to create an amazon bucket, and take care to create a glacier deep archive bucket.

Once you've done that, you can create a

$ aws s3 sync /mnt/seagate/borg-repo s3://my-borg-backups

For restoration

$ aws s3 sync s3://my-borg-backups /mnt/restore-point

I'll show you the final script for the backup

REPOSITORY="/mnt/seagate/borg-repo”
SOURCE="/mnt/ssd/”
BORG_S3_BACKUP_BUCKET="bucket-name”


export BORG_PASSPHRASE='PASSPHRASE'

# Backup to borg repo
borg create -v --stats $REPOSITORY::$(date +%Y-%m-%d-%h) $SOURCE


# Backup to s3
aws s3 sync $REPOSITORY s3://$BORG_S3_BACKUP_BUCKET --storage-class DEEP_ARCHIVE --delete

Finally, I call my script in a cron job every day

If you've made it this far, you now have an automated 3-2-1 backup system with deduplication and an off-site copy.