A Production-Ready Linux Backup Pipeline with restic + systemd timers

#linux #devops #opensource #automation

Backups are only useful if they are automated, verifiable, and restorable.

In this guide, you’ll build a practical backup pipeline on Linux using:

restic for encrypted, deduplicated backups
systemd services + timers for scheduling and reliability
retention + prune + check for long-term maintenance
a restore drill so you know recovery actually works

I’ll use Debian/Ubuntu examples, but this works on any modern Linux distro with systemd.

1) Install restic

# Debian/Ubuntu
sudo apt update
sudo apt install -y restic

# Verify
restic version

2) Create a repository and credentials (securely)

We’ll use a local path as the repository in this example (/srv/restic-repo).
You can later swap this to S3/B2/SFTP/etc.

sudo mkdir -p /srv/restic-repo
sudo chmod 700 /srv/restic-repo

# Create password file (root-only)
sudo install -m 600 /dev/null /etc/restic/passphrase
sudo sh -c 'printf "%s\n" "REPLACE_WITH_A_STRONG_RANDOM_PASSWORD" > /etc/restic/passphrase'

Create an environment file for restic:

sudo tee /etc/restic/env >/dev/null <<'EOF'
RESTIC_REPOSITORY=/srv/restic-repo
RESTIC_PASSWORD_FILE=/etc/restic/passphrase
EOF
sudo chmod 600 /etc/restic/env

Initialize the repo:

sudo --preserve-env=RESTIC_REPOSITORY,RESTIC_PASSWORD_FILE \
  env $(cat /etc/restic/env | xargs) restic init

3) Define what to back up

Create include/exclude lists so backups stay intentional.

sudo tee /etc/restic/includes >/dev/null <<'EOF'
/etc
/home
/var/lib
EOF

sudo tee /etc/restic/excludes >/dev/null <<'EOF'
/home/*/.cache
/var/tmp
/tmp
EOF

4) Backup script (with retention + integrity check)

sudo tee /usr/local/sbin/restic-backup >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

source /etc/restic/env
export RESTIC_REPOSITORY RESTIC_PASSWORD_FILE

# 1) Backup selected paths
restic backup \
  --files-from /etc/restic/includes \
  --exclude-file /etc/restic/excludes \
  --verbose

# 2) Retention policy + prune
# Keep: 7 daily, 4 weekly, 12 monthly snapshots
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune

# 3) Integrity check (metadata/read-data-subset check)
restic check --read-data-subset=5%
EOF

sudo chmod 700 /usr/local/sbin/restic-backup

Why this flow? forget removes snapshots by policy, and prune reclaims unreferenced data. Running check afterward helps catch repository issues early.

5) systemd service + timer units

Service unit

sudo tee /etc/systemd/system/restic-backup.service >/dev/null <<'EOF'
[Unit]
Description=Run restic backup job
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/restic-backup
User=root
Group=root
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
EOF

Timer unit (daily at 02:30, persistent)

sudo tee /etc/systemd/system/restic-backup.timer >/dev/null <<'EOF'
[Unit]
Description=Daily restic backup

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=15m

[Install]
WantedBy=timers.target
EOF

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup.timer
sudo systemctl list-timers --all | grep restic-backup

Persistent=true means if your machine was off at 02:30, the missed run triggers on next boot.

6) Monitor and verify jobs

# Last run logs
journalctl -u restic-backup.service -n 100 --no-pager

# Manual test run (recommended after setup)
sudo systemctl start restic-backup.service

# Snapshot list
sudo env $(cat /etc/restic/env | xargs) restic snapshots

7) Restore drill (don’t skip this)

Create a file, back it up, then restore to a temporary location.

# Example file
echo "backup-restore-test" | sudo tee /etc/restore-drill.txt >/dev/null

# Run backup now
sudo systemctl start restic-backup.service

# Restore latest snapshot into /tmp/restore-test
sudo rm -rf /tmp/restore-test
sudo mkdir -p /tmp/restore-test
sudo env $(cat /etc/restic/env | xargs) restic restore latest --target /tmp/restore-test

# Verify restored file
cat /tmp/restore-test/etc/restore-drill.txt

If this works, your backup system is not just configured — it’s proven.

8) Hardening tips (quick wins)

Store repo and passphrase separately (different failure domains)
Run periodic restic check (full or subset)
Monitor timer failures with systemctl status + logs
Test restore monthly for critical paths
If using remote storage, add network resiliency and alerting

Common pitfalls

Backing up huge cache/temp trees (wastes storage)
Never pruning old snapshots
No restore tests
Running all backup hosts at exactly the same time (avoid thundering herd; use RandomizedDelaySec)