DEV Community

crow
crow

Posted on

Automating Backups for Your AI Projects on Linux

#beginners #productivity #automation #linux

Automating Backups for Your AI Projects on Linux

Step‑by‑step guide (2025)

Why Backup Matters

If you’re a data scientist, hobbyist, or student who’s just started tinkering with machine learning, the biggest headache isn’t the model architecture. It’s the fact that your workstation is a single point of failure. One power outage, one accidental “rm -rf /”, and all your notebooks, datasets, and trained weights are gone.

I built my own AI‑ready Linux workstation for under $800 last year: an AMD Ryzen 5 5600X, an NVIDIA RTX 3060, a fast NVMe SSD, and a secondary SATA drive for backups. After setting up CUDA, LM Studio, and Golem for distributed training, I hit “Run” and realized the real test began: How do I keep all that data safe?

This guide walks you through automating backups on Linux—hardware choices, OS setup, GPU drivers, and a fully‑automated snapshot pipeline—so you can focus on building models instead of fearing data loss.

1. Pick Reliable Hardware (and Save)

Component Why It Matters Affiliate
Motherboard with SATA & NVMe Allows adding a secondary drive later without buying another board. [AFF: Amazon RTX 3060]
Dual‑BIOS / BIOS recovery Prevents boot issues if you accidentally flash the wrong firmware. —
80+ Bronze PSU Reduces risk of sudden power loss that could corrupt drives. —
NVMe SSD (500 GB+) for OS & training data Faster I/O speeds mean less time waiting during backup. [AFF: NVMe SSD]
Secondary SATA SSD or HDD (1‑2 TB) for backups Keeps your primary drive free from wear and acts as a quick restore point. —

Tip: Even on a tight budget, invest in at least one redundant drive. It’s the cheapest way to avoid catastrophic data loss.

2. Install Ubuntu (or Debian‑based distro) with Dual‑Boot for Safety

  1. Create bootable USB – Use Rufus or dd.
  2. Partitioning – Allocate ~300 GB for /home, leaving the rest for OS and swap.
  3. Install Ubuntu 24.04 LTS (or your choice).
  4. Disable Secure Boot if you plan to use custom kernels or drivers that aren’t signed.

Why dual‑boot? A Windows or another Linux install gives you a fallback if the primary OS becomes corrupted.

3. Install NVIDIA Drivers & CUDA Toolkit 

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo ubuntu-drivers autoinstall   # installs the recommended driver (e.g., 535)
Enter fullscreen mode Exit fullscreen mode

Verify with nvidia-smi.

If you’ll use Docker containers that need GPU access, install the NVIDIA Container Toolkit:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt update && sudo apt install -y nvidia-docker2
sudo systemctl restart docker
Enter fullscreen mode Exit fullscreen mode

Pro tip: Keep the driver version aligned with your CUDA toolkit to avoid runtime errors.

4. Set Up LM Studio (or JupyterLab) for Development 

python3 -m venv ~/ai-env
source ~/ai-env/bin/activate
pip install lmstudio==0.x   # replace with latest
Enter fullscreen mode Exit fullscreen mode

Configure GPU usage in your scripts:

device = torch.device("cuda").

Why LM Studio? It bundles a lightweight IDE, GPU monitoring, and quick access to popular models—perfect for rapid prototyping.

5. Deploy Golem (or Similar Distributed Training Framework) 

pip install golem
Enter fullscreen mode Exit fullscreen mode

Configure your node to point to the /home partition where data lives.

Set up a task queue so each job writes checkpoints back to the backup drive, and monitor with golem status.

Why Golem? It turns idle GPUs into compute resources while keeping your local data isolated on the main SSD.

6. Automate Backups: The Core of the Guide

A. Pick a Backup Tool – BorgBackup

Tool Pros Cons
rsync + cron Simple, no extra deps. Manual incremental config.
BorgBackup (borg) Deduplication, compression, encryption. Slight learning curve.
Restic Fast, easy setup, cloud backends. No native deduplication.

We’ll use BorgBackup for its balance of speed and space efficiency.

B. Install Borg 

sudo apt install borgbackup
Enter fullscreen mode Exit fullscreen mode

C. Create a Backup Repository

Mount your secondary SATA SSD at /mnt/backup and run:

mkdir -p /mnt/backup/borg_repo
borg init --encryption=repokey /mnt/backup/borg_repo
Enter fullscreen mode Exit fullscreen mode

You’ll be prompted for a passphrase—keep it safe but accessible.

D. Backup Script

Create /usr/local/bin/ai_backup.sh:

#!/bin/bash
# AI Backup Script – Borg + cron

export BORG_REPO=/mnt/backup/borg_repo
export BORG_PASSPHRASE="YOUR_PASS_PHRASE"

SOURCE="/home"
EXCLUDE=("--exclude=.cache" "--exclude=*.tmp" "--exclude=~/datasets/tmp")

borg create \
  --verbose \
  --filter AME \
  --list \
  --stats \
  --compression lz4 \
  "${BORG_REPO}::$(date +%Y-%m-%d-%H%M%S)" \
  $SOURCE ${EXCLUDE[@]}

# Prune older backups (keep last 7 daily, 4 weekly)
borg prune \
  --list \
  -v \
  --keep-daily=7 \
  --keep-weekly=4 \
  $BORG_REPO
Enter fullscreen mode Exit fullscreen mode

Make it executable:

sudo chmod +x /usr/local/bin/ai_backup.sh
Enter fullscreen mode Exit fullscreen mode

E. Schedule with cron

Edit your crontab:

# AI backup – every day at 2 AM
0 2 * * * /usr/local/bin/ai_backup.sh >> ~/ai-backup.log 2>&1
Enter fullscreen mode Exit fullscreen mode

Now backups run unattended. Check ~/ai-backup.log for any errors.

F. Test Restoration 

borg extract --verbose \
  $BORG_REPO::2025-12-01-020000 \
  /home/user/test_file.txt
Enter fullscreen mode Exit fullscreen mode

Confirm the file matches the original.

7. Optional: Off‑Site Cloud Backup (Backblaze B2)

  1. Create a bucket on Backblaze B2.
  2. Install b2 CLI and authenticate.
  3. Add to your script after the local backup: 
# Upload repo snapshot to B2
b2 upload-file $BORG_REPO /path/to/repo b2://my-ai-backups/$(date +%Y-%m)
Enter fullscreen mode Exit fullscreen mode

This adds an extra layer of protection against hardware failure.

Recap & Call to Action

You now have a complete, automated backup pipeline that protects your AI projects on Linux—from the initial $800 build to nightly snapshots that keep data safe.

What’s your setup? Drop a comment below with your hardware choices, any tweaks you made, or questions you have. And if you’re looking for a GPU upgrade or faster storage, check out our affiliate links:

  • [AFF: Amazon RTX 3060] – great performance for under $300
  • [AFF: NVMe SSD] – fast, reliable storage for your OS and datasets

Happy training—and stay backed up!

Top comments (0)