Originally published on kuryzhev.cloud
Your bash backup script cron job ran fine for eight months — then a cron overlap silently corrupted three weeks of archives before anyone noticed the local staging directory had two aws s3 sync processes writing to it at once. Nobody got paged. There was no error. The exit code was 0. That's the thing about backup scripts: they fail quietly, and by the time you notice, the good backups are already gone. Here are seven things I now bake into every bash backup script cron job before it touches production data.
For reference, here's the full script most of these tips pull from — locking, cleanup, checksums, and S3 upload in one file:
#!/usr/bin/env bash
# backup-to-s3.sh — daily backup with locking, logging, and S3 sync
set -euo pipefail
# --- Config ---
SRC_DIR="/var/www/app/data"
BUCKET="s3://my-backups-bucket/app-data"
LOG_DIR="/var/log/backup"
LOCK_FILE="/var/run/backup-to-s3.lock"
RETENTION_DAYS=30
TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
LOG_FILE="${LOG_DIR}/backup-${TIMESTAMP}.log"
mkdir -p "$LOG_DIR"
# --- Locking: prevent overlapping cron runs ---
exec 200>"$LOCK_FILE"
if ! flock -n 200; then
echo "$(date -Iseconds) Another backup is already running, exiting." >> "$LOG_FILE"
exit 1
fi
# --- Temp workspace + guaranteed cleanup ---
TMP_DIR="$(mktemp -d /tmp/backup.XXXXXX)"
trap 'rm -rf "$TMP_DIR"' EXIT
log() { echo "$(date -Iseconds) $*" >> "$LOG_FILE"; }
log "Starting backup of ${SRC_DIR}"
ARCHIVE="${TMP_DIR}/backup-${TIMESTAMP}.tar.gz"
tar -czf "$ARCHIVE" -C "$(dirname "$SRC_DIR")" "$(basename "$SRC_DIR")"
# Generate checksum before upload
sha256sum "$ARCHIVE" > "${ARCHIVE}.sha256"
log "Archive created: ${ARCHIVE} ($(du -h "$ARCHIVE" | cut -f1))"
aws s3 cp "$ARCHIVE" "${BUCKET}/${TIMESTAMP}/" --storage-class STANDARD_IA
aws s3 cp "${ARCHIVE}.sha256" "${BUCKET}/${TIMESTAMP}/"
log "Upload complete to ${BUCKET}/${TIMESTAMP}/"
# --- Local retention cleanup (be careful with the path!) ---
find "$LOG_DIR" -name "backup-*.log" -mtime +"$RETENTION_DAYS" -delete
log "Backup finished successfully"
Run set -euo pipefail — and know exactly what it doesn't catch
These three flags are the minimum viable safety net for any bash backup script: -e exits on any non-zero command, -u catches undefined variables, and -o pipefail makes a pipeline fail if any command in it fails, not just the last one. What they don't catch: a failing command inside an if cmd; then condition — that's intentional, since bash assumes you're testing the exit code on purpose. The other trap I've hit is command | tee log.txt, which without pipefail reports success even if command died, because tee itself exits 0.
# Without pipefail, this "succeeds" even if tar fails
tar -czf archive.tar.gz /missing/dir | tee backup.log
echo "Exit code: $?" # 0, because tee succeeded
# With set -o pipefail, the real failure surfaces
set -o pipefail
tar -czf archive.tar.gz /missing/dir | tee backup.log
echo "Exit code: $?" # non-zero, tar's actual status
Serialize your bash backup script cron job with flock
If your backup takes 70 minutes and cron fires every 60, you will eventually run two aws s3 sync processes against the same staging directory. I've seen this corrupt a tarball mid-write. The idiomatic fix is flock with a file descriptor, not a lock file check inside the script — that pattern has race conditions. Watch out: if you forget the -n (non-blocking) flag, queued cron jobs pile up waiting for the lock instead of exiting cleanly, and you end up with a backlog of zombie processes an hour later.
# crontab line — flock wraps the script call directly
15 2 * * * flock -n /var/run/backup.lock /opt/scripts/backup-to-s3.sh >> /var/log/backup/cron.log 2>&1
Don't trust logrotate alone on backup script logs
A verbose backup script that logs every file it touches will fill /var/log/backup faster than you'd think — I've had a disk-full incident from exactly this. logrotate handles the rotation, but the config matters. The gotcha is copytruncate: it's necessary if your script holds the log file descriptor open across the run, but it introduces a small race window where lines written between the copy and the truncate get lost. Fine for backup logs, not acceptable for audit trails.
# /etc/logrotate.d/backup
/var/log/backup/*.log {
daily
rotate 14
size 100M
compress
missingok
notifempty
copytruncate
}
Dry-run every S3 sync before it touches production
The --delete flag on aws s3 sync has wiped more production data than any other single CLI option I've dealt with. Every time you touch a sync script, run it with --dryrun first and actually read the output — don't just glance at it. One real incident: a case-sensitivity mismatch between local and remote paths caused --delete to remove files that technically still existed locally, just under a different casing. Also worth knowing: AWS CLI v2 (2.15+) changed default retry/backoff behavior compared to v1, so a script written years ago may behave differently after an OS or CLI upgrade you didn't plan for.
$ aws s3 sync ./local-data s3://my-backups-bucket/app-data --delete --dryrun
(dryrun) delete: s3://my-backups-bucket/app-data/old-file.txt
(dryrun) upload: ./local-data/new-file.txt to s3://my-backups-bucket/app-data/new-file.txt
Use trap for cleanup, not just set -e
set -e stops execution on failure, but it doesn't clean anything up — your lock file, temp directory, or half-finished upload just sit there. I put trap 'cleanup' EXIT immediately after creating the temp workspace, before any command that could fail, so it fires on both the success path and every error/interrupt path. Pair it with mktemp -d instead of a hardcoded /tmp/backup path — otherwise two manual runs or overlapping cron jobs will collide on the same directory.
TMP_DIR="$(mktemp -d /tmp/backup.XXXXXX)"
trap 'rm -rf "$TMP_DIR"' EXIT # runs on success, error, or Ctrl+C
Checksum backups — an exit code of 0 doesn't mean the data is intact
I stopped trusting bare exit codes after a "successful" backup turned out to be a partially uploaded, silently retried file. Generate a sha256sum locally before upload and store it alongside the archive so you can verify integrity later, independent of the CLI's exit status. Watch out: don't compare local MD5 to the S3 ETag for anything uploaded via multipart — aws s3 sync defaults to multipart above 8MB, and the ETag becomes a hash of part hashes, not a plain MD5. On the security side, never hardcode AWS_ACCESS_KEY_ID in the script; use an IAM instance role or a scoped profile limited to s3:PutObject/s3:GetObject on your backup prefix. See the AWS CLI s3 sync documentation for the current multipart threshold behavior.
sha256sum backup-20260728.tar.gz > backup-20260728.tar.gz.sha256
aws s3 cp backup-20260728.tar.gz.sha256 s3://my-backups-bucket/app-data/
Keep the bash backup script cron environment sane
"It works in my terminal but fails in cron" is almost always a PATH problem. Cron runs with a minimal environment — often just /usr/bin:/bin — so any script calling aws, jq, or s3cmd installed under /usr/local/bin will fail with a cryptic "command not found" that never shows up because cron mail is broken or unconfigured on most modern minimal server images. Set PATH explicitly and redirect output to a real log file instead of relying on mail delivery you never set up.
15 2 * * * PATH=/usr/local/bin:/usr/bin:/bin /opt/scripts/backup-to-s3.sh >> /var/log/backup/cron.log 2>&1
None of these fixes are exotic — they're the boring details that separate a bash backup script cron job that quietly rots for months from one that actually protects your data. If you're setting up your own backup automation, start with the full script above, add these seven checks one at a time, and dry-run everything with --delete before it ever touches a production bucket. For more S3 and CLI automation patterns, check out our bash automation archive.
Top comments (0)