DEV Community

Cover image for How to effectively monitor regular backups
CronMonitor profile image Łukasz Maśląg
Łukasz Maśląg for CronMonitor

Posted on • Originally published at cronmonitor.app

How to effectively monitor regular backups

#devops #database #crontab #monitoring

Imagine the following scenario: you created a script in bash to create a backup of a production database, say, an online store. After creating the script and adding it to crontab, everything worked flawlessly. After some time, say a month, the database became corrupted, for example, due to the installation of a faulty plugin. At that moment, you want to retrieve an updated database backup from last night and discover that the last database backup is from two weeks ago. What happened? Everything was working fine.

This nightmare scenario is more common than you might think, and perhaps it has even affected you personally. Scripts added to crontab fail without warning, causing so-called "silent errors." They can be caused by a variety of reasons, such as a full disk, permission changes, network timeouts, expired credentials, or simply a typo after a "quick fix."

The Problem with Unmonitored Backups

Traditional cron jobs have a fundamental flaw: they only report an error when they fail to run. For example, your backup script might fail:

  • Run successfully but exit with errors
  • Exit but generate empty or corrupted files
  • Run but take 10 times longer than expected (a sign of problems)
  • Skip tables due to permission issues

Before you know it, your backup retention period might expire—leaving you without any valid backups.

Monitoring Backup Scripts

The solution is simple: your backup script should actively report its status to an external monitor. Here's how to integrate database backups with CronMonitor.

MySQL/MariaDB Backup Example 

#!/bin/bash

MONITOR_URL="https://cronmonitor.app/api/ping/your-unique-id"
BACKUP_DIR="/backups/mysql"
DATE=$(date +%Y%m%d_%H%M%S)
DB_NAME="production"

# Signal start
curl -s "${MONITOR_URL}/start"

# Perform backup
mysqldump --single-transaction \
    --routines \
    --triggers \
    "$DB_NAME" | gzip > "${BACKUP_DIR}/${DB_NAME}_${DATE}.sql.gz"

# Check if backup was successful and file is not empty
if [ $? -eq 0 ] && [ -s "${BACKUP_DIR}/${DB_NAME}_${DATE}.sql.gz" ]; then
    # Clean old backups (keep last 7 days)
    find "$BACKUP_DIR" -name "*.sql.gz" -mtime +7 -delete

    # Signal success
    curl -s "${MONITOR_URL}/complete"
else
    # Signal failure
    curl -s "${MONITOR_URL}/fail"
    exit 1
fi
Enter fullscreen mode Exit fullscreen mode

PostgreSQL Backup Example 

#!/bin/bash

MONITOR_URL="https://cronmonitor.app/api/ping/your-unique-id"
BACKUP_DIR="/backups/postgres"
DATE=$(date +%Y%m%d_%H%M%S)
DB_NAME="production"

curl -s "${MONITOR_URL}/start"

# Use custom format for flexibility
pg_dump -Fc "$DB_NAME" > "${BACKUP_DIR}/${DB_NAME}_${DATE}.dump"

if [ $? -eq 0 ] && [ -s "${BACKUP_DIR}/${DB_NAME}_${DATE}.dump" ]; then
    # Verify backup integrity
    pg_restore --list "${BACKUP_DIR}/${DB_NAME}_${DATE}.dump" > /dev/null 2>&1

    if [ $? -eq 0 ]; then
        curl -s "${MONITOR_URL}/complete"
    else
        curl -s "${MONITOR_URL}/fail"
        exit 1
    fi
else
    curl -s "${MONITOR_URL}/fail"
    exit 1
fi
Enter fullscreen mode Exit fullscreen mode

Multi-Database Backup with Size Reporting

For more comprehensive monitoring, you can report backup sizes:

#!/bin/bash

MONITOR_URL="https://cronmonitor.app/api/ping/your-unique-id"
BACKUP_DIR="/backups"
DATE=$(date +%Y%m%d)
DATABASES="app_production analytics users"

curl -s "${MONITOR_URL}/start"

FAILED=0
TOTAL_SIZE=0

for DB in $DATABASES; do
    mysqldump --single-transaction "$DB" | gzip > "${BACKUP_DIR}/${DB}_${DATE}.sql.gz"

    if [ $? -ne 0 ] || [ ! -s "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" ]; then
        FAILED=1
        echo "Backup failed for: $DB"
    else
        SIZE=$(stat -f%z "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" 2>/dev/null || stat -c%s "${BACKUP_DIR}/${DB}_${DATE}.sql.gz")
        TOTAL_SIZE=$((TOTAL_SIZE + SIZE))
    fi
done

if [ $FAILED -eq 0 ]; then
    # Report success with metadata
    curl -s "${MONITOR_URL}/complete?msg=Backed%20up%20${TOTAL_SIZE}%20bytes"
else
    curl -s "${MONITOR_URL}/fail"
    exit 1
fi
Enter fullscreen mode Exit fullscreen mode

How to configure CronMonitor to monitor a cron job

  1. Create a new monitor in CronMonitor for the "backup" job
  2. Set a projected schedule (e.g., daily at 2:00 AM)
  3. Configure a grace period—allow enough time for large databases
  4. Configure alerts via email, Slack, or Discord

Key backup monitoring settings:

  • Schedule: Match the server's cron schedule exactly
  • Grace period: Set it longer than the longest projected backup time

Best Practices

1. Verify, don't assume

Always verify that the backup file exists and has content. An empty gzipped file is still a "successful" command.

2. Test restores regularly

Backups are only as good as their restores. Schedule periodic restore tests—and monitor them too.

3. Monitor backup duration

CronMonitor tracks how long each job takes. A sudden increase in backup time often indicates growing data volume or performance issues.

4. Store backups on an external drive/server

Monitoring should include a synchronization step outside the production database server.

# After local backup succeeds
rsync -az "${BACKUP_DIR}/" remote:/backups/ && \
    curl -s "${MONITOR_URL}/complete" || \
    curl -s "${MONITOR_URL}/fail"
Enter fullscreen mode Exit fullscreen mode

5. Document recovery procedures

For example, if you receive an alert on Slack at night about a failed backup attempt, you'll need clear steps on what to do next, not a debugging session.

Conclusion

Database backups are the last line of defense against data loss. They deserve more than a cron job and the hope that everything works and executes correctly as planned. Active monitoring provides immediate information about the problem – while you still have time to resolve it.

Start monitoring your backup scripts today. Your future self (the one who doesn't have to explain data loss to a client) will thank you.

CronMonitor is a simple, developer-friendly cron job monitoring service. Set up your first monitor in under a minute.

Top comments (1)

Collapse
 
deltax profile image
deltax •

This is one of those rare articles that explains a deep systems problem
without over-engineering the explanation.

The key insight here isn’t about backups.
It’s about separating triggers from decision layers.

Cron is intentionally dumb — and that’s a feature.
What matters is that intelligence lives in the application,
with verification, context, and feedback loops.

This framing scales far beyond DevOps:
any reliable system (including AI systems) must be able to
detect silent failure, verify outcomes, and correct itself.

Clear, pragmatic, and structurally sound.
Great work.