Writing an automated Amazon S3 backup script for your MongoDB database using bash and cron in Ubuntu 16.04

Isuru Edirisinghe — Tue, 01 Oct 2019 19:25:09 +0000

While as easy as it may sound to write a simple backup script, there are a few steps that are involved in the process. And might be a tad bit unclear to those who have no prior experience of it. While there may be various methods of taking automated backups of your mongo db, this article aims to outline one such method which I personally prefer myself.

Prerequisites

A properly setup MongoDB server on top of Ubuntu. The method outlined in this tutorial was tested on Ubuntu 16.04 LTS though it may work with other versions as well.
In most cases , cron comes bundled with the most Linux distros. But if not, it should be pretty straightforward as running `sudo apt-get install cron` in a terminal window.
In order to backup to Amazon S3 you need to have aws cli installed and configured. Follow the installation and configuration instructions in the AWS documentation to do so.

Now that we have everything in place, we can start writing the script!

Writing the script

First things first, you need to setup database user credentials and other necessary variables. For now let’s provide DB credentials as variables for simplicity’s sake. Alternatively you could look into other ways to not expose credentials in your script. Also, it is advised to setup a separate database user with only backup privileges so that their is a minimum security risk involved in any case.

Go ahead and add the credentials and other necessary variables to your script

#!/bin/sh

HOST=127.0.0.1 #change to whatever your host IP is
BUCKET=yourbucket #s3 bucket name
DEST=$HOME/tmp #backup directoryDBUSER=yourusername #the mongo username
DBPWD=yourpassword #the mongo users password
DBNAME=yourdbname #the name of the DB to be backup
TIME=`date +%d-%m-%Y-%T` #the time of the backup
TAR=$DEST/../$DBNAME-$TIME.tar.gz #name and dir of backup archive
set -e #this stops execution of script if anything fails

Most of the above code is commented and is self-explanatory. So let’s go ahead and continue with writing the rest of the script.

The next step is to write the `mongodump` command, this may differ depending on your mongo db configuration. If you followed the tutorial linked at the beginning of this article on installing and configuring mongodb, it should look something like the code below (The echo statements are for logging purposes which is needed later. Logs are pretty crucial specially in a production environment, if anything fails these come in handy to find out where things went wrong!)

echo "Backing up $HOST/$DBNAME to s3://$BUCKET/ on $TIME";
mongodump -h $HOST -d $DBNAME -u $DBUSER -p $DBPWD -o $DEST --authenticationDatabase your_auth_db_name
echo "Backup performed successfully";

Next step is to archive the dump folder generated by `mongodump`. We will use the TAR variable we setup previously for this.

echo "Archiving the backup"tar -zcvf $TAR -C $DEST .echo "Done archiving"

Now we upload this archive to our s3 bucket. “a/path/you/desire/” is a relative path of your choice within your s3 bucket. Easy as cake!

echo "Uploading to s3 bucket" $BUCKETaws s3 cp $TAR s3://$BUCKET/a/path/you/desire/

If all went well the backup should now appear on your s3 bucket.
NOTE: Make sure to download and restore the backup on your local database to make sure all your data are there once restored

BUT we still have a few more steps to get through.. We first need to cleanup our backup directory and the following lines handles exactly that!

echo "Cleaning up temp files"
rm -f $TAR    # Remove tar file
rm -rf $DEST  # Remove backup directory
echo "Done cleaning up!"

Now we’re done writing the script. Let’s log it

echo "Backup available at https://s3.amazonaws.com/$BUCKET/a/path/you/desire/$DBNAME-$TIME.tar.gz"

Automating the script

Now let’s automate this script using cron and we’re done!
Firstly, open a crontab using the below command in the terminal (Select a preferred text editor of your choice if prompted, I usually use nano).

crontab -e

Now at the end of the file add the following line. It will set our cron job to run everyday at 01:00 a.m. based on System date/time. You can set it to your preference. A good reference is crontab.guru if you’re not familiar with the notation! And in case you’re wondering, the “>> backup_log” at the end will append all your echo output to a file named backup_log in the same directory as the script.

0 1 * * * path/to/backup/script.sh >> backup_log

It is advised to initially set it to run around every 10 minutes in order to test run whether your script works flawlessly with cron. If your cron job fails in any case, you’d have wasted your time following this tutorial and writing that beautiful work of art(the backup script)!

So that’s it. If you’ve clearly followed the steps, you should have a fine working automated backup cycle up and running now for your mongo db. So go ahead and sip a well-deserved cup of coffee while your script does all the work for you.

Cheers!

DEV Community: Isuru Edirisinghe

Writing an automated Amazon S3 backup script for your MongoDB database using bash and cron in Ubuntu 16.04

Prerequisites

Writing the script

Automating the script