DEV Community: Laurent Lemaire

The Complete Redis Backup Guide (with examples)

Laurent Lemaire — Mon, 15 Aug 2022 08:45:05 +0000

In this article, we’ll see how you a real case example of a Redis backup process, end-to-end.

We'll cover how to configure a Redis backup, which binaries you should use and the most important settings you need to be aware of, as well as how to store your backup remotely and how to restore it.

Let's get started!

What is Redis?

Redis is an open source in-memory key-value store written in C.
Redis stands for Remote Dictionary Server and is used as a database, cache, queue system, and message broker.

Redis is fast because its data is stored in memory, meaning, unlike traditional databases Redis doesn't need to access the disk.
While writing this article I learned that Redis is often called a "Data Structure Server" because it provides data types that are similar to those in modern programming languages.

Some data structures that Redis provides are Strings, Lists, Sets, Hashes, and Sorted Sets.

Redis Data Types →

What is Redis used for?

Even though Redis could be used as your primary Database it's usually not what it is used for.
Here are the most common use cases for Redis:

Cache: Redis is used as a cache, which is a fast way to store data in memory.
Session storage: Redis is used to store session data. Writing and reading data out of Redis is super fast which makes it an ideal candidate for session storage.
Message queue

There are a lot of different use cases for Redis, but these are the most common ones.

Where is Redis database stored?

As stated above Redis is storing its data in memory. But depending on your use case Redis can copy the data on your disk.
This comes obviously handy when you have a large amount of data, and you need to be able to restore it and this is also why you might be needing to back it up.

Redis regularly dumps its data to an RDB file on the disk based on how snapshots are configured.

Redis configuration
The


 file contains your Redis configuration.

The configuration file is located at

 ```/etc/redis/redis.config```

 and straighforward: it's a list of instructions.
You'll find a section called

 ```#### SNAPSHOTTING ####```

 on which you can define if Redis should snapshot its data to the dis and how often it should do it.



```redis.config
save 60 1000

This configuration will make Redis dump the dataset to the disk every 60 seconds if at least 1000 keys are changed.

Learn more about Redis Configuration →

Redis also works with AOF (Append-Only File) which is a way to store the data on the disk by logging all write operations received by the server.
AOS won't be covered in this article but it is worth knowing it exists, especially when you'll need to backup and restore your data.

How to back up Redis data

Making a Redis backup is pretty easy. You'll need to make a fresh copy of the RDB file, compress it and save it somewhere safe.

Redis offers 2 methods to "force" a snapshot:


: This will force a snapshot to be taken synchronously.
-

 ```BGSAVE```

: This will force a snapshot to be taken asynchronously.

The easiest way is to use the

 ```SAVE```

 method but it will block all other operations until the snapshot is done.
Using

 ```BGSAVE```

 will make the server continue to accept commands and will not block other operations but you'll have to figure out when the snapshot is this one is asynchronous.

**So, if you want to make a backup you'll need to do the following:**
1. Review / update your Redis configuration.
2. Create a snapshot for your Redis data (known as a "dump"/"rdb file").
3. Save the RDB file to a remote location

### 1. Review / update your Redis configuration.

You'll need to know where your snapshot file will be generated using the redis-cli command described below.

The default location of your Redis config file is

 ```/etc/redis/redis.config```

.
You can also use this command to find the location of your Redis config file:

 ```redis-cli config get dir```

.

You can find the configuration file here [https://redis.io/topics/config](https://redis.io/topics/config).

### 2. Create a snapshot for your Redis data.

#### <u>Using redis-cli

 ```SAVE```

 command</u>
This method will work synchronously to make a snapshot of your Redis database.
Just ssh into your server and run the following command:

Log in to the database command line interface:

redis-cli




You might have to authenticate to your database:


```shell{promptHost: 127.0.0.1}
auth YOUR_PASSWORD_HERE

SAVE

The output will be something like this:

OK
(1.23s)

You can then exit the redis-cli by typing


.

At this stage, the

 ```RDB```

 file will be saved in

 ```/var/lib/redis/```

 and will be named

 ```dump.rdb```

.

#### <u>Using redis-cli

 ```BGSAVE```

 command</u>

Using the asynchronous dump function, you'll need to make sure you are aware of the end of the process.
One way to do it is to use

 ```inotifywait```

 which will notify you when a change to the dump file is made.

### 3. Automate your Redis backups and store them on AWS S3

As stated in [Redis documentation](https://redis.io/docs/manual/persistence/), it's safe to copy the RDB file even if used by your running server:

> Redis is very data backup friendly since you can copy RDB files while the database is running: the RDB is never modified once produced, and while it gets produced it uses a temporary name and is renamed into its final destination atomically using rename(2) only when the new snapshot is complete.
>
> This means that copying the RDB file is completely safe while the server is running. This is what we suggest:
>
> - Create a cron job in your server creating hourly snapshots of the RDB file in one directory, and daily snapshots in a different directory.
> - Every time the cron script runs, make sure to call the find command to make sure too old snapshots are deleted: for instance you can take hourly snapshots for the latest 48 hours, and daily snapshots for one or two months. Make sure to name the snapshots with date and time information.
> - At least one time every day make sure to transfer an RDB snapshot outside your data center or at least outside the physical machine running your Redis instance.


We'll create a script that will create our dump file, then upload to Amazon S3.

#### <u>Create a shell script that will dump the Redis database</u>


```shell
cd ~
mkdir scripts
cd scripts
nano redis_backup.sh

Copy and paste the script below to it:

#!/bin/bash
rdb_file="/FOLDER_TO_YOUR_REDIS_DUMP_FILE/redis/dump.rdb"
redis_cli="/usr/bin/redis-cli"

DIR=`date +%d-%m-%y`
DEST=~/redis_backups/$DIR
mkdir $DEST

echo save| $redis_cli
exit 1

Send the Redis DUMP to an AWS S3 bucket

Append to


 the following:



```shell
BUCKET_NAME="YOUR_S3_BUCKET_NAME"
aws s3 cp $rdb_file s3://YOUR_S3_BUCKET/redis_backups/ && echo "Backup copied to S3"

Schedule the script to run every day at midnight

First, let's CHMOD the script to make it executable:

chmod +x ~/scripts/db_sync.sh

Then create a cron job to run the script every day at midnight:

crontab -e

0 0 * * * ~/scripts/redis_backup.sh # take a backup every midnight

How to restore a Redis backup

Now that you've made a backup, we'll see how to restore it from a


 file.

**We recommend you first try this on a fresh Redis server**

### Make sure AOF is disabled
AOF stands for Append-Only File, which will instruct Redis to log all operations in a `.aof` file.
Since we're restoring a backup, we need to disable AOF before restoring the data as we don't want Redis to log all these operations.

Open your configuration file (

```redis.config```

) and make sure

 ```appendonly```

 is set to

 ```no```

.

appendonly no




### Stopping the Redis server

Before being able to replace the

 ```dump.rdb```

 file, you'll need to stop the Redis server.



```shell
sudo service redis-server stop

Restoring the Redis database

Prior to restoring the database, you can rename the existing dump.rdb file in order to have a restore point in case something goes wrong.

sudo cp /home/redis/dump.rdb /home/redis/dump.rdb.bak

You can then copy the backup rdb file as follows:

sudo cp /redis_backups/20220810/dump.rdb /home/redis/dump.rdb

And finally make sure to apply the right permissions to the dump.rdb file:

sudo chmod 660 /home/redis/dump.rdb

Re-starting Redis server

sudo service redis-server start

Additional Resources

Official Redis documentation: https://redis.io/docs/manual/persistence/
Service to automate your Redis backups: https://simplebackups.com/redis-backup/

Conclusion

We've seen how to back up your Redis database and restore it, and we've seen how to automate the process.

SimpleBackups will save you a lot of time setting up scripts, ensuring they run without problems, all without code or maintenance.
It will alert you when things go wrong, and allows you to store your backups on many cloud storage services like Google, DigitalOcean, Wasabi, Dropbox, and more…

Cloud Storage Comparison - Best Providers 2021

Laurent Lemaire — Thu, 07 Oct 2021 13:59:58 +0000

We've created a list of all major (and our preferred) providers, together with their prices using the region US-EAST-1 as reference.

If you're looking to store your data on a Cloud Storage Provider, you've got plenty of choices.

Prices are often changing and what each provider offer is different.
Once you know your needs in terms of storage/frequency of access (...), you'll easily be able to identify which provider fits you the best.

This list doesn't contain all the specificities each provider offers as we tried to summarize what most people are looking for when it comes to evaluating an Object Block Storage provider.

If you want to see any additional information in this comparison just let us know.

Object Storage Providers Comparison

Name	Price /GB/month	Min price/month	Price/1000 requests	Price Ingress	Price Egress	Link	Extra
AWS S3	$0.02	-	$0.01	$0.00	$0.09/GB/month (1GB/m included)	https://aws.amazon.com/s3/pricing/	- 12 first months free - 5GB free
AWS S3 - Infrequent access	$0.01	-	$0.01	$0.00		https://aws.amazon.com/s3/pricing/
AWS Glacier	$0.00	-	$0.03	$0.00	$0.01/GB	https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
Wasabi	$0.01	-	$0.00	$0.00	$0.00	https://wasabi.com/cloud-storage-pricing/#cost-estimates
Blackblaze	$0.01	-	$0.00	$0.00	$0.01/GB	https://www.backblaze.com/b2/cloud-storage-pricing.html
DigitalOcean Spaces	$0.02	$5 (250GB included)	$0.00	$0.00	$0.00	https://www.digitalocean.com/pricing/	- Minimum 5$ - 1 TB of outbound transfer (additional 0.01/GB) - Additional storage 0.02/GB
Dropbox	$0.00	$12 (5TB included)	$0.00	$0.00	$0.00	https://www.dropbox.com/plans?tab=work#features	- Reference: Plan standard - Minimum 12$/m
Google Cloud Storage	$0.02	-	$0.01	$0.00	$0.12	https://cloud.google.com/storage/pricing	- Reference: Standard - 5GB free + 5000 operations
FileBase	$0.01	-	$0.00	$0.00	0.0059/GB (1TB included)	https://filebase.com/	- Minimum 5.99/m
Azure Blob Storage	$0.02	-	$0.0065-$0.0005	$0.00	n/a	https://azure.microsoft.com/en-us/pricing/details/storage/blobs/#overview	- Reference: Hot Storage
Exoscale	$0.02	-	$0.00	$0.00	$0.02	https://www.exoscale.com/pricing/#storage
UpCloud	$0.02	-	$0.00	$0.00	$0.01	https://upcloud.com/pricing/#object-storage	- Reference 250GB plan
Vultr	$0.02	-	$0.00	$0.00	$0.01/GB (1TB included)		- Minimu: 5$ (250gb)

This list is maintained and updated monthly - **last update: 20/09/2021)* *

The Ultimate MySQL Database Backup Script

Laurent Lemaire — Sat, 13 Jun 2020 09:43:12 +0000

Any database, needs a backup.

This article is part of “The Ultimate Backup Script” series we are creating to provide you with database backup scripts that not only allow you to create database backups, but also upload the backup dumps to Amazon S3 and automate the process daily.

Why we need a database backup?

One might think why backup is necessary for my database? The answer is simple, backup creates a copy of your physical, logical, and operational data. Which you can store at any safe place such as Amazon S3. This copy comes into use if the running database gets corrupted. Database backup can include files like control files, datafiles, and archived redo logs.

Why Amazon S3 for backup?

For this tutorial, we have chosen Amazon S3 as it is a very common choice. You can do the same thing if you would like to use another clould storage provider. The instructions won’t differ a lot as long as the cloud provider is S3-compatible.

Below we defined some less known terms that we used in the article:

What is Cron?

Cron is a software utility that offers time-based job scheduling. It supports Unix computer operating systems. To set up software environments, the developer uses Cron. He/she schedules commands or shell scripts so that they run at chosen times. It could be daily, once a week, or any interval as desired.

What is Chmod?

The chmod a short command of ‘change mode’ enables the admin to set rules for file handling. In other words, with the help of a “chmod” system call. An administrator can change the access permissions of file system objects.

Database Backup Script for MySQL and Dumping to Amazon S3

You can automate the creation of backup and storing it to Amazon S3 within a few minutes. Below bullets brief about what you are going to learn in this part of the article:

Create a script that automates the MySQL backup directory creation
Upload/sync the backups with Amazon S3
Cron will run this command every day (to back up)

Step 1: Generating a shell script which will dump the MySQL database

cd ~
mkdir scripts
cd scripts
nano db_backup.sh

Copy and paste the script below to it

#!/bin/bash
DIR=`date +%d-%m-%y`
DEST=~/db_backups/$DIR
mkdir $DEST

mysqldump -h mysql_hostname -u mysql_user  -p"mysql_password" database_name > dbbackup.sql

Now chomd the script to allow it to for execution

chmod +x ~/scripts/db_backup.sh

Step 2: Creating the shell script which sync the backups with Amazon S3

nano db_sync.sh

Copy and paste the script below to it

#!/bin/bash
/usr/local/bin/aws s3 sync ~/db_backups s3://my-bucket-name

Now chmod the script to allow it for execution

chmod +x ~/scripts/db_sync.sh

Step 3: Creating the folder in Amazon S3 for the database dumps

cd ~
mkdir db_backups

Step 4: Time to configure the AWS CLI

Before installing the AWS CLI you need to installpython-pi. Type the following commands:

apt-get update
apt-get -y install python-pip
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"

Install the AWS CLI

Type the following command:

pip install awscli

Step 5: Time to set up AWS key & Secret

Configuration and credential file settings

cd ~
mkdir .aws
nano ~/.aws/config

Paste in key_idand secret_access_key as shown below

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Step 6: Set up the Cron (to automate the process)

crontab -e

Paste the below commands at the bottom to automate the process

0 0 * * * ~/scripts/db_backup.sh # take a backup every midnight
0 2 * * * ~/scripts/db_sync.sh # upload the backup at 2am

This way the backup script will run and also sync with Amazon S3 daily.

Conclusion

Hence, by using these scripts you can achieve 3 goals:

Creating the database backup via a shell script
uploading the dump to Amazon S3
also automating this process using Cron.

Have you tried SimpleBackups.io yet?

SimpleBackups will save you a lot of time setting up scripts and ensuring they run without problems. It will alert you when things go wrong, and allows you store your backups on many cloud storage services like Google, DigitalOcean, Wasabi, Dropbox, and more…