Migrate Redis to AWS ElastiCache

How to Migrate Redis from One Server to Another

Starting from Redis version 5.0.0, Redis supports the REPLICAOF command, which assists in migrating data by making the destination Redis server a replica of the source Redis server. You can perform this migration with just a few simple steps:

Step 1: Make the destination Redis server a replica of the source Redis server by running the following command:

REPLICAOF host port

Step 2: Transform the destination Redis server into a master server by executing the following command:

REPLICAOF NO ONE

That's it!

AWS ElastiCache does not support REPLICAOF

When migrating Redis, we encountered a struggle as AWS Elasticache restricts access to certain commands, including "MIGRATE" and "REPLICAOF".

Solution

I discovered a repository written in Go that helped us address this issue by running a one-time application.
It appeared promising, but it hadn't been updated for several years. I identified some problems:

• It used LPOP, which resulted in the removal of list elements from the "From Redis Instance."
• It ran in a single thread, causing significant delays in migrating large records.
• It did not support certain key types, including Set and SortedSet.

I forked the repository and made the necessary fixes & improvements.

Usage

Step 1: Create config file
config.yaml

old_redis:
  host: localhost # IP redis server
  port: 6379 # Port redis server
  password: "" # Password of redis server, leave empty if there is no password

new_redis:
  host: localhost
  port: 6380
  password: ""

migration_databases: [0,1,2,3,4,5,6,7,8] # Databases list which needs to be migrated
concurrent_workers: 5
clear_before_migration: true

Step 2: Run using Docker

docker run \
--rm \
-v /path/to/config.yaml:/data/config.yaml \
 huanttok/redis-migrator migrate \
 --config.file=/data/config.yaml \
--log.level=debug

(To call localhost while using Docker, use host.docker.internal instead.

Source code

huantt / redis-migrator

This tool is designed to migrate Redis database keys from one server to another, addressing the pain point of missing 'REPLICAOF' on AWS ElastiCache

About

The Redis Migrator is a Golang-based tool designed to migrate database keys from one Redis cluster to another.

Since Redis 5.0.0, you can use REPLICAOF command to replicate data from one redis to another.

This application is developed for the older version of Redis or AWS ElastiCache instances that do not support REPLICAOF command (AWS docs).

(This repo is forked from opstree/redis-migration. The original repo has been inactive for over 3 years, so I decided to continue its development without creating pull requests.)

Supported key types:

• String
• Hash
• List
• Set
• Sorted Set

Usage

Installation

Using Go

go install github.com/huantt/redis-migrator@latest

Using Docker

docker run \
--rm \
-v /path/to/migrate.yaml:/data/migrate.yaml \
 huanttok/redis-migrator migrate \
 --config.file=/data/migrate.yaml \
--log.level=debug

To call localhost while using Docker, use host.docker.internal instead.

Configuration

For using redis-migrator, we have to create a configuration file and provide some needful information to it. An example…

View on GitHub