DEV Community: Pavel Evstigneev

Copy Redis Data to New Server

Pavel Evstigneev — Thu, 04 Aug 2022 19:28:08 +0000

Sometimes we need to copy redis data to new instance

I know 3 simple ways to do it:

continuous migration tools
simple script to copy data
redis master-slave replication

Other options, more complex:

Redis HA & sentinel solution - it will require rebuild redis
https://github.com/Netflix/dynomite - replication and routing

redis-migrate-tool

https://github.com/vipshop/redis-migrate-tool

It monitors changes in source server/cluster and apply them in destination server. Useful when we migrate app that actively writes to redis to use new redis instance. There are other similar tools, but this one works and easy to configure.

Check README for supported types, It's written in C, easier to run it with docker

docker pull ahmadgh21/redis-migrate-tool
docker run -it ahmadgh21/redis-migrate-tool bash
# inside container
mkdir /tmp
chmod 777 /tmp
apt-get update
apt install nano
nano /etc/redis-migrate-tool/rmt.conf

Config example

[source]
type: single
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
servers :
- <source IP>:6379 # <--- change this

[target]
type: single
servers:
- <destination IP>:6379 # <--- change this

[common]
listen: 0.0.0.0:8888

run migrate-tool inside container

/opt/redis-migrate-tool -c /etc/redis-migrate-tool/rmt.conf

Make own script

This script simply copy all keys from server A to server B (it doesn't support all data types, find it in code)

require 'redis'

puts "Connecting to redis://#{ARGV[0]} and redis://#{ARGV[1]}"

redis_src = Redis.new(url: "redis://#{ARGV[0]}")
redis_dest = Redis.new(url: "redis://#{ARGV[1]}")

dest_keys = redis_dest.keys("*")

redis_src.keys("*").each do |key|
  next if dest_keys.include?(key)
  type = redis_src.type(key)

  if type == "hash"
    value = redis_src.hgetall(key)
    puts "HASH   #{key} : #{value}"
    redis_dest.mapped_hmset(key, value)
  elsif type == "list"
    values = redis_src.lrange(key, 0, -1)
    puts "LIST.  #{key} : #{values}"
    redis_dest.lpush(key, values)
  elsif type == "set"
    values = redis_src.smembers(key)
    puts "SET    #{key} : #{values}"
    redis_dest.sadd(key, values)
  else
    data = redis_src.get(key)
    puts "STRING #{key} : #{data}"
    redis_dest.set(key, data)
  end
end

puts "Complete"

Make sure you have ruby redis library installed (gem install redis)

Run the script

ruby redis_copy.rb source_IP destination_IP

Setup redis replication

I think this is most proper option but also require more preparation

on master we need to update config

protected-mode no
requirepass 1234abcd

on replica server we need to put in config:

replicaof <master ip> 6379
masterauth 1234abcd

restart master, then replica, check logs

check status in redis-cli

INFO replication

more docs:

Pros and Cons

Simple script - It copy values once, can be used for not critical data such as cache, rate limit, and for rarely updated data such as configuration

I usually run it once before updating apps to connect to new server, and once again after to make sure all data is in sync

redis-migrate-tool - Constantly replicates all data from source to destination. For apps that constantly write updates to redis, e.g. user sessions, temporary important values.

But it doesn't replicate writes from destination server back to source. To avoid situation when one process write/update to new server and other process can't find it in old server, we need to restart all our apps at one time with updated config. (I usually scale app down to 1 replica during migration)

redis replication - replica is not writable, migration may require short downtime or we need to modify code to handle it smoothly. In our code we can try to save data to new server, and if got "not writable" error then save to old server

Have a smooth redis migration to everyone who's doing it now :)

Making portable binary for MacOS

Pavel Evstigneev — Sat, 26 Sep 2020 13:57:17 +0000

Let’s say we want to take some binary and prepare it for distribution (unsigned), here is how we can make it portable with example of psql

Copy executable file

# cp /Applications/Postgres.app/Contents/Versions/latest/bin/psql .

Check linked libraries

# otool -L psql
psql:
/Applications/Postgres.app/Contents/Versions/13/lib/libpq.5.dylib (compatibility version 5.0.0, current version 5.13.0)
/usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

We can see psql has dependency on libpq.5.dylib

Copy required dylib’s

# cp /Applications/Postgres.app/Contents/Versions/latest/lib/libpq.5.13.dylib .

Check dependencies of `libpq.5.13.dylib`

# otool -L libpq.5.13.dylib
libpq.5.13.dylib:
/Applications/Postgres.app/Contents/Versions/13/lib/libpq.5.dylib (compatibility version 5.0.0, current version 5.13.0)
/Applications/Postgres.app/Contents/Versions/13/lib/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
/Applications/Postgres.app/Contents/Versions/13/lib/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

Copy `libssl.1.1.dylib`, `libcrypto.1.1.dylib` and their dependencies (if any)

# ls
libcrypto.1.1.dylib
libssl.1.1.dylib
libpq.5.13.dylib
psql

Use install_name_tool to change dependencies paths to `@loader_path/...`.

More details about @loader_path

install_name_tool -change /Applications/Postgres.app/Contents/Versions/13/lib/libssl.1.1.dylib @loader_path/libssl.1.1.dylib ./libpq.5.13.dylib

install_name_tool -change /Applications/Postgres.app/Contents/Versions/13/lib/libcrypto.1.1.dylib @loader_path/libcrypto.1.1.dylib ./libpq.5.13.dylib

install_name_tool -change /Applications/Postgres.app/Contents/Versions/13/lib/libcrypto.1.1.dylib @loader_path/libcrypto.1.1.dylib ./libssl.1.1.dylib

install_name_tool -change /Applications/Postgres.app/Contents/Versions/13/lib/libpq.5.dylib @loader_path/libpq.5.13.dylib ./psql

Try to run psql

If you run this on Macos Catalina, it will be killed by system

# ./psql
[1]    2490 killed     ./psql

It happens because those binaries was signed but modified them, and signature check started to fail

Let’s remove code signing

codesign --remove-signature *

If binary is not signed we may need to remove quarantine flag

xattr -d com.apple.quarantine *

Try again

# ./psql
psql (13.0, server 10.14)
Type "help" for help.
myname=#

It works 👏

I use this technique in Postbird.app - Free Cross-platform Desktop Client for PostgreSQL

Capture Javascript Async Stack Traces

Pavel Evstigneev — Sat, 01 Jun 2019 17:59:14 +0000

Javascript stacktrace do not include async calls, that sucks.

Example:

var sayHi = () => {
  setTimeout(() => {
    console.log('Hi there');
    thisLineBreaks();
  }, 1000);
};

sayHi();

Output:

$ node try_async.js
Hi there
/my_code/try_async.js:4
    thisLineBreaks();
    ^

ReferenceError: thisLineBreaks is not defined
    at Timeout.setTimeout [as _onTimeout] (/my_code/try_async.js:4:5)
    at listOnTimeout (timers.js:324:15)
    at processTimers (timers.js:268:5)

We can only see error details inside sayHi function and we don't know where it was called.

Solution

To solve this, we can capture stack trace before async part and append it to actual error.

var sayHi = () => {
  var startStack = new Error().stack;
  setTimeout(() => {
    try {
      console.log('Hi there');
      thisLineBreaks();
    } catch (error) {
      error.stack = error.stack + "\n" +
        startStack.substring(startStack.indexOf("\n") + 1);
      throw error;
    }
  }, 1000);
};

sayHi();

And now we getting it!

node try_async.js
Hi there
/my_code/try_async.js:9
      throw error;
      ^

ReferenceError: thisLineBreaks is not defined
    at Timeout._onTimeout (/my_code/try_async.js:6:7)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7)
    at sayHi (/my_code/try_async.js:2:20)
    at Object.<anonymous> (/my_code/try_async.js:14:1)           <<<< HERE
    at Module._compile (internal/modules/cjs/loader.js:774:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:785:10)
    at Module.load (internal/modules/cjs/loader.js:641:32)
    at Function.Module._load (internal/modules/cjs/loader.js:556:12)
    at Function.Module.runMain (internal/modules/cjs/loader.js:837:10)
    at internal/main/run_main_module.js:17:11

It show where my function was called — /my_code/try_async.js:14

Promises

Now days most of libraries uses promises and we can easily apply it for every function that returns promise object:

var trackPromise = async (func) => {
  var startStack = new Error().stack;
  try {
    return await func();
  } catch (error) {
    error.stack = error.stack + "\n" +
      startStack.substring(startStack.indexOf("\n") + 1);
    throw error;
  }
}

// will make complete stack traces
var rows = trackPromise(() => {
  return db.query('select * from something');
});

Or modify async functions:

// patch method in object to keep stack traces
var addStackTracker = (proto, methodName) => {
  var originalMethod = proto[methodName];

  proto[methodName] = function(...args) {
    var startStack = new Error().stack;
    return new Promise((resolve, reject) => {
      originalMethod.apply(this, args).then(res => { resolve(res); })
        .catch(error => {
          error.stack = error.stack + "\n" + startStack.substring(startStack.indexOf("\n") + 1);
          reject(error);
        });
    });
  }
}

addStackTracker(DatabaseClient.prototype, 'query')

var db = new DatabaseClient()
db.query('...') // will have full stack

now my dev life became easier.

Note: if your function call multiple async functions inside, than this approach will keep only stack before it enter function and stack for last async call