DEV Community

loading...
AWS Community Builders

How to copy data between DynamoDb tables

aklm10barca profile image Arun Kumar ・2 min read

Steps

  • Spin up an EC2 instance and execute the below.
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
. ~/.nvm/nvm.sh
nvm install node
npm i copy-dynamodb-table
Enter fullscreen mode Exit fullscreen mode
  • Update EC2 IAM role to grant Dynamodb access to source/target table. If the table is encrypted by Custom KMS, then you need to grant permission to EC2 IAM role as well.
node ./test.js 
Enter fullscreen mode Exit fullscreen mode

—> where test.js contains the following

#### start code ####
var copy = require('copy-dynamodb-table').copy

copy({
    source: {
      tableName: 'source-dynamodb-table', // required
    },
    destination: {
      tableName: 'demo-app-kefta-999-dynamo', // required
    },
    log: true, // default false
    create : false // create destination table if not exist
    schemaOnly : false // if true it will copy schema only -- optional
  },
  function (err, result) {
    if (err) {
      console.log(err)
    }
    console.log(result)
  })
#### end code ####
Enter fullscreen mode Exit fullscreen mode

Adding AWS Config

var copy = require('copy-dynamodb-table').copy
var globalAWSConfig = { // AWS Configuration object http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#constructor-property
accessKeyId: 'AKID',
secretAccessKey: 'SECRET',
region: 'eu-west-1'
}
copy({
config: globalAWSConfig, // config for AWS
source: {
tableName: 'source_table_name', // required
},
destination: {
tableName: 'destination_table_name', // required
},
log: true, // default false
create : true // create destination table if not exist
},
function (err, result) {
if (err) {
console.log(err)
}
console.log(result)
})
Enter fullscreen mode Exit fullscreen mode

AWS Config for each table (cross region)

var copy = require('copy-dynamodb-table').copy
var globalAWSConfig = { // AWS Configuration object http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#constructor-property
accessKeyId: 'AKID',
secretAccessKey: 'SECRET',
region: 'eu-west-1'
}
var sourceAWSConfig = {
accessKeyId: 'AKID',
secretAccessKey: 'SECRET',
region: 'eu-west-1'
}
var destinationAWSConfig = {
accessKeyId: 'AKID',
secretAccessKey: 'SECRET',
region: 'us-west-2' // support cross zone copying
}
copy({
config: globalAWSConfig,
source: {
tableName: 'source_table_name', // required
config: sourceAWSConfig // optional , leave blank to use globalAWSConfig
},
destination: {
tableName: 'destination_table_name', // required
config: destinationAWSConfig // optional , leave blank to use globalAWSConfig
},
log: true,// default false
create : true // create destination table if not exist
},
function (err, result) {
if (err) {
console.log(err)
}
console.log(result)
})
Enter fullscreen mode Exit fullscreen mode

Note:

  • If source.config or destination.config value is undefined , the module will use the globalAWSConfig.
  • If globalAWSConfig value is undefined the module will extact AWS config from environment variables.
  • Increase Write capacity for your dynamodb table temporarily until the copying is finished so you can get the highest copying speed.

Reference

[https://www.npmjs.com/package/copy-dynamodb-table]

This module will allow you to copy data from one table to another using very simple API, Support cross zone copying.

Discussion (1)

Collapse
psiho profile image
Mirko Vukušić • Edited

Needed something similar few years ago :) when I was migrating Cognito and Dynamodb tables. Did it manually then with simple scripts. I put the solution to github.com/psiho/cognito-dynamodb-... if anybody needs it

Forem Open with the Forem app