CouchBase Mirror Data Records

Dameng — Wed, 05 Jul 2023 04:52:00 +0000

Hi everyone, recently, I have been worked on the couchbase with a request of data mirror from one couchbase database to another.

Basically, what I have done can be summarised as: we try to export original data and use a NodeJS script to generate a query for all the original data and finally run the query to inject all data from original data (One word: Using query to copy and paste data)

Here is the query generation script:

// This script can only be suitable for the data records which is less than 2000;

const fs = require("fs");

function convert(collectionName, sourceFile, query, resultFile) {
  fs.readFile(sourceFile, "utf8", (err, data) => {
    if (err) {
      console.log("Source file read failed:", err);
      return;
    }

    const json = JSON.parse(data);

    const response = json.reduce((acc, cur, index) => {
      const key = cur[collectionName].id.toString();
      const value = cur[collectionName];
      const valueString = JSON.stringify(value);

      cur = ('("' + key + '", ' + valueString + ")").replace(/\\/g, "");

      comma = index === json.length - 1 ? "" : ", ";

      acc += cur + comma;

      return acc;
    }, "");

    if (response) {
      const final = `${query} ${response}`;
      fs.writeFile(resultFile, final, function (err) {
        if (err) return console.log(err);
        console.log(`Generated data have been saved into ${resultFile}`);
      });
    }
  });
}

Here is the example of how to do it:

Step 1: we need to go to the original data source we want to copy from (The couchbase bucket -> collection -> document).

Just provide an example here, if I want to mirror data for test-bucket (bucket name) -> test-collection (collection name) -> test-documents (document name)

Step 2: we click the Query tab on the left side menu bar

Step 3: we run a query: SELECT * FROM test-bucket.test-collection.test-documents

Step 4: we select EXPORT button on the top right corner and you will see a JSON file export modal window:

Step 5: we name the file name as test-documents

Then Click Save button to export data as JSON file

Step 6: You can open the convert.js script I have attached below and you will see this setup from line 39 to line 45 below:

convert(
  "test-documents", // the collection name
  "./test-documents.json", // the file you have exported from CouchBase (Step 5)
  "INSERT INTO `test-bucket`.`test-collection`.`test-documents`(KEY, VALUE) VALUES ", // the target bucket -> collection -> document 
  "test-documents-query.txt" // the final generated result query for data mirror / injection usage 

);

Step 7: Modify the script provided in above (Step 6) based on your specific requirements.

As an example here, we want to copy the one couchbase database test-documents (document name) data to another couchbase database test-documents (document name), we can run above script for generating the data injection /mirror query.

run

node convert.js

command in your terminal within the current folder path and you will be able to generate the new query text file called as test-documents-query.txt.

Step 8: We go to the destination / target document (eg: test-documents) and go to the Query tab (Same as Step 2) and paste the new generated query inside the Query Editor

Step 9 [Optional]: If you just created a new document for UK couchBase, you need to run this query first to avoid some index errors which will be reported by couchbase

CREATE PRIMARY INDEX ON `default`:`test-bucket`.`test-collection`.`test-documents`

*** If you already have document primary index setup already, you can skip Step 9. ***

Step 10: You will open the final generated data injection text file: test-documents-query.txt and copy all the contents and paste into the Query Editor

Step 11: Click the Execute button and starting injecting the data from one test-documents to another 🚀🚀

Some Additional Tips:

Before mirror the data, better to try to use the script locally (localhost:3000), so you will be able to get familiar with and understand this script before actually use it
Always ensure the bucket name, collection name and document name are entered correctly. Otherwise, you might inject the data in a wrong document

Finally, as a reference,

You might be able to find the work I have been working on:
https://github.com/DamengRandom/n1ql-query-generator

Thank you very much for reading, hopefully it could help a little with your couchbase data mirror process.

DEV Community: Dameng

CouchBase Mirror Data Records