Multiple databases in Big Data projects

Ayoub Oudmane — Sun, 23 Jun 2019 22:59:13 +0000

I worked in multiple NodeJS projects that had to be connected to multiple databases and softwares at the same time. Every time I start a new project, I first need to write the code that configures clients of databases (MongoDB, ElasticSearch, Redis...), make sure it connected successfully and then move on to what I want to do.

The problem

The problem is that every client has it's own way to configure a client/connection, plus it's own way of checking if the connection was successful or not.

mongodb you check with a callback(error, client) (supports Promises too).
elasticsearch you initiate the client, then you going to need to call client.ping() to know if it works
redis you need to listen to connect, error events

An idea

I need to make sure that I'm connected to all services before I start what I want to do. When I write code, I prefer working with Promises than callbacks, so I thought about wrapping the configuration step in a Promise that resolves the client/connection instance when it succeeds, and rejects the error when it fails like the example bellow :

import mongodb from 'mongodb'
import elasticsearch from 'elasticsearch'
import redis from 'redis'

Promise.all([
  // mongodb
  new Promise(
    (resolve, reject) => {
      mongodb.MongoClient.connect(mongodbURL, function (err, client) {
        if (err)
          reject(err)
        else
          resolve(client.db(dbName))
      })
    }
  ),
  // elasticsearch
  new Promise(
    (resolve, reject) => {
      var client = new elasticsearch.Client({
        host: 'localhost:9200'
      })
      client.ping({
        // ping usually has a 3000ms timeout
        requestTimeout: 1000
      }, function (error) {
        if (error)
          reject(error)
        else
          resolve(client)
      })
    }
  ),
  // redis
  new Promise(
    (resolve, reject) => {
      var client = redis.createClient()
      client.on("error", function (error) {
        reject(error)
      })
      client.on("connect", function (error) {
        resolve(client)
      })
    }
  )
]).then(
  ([mongodbClient, elasticsearchClient, redisClient]) => {
    // here I write what I want to do
  }
)

The solution above worked for Me when I wrote scripts that are written in one file. I like to refactor my projects into multiple files/modules when it gets complicated, for example an API with express that has multiple routes, I'de prefer to write them separately, it makes it easy to know where to look while debugging.

Now,

How am I going to access the clients from other files ?

With the express example, we can use a middleware to include the clients in req and access it in each route easily, but this is just an example of a project, what to do when we don't have middlewares as an option ?

To be honest you can figure it out, it depends on your project, what you want to do and how you going to do it, passing the clients as a parameter when you call other functions, pass them to constructors when you initiate objects, you're always going to need to decide where to pass them.

I'm a lazy developer, I want to focus on working on the solution, and I hate making it more complicated with the clients baggage. I wanted something that will be easy to setup and can be used everywhere !

Here is what I have decided to do:

the solution

I defined this interface to be followed while wrapping a database/software client

class DriverInterface {
  // methods

  // configureWithName is to support multiple configurations of the same software
  static configureWithName(name, ...clientOptions) // return Promise<client,error>

  // this just an alias that calls this.configureWithName('default', ...clientOptions)
  static configure(...clientOptions) // return Promise<client,error>

  // get client by name
  static getClient(name) // returns client

  // properties
  static get client() // an alias to this.getClient('default')

  static get clients() // returns all clients Map<string,client>
}

I started with mongodb and published it at npm as @oudy/mongodb which can be used like this

Example

import MongoDB from '@oudy/mongodb'

MongoDB.configure('test', 'mongodb://localhost:27017').then(
 database => {
   const users = database.collection('users').find()
 }
)

Also if your project is refactored into multiple files/modules you can access the client using MongoDB.client

Example

// models/getUsers.js
import MongoDB from '@oudy/mongodb'

export default getUsers(limit = 20, skip = 0) {
  return MongoDB.client
    .collection('users')
    .find()
    .limit(limit)
    .skip(skip)
    .toArray()
}

Multiple databases

You can use @oudy/mongodb to connect easily with multiple databases

Example

import MongoDB from '@oudy/mongodb'

Promise.all([
  MongoDB.configureWithName('us', 'myproject', 'mongodb://us_server:27017'),
  MongoDB.configureWithName('eu', 'myproject', 'mongodb://eu_server:27017')
]).then(
 ([US_region, EU_region]) => {
   // get from US
   US_region.collections('files').find().forEach(
     file => {
       // do our changes and insert to v2
       EU_region.collections('files').insertOne(file)
     }
   )
 }
)

If you want access to the us or eu databases from other files you can use MongoDB.getClient()

Example

// models/files.js
import MongoDB from '@oudy/mongodb'

export default getFiles(region, limit = 20, skip = 0) {
  return MongoDB.getClient(region)
    .collection('files')
    .find()
    .limit(limit)
    .skip(skip)
    .toArray()
}

Now, what's next

I've implemented the same Interface on other packages @oudy/elasticsearch, @oudy/mysql, @oudy/amqp, @oudy/redis. I'm still working on documenting them properly.
We've been working with databases this way for 2 years now on multiple projects specially at CRAWLO (Leading a big data based software that helps e-commerce websites to increase their sales by monitoring internal and external factors).

I published the repository here github.com/OudyWorks/drivers. please check it out and consider contributing if you have suggestions or found bugs.

This is just one of the cool projects I've built (I think it's cool :D), based on this I've made other packages for building restful APIs, GraphQL servers and even web apps. They're already public in here github.com/OudyWorks (not documented yet). I'm planning to document them and write more articles to explain the story behind why I've made them.

I'm sorry for any typos you may find, this is my first time publishing an article about my work and I'm just so excited to share with you what I've been working on.

Please, feel free to leave comments bellow and follow Me if you're are interested in cools projects in NodeJS.

DEV Community: Ayoub Oudmane

Multiple databases in Big Data projects

The problem

An idea

How am I going to access the clients from other files ?

the solution

Multiple databases

Now, what's next