DEV Community: Julius Ngwu

Message Queue in Nodejs with BullMQ and Redis

Julius Ngwu — Sun, 24 Sep 2023 15:10:33 +0000

After digesting this article, you will come to understand that a portion of the latency within your application can be attributed to the manner in which you execute certain tasks. In the context of Node.js, the event loop is tasked with managing the execution of asynchronous operations, all within a single thread. Consequently, it is advisable to steer clear of the main thread when executing CPU-intensive logic, and this is the primary focus of the entire article. I recommend not only reading through it but also actively implementing the project to fully leverage the article’s insights.

Message queues are a vital part of modern serverless and microservices architectures. They facilitate asynchronous communication between services, enhancing an application’s performance, reliability, and scalability. One of their key advantages is the ability to handle time-consuming tasks separately, allowing various components of an application to operate independently for increased flexibility and scalability.

However, managing message queues can be challenging, especially when dealing with a high volume of messages in a distributed environment. This is where BullMQ, a Node.js library built on Redis, comes into play. BullMQ offers a fast and robust queue system that simplifies the management of message queues, making it an effective solution for addressing these challenges.

In the context of a practical scenario, consider a situation where a lecturer uploads result sheets to a school portal, and the application needs to process and email individual students. BullMQ enables seamless handling of this process without requiring the lecturer’s presence. By validating the result sheet format and acknowledging the upload, the BullMQ worker, if appropriately configured, can execute the tasks in a separate thread, ensuring the main thread remains unburdened.

In upcoming discussions, we will explore how to leverage BullMQ, Redis, and Node.js to implement such solutions efficiently.

Introduction to Node.js

Node.js is a JavaScript runtime environment that allows server-side scripting, offering non-blocking, event-driven capabilities for building scalable and efficient network applications.

Introduction to BullMQ and Redis

BullMQ docs defined BullMQ as a Node.js library that implements a fast and robust queue system built on top of Redis that helps in resolving many modern-age micro-services architectures.

Redis is an open-source, in-memory data store and caching system known for its speed and versatility, often used to store key-value data and support various data-intensive applications.

About the Project

In this tutorial, we will construct a basic queue using Bullmq, Redis, and Node.js. We’ll establish an endpoint that can receive both a username and a CSV file. While the application promptly responds to the user, it simultaneously initiates a background job responsible for performing the following tasks:

Converting the CSV data into JSON format.
Analyzing and organizing the content based on dates.
Creating a unique key associated with the username and assigning the grouped data to the respective user.
Generating a .json file containing the modified data.

Prerequisites

Make sure you have Redis running on your machine
Have Node.js installed on your machine
Download Employment-indicators-Weekly

Setting up a Node Express.js Server

Run npm init --y
Add the following script to your package.json

    {
      "name": "bullmq",
      "version": "1.0.0",
      "description": "",
      "main": "index.js",
      "scripts": {
        "start": "node './src/index.js'",
        "test": "echo \"Error: no test specified\" && exit 1"
      },
      "keywords": [],
      "author": "",
      "license": "ISC",
      "dependencies": {
        "bullmq": "^4.11.1",
        "csv-parse": "^5.5.0",
        "express": "^4.18.2"
      }
    }

Now, run npm i

Please bear in mind that the directory and file structure I’ve employed here is purely a matter of preference. You have the flexibility to opt for an alternative structure that suits your needs.

Inside your main directory, create a srcdirectory.
In your src directory create an index.js file and add the code below:

    const express = require('express');
    const path = require('path');

    const addJobToQueue = require('./bullmq/queue');

    const app = express();
    const PORT = 3001;

    app.use(express.json());

    const csvFilePath = path.join(__dirname, './employment_indicators.csv');

    app.post('/', async (req, res) => {
      const { userName } = req.body;
      const data = { jobName: 'csvJob', userName, csvFilePath };
      const job = await addJobToQueue(data);

      return res.status(201).json({ jobId: job.id });
    });

    app.listen(PORT, async function onListen() {
      console.log(`Server is up and running on port ${PORT}`);
    });

This code initializes your fundamental Express server. We’ll retrieve the userName from the request body and hardcode the CSV data for the sake of simplicity and clarity in the application.

In BullMQ docs, we were told that BullMQ is based on 4 classes that together can be used to resolve many different problems. These classes are Queue, Worker, QueueEvents, and FlowProducer.

You may have observed that we imported the addJobToQueue function at this point, which brings us to our initial encounter with the Bullmq class, Queue.

Queue

Now, move the CSV file you downloaded into this src then inside the src create the bullmq/config.js directory and add the code below:

    const CONNECTOR = {
      host: 'localhost',
      port: 6379,
    };

    const DEFAULT_REMOVE_CONFIG = {
      removeOnComplete: {
        age: 3600,
      },
      removeOnFail: {
        age: 24 * 3600,
      },
    };
    module.exports = { CONNECTOR, DEFAULT_REMOVE_CONFIG };

We also create a function that can be used to add jobs to the queue from an endpoint handler. Now create bullmq/queue.js file and add the code below:

    const { Queue } = require('bullmq');
    const configModule = require('./config');
    const setUpWorker = require('./worker');

    const myQueue = new Queue('JOBS', {
      connection: configModule.CONNECTOR,
    });
    myQueue.setMaxListeners(myQueue.getMaxListeners() + 100);

    setUpWorker();

    const addJobToQueue = (data) => {
      return myQueue.add(data.jobName, data, configModule.DEFAULT_REMOVE_CONFIG);
    };

    module.exports = addJobToQueue;

You notice that we imported a function called setUpWorker right? That leads us to another two classes in Bullmq called Worker, and QueueEvents.

Worker and QueueEvents

Now create a bullmq/worker.js file and add the code below:

    const { Worker } = require('bullmq');
    const path = require('path');

    const configModule = require('./config');
    let worker;

    const processorPath = path.join(__dirname, 'processor.js');

    const setUpWorker = () => {
      worker = new Worker('JOBS', processorPath, {
        connection: configModule.CONNECTOR,
        autorun: true,
      });

      worker.on('active', (job) => {
        console.debug(`Processing job with id ${job.id}`);
      });

      worker.on('completed', (job, returnValue) => {
        console.debug(`Completed job with id ${job.id}`, returnValue);
      });

      worker.on('error', (failedReason) => {
        console.error(`Job encountered an error`, failedReason);
      });
    };

    module.exports = setUpWorker;

You may have observed how the worker and the queue share the same name and connection information, which is a requisite for the worker to commence job consumption from the queue. With the event listeners incorporated into our queue.js, we can now receive updates on the job statuses.

Additionally, you might have noticed this line of code:

    const processorPath = path.join(__dirname, 'processor.js');

This line refers to our processor, where we handle resource-intensive tasks like database operations or interactions with third-party APIs. The objective is to ensure that our workers execute these jobs in a separate thread.

By providing the file path as the second parameter to the worker, BullMQ will execute the function exported from the specified file within a dedicated thread. This approach guarantees that the main thread remains unburdened by the CPU-intensive workload handled by the processor.

Now create bullmq/processor.js file and add this code below:

    const fs = require('fs');
    const { parse } = require('csv-parse');
    const { promisify } = require('util');

    const sleep = promisify(setTimeout);

    const fileName = 'employment_indicators.json';

    if (fs.existsSync(fileName)) {
      fs.unlinkSync(fileName);
    }

    const jobProcessor = async (job) => {
      await job.log(`Started processing job with id ${job.id}`);
      // TODO: do your CPU intense logic here
      await extractCSVData(job?.data);

      await job.updateProgress(100);
      return 'DONE';
    };

    module.exports = jobProcessor;

    const extractCSVData = async (jobData) => {
      try {
        let result = [];

        await sleep(10000);

        fs.createReadStream(jobData.csvFilePath)
          .pipe(
            parse({
              columns: true,
            })
          )
          .on('data', async (data) => {
            result.push(data);
          })
          .on('error', async (error) => {
            console.log(error, '????');
          })
          .on('end', () => {
            result = result.reduce((acc, item) => {
              const key = item.Week_end;

              acc[key] = acc[key] || [];
              acc[key].push(item);
              return acc;
            }, {});
            const writeData = { [jobData.userName]: result };
            fs.writeFileSync(fileName, JSON.stringify(writeData));
          });
      } catch (error) {
        console.log(error);
      }
    };

Notice how we called our extractCSVData function in the jobProcessor. At this point, your file structure will look like this:

You can now start your server by running npm start

Then go ahead and make a postman call to the endpoint. Your postman should look like this:

FlowProducer

The FlowProducer class allows to add jobs with dependencies between them in such a way that it is possible to build complex flows. Note: A flow is a tree-like structure of jobs that depend on each other. Whenever the children of a given parent are completed, the parent will be processed, being able to access the children’s result data. All Jobs can be in different queues, either children or parents.

You can access this project via the following link on GitHub

Conclusion

Just as Balázs Tápai right said, In an ExpressJS application, executing computationally intensive tasks on the main thread may lead to unresponsive or sluggish endpoints. Transitioning these tasks to a separate thread can effectively mitigate performance bottlenecks, and employing BullMQ can prove to be a valuable asset in this regard.

Resources

Balázs Tápai
Bullmq.io

Automated Web Scraping with Node.js and Puppeteer

Julius Ngwu — Sun, 24 Sep 2023 14:56:31 +0000

While working on a certain web application, I encountered a situation where I needed to synchronize my app’s data with constantly changing data from another website. During my quest to find a solution to this challenge, I came across several fascinating approaches that could effectively address the issue. However, I’d like to delve into the one that captured my interest the most. If you’ve been following my article series, you’ll notice my preference for sharing knowledge through practical demonstrations, guiding you through each step of the process. Before we embark on this journey, please ensure that you have Node.js installed on your computer. You can download node from here: download

Puppeteer is a Node.js library that provides a high-level API to control Chrome/Chromium over the DevTools Protocol. It’s capable of:

Generate screenshots and PDFs of web pages.
Crawl and capture pre-rendered content for Single-Page Applications (SPA).
Automate actions like form submissions, UI testing, and keyboard input.
Develop an automated testing setup utilizing modern JavaScript and browser capabilities.
Record timeline traces to diagnose performance problems.
Conduct testing for Chrome Extensions.

Today, our objective is to put into action the following:

Capture a screenshot of a given web URL
Generate a PDF document from any web application.
Automate the extraction of product names, prices, and product image URLs from a renowned e-commerce platform known as Orla Africa, and save this information in a .json file.

Setup

Create a directory and name it web_scaping, then open it on your IDE(vs code preferably)
Run the code below in your terminal to initialize npm in your project npm init --y
Also, run the code below to install Puppeteer: npm i puppeteer
On the root directory, create a file called scraper.js
Inside the scraper.js file add the code below.

    const puppeteer = require('puppeteer');

    async function runScraper() {
      const browser = await puppeteer.launch({ headless: 'old' });
      try {
        const page = await browser.newPage();

        await page.goto('https://orla.africa/');

        //add the magic codes here
      } catch (e) {
        console.error(e);
      } finally {
        await browser.close();
      }
    }
    runScraper();

In the code snippet provided, we initiated the browser, created a new empty page, and subsequently directed the page to a specific URL. We then closed the browser before proceeding to invoke the functions necessary to accomplish some of the tasks outlined earlier. At this juncture, we are prepared to utilize these functions to fulfill our objectives.

1. Capture a screenshot of a given web URL

We are going to capture a full-page screenshot of Orla Africa. To accomplish this, make the following adjustments to the code in the runScraper function, as shown below:

    const puppeteer = require('puppeteer');


    async function runScraper() {
    const browser = await puppeteer.launch({ headless: 'old' });
    try {
    const page = await browser.newPage();


    await page.goto('https://orla.africa/');


    await new Promise((r) => setTimeout(r, 2000));


    await page.screenshot({ path: 'orla.png', fullPage: true });


    } catch (e) {
    console.error(e);
    } finally {
    await browser.close();
    }
    }
    runScraper();

Notice we have added a delay with a promise to allow the page to load properly before taking a screenshot.

Then on your terminal run: node ./scraper.js

You will notice that a new orla.png file has been added to your root directory.

2. Generate a PDF document from any web application.

Just as we created a .png image of Orla Africa, we can similarly generate a PDF of the same website. To do this, make the following modifications to your code as outlined below:

    const puppeteer = require('puppeteer');


    async function runScraper() {
    const browser = await puppeteer.launch({ headless: 'old' });
    try {
    const page = await browser.newPage();


    await page.goto('https://orla.africa/');


    await new Promise((r) => setTimeout(r, 2000));


    await page.pdf({ path: 'orla.pdf', format: 'A4' });
    } catch (e) {
    console.error(e);
    } finally {
    await browser.close();
    }
    }
    runScraper();

Run node ./scraper.js and notice that the orla.pdffile has been added to your root directory. The PDF version might be a bit different from the PNG version.

3. Automate the extraction of product names, prices, and product image URLs from a renowned e-commerce platform known as Orla Africa, and save this information in a .json file.

Now, we’ve arrived at the exciting phase where your understanding of the Document Object Model (DOM) will be immensely valuable. We’ll closely examine the elements of Orla Africa by utilizing the Chrome inspect tool. This exploration is aimed at comprehending the organization of HTML and CSS classes, enabling us to devise a strategy for retrieving the desired data. Presented below is the layout of the trending products on Orla Africa:

In this context, we can observe that the collection of trending products is encapsulated within a CSS class named _overflowCard_1cymp_30, while each individual product is enclosed within a CSS class named _productCard_1cymp_12. Furthermore, we note that the product's name can be located within an h4 tag, the product image URL resides in an img tag, and the price is contained within a strong tag. Now, let's proceed with the automation of this procedure.

Step 1:

Modify scraper.js file as seen below:

    const puppeteer = require('puppeteer');
    const fs = require('fs');


    const fileName = 'orla_data.json';


    if (fs.existsSync(fileName)) {
    fs.unlinkSync(fileName);
    }


    async function runScraper() {
    const browser = await puppeteer.launch({ headless: 'old' });
    try {
    const page = await browser.newPage();


    await page.goto('https://orla.africa/');
    console.log('Web scraping started...🎬');


    await new Promise((r) => setTimeout(r, 20000));


    const title = await page.evaluate(() => document.title);
    const gadgets = await page.$$eval(
    '._overflowCard_1cymp_30 ._productCard_1cymp_12',
    (gadgets) =>
    gadgets.map((item) => ({
    title: item.querySelector('h4').innerText,
    imageUrl: item.querySelector('img').src,
    price: item.querySelector('strong').innerText,
    }))
    );


    const orla_data_json = { title: title, products: gadgets };


    fs.writeFileSync(fileName, JSON.stringify(orla_data_json));


    console.log('Web scraping completed successfully ✅✅✅');
    } catch (e) {
    console.error(e);
    } finally {
    await browser.close();
    }
    }
    module.exports = runScraper;

Step 2:

On your root directory, create a server.js file and populate it with the code below:

    const express = require('express');


    const runScraper = require('./scraper');


    const app = express();
    const PORT = 8000;


    app.listen(PORT, async function onListen() {
    runScraper();
    console.log(`Server is up and running on port ${PORT}`);
    });

Step 3:

Update your package.jsonfile as seen below:

    {
    "name": "web-scraping",
    "version": "1.0.0",
    "description": "",
    "main": "scrapper.js",
    "scripts": {
    "start": "node './server.js'",
    "test": "echo \"Error: no test specified\" && exit 1"
    },
    "keywords": [],
    "author": "",
    "license": "ISC",
    "dependencies": {
    "express": "^4.18.2",
    "puppeteer": "^21.1.1"
    }
    }

Step 4:

On your terminal, run npm start
You will observe that a new file called orla_data.json has been added to your directory. It contains the Orla Africa web title and a long list of trending product details.

In this context, the JSON file is acting as our data repository. It’s up to you to determine how you’d like to utilize this data. You have the flexibility to store it in your database, share it via email, compare it with data from other platforms, update pricing records in your application, and much more. What’s particularly intriguing is that you can automate this process to run at specified intervals. To set it to run every 24 hours, modify the way you invoke the runScraper function as illustrated below:

setInterval(runScraper, 1000 * 60 * 60 * 24);

You can access this project via the following link on GitHub

Congratulations on the successful implementation of an automated web scraper. I appreciate your time and dedication in reading and collaborating on this project. Please consider following me for more captivating articles, and be sure to show your support by leaving a like and leaving a comment.

References

Puppeteer
Traversy Media

My Reflections on Andela’s EPIC Values.

Julius Ngwu — Sun, 24 Sep 2023 14:43:46 +0000

My Reflections on Andela’s EPIC Values.

EPIC! an acronym for Excellence, Passion, Integrity, and Collaboration. Is obvious that the secret behind Andelans success is their ability to uphold these four values. Let’s go through it bit by bit.

The ‘E’ in EPIC stands for excellence

I have always believed that one has attended excellence when he stands out in a competition. This has left me with the mindset that even if the winner didn’t perform so great, as long as he/she emerged highest, that makes the person an excellent individual.
But from Andela point of view, excellence means when what is to be done is executed to it’s very best. Even if you are the only one doing it, it is expected that the best outcome is delivered. And the quality should not be location relative but should retain the same worth where ever is presented.

The ‘P’ in EPIC stands for Passion

Most times people only have passion for what they can do with ease. They believe that they can’t be successful with anything that gives them trouble to achieve. But here in Andela, is completely different. Andela believes that everyone can make a difference when given the opportunity. Every Andelan is always eager to learn or teach. The long for new knowledge and never shy away from challenges. Here, challenge already got a new name “Blocker” and everyone spend time unblocking and looking for more blockers.
I have come to understand that true satisfaction comes when you decide to add more value to what you already know and discover more of what you can learn.

The ‘I’ in EPIC stands for Integrity

Integrity here at Andela is the most upheld value amongst the four values of the EPIC acronym. Andela starts from the first stage of application to advise applicants on the importance of being yourself. They even have an acronym “YOYO” meaning you own your own. Andela believes that you should learn from people not imitate them. Andela insists that you live your own life not someone else life. They frown at plagiarism so much that they can even lay-off their best staff if they should be a victim of plagiarism.

The ‘C’ in EPIC stands for Collaboration

I use to think that a programmer should always be indoor writing code and debugging. I also believed that when you encounter a blocker, You should remain on your own until you are able to figure it out. I also believed that what makes a programmer great is his ability to write codes only he can understand.
But Andela thinks different. Andela believes that no man is an Island of knowledge. Every person’s opinion should be put into considerations. That’s why in Andela eventually everything is grouped into teams. Andela has an uncountable number of resources online and many learning platform eg: Andela learning community (ALC). They believe that nobody is too big to need help.
And that we can attain excellence easier when we work as a team.

Connecting to Multiple Databases with Nodejs and Sequelize

Julius Ngwu — Wed, 02 Aug 2023 17:41:15 +0000

I know why you are here. You have reached a point where you wish to interact with multiple databases using your Sequelize library. However, you recently noticed that the Sequelize documentation mentions that

If you're connecting to the database from a single process, you should create only one Sequelize instance. Sequelize will set up a connection pool on initialization.

Consequently, it implies that we must generate a new instance for each database we intend to establish a connection. Your goal is to circumvent the need for creating numerous instances, which led you to seek assistance. Additionally, you are curious about the scarcity of engineers who have written about this crucial subject. I, too, share your wonderment.

Great news awaits you! You've come to the perfect location. Below, I present a comprehensive, step-by-step guide on how to connect multiple databases without the need to create multiple instances.

I wanted to start with how Sequelize is installed but a former colleague has already done a great job on Getting Started with Sequelize and Postgres.. If you haven't yet created a Node.js and Sequelize project, I encourage you to click the link for detailed instructions. However, if you're already working on one, feel free to disregard the link.

Let us assume you are already working with a Postgres database called Customer and you want to add these 4 more Postgres(any database that is supported by Sequelize) databases to your Nodejs application:

Wallet
Auth
Merchant
Role

Step 1:

Create a .env file and populate it as seen below:



PORT=
API_VERSION=

DB_USERNAME=
DB_PASSWORD=
DB_NAME=
DB_HOST=
DB_PORT=

DB_CUSTOMER_URL=
DB_CUSTOMER_USERNAME=
DB_CUSTOMER_PASSWORD=
DB_CUSTOMER_NAME=
DB_CUSTOMER_HOST=

DB_WALLET_URL=
DB_WALLET_USERNAME=
DB_WALLET_PASSWORD=
DB_WALLET_NAME=
DB_WALLET_HOST=

DB_ROLE_URL=
DB_ROLE_USERNAME=
DB_ROLE_PASSWORD=
DB_ROLE_NAME=
DB_ROLE_HOST=

DB_MERCHANT_URL=
DB_MERCHANT_USERNAME=
DB_MERCHANT_PASSWORD=
DB_MERCHANT_NAME=
DB_MERCHANT_HOST=

DB_AUTH_URL=
DB_AUTH_USERNAME=
DB_AUTH_PASSWORD=
DB_AUTH_NAME=
DB_AUTH_HOST=

Step 2:

In the config directory, modify your config.ts (config.js if you are not using typescript) to this:



import config from "./index";

const { secrets, dbVariables } = config();

module.exports = {
  development: {
    username: secrets.username,
    password: secrets.password,
    database: secrets.name,
    host: secrets.host,
    port: secrets.port,
    dialect: "postgres",
    dialectOptions: {
      bigNumberStrings: true,
    },
    databases: {
      Customer: {
        username: dbVariables.dbCustomerUsername,
        password: dbVariables.dbCustomerPassword,
        database: dbVariables.dbCustomerName,
        host: dbVariables.dbCustomerHost,
        port: 5432,
        dialect: "postgres",
        dialectOptions: {
          bigNumberStrings: true,
        },
      },
      Auth: {
        username: dbVariables.dbAuthUsername,
        password: dbVariables.dbAuthPassword,
        database: dbVariables.dbAuthName,
        host: dbVariables.dbAuthHost,
        port: 5432,
        dialect: "postgres",
        dialectOptions: {
          bigNumberStrings: true,
        },
      },
      Wallet: {
        username: dbVariables.dbWalletUsername,
        password: dbVariables.dbWalletPassword,
        database: dbVariables.dbWalletName,
        host: dbVariables.dbWalletHost,
        port: 5432,
        dialect: "postgres",
        dialectOptions: {
          bigNumberStrings: true,
        },
      },

      Merchant: {
        username: dbVariables.dbMerchantUsername,
        password: dbVariables.dbMerchantPassword,
        database: dbVariables.dbMerchantName,
        host: dbVariables.dbMerchantHost,
        port: 5432,
        dialect: "postgres",
        dialectOptions: {
          bigNumberStrings: true,
        },
      },
      Role: {
        username: dbVariables.dbRoleUsername,
        password: dbVariables.dbRolePassword,
        database: dbVariables.dbRoleName,
        host: dbVariables.dbRoleHost,
        port: 5432,
        dialect: "postgres",
        dialectOptions: {
          bigNumberStrings: true,
        },
      },
    },
  },
};

Here we have succeeded in declaring 4 more databases with their credentials. Now you must have noticed that I imported an index file at the beginning of this file. It’s not compulsory. I always like to have a separate file for my secret keys. Just hold on, we will create that index file soon. Now speaking of keeping the code clean, let's create a helper file that will read and declare all our db environment variables by looping through them.

Step 3:

In your helpers directory, create a dbConfig.ts file and enter the following lines of codes:



export const fetchAllDbConfig = () => {
  const DBs = {
    CUSTOMER: "Customer",
    WALLET: "Wallet",
    ROLE: "Role",
    MERCHANT: "Merchant",
    AUTH: "Auth",
  };

  const dbVariables = {};
  for (let [key, value] of Object.entries(DBs)) {
    dbVariables[`db${value}URL`] = process.env[`DB_${key}_URL`] || "";
    dbVariables[`db${value}Name`] = process.env[`DB_${key}_NAME`] || "";
    dbVariables[`db${value}Host`] = process.env[`DB_${key}_HOST`] || "";
    dbVariables[`db${value}Password`] = process.env[`DB_${key}_PASSWORD`] || "";
    dbVariables[`db${value}Username`] = process.env[`DB_${key}_USERNAME`] || "";
  }
  return dbVariables;
};

Step 4:

Now let us take a look at this index file we imported earlier.
Back to creating the index file we imported in the config.ts file. In that same config directory create an index.ts file and add the following lines to it.



import dotenv from "dotenv";
import path from "path";
import { fetchAllDbConfig } from "../helpers/dbConfig";

// describes a secrets object
type Secrets = Readonly<{
  env: string;
  version: string;
  port: string;
  secrets: {
    name: string;
    host: string;
    port: number;
    username: string;
    password: string;
  };
  dbVariables: {
    dbCustomerURL: string;
    dbCustomerName: string;
    dbCustomerHost: string;
    dbCustomerPassword: string;
    dbCustomerUsername: string;

    dbWalletURL: string;
    dbWalletName: string;
    dbWalletHost: string;
    dbWalletPassword: string;
    dbWalletUsername: string;

    dbRoleURL: string;
    dbRoleName: string;
    dbRoleHost: string;
    dbRolePassword: string;
    dbRoleUsername: string;

    dbAuthURL: string;
    dbAuthName: string;
    dbAuthHost: string;
    dbAuthPassword: string;
    dbAuthUsername: string;

    dbMerchantURL: string;
    dbMerchantName: string;
    dbMerchantHost: string;
    dbMerchantPassword: string;
    dbMerchantUsername: string;
  };
}>;

const env = process.env.NODE_ENV || "development";
let envfile: string;

switch (env) {
  case "production":
    envfile = ".env";
    break;
  case "test":
    envfile = ".env.test";
    break;
  case "development":
  default:
    envfile = ".env.local";
    break;
}

const envpath: string = path.join(__dirname, "../..", envfile);
let cache: Secrets | any;

export default function config() {
  if (!cache) {
    dotenv.config({ path: envpath });
    cache = Object.freeze({
      env,
      version: process.env.API_VERSION || "v1",
      port: process.env.PORT || "3300",
      secrets: {
        name: process.env.DB_NAME || "",
        host: process.env.DB_HOST || "",
        port: +(process.env.DB_PORT || "") || 25,
        username: process.env.DB_USERNAME || "",
        password: process.env.DB_PASSWORD || "",
      },
      dbVariables: fetchAllDbConfig(),
    });
  }
  return cache;
}

Here we have declared our variables in a more clean format. Also, notice how we imported fetchAllDbConfig from our helpers/dbConfig we created earlier.

Step 5:

Carefully edit the models/index.js file as follows:



"use strict";

const fs = require("fs");
const path = require("path");
const Sequelize = require("sequelize");
const basename = path.basename(__filename);
const env = process.env.NODE_ENV || "development";
const config = require(path.resolve("build/config", "config.js"))[env];

const db = {};
const databases = Object.keys(config.databases);

for (let i = 0; i < databases.length; ++i) {
  let database = databases[i];
  let dbPath = config.databases[database];
  if (config.use_env_variable) {
    db[database] = new Sequelize(process.env[config.use_env_variable], config);
  } else {
    db[database] = new Sequelize(
      dbPath.database,
      dbPath.username,
      dbPath.password,
      dbPath
    );
  }
}

for (let i = 0; i < databases.length; ++i) {
  let database = databases[i].toLowerCase();
  fs.readdirSync(`${__dirname}/${database}`)
    .filter((file) => {
      return (
        file.indexOf(".") !== 0 && file !== basename && file.slice(-3) === ".js"
      );
    })
    .forEach((file) => {
      const model = require(path.join(`${__dirname}/${database}`, file))(
        db[databases[i]],
        Sequelize.DataTypes
      );

      db[model.name] = model;
    });
}

Object.keys(db).forEach((modelName) => {
  if (db[modelName].associate) {
    db[modelName].associate(db);
  }
});

module.exports = db;

The "db" object currently holds connection information for all databases specified in our config.js file. However, it lacks details about the models in the databases, which results in incorrect SQL queries. To rectify this, we must read the model information from our model files and incorporate it into the "db" object.

Please be attentive as we proceed. To achieve this, we must modify the folder structure as depicted below:

Step 6:

Restructure your folder tree to look as seen below.

Step 7:

Create new Sequelize options files similar to our .sequelizerc file for every database. So let us create these two files in the directory of our package.json file:

.sequelize-customer
.sequelize-auth
.sequelize-wallet
.sequelize-role
.sequelize-merchant

In each file, we can define in which directory our config, models, migrations, and seeders are stored. Do this for all 5 files and remember to change the names. Below is an example of how the .sequelize-customer file will look.



const path = require("path");

module.exports = {

  config: path.resolve("build/config", "config.js"),

  "models-path": path.resolve("src/db/models"),

  "seeders-path": path.resolve("src/db/models/customer/seeders"),

  "migrations-path": path.resolve("src/db/models/customer/migrations"),

};

Step 8:

In your package.json, add these migration scripts as follows:



{

    "customer:migrate": "sequelize --options-path ./.sequelize-customer --env development db:migrate",

    "customer:seed": "sequelize --options-path ./.sequelize-customer --env development db:seed:all",

</span><span class="nl">"wallet:migrate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sequelize --options-path ./.sequelize-wallet --env development db:migrate"</span><span class="p">,</span><span class="w">
</span><span class="nl">"wallet:seed"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sequelize --options-path ./.sequelize-wallet --env development db:seed:all"</span><span class="p">,</span><span class="w">

</span><span class="nl">"merchant:migrate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sequelize --options-path ./.sequelize-merchant --env development db:migrate"</span><span class="p">,</span><span class="w">
</span><span class="nl">"merchant:seed"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sequelize --options-path ./.sequelize-merchant --env development db:seed:all"</span><span class="p">,</span><span class="w">

</span><span class="nl">"role:migrate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sequelize --options-path ./.sequelize-role --env development db:migrate"</span><span class="p">,</span><span class="w">
</span><span class="nl">"role:seed"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sequelize --options-path ./.sequelize-role --env development db:seed:all"</span><span class="p">,</span><span class="w">

</span><span class="nl">"auth:migrate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sequelize --options-path ./.sequelize-auth --env development db:migrate"</span><span class="p">,</span><span class="w">
</span><span class="nl">"auth:seed"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sequelize --options-path ./.sequelize-auth --env development db:seed:all"</span><span class="p">,</span><span class="w">



}

Download from Github Repo

You can clone this template from github repo

Step 9:

Congratulations. You did it. You can now access all five different databases from one instance.

Please don’t forget to leave a like and comment.

DEV Community: Julius Ngwu

Message Queue in Nodejs with BullMQ and Redis

Introduction to Node.js

Introduction to BullMQ and Redis

About the Project

Prerequisites

Setting up a Node Express.js Server

Queue

Worker and QueueEvents

FlowProducer

Conclusion

Resources

Automated Web Scraping with Node.js and Puppeteer

Setup

1. Capture a screenshot of a given web URL

2. Generate a PDF document from any web application.

3. Automate the extraction of product names, prices, and product image URLs from a renowned e-commerce platform known as Orla Africa, and save this information in a .json file.

Step 1:

Step 2:

Step 3:

Step 4:

References

My Reflections on Andela’s EPIC Values.

My Reflections on Andela’s EPIC Values.

Connecting to Multiple Databases with Nodejs and Sequelize

Step 1:

Step 2:

Step 3:

Step 4:

Step 5:

Step 6:

Step 7:

Step 8:

Download from Github Repo

Step 9:

References: