DEV Community: Williams

How to Connect Your Nest.js App to AWS DocumentDB: A Step-by-Step Guide

Williams — Thu, 28 Nov 2024 22:29:16 +0000

Setting up your NestJS application to connect with Amazon DocumentDB can be challenging. You might face issues with TLS/SSL configuration or security groups.
This guide will walk you through the process smoothly, ensuring a hassle-free setup.

Understanding the Connectivity requirement of DocumentDb

Before creating your DocumentDB database, it's important to know that Amazon DocumentDB is accessible by Amazon EC2 instances and other AWS services within the same VPC. You can connect also from different VPCs within the same AWS Region or across Regions using VPC peering.

If you need to access DocumentDB from outside AWS, you can use SSH tunneling.

1. Creating a New Amazon DocumentDB Cluster

Step 1: Access the AWS Console

Login to your AWS console.
In the search bar at the top of the console, type "DocumentDB" and select Amazon DocumentDB from the search results.

Step 2: Begin Cluster Creation

On the Amazon DocumentDB page, click Create Cluster or Create your first cluster if this is your first time setting up DocumentDB.

Step 3: Configure Cluster Settings: Fill in the required information

DB Cluster Identifier: Provide a unique name for your cluster.
Authentication Method: Enter the necessary credentials (username and password) for your cluster.

Step 4: Finalize and Create the Cluster

After filling in the required details, review your configuration settings
Click Create Cluster to initiate the cluster creation process.

AWS will take a few minutes to provision your new DocumentDB cluster. Once completed, you'll have a fully functional cluster ready for use.

2. Set Up a Security Group to Allow Connectivity to the DocumentDB Cluster Within the VPC

Next, we'll need to configure security groups to allow our Nest.js application to connect to the DocumentDB cluster within the VPC.

Step 1: Create a Security Group

In your AWS Console, Search for Security Groups

Click Create Security Group.
Provide a Name and Description for the security group
Ensure you select the appropriate VPC where your DocumentDB cluster reside

Step 2: Configure Inbound Rules

Under the Inbound rules tab, click Add Rule.
Set the Type to Custom TCP
For the Port Range, enter 27017 (the default port for DocumentDB)
In the Source field, specify Anywhere or 0.0.0.0/0 this would allow us to connect from anywhere within the vpc

Step 3: Configure Outbound Rules

Under the Outbound rules tab, click Add Rule.
Set the Type to Custom TCP
For the Port Range, enter 27017 (the default port for DocumentDB)
In the Source field, specify Anywhere or 0.0.0.0/0

Once you've configured the rules, click Create Security Group. Your new security group is now ready to allow traffic to and from your DocumentDB cluster.

Step 4: Modify DocumentDb Security Group

Go back to the Amazon DocumentDB service in the AWS Console
Select your newly created cluster.
On the VPC Security groups section replace the current security group with the new one you created.
Click Continue to proceed.

Check the Apply Immediately box to enforce the changes without delay.
Click Modify cluster to finalize the update.

The above steps will update your cluster's security settings, ensuring your DocumentDB is fully configured and ready for use.

3. Creating Our Nest.js application and Connecting to DocumentDb

Step 1: Let's create a new NestJS project

nest new document-db-app

I would Open the project with vs code and install the necessary dependencies.

npm install @nestjs/mongoose mongoose @nestjs/config

Step 4: Download the Amazon DocumentDB Certificate Authority (CA) certificate

Navigate to your DocumentDB dashboard and click the Connectivity & Security tab. Find and copy the link to download the Amazon DocumentDB CA certificate. Save this file in the root of your project.

Step 5: Setup Connection to Document db

//src/app.module.ts

import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { MongooseModule } from '@nestjs/mongoose';
import * as path from 'path';

@Module({
  imports: [
    ConfigModule.forRoot(),
    MongooseModule.forRootAsync({
      imports: [ConfigModule],
      useFactory: async (configService: ConfigService) => ({
        uri: configService.get('DOCUMENTDB_URI'),
        tls: true,
        tlsCAFile: path.resolve('global-bundle.pem'),
      }),
      inject: [ConfigService],
    }),
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

In the app.module.ts file, we connect our NestJS application to DocumentDB using MongooseModule. Inside MongooseModule, we call the forRootAsync method to configure the connection asynchronously.

In forRootAsync, we define a useFactory function, which retrieves the database URI from environment variables using Config Module. We also set tls: true option to enable secure communication and specify the tlsCAFile option to point to our downloaded CA certificate global-bundle.pem, ensuring that the connection is authenticated.

By doing this, we set up a secure and reliable connection to DocumentDB, allowing our NestJS application to interact with the database safely.

Step 6: Setting Up the Environment Variables
Create a .env file in the project root and add the DocumentDb connection string

DOCUMENTDB_URI=mongodb://username:password@<your-documentdb-endpoint>:27017/<your-database-name>?tls=true&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false

Note: The above setup assumes you have your app running in the same VPC as your Documentdb cluster. Amazon DocumentDB does not support direct connections from outside the VPC for security reasons.

If you need to connect from your local machine for testing or development purposes you need to setup ssh tunneling through an EC2 instance. For a step-by-step guide on how to set this up, refer to this article on Connecting to DocumentDB via SSH tunneling.

Step-by-Step Guide to Implementing Node.js Audit Trail

Williams — Thu, 04 Jan 2024 16:11:29 +0000

When developing applications, both user experience and security is paramount . One challenge is allowing users with different levels of access to use the system smoothly while also keeping a detailed record of their actions. This becomes particularly tricky when we want to track every move users make for accountability and security reasons. That's where audit trailing comes in – it's like a logbook of events that happen in our application, covering things like data changes, how the application responds, and which users are involved. It's basically our way of keeping tabs on everything that happens in the application. Lets take a look at how we can solve this challenge building our own application

Implementation

Let's begin our project by initializing a new Node.js application and configure typescript.

npm init -y && tsc --init

Next, let's install the necessary dependencies including express, typescript, nodemon, ts-node and @types/express

npm i express && npm i typescript @types/express nodemon ts-node -D

Let's edit our package.json file to add our start script .

  "scripts": {
    "start": "nodemon src/index.ts"
  },

Create a src directory and add an index.ts file to kick off our application.

//src/index.ts
import express, { Request, Response } from 'express';
const app = express();

app.get('/', (req: Request, res: Response)=> {
    try {
        return res.status(200).json({
            msg: 'Success'
        })
    } catch (error) {
        return res.status(500).json({
            msg: 'Something went wrong, Try again !'
        })
    }
});

app.listen(5000, ()=> {
    console.log('App is running')
})

We have successfully laid the foundation for our Express application, let's proceed to create a middleware function for robust audit logging. This middleware will seamlessly track every incoming request and the corresponding response, providing a comprehensive audit trail for accountability and security.

//src/audit-log.middleware.ts

import { NextFunction, Request, Response } from "express";

export const auditLoggerInterceptResponse = async (req: Request, res: Response, next: NextFunction) => {

    //Save the initial res.json method to a variable for later use
    const originalJson = res.json;

    //Overide the res.json method with a custom implementation
    res.json = function (body: any) {

         // Create a payload capturing relevant information about the request and response
            const payload = {
                url: req.originalUrl,
                method: req.method,
                body: req.body,
                params: req.params,
                headers: req.headers,
                statusCode: res.statusCode,
                response: body,
            };


            console.log(payload);

        //Save Payload
        /* 
            Implement database logic
            Note: Make use of Promise chaining for promises . Using async await
            would not match the return type of res.json. This may result in an error
        */

        // Call the original `res.json` method to send the response
        return originalJson.call(this, body);
    };

    // Move to the next middleware or route handler in the Express middleware stack
    next();
};

Let me break down how this function works

1 . Save the Initial res.json Method:

const originalJson = res.json;
The original res.json method is saved to the variable originalJson for later use.

2 . Override res.json with Custom Implementation:

res.json = function (body: any) {...};
The res.json method is overridden with a custom implementation inside the middleware.
This custom implementation captures relevant information about the incoming request (req) and the outgoing response (res).
It creates a payload object containing details such as the request URL, method, body, parameters, headers, response status code, and response body.

3 . Database Logic Reminder:

A comment reminds developers to implement database logic for saving the payload.
It suggests using Promise chaining for promises, as using async/await might not match the return type of res.json.

** Note: For simplicity, the implementation above omits the database logic section. In a real-world scenario, you would have to integrate your preferred database logic within the designated section to ensure a tailored and robust audit trail for your application **

4 . Call the Original res.json Method:

return originalJson.call(this, body);
The original res.json method is called to ensure that the response is sent to the client as intended.

5 . Move to the Next Middleware or Route Handler:

next();
The next() function is called to move to the next middleware or route handler in the Express middleware stack.

In a nutshell this middleware modifies the res.json function so that when called in our routes it would perform our custom implementation. This tweak acts as a potent tool for improving our application's audit trail, incorporating custom actions during the response phase.

Let's proceed to call our middleware in our index.ts as a global middleware

//src/index.ts

import express, { Request, Response } from 'express';
import { auditLoggerInterceptResponse } from './audit-log.middleware';
const app = express();

app.use(auditLoggerInterceptResponse);

app.get('/', (req: Request, res: Response)=> {
    try {
        return res.status(200).json({
            msg: 'Success'
        })
    } catch (error) {
        return res.status(500).json({
            msg: 'Something went wrong, Try again !'
        })
    }
});

app.listen(5000, ()=> {
    console.log('App is running')
})

Concluding our exploration, making a request to the GET route reveals the middleware function's response in our console, showcasing the powerful audit logging capabilities we've integrated. This hands-on experience solidifies the significance of our middleware as a crucial tool for observing user interactions, ultimately bolstering security and accountability within our Node.js application.

Github Repository: Repository Link

Worker Threads Nodejs

Williams — Fri, 07 Jul 2023 10:43:36 +0000

The worker thread module in Node.js allows us to run JavaScript code in parallel, preventing CPU-intensive operations from blocking the main thread of our application. Before delving deeper into this module, it's important to understand a few key concepts. Familiarity with these concepts will help us use and benefit from worker threads effectively.

CPU Core

A CPU core serves as the fundamental unit responsible for executing instructions within a computer system. When exploring computing systems, it is essential to distinguish between single-core and multi-core systems. In a multi-core system, there are multiple CPU cores available, each capable of independent execution. Worker threads in Node.js can be distributed across multiple CPU cores to achieve parallel execution and leverage the full processing power of the system.

In contrast, a single-core system has only one CPU core available for executing instructions. When worker threads are introduced in such a system, they run alongside the main thread but do not achieve true parallelism. Instead, a technique called time-sharing is used. This means that when a worker thread is scheduled to run, it occupies the sole CPU core, causing the main thread and other threads to wait until the core becomes available again. As a result, the main thread gets blocked, leading to decreased application performance.

Threads

Threads play a vital role in dividing the execution of a program into concurrent tasks, enabling both concurrency and parallelism. Essentially, threads serve as the smallest units of execution within a program. However, Node.js, which is primarily single-threaded, introduces the concept of threads through the underlying libuv library. This powerful library handles asynchronous I/O operations and manages the event loop efficiently. It achieves this by utilizing a dynamic thread pool that adapts based on system conditions and configurations.

On the other hand, the V8 engine, responsible for executing JavaScript code in Node.js, employs multiple threads internally for tasks like garbage collection and other optimizations. These threads are managed by the V8 engine itself and are not directly exposed or configurable within Node.js. Overall, the combined use of threads, the libuv library, and the V8 engine enables Node.js to handle concurrent and parallel tasks effectively, enhancing performance and responsiveness.

Worker Thread

We have covered the fundamental concepts . Now, we can proceed to learning how to utilize worker threads in our applications.

To initialize a new Node.js application, run the following command to generate a package.json file with default settings:

npm init -y

By default, Node.js comes bundled with the worker thread module. To create a simple server, let's install Express

npm i express

In our package.json we can go ahead and create a start script

//package.json

"scripts": {
  "start": "node index.js"
}

Your updated package.json file should look like this 👇👇

Now let's create a simple program that simulates blocking the main thread. This program will loop one million times and write its output to a file.

//index.js

const fs = require('node:fs');
const express = require('express');
const app = express();

app.get('/write', async (req, res) => {
     try {
        for(let i = 0; i < 1000000 ; i++){
            fs.appendFileSync( 'test.txt',` ${i} `);

        }
        return res.status(200).json("Done");
     } catch (error) {
        return res.status(500).json("Error Occured ");
     }
});

app.get('/' , (req , res)=> {
    try {
        return res.status(200).json("Health check route");
    } catch (error) {
        return res.status(500).json("Error Occured ");
    }
})

app.listen(3001 , ()=> {
    console.log(" App is running in 3001 ");
});

The code inside the /write route will block the main thread when executed. This means it will consume all available resources, preventing other incoming requests from being processed.

To fix this issue and avoid blocking the main thread , we can utilize the worker thread .

Let's modify our code to use the worker thread . Let's create a separate file called 'worker.js'

//worker.js

const fs = require('fs');
const { parentPort } = require('worker_threads');
const {
    workerData
} = require("worker_threads");

const count = workerData

for(let i = 0; i < count ; i++){
    fs.appendFileSync( 'test.txt',` ${i} `);
}

parentPort.postMessage(`Done`)

In the worker.js file, we import the required modules fs, parentPort, and workerData from the worker_threads module.
The workerData variable allows passing data from the main thread to the worker thread.
Inside the worker thread, we perform the expensive operation, which in this case is appending a sequence of numbers to the test.txt file.
After the operation is completed, we use parentPort.postMessage() to send a message back to the main thread, indicating that the task is done.

We can now proceed to modify our index.js

//index.js

const express = require('express');
const app = express();
const {Worker} = require('worker_threads');

app.get('/write', async (req, res) => {
     try {
      const worker = new Worker('./worker.js' , {workerData : 1000000});
      worker.on("message", msg => {
        return  res.status(200).json(msg);
      });
      worker.on("error", err => console.error(err));

     } catch (error) {
        console.log(error);
        return res.status(500).json("Error Occured ");
     }
});

app.get('/' , (req , res)=> {
    try {
        return res.status(200).json("Health check route");
    } catch (error) {
        return res.status(500).json("Error Occured ");
    }
})

app.listen(3001 , ()=> {
    console.log(" App is running in 3001 ");
});

In the updated index.js file , When a GET request is made to the /write route a new worker thread is created using the Worker class from the worker_threads module
The workerData option is used to pass the value 1000000 to the worker thread.
Two event listeners, message and error, are added to the worker thread. The message listener handles messages sent from the worker thread, while the error listener handles any errors that occur.

With this implementation, the /write route will spawn a worker thread to execute the blocking loop. The main thread remains responsive to handle other incoming requests, improving the overall performance and responsiveness of the application.

How to publish your own npm package - Typescript

Williams — Sat, 04 Mar 2023 23:34:37 +0000

Introduction

NPM (Node Package Manager) is a popular JavaScript package manager. It is one of the largest software registries with over a million packages. It is an excellent way for developers to share reusable code with other developers. This article will explain how to write and publish an NPM package. We will be using TSDX to bootstrap our package. TSDX is a zero-config CLI that helps you develop, test, and publish modern TypeScript packages with ease.

Initialize Our Package

npx tsdx create <Your project name>

select basic and continue

Let's write our package

// src/index.ts

class MyInfo {

  name: string;
  age: number;

  constructor(name: string, age: number) {
    this.name = name;
    this.age = age;
  }

  myName() {
    return `Hi there my name is ${this.name}`
  }

  myAge() {
    return `I am ${this.age} years old`
  }

  yearOfBirth() {
    return 2023 - this.age
  }

}

export default MyInfo

This is a simple package that takes in the name and age when you create a new instance of the class MyInfo, The instance would have the functions in the class (myName, myAge, yearOfBirth).

We can go ahead and publish our package, Let's go ahead and publish locally for testing.

//Publish locally

npm link

We can now make use of the above created package in our projects locally

//Make use of the package locally

npm link <Your project name>

import MyInfo from 'Your Project name'

const myInfo = new MyInfo('Williams' , 10);

console.log(myInfo.myName()); // output: "Hi there my name is Williams"
console.log(myInfo.myAge()); // output: "I am 10 years old"
console.log(myInfo.yearOfBirth()); // output: 2013

Publish to the NPM Registry

Now that we have our package written and tested locally, we can publish it to the NPM registry for others to use.

First, you'll need to create an account on the NPM website if you don't already have one. Once you've done that, you can log in to your account from the command line using the npm login command. This will prompt you for your NPM username and password.

npm login

Next, navigate to the root directory of your project and run the npm publish command. This will package up your code and upload it to the NPM registry.

npm publish

Congratulations! Your package is now published on the NPM registry and is available for others to use. They can install it in their own projects using the npm install command.

In this article, we've learned how to write and publish an NPM package using TSDX. We've covered how to create a new project, write some code, and publish the package to the NPM registry. With these tools and techniques, you can share your own reusable code with others.

Introduction to Node.js Buffer Module

Williams — Mon, 13 Feb 2023 11:16:52 +0000

Introduction?

The Buffer module in Node.js is a way to handle binary data. In order to understand the Buffer module, we first need to have a good understanding of binary data and character sets.

Binary Data

Binary data is data that can only take on two possible states, which are 0 and 1. This is the language that computers understand, and all your images, videos, and text are stored as binary data in the form of 0s and 1s at the core of your computer.

Character Sets

A character set, also known as an encoding system, is a system that lets computers know how to recognize characters such as letters, numbers, punctuation marks, and whitespace. The Unicode character set is a widely used encoding system, where characters have decimal representations that are then converted into binary numbers that computers can understand. You can find all the Unicode characters and their decimal, binary, and hexadecimal representation at https://unicodelookup.com/.

Let's dive right into the Node.js Buffer module, we now have a clear understanding of what binary data are and character sets are . We would covering just a few methods to give us the basic understanding of this module

import { Buffer } from 'buffer'

const buf1 = Buffer.alloc(10);

In example above we created an empty buffer container with the length of 10 (10 byte of data) . We can then write to this buffer container using the write method

buf1.write('Hello' , 'utf-8')

the write method takes in two arguments , the first is a string and the second is the encoding type . the written byte ('Hello') cannot exceed the length (in bytes) of the buffer container buf1.

import { Buffer } from 'buffer'

const buf2 = Buffer.from('string' , 'utf-8');

Even better We can create a buffer filled with the specified string, array, or buffer using the Buffer.from() method. The size of the buffer, in bytes, is determined by the length of the data passed as the first argument

Let Convert the above created buffer back to string

console.log(buf2.toString());

In the example above, the buffer buf2 was created from a string using the Buffer.from() method. The buf2.toString() method was then called to convert the binary data in the buffer back into a string, which was logged to the console.

The encoding type argument is optional, and if not specified, the default encoding of 'utf-8' is used. Other supported encoding types include 'ascii', 'utf16le', 'ucs2', 'base64', 'latin1', and 'binary'.

In addition to the methods covered in this article, there are many more available in the Buffer module. For a full list of all the methods and more in-depth explanations, you can visit the official Node.js documentation at https://nodejs.org/api/buffer.html.

The purpose of this article was to provide a comprehensive overview of the Buffer module in Node.js, and give you a solid understanding of how it works. Whether you are a beginner or an experienced developer, understanding the Buffer module is an important aspect of working with binary data in Node.js. With the knowledge gained from this article, you now have a foundation to build upon and explore the full range of capabilities offered by the Buffer module.