DEV Community: Chiamaka Ojiyi

Simplify Your Microservice Debugging Workflow with an Automated Startup Script

Chiamaka Ojiyi — Mon, 19 Jun 2023 16:32:19 +0000

Microservice architecture offers development teams scalability and flexibility, but debugging microservices can be a daunting task, especially when dealing with multiple interconnected services. Manually starting up each service and ensuring proper communication can consume valuable time and effort.

In this article, I will show you how to streamline your microservice debugging process by automating the startup with a script.

Typically, when testing the flow of a microservices project, manually starting each service in separate terminals and monitoring the logs becomes inefficient for debugging purposes.

However, there's a better way. You can create a script to automate the startup process.

Let's consider a project consisting of the following microservices:

User microservice
Billing microservice
Consumer microservice
Reports microservice

If you need to debug an issue in the Reports microservice, the manual approach would involve starting up each service that communicates with it and switching between multiple terminals to follow the logs. This method is tedious and time-consuming.

To automate this process, we'll create a shell script that starts all the microservices with a single command, consolidating the logs into a single terminal for efficient debugging.

Assuming all the microservices are in the same directory, follow these steps:

In your terminal, navigate to the directory containing all the microservice folders and create a shell script with the following command:

touch startup.sh

Open the startup.sh script in your preferred editor and add the following code:

#!/bin/bash

# Start Microservice A
cd /path/to/microservice-a
echo "Starting Microservice A..."
command_to_start_microservice_a >> /dev/tty

# Start Microservice B
cd ../path/to/microservice-b
echo "Starting Microservice B..."
command_to_start_microservice_b >> /dev/tty

# Start Microservice C
cd ../path/to/microservice-c
echo "Starting Microservice C..."
command_to_start_microservice_c >> /dev/tty

# Add more services as needed

cd ../

# Wait for user input to shut down services

echo "Press any key to stop all microservices..."

read -n 1 -s

killall node

In the above script, each startup command is preceded by an echo statement indicating the microservice being started. The >> /dev/tty redirects the output of each startup command to the terminal, allowing you to view the logs and verify that the services started successfully.

Make sure to update the script by replacing /path/to/microservice-x with the actual paths to your microservice repositories, and command_to_start_microservice-x with the respective startup commands for each microservice.

Save the script and make it executable by running the following command in your terminal:

chmod +x startup.sh

By executing this script, you can start all your microservices simultaneously. When you're ready to stop them, simply press any key in the terminal where the script is running. The script will then send termination signals, effectively shutting down the microservices.

Automating the startup process of your microservices simplifies your debugging workflow, eliminates the need for manual service startup, and allows you to focus more on actual debugging tasks.

In conclusion, by leveraging an automated startup script, you can streamline your microservice debugging process, reduce time and effort, and enhance overall productivity.

Implementing SMS-enabled Two-Factor Authentication using NestJS, Twilio and Prisma

Chiamaka Ojiyi — Mon, 08 May 2023 13:45:40 +0000

Introduction

When building robust and secure applications, implementing an additional layer of security is strongly recommended. The username and password are a basic first step in the security architecture of digital applications. This first layer of security is prone to attacks and can lead to account breaches when it is the only barrier sitting in front of restricted resources.

Two-factor authentication acts as a second barrier to accessing protected resources. It requires that a user provides additional information after the username-password identity declaration step has been fulfilled. Two-factor authentication can be achieved by requiring a user to enter a one-time passcode or fingerprint.

In this article, we will be going over how to implement an SMS-based two-factor authentication in a NestJS backend project. We will be using Twilio as the SMS provider for sending out a time-based one-time passcode to the user's phone number. If you are coming from the Express framework but want to get started with NestJS, this is a good guide to get a feel of how things are done in NestJS.

Prerequisites

To follow along with this guide, you should have Node.js and Postgres installed on your computer. The project source code is written in Typescript. However, a knowledge of Javascript will suffice to understand what's going on.

Project architecture

We will be building a couple of REST endpoints in this project. Below is a high-level overview of the project structure.

The user signs up on the application and proceeds to log in. A JSON web token is used to authenticate the user upon successful login.

The user can enable two-factor authentication on their account. To enable 2FA, they need to first verify their phone number since they will be receiving their one-time password on that number.

Once 2FA is enabled on their account, the user will be required to enter a time-based OTP once they have successfully fulfilled the username-password component of the login process.

Setting up the Nest project

Compared to Express, NestJS shines when it comes to project organization and setup, thanks to its robust CLI. In your terminal, run the command below to install the NestJS CLI globally.

npm install -g @nestjs/cli

Once installed, we can use the CLI to generate a new nest project.

nest new project-name

Our newly scaffolded project will have a structure like this:

project-name
|-- node_modules
|-- src
|   |-- app.controller.spec.ts
|   |-- app.controller.ts
|   |-- app.module.ts
|   |-- app.service.ts
|   |-- main.ts
|-- test
|-- .eslintrc.js
|-- .gitignore
|-- .prettierrc
|-- nest-cli.json
|-- package-lock.json
|-- package.json
|-- README.md
|-- tsconfig.build.json
|-- tsconfig.json

NestJS has a module-oriented approach to project development. This means that each feature is developed as a module that comprises a controller and a service. This approach makes projects clean, organized, and easy to maintain.

Let's start our project on a clean slate by deleting files we won't be needing. Please delete the files: app.controller.spec.ts, app.controller.ts, and app.service.ts and remove their references in app.module.ts.

Setting up utilities

In this section, we will set up some utilities such as the project base URL, logging middleware, and custom exception filter which will be used throughout the application.

Setting global prefix in NestJS

In the src/main.ts, we can set a global prefix for all our endpoints by adding our preferred prefix inside the application bootstrap function.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';

async function bootstrap() {
    const app = await NestFactory.create(AppModule);
    app.setGlobalPrefix('api/v1');
    await app.listen(3000);
}

bootstrap();

Logging middleware

NestJs has an inbuilt logging package that we can customize to log requests to our endpoints. We will create a global middleware that will log incoming requests, request methods, status codes, request content length, and user agent.

In your src folder, create the logger middleware in this path: src/common/utils/logger.ts.

import { Injectable, Logger, NestMiddleware } from '@nestjs/common';
import { Request, Response, NextFunction } from 'express';

@Injectable()

export class LoggerMiddleware implements NestMiddleware {
    private logger = new Logger('HTTP');

    use(request: Request, response: Response, next: NextFunction): void {
        const { ip, method, originalUrl } = request;
        const userAgent = request.get('user-agent') || '';

        response.on('finish', () => {
        const { statusCode } = response;
        const contentLength = response.get('content-length');
        this.logger.log(`${method} ${originalUrl} ${statusCode} ${contentLength} - ${userAgent} ${ip}`);
        });

        next();

    }

}

We will then register our logger middleware in our app module. In src/app.module.ts, import the logger middleware like so:

import { Module, NestModule, MiddlewareConsumer } from '@nestjs/common';
import { LoggerMiddleware } from './common/utils/logger';

@Module({
    imports: [],
    controllers: [],
    providers: [],
})

export class AppModule implements NestModule {
    configure(consumer: MiddlewareConsumer) {
    consumer.apply(LoggerMiddleware).forRoutes('*');

    }
}

Custom exception filter

NestJs has a built-in exception filter that automatically sends a response to the client for all unhandled errors. This global filter sends a generic response in this format:

{
  "statusCode": 500,
  "message": "Internal server error"
}

This response doesn't let the client know what went wrong and that is why you should create a custom filter that propagates the actual error to the client.

First, we'll create an interface for our custom error filter in this path: src/common/interfaces/error.interface.ts. I don't like slapping a 500 status code if I can help it, so this interface will extend the Error object and we can give an appropriate status code for errors.

export interface ResponseError extends Error {
    statusCode?: number;
}

Create the custom filter in this path: src/common/filters/custom-exception.filter.ts.

import {
  ExceptionFilter,
  Catch,
  ArgumentsHost,
  HttpException,
} from '@nestjs/common';
import { Request, Response } from 'express';
import { ResponseError } from '../interfaces/error.interface';

@Catch()
export class CustomExceptionFilter implements ExceptionFilter {
  catch(error: ResponseError, host: ArgumentsHost) {
    const ctx = host.switchToHttp();
    const response = ctx.getResponse < Response > ();
    const request = ctx.getRequest < Request > ();

    if (error instanceof HttpException) {
      response.status(error.getStatus()).json({
        statusCode: error.getStatus(),
        message: error.message,
        error: error.name,
      });
    } else {
      const status = error?.statusCode || 500;
      response.status(status).json({
        statusCode: status,
        message: error.message,
        error: 'Internal Server Error',
      });
    }
  }
}

Setting up Postgres database

Now we'll create our database using psql, a command line utility for interacting with Postgres. Start up psql in your terminal by running the command:

psql -U postgres

The command we ran connects us to Postgres with the default user. We can then create a database by running the following command:

CREATE DATABASE two-fa-demo;

Setting up Prisma

In this section, we will be connecting to our Postgres database using the Prisma ORM. Prisma simplifies our interaction with the database and generates migrations for every model update we make.

🔥 Tip: If you're using Visual Studio Code, I recommend installing the Prisma extension to enable syntax highlighting and linting for .prisma files.

Install the Prisma CLI as a development dependency.

npm install -D prisma

Initialize Prisma in your project by running the command below in your terminal.

npx prisma init

The command above creates a Prisma folder in our project's root directory. The Prisma folder contains the schema.prisma file which we'll use to define the models for our database tables. When we migrate our model to our database, Prisma will generate a migration file that will map our model to our database.

You should also notice that Prisma created a .env file in our project root directory. Update the database URL with our Postgres database connection string.

DATABASE_URL="postgres://postgres@localhost:5432/two-fa-demo"

In the prisma/schema.prisma file, make sure that the data source provider is set to postgresql.

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

Defining models

We will be creating 2 tables in this project: A User table and an Otp table. Update the prisma/schema.prisma file with the models:

model User {
  id        String   @id @default(uuid())
  email     String   @unique
  phone     String   @unique
  password  String
  firstName String
  lastName  String
  twoFA     Boolean  @default(false)
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
  Otp       Otp[]
  isPhoneVerified Boolean @default(false)
}

model Otp {
  id        String   @id @default(uuid())
  owner     User     @relation(fields: [userId], references: [id])
  userId    String
  code      String
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
  expiresAt DateTime @db.Timestamp(5)
  useCase   UseCase
}

enum UseCase {
  LOGIN
  D2FA
  PHV
}

We declare a one-to-many relationship between the User table and the Otp table. A user can have many OTPs. When a user is created in our application, 2FA is disabled by default and their phone number is set as unverified.

The OTPs generated in our application will be valid for 5 minutes and will have 3 use cases:

LOGIN: This is for OTPs that are sent to users who have 2FA-enabled accounts
D2FA: When a user decides to disable two-factor authentication on their account, we will send an OTP to their phone number. If they verify this, we will disable 2FA on their account. This use case addresses that.
PHV: This is for OTPs that are sent to the user's phone number when they want to verify their phone number on the application.

Now that we have defined our models, we can run the following command to generate and run a migration against our database.

npx prisma migrate dev --name init

With the above command, Prisma does the following:

Generates an SQL migration file in the prisma/migrations/<migration-folder>.
Runs the migration against the database to create the tables according to the defined models.
Generates the Prisma client, a type-safe query builder adapted to our models. You'll notice that our package.json has been updated with a new dependency: @prisma/client.

Whenever you update the prisma/schema.prisma file, it is required that you regenerate the Prisma client with the command *npx prisma generate*. This way the client is adapted to the new changes in your schema file.

Creating Prisma service

We will now encapsulate the Prisma client in its service. The idea of creating a service for the Prisma client is to isolate it from the rest of our application code, making our code more organized and maintainable. The Prisma service will be used to create an instance of the Prisma client and query our database.

In your terminal, run the following command:

nest generate module prisma
nest generate service prisma

When the above command is run, the Nest CLI does the following:

Creates the files src/prisma/prisma.module.ts and src/prisma/prisma.service.ts.
Updates the app.module.ts file by importing the Prisma module and adding it to the imports array.

In the Prisma service file, we'll create a PrismaService class that extends the Prisma Client. Update the Prisma service file with the code:

import { INestApplication, Injectable, OnModuleInit } from '@nestjs/common';
import { PrismaClient } from '@prisma/client';

@Injectable()
export class PrismaService extends PrismaClient implements OnModuleInit {
  async onModuleInit() {
    await this.$connect();
  }

  async enableShutdownHooks(app: INestApplication) {
    this.$on('beforeExit', async () => {
      await app.close();
    });
  }
}

The onModuleInit method ensures that a connection to the database is established when the Prisma module is initialized. The enableShutdownHooks method ensures that our application gracefully shutdowns whenever the beforeExit event is triggered.

Next, import the PrismaService in the Prisma Module file and add the PrismaService to the providers and exports array. The Prisma module file should look like this:

import { Module } from '@nestjs/common';
import { PrismaService } from './prisma.service';

@Module({
  providers: [PrismaService],
  exports: [PrismaService],
})
export class PrismaModule {}

Implementing sign up

In this section, we'll be creating everything regarding signing up to the application. In your terminal run these commands to create a module, controller, and service for account creation on the application:

nest generate module account
nest generate controller account
nest generate service account

Notice that Nest CLI has:

Created the files src/account/account.controller.ts, src/account/account.service.ts, src/account/account.module.ts.
Imported the account controller and service into the account module and updated the account module controller and provider arrays.

Create validation pipe

Next, we'll create a pipe for validating incoming requests. We'll be using Joi for request body validation. Install the joi npm package like so:

npm i joi

Pipes are classes that sit in front of a controller route handler. NestJS uses pipes to validate or transform the data on incoming requests. We'll create a custom pipe that will use Joi to validate incoming requests.

Create the custom pipe in this path: src/common/pipes/joi.ts.

import {
  PipeTransform,
  Injectable,
  ArgumentMetadata,
  BadRequestException,
} from '@nestjs/common';
import { ObjectSchema } from 'joi';

@Injectable()
export class JoiValidationPipe implements PipeTransform {
  constructor(private schema: ObjectSchema) {}

  transform(value: any, metadata: ArgumentMetadata) {
    const { error } = this.schema.validate(value);
    if (error) {
      const errorMessage = error.details[0].message.replace(/"/g, '');
      throw new BadRequestException(errorMessage);
    }
    return value;
  }
}

Our custom pipe takes a joi schema as an argument. The joi schema is a representation of data that is acceptable for a route. Whenever an incoming request fails the joi validation, we throw a 400 error and propagate the error message to the client.

Create joi sign-up schema

In the path src/common/joi-schema/signup.ts, create the schema for sign-up.

import * as Joi from 'joi';

export const signupSchema = Joi.object({
  email: Joi.string().email().required(),
  phone: Joi.string().required(),
  password: Joi.string().required(),
  firstName: Joi.string().required(),
  lastName: Joi.string().required(),
});

Our sign-up route will be expecting a post request with a body that meets the rules outlined in the joi schema.

Also, we'll create a dto class to represent the expected data for our sign-up route. This dto class will be used to infer the type of the incoming request's body. In src/account/dto/create-user.dto.ts add this code:

export class CreateUserDto {
  email: string;
  phone: string;
  password: string;
  firstName: string;
  lastName: string;
}

Create sign-up service

In src/account/account.service.ts, we'll create the service method for signing up.

import { Injectable } from '@nestjs/common';
import { PrismaService } from 'src/prisma/prisma.service';
import { CreateUserDto } from './dto/create-user.dto';
import { hashPassword } from '../common/utils/passwordHasher';
import _ from 'underscore';

@Injectable()
export class AccountService {
  constructor(private prisma: PrismaService) {}

  async signUp(createUserDto: CreateUserDto) {
    createUserDto.password = await hashPassword(createUserDto.password);
    const user = await this.prisma.user.create({ data: createUserDto });
    return _.omit(user, 'password');
  }
}

We import the Prisma service, inject it into the account service class and use it to create a new user record in the database. I created a utility to hash the user's password before the user account is created. Once the user account is created, I use the underscore package to omit the password from the user record before sending it back to our route controller.

Install some dependencies we'll be needing.

npm i bcrypt
npm i underscore

Create the password-hasher utility in src/common/utils/passwordHasher.ts.

import * as bcrypt from 'bcrypt';

export const hashPassword = async (password: string): Promise<string> => {
  const hash = await bcrypt.hash(password, 10);
  return hash;
};

Create sign-up route handler

Update the src/account/account.controller.ts file to look like this:

 import {
  Controller,
  Post,
  Body,
  UsePipes,
} from '@nestjs/common';
import { AccountService } from './account.service';
import { CreateUserDto } from './dto/create-user.dto';
import { signupSchema } from '../common/joi-schema/signup';
import { JoiValidationPipe } from '../common/pipes/joi';

@Controller('account')
export class AccountController {
  constructor(private readonly accountService: AccountService) {}

  @Post('signup')
  @UsePipes(new JoiValidationPipe(signupSchema))
  create(@Body() createUserDto: CreateUserDto) {
    return this.accountService.signUp(createUserDto);
  }
}

The create method in the account controller accepts a post request whose body will be validated by our custom pipe. Notice how we pass the signup schema to the pipe using the @UsePipes decorator. We extract the body from the request using the @Body decorator and finally pass it on to the service method. NestJS will automatically send a 201 status code when the request resolves successfully.

Our sign-up route will be accessed at the path api/v1/account/signup. Notice that the @Controller decorator is passed a parameter: account. Now every route created in this controller class will be prefixed with /account.

Implementing login

Our login will take this flow:

User will enter email and password
User details will be validated
If correct, we generate a JSON web token based on the user credentials and return the token to the client.

First, we'll create an auth module to handle login and authorization-related functionality. Run the below commands in your terminal to create the auth module, controller, and service files.

nest generate module auth
nest generate controller auth
nest generate service auth

Register the JWT module

In your .env, add a secret for your JSON web token. This will be used in creating and validating our JWTs.

JWT_SECRET=twofademo

Create a constants file to easily access the JWT secret. In src/common/utils/constants.ts, add this code:

export const constants = {
  jwtSecret: process.env.JWT_SECRET
};

NestJS has a built-in module that makes it easy to handle creating JWTs. Register the NestJS JWT module in the auth module:

import { Module } from '@nestjs/common';
import { JwtModule } from '@nestjs/jwt';
import { AuthController } from './auth.controller';
import { AuthService } from './auth.service';
import { PrismaModule } from '../prisma/prisma.module';
import { constants } from '../common/utils/constants';

@Module({
  controllers: [AuthController],
  providers: [AuthService],
  imports: [
    PrismaModule,
    JwtModule.register({
      global: true,
      secret: constants.jwtSecret,
      signOptions: { expiresIn: '2 days' },
    }),
  ],
})
export class AuthModule {}

Validation for login route

Create the login dto in this path: src/auth/dto/login.dto.ts and add this code:

export class LoginDto {
  email: string;
  password: string;
}

Create the login joi schema in this path: src/common/joi-schema/login.ts :

import * as Joi from 'joi';

export const loginSchema = Joi.object({
  email: Joi.string().email().required(),
  password: Joi.string().required(),
});

Set up login service handler

In src/auth/auth.service.ts, add the code:

import { HttpStatus, Injectable } from '@nestjs/common';
import { JwtService } from '@nestjs/jwt';
import { HttpException } from '@nestjs/common';
import { Prisma } from '@prisma/client';
import { Request } from 'express';
import { PrismaService } from '../prisma/prisma.service';
import { LoginDto } from './dto/login.dto';
import { verifyPassword } from '../common/utils/passwordHasher';

@Injectable()
export class AuthService {
  constructor(private prisma: PrismaService, private jwtService: JwtService) {}

  async login(loginDto: LoginDto) {
    const user = await this.prisma.user.findUnique({
      where: { email: loginDto.email },
    });
    if (!user) {
      throw new HttpException(
        'Invalid email or password',
        HttpStatus.BAD_REQUEST,
      );
    }
    const validPassword = await verifyPassword(
      loginDto.password,
      user.password,
    );

    if (!validPassword) {
      throw new HttpException(
        'Invalid email or password',
        HttpStatus.BAD_REQUEST,
      );
    }

    if (!user.twoFA) {
      const payload = {
        email: user.email,
        first_name: user.firstName,
        last_name: user.lastName,
        sub: user.id,
      };
      return {
        success: true,
        access_token: await this.jwtService.signAsync(payload),
      };
    }
  }
}

In the login method of the AuthService class, we check if the user exists in our database. If true, we verify their password. If they do not have 2FA enabled on their account, we immediately generate a JWT using their credentials. If the user doesn't exist in the database, we throw an error message with a 400 status code.

Create verifyPassword function in the passwordHasher utility file we previously created.

export const verifyPassword = async (password, hash): Promise<boolean> => {
  const isMatch = await bcrypt.compare(password, hash);
  return isMatch;
};

Set up login route handler

Set up the route handler for login by adding the code below to the AuthController file.

import {
  Body,
  Controller,
  HttpCode,
  Post,
  UsePipes,
} from '@nestjs/common';
import { Request } from 'express';
import { AuthService } from './auth.service';
import { LoginDto } from './dto/login.dto';
import { loginSchema } from '../common/joi-schema/login';
import { JoiValidationPipe } from '../common/pipes/joi';
import { tokenSchema } from '../common/joi-schema/token';

@Controller('auth')
export class AuthController {
  constructor(private readonly authService: AuthService) {}

  @UsePipes(new JoiValidationPipe(loginSchema))
  @HttpCode(200)
  @Post('login')
  login(@Body() loginDto: LoginDto) {
    return this.authService.login(loginDto);
  }

}

The login route can be accessed at the path: api/v1/auth/login. Successful requests resolve with a status code of 200. In the login method, we extract the body of the request and pass it on to our service handler method.

Phone verification

Before users can enable two-factor authentication on their account, their phone number must be verified on the application. In this section, we will be using Twilio to send a verification OTP to their registered phone number.

Set up Twilio

Sign up for an account on the Twilio website. Once signed up on Twilio, you'll get a phone number that you'll use to send out SMS. Also, grab your Twilio Account SID and Twilio Auth Token from your Twilio console. Update your .env with your new credentials like so:

TWILIO_AUTH_TOKEN=
TWILIO_ACCOUNT_SID=
TWILIO_PHONE_NUMBER=

Install the Twilio npm package which we'll use to access the Twilio API.

npm i twilio

Update the constants utility file previously created to look like this:

export const constants = {
  jwtSecret: process.env.JWT_SECRET,
  twilioAuthToken: process.env.TWILIO_AUTH_TOKEN,
  twilioAccountSID: process.env.TWILIO_ACCOUNT_SID,
  twilioPhoneNumber: process.env.TWILIO_PHONE_NUMBER,
};

Next, we'll create a function to handle sending out SMS. Create the function in this file path: src/common/utils/twilio.ts.

import { Twilio } from 'twilio';
import { constants } from './constants';

const { twilioAccountSID, twilioAuthToken, twilioPhoneNumber } = constants;
const client = new Twilio(twilioAccountSID, twilioAuthToken);

export const sendSMS = async (phoneNumber: string, message: string) => {
  try {
    const smsResponse = await client.messages.create({
      from: twilioPhoneNumber,
      to: phoneNumber,
      body: message,
    });
    console.log(smsResponse.sid);
  } catch (error) {
    error.statusCode = 400;
    throw error;
  }
};

Create Auth Guard

In NestJS, guards are classes that are used to determine if a request has the necessary authorization to access a resource. Our phone verification route will be a protected route. Hence, requests to this route must carry a bearer token from a signed-in user.

Create the auth guard in this path: src/common/guards/auth.guard.ts.

import {
  CanActivate,
  ExecutionContext,
  Injectable,
  UnauthorizedException,
} from '@nestjs/common';
import { JwtService } from '@nestjs/jwt';
import { constants } from '../utils/constants';
import { Request } from 'express';

@Injectable()
export class AuthGuard implements CanActivate {
  constructor(private jwtService: JwtService) {}

  async canActivate(context: ExecutionContext): Promise<boolean> {
    const request = context.switchToHttp().getRequest();
    const token = this.extractTokenFromHeader(request);
    if (!token) {
      throw new UnauthorizedException();
    }
    try {
      const payload = await this.jwtService.verifyAsync(token, {
        secret: constants.jwtSecret,
      });
      // 💡 We're assigning the payload to the request object here
      // so that we can access it in our route handlers
      request['user'] = payload;
    } catch {
      throw new UnauthorizedException();
    }
    return true;
  }

  private extractTokenFromHeader(request: Request): string | undefined {
    const [type, token] = request.headers.authorization?.split(' ') ?? [];
    return type === 'Bearer' ? token : undefined;
  }
}

In the auth guard, we have a private method that extracts the bearer token from the authorization header of the incoming request and a canActivate method that validates the bearer token. If the token is not valid, we throw a 401 error else we append a user property to the request and pass the request on to its route handler.

Send phone verification OTP

The phone verification request will be sent to a post endpoint. Create a joi validation schema for the request in this path: src/common/joi/verify-phone.ts.

import * as Joi from 'joi';

export const verifyPhoneSchema = Joi.object({
  verify: Joi.boolean().valid(true).required(),
});

The schema shows that the endpoint expects a request with a verify key in the request body.

In the AccountController file, add the route handler method for phone verification.

 import {
  Controller,
  Post,
  Body,
  UsePipes,
  UseGuards,
  HttpCode,
  Req,
} from '@nestjs/common';
import { Request } from 'express';
import { AccountService } from './account.service';
import { CreateUserDto } from './dto/create-user.dto';
import { signupSchema } from '../common/joi-schema/signup';
import { JoiValidationPipe } from '../common/pipes/joi';
import { AuthGuard } from '../common/guards/auth.guard';
import { verifyPhoneSchema } from '../common/joi-schema/verify-phone';

@Controller('account')
export class AccountController {
  constructor(private readonly accountService: AccountService) {}

  ...

  @HttpCode(200)
  @Post('phone/verify')
  @UsePipes(new JoiValidationPipe(verifyPhoneSchema))
  @UseGuards(AuthGuard)
  verifyPhone(@Body() _body, @Req() request: Request) {
    return this.accountService.verifyPhone(request);
  }
}

The request will be sent to the path: api/v1/account/phone/verify. We use the @UseGuards decorator to ensure that only authorized requests can access this route. Once the request successfully passes the guard validation, we pass the entire request to the account service method. The reason for this is that we need to access the user property on the request. This way, we know the user who is requesting a phone verification.

Create the service method for phone verification and update your imports.

...
import { HttpException, HttpStatus, Injectable } from '@nestjs/common';
import { Request } from 'express';
import { generateOTP } from '../common/utils/codeGenerator';
import { Prisma } from '@prisma/client';
import { getExpiry, isTokenExpired } from '../common/utils/dateTimeUtility';
import { sendSMS } from '../common/utils/twilio';

@Injectable()
export class AccountService {
  constructor(private prisma: PrismaService) {}

    ...
  async verifyPhone(req: Request) {
      const userDetails = req['user'];
      const user = await this.prisma.user.findUnique({
        where: { id: userDetails.sub },
      });
      if (!user) {
        throw new HttpException('User not found', HttpStatus.NOT_FOUND);
      }
      if (user.isPhoneVerified) {
        return { success: true };
      }
      const otp = generateOTP(6);
      const otpPayload: Prisma.OtpUncheckedCreateInput = {
        userId: user.id,
        code: otp,
        useCase: 'PHV',
        expiresAt: getExpiry(),
      };

      await this.prisma.otp.create({
        data: otpPayload,
      });
      await sendSMS(
        user.phone,
        `Use this code ${otp} to verify the phone number registered on your account`,
      );
      return { success: true };
    }
  }
}

In the verifyPhone method, we extract the user property on the request and check if the user indeed exists on the database. If the user is unverified, we generate a 6-digit code, create a record in our Otp table with the use case of PHV and send the user an sms containing an Otp that will expire in 5 minutes.

Create the utility functions generateOTP and getExpiry.

//src/common/utils/codeGenerator.ts
export const generateOTP = (n: number): string => {
  const digits = '0123456789';
  let otp = '';

  for (let i = 0; i < n; i++) {
    otp += digits[Math.floor(Math.random() * digits.length)];
  }

  return otp;
};

Import the moment npm package by running npm install moment.

//src/common/utils/dateTimeUtility.ts
import * as moment from 'moment';
export const getExpiry = () => {
  const createdAt = new Date();
  const expiresAt = moment(createdAt).add(5, 'minutes').toDate();
  return expiresAt;
};

export function isTokenExpired(expiry: Date): boolean {
  const expirationDate = new Date(expiry);
  const currentDate = new Date();
  return expirationDate.getTime() <= currentDate.getTime();
}

Verify PHV OTP

The client will make a request to verify the OTP the user received for phone verification. Create the joi schema for this endpoint in this path: src.common/joi/token.ts.

import * as Joi from 'joi';

export const tokenSchema = Joi.object({
  token: Joi.string().required(),
});

Create a route handler method for this verification in the AccountController.

@Controller('account')
export class AccountController {
  constructor(private readonly accountService: AccountService) {}

  ...

  @UsePipes(new JoiValidationPipe(tokenSchema))
  @UseGuards(AuthGuard)
  @HttpCode(200)
  @Post('phone/verify/token')
  validatePhoneVerification(@Body() _body, @Req() request: Request) {
    return this.accountService.validatePhoneVerification(request);
  }
}

Create the service method to handle this verification in the AccountService class.

@Injectable()
export class AccountService {
  constructor(private prisma: PrismaService) {}

    ...
  async validatePhoneVerification(req: Request) {
    const {
      body: { token },
    } = req;
    const userDetails = req['user'];
    // find otp record
    const otpRecord = await this.prisma.otp.findFirst({
      where: { code: token, useCase: 'PHV', userId: userDetails.sub },
    });
    if (!otpRecord) {
      throw new HttpException('Invalid OTP', HttpStatus.NOT_FOUND);
    }
    // check if otp is expired
    const isExpired = isTokenExpired(otpRecord.expiresAt);
    if (isExpired) {
      throw new HttpException('Expired token', HttpStatus.NOT_FOUND);
    }
    // update user isPhoneVerified to true
    await this.prisma.user.update({
      where: { id: userDetails.sub },
      data: { isPhoneVerified: true },
    });

    // delete the otp record
    await this.prisma.otp.delete({ where: { id: otpRecord.id } });
    return { success: true };
  }

}

The service method does the following:

Extracts the token from the request body and checks if it exists in the database.
Checks the token expiry if it exists
Update the isPhoneVerified field in the user record to true if the token is valid
Deletes the OTP record from the database

Setting 2FA

In this section, we'll create an endpoint to enable or disable two-factor authentication on the user account.

Create the joi schema for this endpoint in src/common/joi-schema/set2fa.ts.

import * as Joi from 'joi';

export const set2faSchema = Joi.object({
  set_2fa: Joi.boolean().required(),
});

Create the route handler method in AccountController.

@Controller('account')
export class AccountController {
  constructor(private readonly accountService: AccountService) {}

 ...

  @UsePipes(new JoiValidationPipe(set2faSchema))
  @UseGuards(AuthGuard)
  @HttpCode(200)
  @Post('set/twofa')
  enableTwoFA(@Body() _body, @Req() request: Request) {
    return this.accountService.setTwoFA(request);
  }
}

The request will be sent to the path: api/v1/account/set/twofa. Create the service method for this route in the AccountService class.

@Injectable()
export class AccountService {
  constructor(private prisma: PrismaService) {}

  ...

  async setTwoFA(req: Request) {
    const {
      body: { set_2fa },
    } = req;
    const userDetails = req['user'];

    const user = await this.prisma.user.findUnique({
      where: { id: userDetails.sub },
    });

    if (!user) {
      throw new HttpException('User not found', HttpStatus.NOT_FOUND);
    }

    if (user.twoFA === set_2fa) {
      return { success: true };
    }

    if (user.twoFA && set_2fa == false) {
      // generate otp to disable MFA, create otp record, send sms and return
      const otp = generateOTP(6);
      const otpPayload: Prisma.OtpUncheckedCreateInput = {
        userId: user.id,
        code: otp,
        useCase: 'D2FA',
        expiresAt: getExpiry(),
      };

      await this.prisma.otp.create({
        data: otpPayload,
      });
      await sendSMS(
        user.phone,
        `Use this code ${otp} to disable multifactor authentication on your account`,
      );
      return { success: true };
    }

    await this.prisma.user.update({
      where: { id: user.id },
      data: { twoFA: set_2fa },
    });

    return { success: true };
  }
}

This service method does the following:

If the set_2fa parameter of the request body is true and the user does not have 2FA enabled, two-factor authentication is enabled on their account.
- If the set_2fa parameter of the request body is false and the user already has 2FA enabled, an OTP with the use case D2FA is sent to their phone number. The OTP will be validated on a separate endpoint before 2FA is disabled on the user account.

Disabling 2FA

To disable two-factor authentication on their account, the user will receive an OTP. This OTP will be validated before 2FA is finally disabled on their account. We'll create the route handler and service method for this.

@Controller('account')
export class AccountController {
  constructor(private readonly accountService: AccountService) {}

 ...

  @UsePipes(new JoiValidationPipe(tokenSchema))
  @UseGuards(AuthGuard)
  @HttpCode(200)
  @Post('disable-twofa/verify')
  disable2FAVerification(@Body() _body, @Req() request: Request) {
    return this.accountService.disable2FAVerification(request);
  }
}

@Injectable()
export class AccountService {
  constructor(private prisma: PrismaService) {}

  ...
  async disable2FAVerification(req: Request) {
      const {
        body: { token },
      } = req;
      const userDetails = req['user'];
      const otpRecord = await this.prisma.otp.findFirst({
        where: { code: token, useCase: 'D2FA', userId: userDetails.sub },
      });
      if (!otpRecord) {
        throw new HttpException('Invalid OTP', HttpStatus.NOT_FOUND);
      }
      const isExpired = isTokenExpired(otpRecord.expiresAt);
      if (isExpired) {
        throw new HttpException('Expired token', HttpStatus.NOT_FOUND);
      }
      await this.prisma.user.update({
        where: { id: userDetails.sub },
        data: { twoFA: false },
      });

      await this.prisma.otp.delete({ where: { id: otpRecord.id } });
      return { success: true };
    }
}

To verify the OTP sent for disabling 2FA, a post request will be sent toapi/v1/account/disable-twofa/verify. In the disable2FAVerification method, we check if the OTP exists in the database and if it does, we check if it has expired. If the OTP is still valid, we update the user record in the database by setting the twoFA field to false. Finally, we delete the OTP record from the database.

Login flow for 2FA enabled account

Recall that in the login method of the AuthService class, if the user does not have two-factor authentication enabled, we immediately generate a JWT and send it back to the client. We'll now update the login method to cater to users who have enabled 2FA. Update the login method of the AuthService class:

@Injectable()
export class AuthService {
  constructor(private prisma: PrismaService, private jwtService: JwtService) {}

  async login(loginDto: LoginDto) {
    const user = await this.prisma.user.findUnique({
      where: { email: loginDto.email },
    });
    if (!user) {
      throw new HttpException(
        'Invalid email or password',
        HttpStatus.BAD_REQUEST,
      );
    }
    const validPassword = await verifyPassword(
      loginDto.password,
      user.password,
    );

    if (!validPassword) {
      throw new HttpException(
        'Invalid email or password',
        HttpStatus.BAD_REQUEST,
      );
    }

    if (!user.twoFA) {
      const payload = {
        email: user.email,
        first_name: user.firstName,
        last_name: user.lastName,
        sub: user.id,
      };
      return {
        success: true,
        access_token: await this.jwtService.signAsync(payload),
      };
    }

    const otp = generateOTP(6);
    const otpPayload: Prisma.OtpUncheckedCreateInput = {
      userId: user.id,
      code: otp,
      useCase: 'LOGIN',
      expiresAt: getExpiry(),
    };
    await this.prisma.otp.create({
      data: otpPayload,
    });
    await sendSMS(
      user.phone,
      `Use this code ${otp} to finalize login on your account`,
    );
    return { success: true };
  }
}

Now for accounts with 2FA enabled, a 6-digit OTP with a use case of LOGIN will be sent to their phone number. The OTP will be validated on a separate endpoint and if it is valid, a JWT will be sent back to the client.

Verify LOGIN OTP

Update the AuthController to include a route handler method which will be used to validate the login OTP.

@Controller('auth')
export class AuthController {
  constructor(private readonly authService: AuthService) {}

  ...

  @UsePipes(new JoiValidationPipe(tokenSchema))
  @HttpCode(200)
  @Post('login/verify/token')
  verifyLogin(@Body() _body, @Req() request: Request) {
    return this.authService.verifyLogin(request);
  }
}

The login OTP is sent as a post request to api/v1/auth/login/verify/token. Create a service method for this route handler in the AuthService class.

@Injectable()
export class AuthService {
  constructor(private prisma: PrismaService, private jwtService: JwtService) {}
  async verifyLogin(req: Request) {
    const {
      body: { token },
    } = req;
    const otpRecord = await this.prisma.otp.findFirst({
      where: { code: token, useCase: 'LOGIN' },
    });
    if (!otpRecord) {
      throw new HttpException('Invalid OTP', HttpStatus.NOT_FOUND);
    }
    const isExpired = isTokenExpired(otpRecord.expiresAt);
    if (isExpired) {
      throw new HttpException('Expired token', HttpStatus.NOT_FOUND);
    }
    const user = await this.prisma.user.findUnique({
      where: { id: otpRecord.userId },
    });

    if (!user) {
      throw new HttpException('Invalid OTP', HttpStatus.NOT_FOUND);
    }
    const payload = {
      email: user.email,
      first_name: user.firstName,
      last_name: user.lastName,
      sub: user.id,
    };
    return {
      success: true,
      access_token: await this.jwtService.signAsync(payload),
    };
  }
}

The verifyLogin service method does the following:

Checks if the OTP exists in the database.
If OTP exists, it checks if it is expired.
If OTP is valid, a JWT is generated based on the user credentials and returned to the client.

Conclusion

In conclusion, implementing two-factor authentication (2FA) can significantly enhance the security of an application. By requiring users to provide an additional verification factor, such as a time-based code from their mobile device, you can reduce the risk of unauthorized access and protect sensitive data.

In this article, we have covered various aspects of implementing 2FA in a NestJS project, including sign-up, login, authentication, request validation, and route protection. We have also discussed how to enable and disable 2FA for users.

By using NestJS's built-in features such as pipes and guards, we can ensure that our implementation is robust and secure. It is important to note that while 2FA adds an extra layer of security, it is not a replacement for strong passwords and other security best practices.

🔥 Project GitHub repository
🔥 Postman documentation for endpoints

Thank you for reading. Feel free to share your thoughts in the comments.

Git Hooks: The Powerful Tool You're Probably Not Using (But Should Be)

Chiamaka Ojiyi — Sun, 26 Mar 2023 19:56:51 +0000

Introduction

Git tops the list of version control software, allowing developers to collaborate and keep track of changes to source code. Beyond the conventional git workflow of staging changed files, creating commits, and pushing those changes, there is a lot more you can do with Git to 10x your productivity. In this article, we will explore how to leverage Git hooks, a powerful feature available in Git, to automate repetitive tasks and enforce coding standards throughout a repository.

What are Git Hooks

Git hooks are customizable scripts located in the .git/hooks directory. These scripts are executed when certain events occur. Git hooks are built into Git, meaning they exist if you installed Git. The git hooks directory is populated with sample shell scripts with the file extension .sample. These scripts have some default code in them. However, you can customize these scripts to adapt them to your project's needs.

Types of Git Hooks

There are two groups of Git hooks: client-side hooks and server-side hooks.

Client-side Hooks

Client-side Git hooks are triggered by events that occur only in the local repository. An example of a client-side git hook is the pre-commit hook which you can set up to execute before any change is committed to your local repository. The client-side hooks available in Git include:

pre-commit
prepare-commit-msg
commit-msg
post-commit
pre-rebase
post-rewrite
post-checkout
post-merge
pre-push

Server-side Hooks

Server-side hooks, as the name implies, are hooks that run on the remote Git repository server. An example of a server-side Git hook is the pre-receive hook. This hook is triggered before any changes are accepted into the remote repository, allowing you to enforce specific checks on incoming code changes. The server-side hooks available in Git include:

pre-receive
update
post-receive

Using Git Hooks in your Workflow

Pre-Commit Hook

The pre-commit hook is a script that Git executes before you commit any code to your repository. Here are some scenarios where you can adopt pre-commit hooks to automate your workflow:

Code formatting

Imagine you had to check out from production real quick to make a hot fix and you forgot to format your code before committing and pushing. Or you can't make up your mind about whether to use single quotes or double quotes, so you end up using both in your code. If that sounds like you, the good news is that you can set up pre-commit hooks that will format your code and enforce a consistent style before you commit new changes to your repository.

Here's how to set up a pre-commit hook that uses Prettier to format code before you make a commit in a Node.js project.

You should have prettier installed in your project as a dev dependency.

npm install prettier --save-dev

Create a .prettierrc configuration file in your project's root directory and add your settings.

{

"trailingComma": "all",

"tabWidth": 2,

"semi": true,

"singleQuote": true,

"jsxSingleQuote": true,

"printWidth": 80,

"arrowParens": "always"

}

In the .git/hooks directory, open the pre-commit.sample file. Replace the default code in the file with the code below and rename the file to pre-commit.

#!/bin/sh

# Run prettier on staged files

git diff --name-only --cached --diff-filter=ACM | grep '\.jsx\?$' | xargs ./node_modules/.bin/prettier --write

# Add back the modified/prettified files to staging

git add .

So what exactly does this script do? Glad you asked.

We first tell Git to list the names of all the files that have been staged for the commit. To select the files we want to pass to prettier for formatting, we do grep '\.jsx\?$'. This filters the list to include only files with the .js or .jsx file extension (you can adjust this to match your project's file extensions). We then pipe this filtered list of file names to the prettier command, which formats each file and writes the changes back to disk. Because the files are formatted, Git detects new changes in them so we have to stage them again by running git add .

Next, we have to make the pre-commit hook executable. Do this by running this command in your terminal.

chmod +x .git/hooks/pre-commit

Now, whenever you try to commit a new change, Git will execute the pre-commit hook and format your code using Prettier.

Code Linting

You can also use a pre-commit hook to run a linting tool on your code before new changes are committed. This is a great way to catch errors and ensure that only quality is committed to your codebase.

Here's how to set up a pre-commit hook that uses ESlint to lint code before you make a commit in a Node.js project.

You should have ESlint installed in your project as a dev dependency.

npm install eslint --save-dev

Create a .eslintrc.js configuration file in your project's root directory and add your linting settings.

module.exports = {
  env: {
    node: true,
    es6: true,
  },
  extends: [
    'eslint:recommended',
    'plugin:node/recommended',
  ],
  plugins: [
    'node',
  ],
  parserOptions: {
    ecmaVersion: 2018,
  },
  rules: {
    'no-console': 'off',
    'node/no-unpublished-require': 'off',
    'node/no-unsupported-features/es-syntax': 'off',
  },
};

// sample eslint settings

Set up your pre-commit hook to run a code-linter by adding this code to the script.

#!/bin/sh

# Find all staged files with a .js, .jsx, or .ts file extension
files=$(git diff --name-only --cached | grep -E '\.(js|jsx|ts)$')

# Lint each staged file with ESLint
if [ -n "$files" ]; then
  echo "Linting staged files with ESLint..."
  echo "$files" | xargs ./node_modules/.bin/eslint
  if [ $? -ne 0 ]; then
    echo "ESLint failed. Aborting commit."
    exit 1
  fi
fi

Just like the previous pre-commit hook we saw, this script starts by finding all staged files that have a .js, .jsx, or .ts file extension using git diff. If there are staged files with one of those extensions, the script then lints each of those files using ESLint. If ESLint reports any errors, the commit is aborted and an error message is printed in the console. If ESLint reports no errors, the script completes and allows the commit to proceed.

Branch Naming Convention

Some teams maintain a convention for naming branches in a repository. A common convention would be naming a branch with the ticket number and ticket title.

Here's an example of a branch name following this convention: STO-120-signup-bug-fix. The STO stands for story. Your team may be following a different convention. You can enforce this convention in your repository using a pre-commit hook.

#!/bin/bash



# Define the pattern to match

PATTERN="^STO-[0-9]+-[a-zA-Z0-9_-]+$"



# Check the current branch name

BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD)



# Check if the branch name matches the pattern

if ! [[ "$BRANCH_NAME" =~ $PATTERN ]]

then

echo "Branch name must match pattern: $PATTERN"

exit 1

fi

Pre-Push Hook

Pre-push hooks are executed before code changes are pushed to the remote repository. A simple yet good scenario for adopting a pre-push hook in your workflow will be to use it to validate commit messages. Following best practices when writing commit messages can distinguish you as a developer who prioritises quality and is keen on detail.

A good commit message should be short and follow this structure:

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

A quick break down of a good commit message structure:

It must have a type. Some commit types include build:, chore:, ci:, docs:, style:, refactor:, perf:, test:
Optional scope: This is to provide some additional context to the commit.
Description of the commit
An optional body and optional footer.

To create a pre-push git hook that will enforce commit message best practices in a Node.js project, follow these steps:

Navigate to the pre-push.sample file in the .git/hooks directory in your project.
Rename the file to pre-push without any file extension.
Paste in this code into the file:

#!/usr/bin/env node

const { execSync } = require('child_process');

const commitMsgRegex = /^(feat|fix|docs|style|refactor|test|chore|build)(\(.+\))?: .{1,72}$/;

const validateCommitMsg = (commitMsg) => {
  if (!commitMsgRegex.test(commitMsg)) {
    console.error(
      '\nInvalid commit message format. Please use the following format:\n' +
      'type(scope?): subject\n\n' +
      'where type can be one of the following:\n' +
      'feat: A new feature\n' +
      'fix: A bug fix\n' +
      'docs: Documentation only changes\n' +
      'style: Changes that do not affect the meaning of the code\n' +
      'refactor: A code change that neither fixes a bug nor adds a feature\n' +
      'test: Adding missing tests or correcting existing tests\n' +
      'chore: Changes to the build process or auxiliary tools and libraries\n' +
      'build: Changes that affect the build system or external dependencies\n\n' +
      'And subject should be no longer than 72 characters.'
    );
    process.exit(1);
  }
};


const main = () => {
  const localRef = process.argv[2];
  const localSha = process.argv[3];
  const remoteRef = process.argv[4];
  const remoteSha = process.argv[5];

  const commits = execSync(`git log ${remoteSha}..${localSha} --pretty=format:%s`).toString().trim().split('\n');

  commits.forEach((commitMsg) => {
    validateCommitMsg(commitMsg);
  });

};

main();

The pre-push hook uses the Node.js child_process module to scan the commit messages before changes are pushed to the remote repository. It checks that the commit message is not longer than 72 characters and that it is started with a type which can be any of these types: feat|fix|docs|style|refactor|test|chore|build. Commit messages that don't conform to the rules will trigger an error and the changes will not be pushed to the remote branch.

Git Hooks and Version Control

It is important to note that the .git/hooks directory where the hook scripts are stored is not tracked by git. Consequently the hooks that you have set up will not be available to other developers who clone your project.

However, if you want to make the hooks available to other developers working on the project, you need to create a separate directory which will hold the git hooks that your project requires. Here are the steps to follow if you want git to track your hook scripts:

Create a hooks directory in your project's directory.
Add your Git hook scripts to your newly created hooks directory.
Create a symbolic link between the .git/hooks and the hooks directory in your project's parent directory.

ln -s ../../hooks .git/hooks

Finally, commit the hooks directory to your Git repository.

Now, whenever Git runs a hook, it will look in the hooks directory in your project.

Note that every developer who clones your project will have to create a symbolic link between the hooks directory and the .git/hooks directory on their local machine. This is will ensure that Git looks in the project's hooks directory when running hooks. You can add this information to your project's README so that developers working in the repository are aware of it.

Conclusion

In this article we explored Git hooks, the customisable built-in scripts that are automatically run by Git when certain actions occur in a repository. We also explored some scenarios where Git hooks can be leveraged to enforce coding standards, catch errors and ensure that only quality changes are committed to the code base.

If you haven't started using git hooks in your project, I encourage to add it to your arsenal of development tools. It is a powerful tool that can increase your efficiency, streamline your workflow and improve the quality of your code base.

Thank you for reading. Feel free to share your thoughts in the comments.

Implementing Joins in MongoDB using the $lookup operator

Chiamaka Ojiyi — Mon, 20 Feb 2023 16:56:14 +0000

A join is a statement used in relational database systems to combine data from two or more tables based on a commonly shared column. MongoDB is however a non-relational database system which stores data as documents in collections, which are the equivalent of tables in SQL databases.

MongoDB is a document-oriented database which is great for storing unstructured data. Though non-relational in design, MongoDB has some operators that allow us to perform some join operations. Using the $lookup operator, we'll dive into how to implement a join operation in MongoDB.

Dependencies
You'll need to have MongoDB and MongoDB compass to follow this tutorial.

MongoDB's $lookup Operator

The $lookup operator provides us with a way to perform a left-outer join.

A left-outer join is an operation that returns all data from the left table and only matching data from the right table.

In the context of MongoDB, a left-outer join returns all data from the left collection and any matching data from the right collection. If there is no matching data in the right collection, only data from the left collection is returned.

The $lookup operator is used as a stage in the aggregation pipeline. In MongoDB, aggregation is the process of analysing data by passing it through different stages of operations. A strong use for aggregation is when we want to perform analytics and create reports based on data that spans across different collections.

Left Outer Joins using the $lookup operator

Head over to your terminal and start the mongo shell with the command

mongosh

Initialize a database named business with the command

use business

Create 2 collections which we will be using to demonstrate joins implementation via the $lookup operator.

db.createCollection('company')

db.createCollection('service')

Next, we'll seed our collections with data.

db.company.insertMany([
        {
            "_id": "886076c1-af54-46bf-9afd-d50c50543456",
            "companyName": "Thoughtstorm",
            "email": "ksoanes0@un.org",
            "isRegistered": true
        },
        {
            "_id": "4f6f50a8-51bf-4435-89ed-f421707bea9e",
            "companyName": "Skilith",
            "email": "bovesen1@miitbeian.gov.cn",
            "isRegistered": true
        },
        {
            "_id": "bd69948e-904c-4f56-a129-945ea829108e",
            "companyName": "Yozio",
            "email": "mshouler2@hc360.com",
            "isRegistered": true
        },
        {
            "_id": "818d73bc-46f8-497d-9ae4-ac71508f81f1",
            "companyName": "Kayveo",
            "email": "mbyfield3@dell.com",
            "isRegistered": false
        },
        {
            "_id": "f6fc009a-ac8e-433a-b47e-855abe246f1b",
            "companyName": "Chatterbr_idge",
            "email": "lhalfacree4@github.com",
            "isRegistered": false
        },
        {
            "_id": "e15015cc-0e9a-4fd9-9246-b51dc6f22f89",
            "companyName": "Gabtype",
            "email": "aurlin5@yandex.ru",
            "isRegistered": false
        },
        {
            "_id": "90900c9b-3b1e-4151-829c-0ca365220188",
            "companyName": "Bubbletube",
            "email": "pbaynham6@printfriendly.com",
            "isRegistered": false
        },
        {
            "_id": "7a1f34b7-6232-4ec3-a554-f5b3aef31231",
            "companyName": "Trilith",
            "email": "skinnaird7@ifeng.com",
            "isRegistered": false
        },
        {
            "_id": "f24f0151-1ae5-4724-aa97-d43fe8d3f854",
            "companyName": "Katz",
            "email": "lsimeoli8@ftc.gov",
            "isRegistered": true
        },
        {
            "_id": "cef57a11-acac-416d-ab3a-096a17722a84",
            "companyName": "Livetube",
            "email": "hhounsom9@usatoday.com",
            "isRegistered": false
        }
    ])

db.service.insertMany([
        {
            "_id": "11f314d4-69b0-4abe-8bac-3025ebfc38ee",
            "companyId": "886076c1-af54-46bf-9afd-d50c50543456",
            "serviceName": "Rebar & Wire Mesh Install",
            "isActive": true
        },
        {
            "_id": "08166e38-399b-4c8f-a5e8-c48e397538be",
            "companyId": "4f6f50a8-51bf-4435-89ed-f421707bea9e",
            "serviceName": "Wall Protection",
            "isActive": true
        },
        {
            "_id": "af3e867b-633a-44b3-8a8d-9ddda2320bba",
            "companyId": "bd69948e-904c-4f56-a129-945ea829108e",
            "serviceName": "Epoxy Flooring",
            "isActive": true
        },
        {
            "_id": "662883be-0e0a-4e80-92cc-7a45d8984006",
            "companyId": "818d73bc-46f8-497d-9ae4-ac71508f81f1",
            "serviceName": "Masonry",
            "isActive": true
        },
        {
            "_id": "5bec5ef0-6449-40c5-b097-27b330415757",
            "companyId": "f6fc009a-ac8e-433a-b47e-855abe246f1b",
            "serviceName": "Plumbing & Medical Gas",
            "isActive": true
        },
        {
            "_id": "fe04f4f1-27e1-421b-abe8-06f415136421",
            "companyId": "e15015cc-0e9a-4fd9-9246-b51dc6f22f89",
            "serviceName": "Casework",
            "isActive": true
        },
        {
            "_id": "0d2db760-9b5c-4fa2-be0f-f6acb251e7de",
            "companyId": "90900c9b-3b1e-4151-829c-0ca365220188",
            "serviceName": "Exterior Signage",
            "isActive": true
        }
    ])

Some points about the data we just added to our collections.

There are 10 companies. 4 out of the 10 companies are registered.
There are 7 services offered by these companies and all services are active.

Now the Analysis

Say we want to create a report that shows us only services that offered by registered companies.

How would we go about that?

Because the data we need spans across two collections, the company and service collection, we would need to do a join to generate this report.

The common factor between these two collections is the _id of the company collection and this is referenced as the companyId in the service collection.

For the visual clarity and ease of understanding, I recommend that you use MongoDB's Compass to implement the joins between the two collections.

In Compass, connect to the business database on localhost.

On the left menu pane, you should see the business database and the two collections we created.

Select the service collection.

In the service collection view, select the Aggregations tab.

In the first stage of the Aggregations view, select the $lookup operator.

Break down of the lookup stage syntax

The $lookup stage has the syntax:

{
  from: collection,
  localField: field,
  foreignField: field,
  as: result
}

The from key represents the collection we wish to join to. In this case, it is the company collection.

The localField key represents the field/column in our input collection. We are carrying out this join operation from the service collection which is our input collection. Therefore the localField in this case will be the companyId.

The foreignField key is the field from the collection we want to join. In our case, the foreignField is the _id field in the company collection.

The as key represents a variable which will hold the result of the join operation. The result of the join operation is an array. You are free to give the variable a name of your choice.

In Compass, fill out the $lookup syntax like this

{
  from: "company",
  localField: "companyId",
  foreignField: "_id",
  as: "registeredServices"
}

In the preview pane on Compass, you can see that we have 7 documents from this join operation. This operation fetches both companies that are registered and unregistered. But we want only the services that belong to registered companies.

Adding a second stage to the aggregation pipeline

To filter the data from the first aggregation stage so that we have only services offered by registered companies, we can use the $match operator.

The $match operator filters documents based on a specified condition.

In Compass, add a second stage in the aggregation pipeline and select the $match operator.

{
  'registeredServices':{
    $elemMatch: {
      'isRegistered': true
    }
  }
}

What's going on?

We use the $match operator to target the array registeredServices. Then we use the $eleMAtch operator to specify a condition that we only need objects that have the isRegistered field set as true.

Run the pipeline in Compass and you'll see that we now have only 3 documents which represents services that are offered by registered companies.

Conclusion

In this article we learned how to implement a join in MongoDB using the $lookup operator. The $lookup operator performs a left-outer join and with it we can conduct analyses on data that span across different collections in a database.

Thank you for reading this far. Feel free to share your thoughts in the comments.

How to Subscribe to and Receive Push Notifications from YouTube’s API using Typescript and Nodejs

Chiamaka Ojiyi — Mon, 08 Nov 2021 16:25:11 +0000

Youtube’s API provides developers with a way to get push notifications for when specific changes occur on a youtube channel. Youtube’s API sends out these notifications through a PubSubHubbub webhook protocol.

Webhooks are used to send near-real-time data to applications.

Let’s say you have an application that sends out emails to a mailing list and you only want to send out these emails whenever Beyonce posts a video on her channel. To do this, you can use Youtube’s webhooks to subscribe to push notifications on Beyonce’s channel. Whenever Beyonce posts a video, Youtube will tunnel down a request to your server to inform you of this event.

In order to get the push notifications from Youtube’s webhook, you need to provide Youtube with a URL which you will use to receive the data for whenever a new video is uploaded on a specific channel. This URL is called a webhook call-back URL.

Youtube only sends out push notifications for the following events on a channel:

New video is uploaded
A video’s title is updated
A video’s description is updated

Youtube sends out the push notifications to your call-back URL in xml format.

In this tutorial, I’ll show you how you can set up a call-back server to subscribe to and receive push notifications from Youtube’s API using typescript and nodejs.

Project set up

In your project folder, run npm init -y in your terminal to create a package.json file

Run npx tsc --init to create a tsc file. This is a typescript configuration file.

In the script section of your package.json, add this bit of code for running the application.

"dev": "tsc",
"start": "ts-node index"

Install dependencies

Run npm i @types/express, dotenv, express, ts-node, typescript, youtube-notification to install dependencies.

Let's code the application

Create a file and name it index.ts.

import express from 'express';
import dotenv from "dotenv";
dotenv.config();

const YouTubeNotifier = require('youtube-notification');
const app = express();
const port = 6050;
const baseUrl = "https://strong-yak-75.loca.lt"
let channelId = process.env.CHANNEL_ID;

export const notifier = new YouTubeNotifier({
    hubCallback: `${baseUrl}/youtube/notifications`,
});

app.use("/youtube/notifications", notifier.listener());

app.listen(port, () => {
    console.log(`App listening at http://localhost:${port}`)

})

notifier.subscribe(channelId);

notifier.on('subscribe', data => {
    console.log('Subscribed');
    console.log(data);
});

notifier.on('notified', data => {
    console.log('New Video');
    console.log(data);
});

Create a .env file in the root of your project and add your youtube channel ID there.

Add the channel ID in this format:
CHANNEL_ID="YOUR_CHANNEL_ID_GOES_HERE"

To get the channel ID of a youtube channel, navigate to the channel on youtube. The browser address bar should have a URL in the format below.

https://www.youtube.com/channel/UC8butISFwT-Wl7EV0hUK0BQ

This UC8butISFwT-Wl7EV0hUK0BQ part is the channel ID.

To get the notifications from Youtube, your application will need to be deployed in order to have a https forwarding URL. Since we are still developing locally, we'll use a secondary service called Localtunnel.

Locatunnel allows us to easily share a web service on our local development machine with the world.

To start your server, go to your terminal and run:
npm start

While your server is running, open another terminal window and run:

npx localtunnel --port "ADD_YOUR_PORT_NUMBER_HERE"

A https URL will be printed on your terminal. Copy the URL and add it as the baseUrl in your index file. Do not close the terminal running localtunnel.

Restart your server to reflect the edit you just made. You should have two terminals up.

Now whenever a new video is uploaded to the specific youtube channel, the details of the video will be logged to the terminal where your serve is currently running on.

Some helpful debugging tips

You may need to diagnose your subscription to be sure you are subscribed or if errors occur when Youtube tries to send you push notifications.

To run this diagnosis, navigate to the PubSubHubBub page.

Go to the Subscriber Diagnostics section of the page. Add your call-back URL and the topic URL.

Topic URL:

https://www.youtube.com/xml/feeds/videos.xml?channel_id=YOUR_CHANNEL_ID

Click "Get Info" to run the diagnosis.

Finally, here's the Github repo for this article. You are welcome to fork and star the repo.

Thank you for reading.