DEV Community: Chetan Patil

Node.js Express Sequelize Boilerplate - Exploring the Directory Structure

Chetan Patil — Mon, 29 Jul 2024 18:25:43 +0000

Welcome to an engaging blog focused on building a robust and efficient Node.js RESTful service using Express, PostgreSQL, and Sequelize.

This blog will explore the fundamental concepts, techniques, and tools necessary to build a high-performing Node.js service. We will begin by laying the foundation for our RESTful service, covering topics such as setting up Express as our web framework, integrating PostgreSQL as the database, and leveraging Sequelize as our ORM for seamless database interactions.

Get ready to embark on an exciting journey as we kick off this blog with focus on creating a comprehensive boilerplate structure for a Node.js RESTful service using Express, PostgreSQL, and Sequelize.

Let’s start

Finding or creating a solid baseline repository structure was time-consuming and challenging when starting work on a new backend service. I struggled to locate proper resources, to begin with, leading me to create my own boilerplate code based on my learnings.

In this blog post, I aim to share and explain this boilerplate code structure to the best of my abilities, in the hope that it can assist others facing similar challenges.

Why folder structure is important?

Organization and maintainability: A well-defined folder structure keeps your codebase organized and makes it easier to manage and locate files related to specific functionalities or modules.
Code readability and understanding: A consistent folder structure enhances code readability and helps developers understand the architecture of the service, making it easier to navigate and locate relevant files.
Scalability and modularity: A well-thought-out folder structure supports the scalability and modularity of your service, allowing for easy addition of new features or modules without disrupting the existing codebase.
Reusability and code sharing: A logical folder structure promotes code reusability and sharing by providing a dedicated location for reusable components, utilities, or middleware that can be easily imported and utilized across different parts of the service.
Testing and debugging: A well-structured folder layout aids in organizing test files and resources, making it simpler to write and run tests against specific modules or functionalities. It also helps identify relevant files and their dependencies during debugging.
Collaboration and onboarding: A consistent folder structure promotes collaboration and facilitates onboarding of new team members by reducing confusion and enabling faster adaptation to the codebase. It provides a clear project layout that helps new developers find their way around the service.

Directory Structure

Here is a depiction of the initial directory structure:

src

The source directory contains the main application code.

app.js: The entry point of the application that initializes and starts the server.
server.js: Sets up and starts the Node.js server.

config

This directory holds configuration files for various aspects of the application.

controllers

Contains controller files that handle specific business logic for different routes.

middlewares

Holds middleware files responsible for handling specific tasks during request processing. Like, error-handler, generic response handler.

migrations

Contains database migration files that manage changes to the database schema. These files are generated via sequelize migration functionality.

models

Holds model files that define the database schema and interact with the database using Sequelize.

routes

Contains route files that define the API routes and their associated controller functions. These can include version based nested sub-directories and routes.

services

Contains service files that encapsulate complex business logic and interact with the models.

utils

Holds utility files that provide common functionalities used throughout the application. Like, graceful shutdown, etc.

validations

Contains validation schema files for request data validation.

tests

To ensure simplicity and easy navigation, we maintain the same file structure as the src directory. Additionally, we include a fixtures directory specifically designed to store test data required for executing tests.

Summary

This well-structured directory layout promotes code maintainability, scalability, and readability. By following this structure, developers can easily navigate the codebase, locate relevant files, and efficiently build and enhance the Node.js application.

Conclusion

In conclusion, the main emphasis of this article was on creating a solid foundation for Node.js services by implementing a boilerplate structure. The utilization of this boilerplate significantly minimizes the time and effort typically required to search for and construct the fundamental folder structure when starting a new service. Consequently, developers can allocate more resources and concentrate on crafting the essential business logic that is crucial for attaining the desired functionality. This streamlined approach enhances efficiency and allows for a more focused and productive development process.

“Thank you for reading this blog on creating a solid foundation for Node.js services. We hope you found the information helpful in streamlining your development process.”

The source code referred to in this article is available at the following GitHub repository: node-pg-sequelize-boilerplate.

Level Up Your Node.js Testing with Native Test Runner and Mocks: A BigQuery Example

Chetan Patil — Thu, 25 Jul 2024 17:59:50 +0000

In today's software development, testing is not just good practice, it's necessary, in my opinion.

I'm excited to share some valuable insights on testing in Node.js! Through my own experiences, I've discovered some practical tips and techniques that I believe will be helpful to the developer community.

Before Node.js 20, developers relied on external testing tools like Jest and Mocha to ensure code quality. But with the arrival of Node.js 20, the game changed. This tutorial will explore the exciting new world of Node.js's built-in test runner, exploring powerful features like mocks.

Native Node.Js Test Runner Offers:

Simplicity (Easy setup, no extra depenndency)
Integration (Seamless integration with Node.js features)
Efficiency (Fast execution)

"Node.js's native test runner simplifies ESM testing and mocking, making it a better choice over Jest for modern projects."

From Theory to Practice: Mocking BigQuery Calls in Node.Js

We covered fundamentals of Node.Js native test runner, now let's get our hands dirty with real world scenario.

In this practical example, we'll walk through how to leverage the node:test module and its mocking capabilities to thoroughly test a Node.js module responsible for interacting with Google BigQuery. We'll focus on the @google-cloud/bigquery library and demonstrate how to simulate its behavior, ensuring our code is resilient and functions correctly without making any real BigQuery calls during testing.

Creating BigQuery Service

Let's assume to have task about querying data from BigQuery dataset in stream mode.



import { BigQuery } from '@google-cloud/bigquery';

export const createBigQueryClient = () => new BigQuery();

const query = async ({ query, emailId, country }, bigquery = createBigQueryClient()) => {
  const output = [];

  const options = {
    query: query,
    params: {
      emailId,
      country,
    },
  };

  return new Promise((resolve, reject) => {
    bigquery.createQueryStream(options)
      .on('error', reject)
      .on('data', (row) => {
        output.push(row);
      })
      .on('end', () => {
        resolve(output); // Resolve the promise when the stream ends
      });
  });
}

export {
  query,
};

In this code, you must have observed export for createBigQueryClient.

Why Export createBigQueryClient?

Wondering why there is such simple function export:



export const createBigQueryClient = () => new BigQuery();

Here's the reasoning:

Testability: This function is key to easily mocking the BigQuery client in tests. By abstracting the creation of the client, it becomes easy to swap it out for a mock during testing.

Potential Configuration: While not used in this example, imagine needing to pass authentication option or other setting. Having this function makes such future cases easy for implementation without changing core login for query function.

Soften BigQuery with Mocks: The Tests

Testing bigquery.js service directly would require live BigQuery project running, which is costly and not always controllable. Here mocking comes into picture and Node.Js native test runner gives us that functionality in simplest way.

The Power of node:test



import assert from 'node:assert';
import { describe, it, beforeEach, after, mock } from 'node:test';
import { query } from './bigquery.js';

describe('BigQuery Module', () => {
  let mockBigQuery;

  beforeEach(() => {
    mockBigQuery = {
      createQueryStream: mock.fn(() => ({
        on: mock.fn(function (event, callback) {
          if (event === 'error' && mockBigQuery.error) {
            // @ts-ignore
            callback(mockBigQuery.error);
          } else if (event === 'data' && mockBigQuery.data) {
            mockBigQuery.data.forEach(callback); // Stream sends individual objects
          } else if (event === 'end') {
            // @ts-ignore
            callback();
          }
          return this; // Return 'this' for chaining

        }),
      })
      ),
    };
  });

  after(() => {
    mock.restoreAll();
  });

  const tests = {
    success: [
      {
        name: 'should execute a query and return results',
        input: {
          query: 'SELECT * FROM dataset.table WHERE emailid = @emailid AND country = @country',
          emailid: 'chetan@example.com',
          country: 'IN'
        },
        mockData: [{ id: 1, name: 'Chetan' }],
        expectedOutput: [{ id: 1, name: 'Chetan' }],
        expectedError: null,
      },
      {
        name: 'should handle empty result set',
        input: {
          query: 'SELECT * FROM `project.dataset.table` WHERE 1=2', // Always false condition
          emailid: 'random@example.com',
          country: 'CN',
        },
        mockData: [],
        expectedOutput: [],
        expectedError: null,
      },
    ],
    error: [
      {
        name: 'should reject the promise on BigQuery error',
        input: {
          query: 'INVALID SQL QUERY',
          emailId: 'ch@example.com',
          country: 'AT',
        },
        mockData: null,
        expectedOutput: null,
        expectedError: new Error('Simulated BigQuery Error'),
      },
    ],
  };

  tests.success.forEach((t) => {
    it(t.name, async () => {
      mockBigQuery.data = t.mockData;

      const result = await query(t.input, mockBigQuery);
      assert.deepStrictEqual(result, t.expectedOutput);
    });
  });

  tests.error.forEach((t) => {
    it(t.name, async () => {
      mockBigQuery.error = t.expectedError;

      await assert.rejects(async () => {
        await query(t.input, mockBigQuery);
      }, {
        message: t.expectedError.message,
      });
    });
  });
});

Key element

mock.fn(): A powerful tool from node:test to create mock functions. We use it to stub out the createQueryStream method of the BigQuery client.

Mock implementation of createQueryStream simulates data streaming and errors through callbacks.

"Using a table-driven test format helps minimize code duplication and efficiently manage new test scenarios."

Running the Tests

Executing tests is as simple as running the following command in terminal:



node --test

Node.js will automatically discover and execute test files withing project.
The output looks like:

Why Node.Js Native Test Runner?

Built-in Goodness: Embrace the simplicity of using Node.js's native testing capabilities.

Streamlined Workflow: No need for external dependencies, making your project setup cleaner.

Improved Readability: node:test encourages well-structured tests that are easy to understand and maintain.

Future-Proofing: Align your testing practices with the future of Node.js development.

Time to Test Smarter, Not Harder

This article aimed to introduce you to the Node.js native test runner and demonstrate how to write test functions with straightforward mocking, similar to other testing frameworks.

Give native testing a try in your next project and see how easy it is to write clean, efficient tests with built-in mocking. Your code will be more reliable and easier to maintain.

I'm passionate about learning, sharing and helping developers, so I've started a collection of Node.js native test runner examples at https://github.com/TheCodersStudio/node-native-test-runner.

Happy coding!

Automating Database Migrations with Flyway and GitHub Actions

Chetan Patil — Mon, 01 May 2023 12:28:26 +0000

Introduction

When building an application, the database is an integral part of the overall architecture. Over time, as requirements change and new features are added, database schema changes are inevitable. As a result, it becomes crucial to have a reliable and efficient way to manage database schema changes. This is where database migration tools come in.

This blog post will cover how to set up automated database migrations using Flyway and GitHub Actions.

Prerequisites

Certainly! Before getting started with Flyway and setting up automated database migrations using GitHub Actions, it's important to ensure that you have all the necessary prerequisites in place. Here's some more information on each of the prerequisites mentioned in the original post, along with some additional resources to help you get started:

Database

A relational database management system such as MySQL or PostgreSQL.

Flyway

Flyway is an open-source database migration tool that simplifies database schema management. It is a Java-based tool that can be easily integrated into your application's build process or deployed separately. Flyway tracks the evolution of your database schema over time, making it easier to manage changes and track updates.

It provides a simple yet powerful framework for managing database changes, allowing you to version and execute database scripts in a reliable and repeatable manner.

You can download Flyway from the official website: https://flywaydb.org/download/.

Here are some additional resources to help you get started with Flyway:

Flyway Documentation

Getting started

To begin, create a new directory with any name of your choosing, then create two sub-directories within it called conf and sql. This will serve as the working directory for your Flyway migration.

Example

mkdir flyway-migrator

cd flyway-migrator

mkdir conf sql

Setting up Flyway

After successfully downloading and installing Flyway, the next step is to configure it with your database connection details. This can be achieved by creating a configuration file named flyway.conf in the conf directory that you previously created.

To set up Flyway, follow these steps:

Open the flyway.conf file using a text editor of your choice.
Configure the following properties in the flyway.conf file:

Parameter	Mandatory	Description
`flyway.url`	Yes	the JDBC URL for your database
`flyway.user`	Yes	the username for your database
`flyway.password`	Yes	the password for your database
`flyway.schemas`	Yes	the schema(s) to apply migrations to
`flyway.baselineVersion`	No	the baseline version to use for existing databases

For more detailed information about the configuration options available in Flyway, you can refer to the official Flyway documentation at here.

Here's an example flyway.conf file:

flyway.url=jdbc:postgresql://localhost:5432/mydatabase
flyway.schemas=public
flyway.user=myuser
flyway.password=mypassword
flyway.validateMigrationNaming=true

Note that you'll need to replace mydatabase, myuser, and mypassword with the appropriate values for your database setup.

Once you've configured Flyway with your database connection details, you're ready to start using it to manage your database migrations. You can run Flyway using the flyway command line tool and specify the configuration file using the -configFiles option, which we will see in the next steps.

Running the migration

After you have configured Flyway with your database connection details and placed your migration scripts in the sql directory, you can run the migration using the flyway migrate command. However, to ensure that Flyway uses the correct configuration file, you need to specify the path to the flyway.conf file using the -configFiles option.

From the root directory of the cloned repository, execute the following command in the terminal:

flyway -configFiles="./conf/flyway.conf" migrate

This command will trigger Flyway to execute all the migration scripts located in the directory specified in the configuration file. Flyway will keep track of which scripts have already been executed and will only execute the new ones. Once the migration is complete, Flyway will update the schema version in the database to reflect the latest version of the schema.

Automation

After successfully setting up and configuring Flyway for local database migration, the next step is to automate this process. One way to achieve this is by utilizing GitHub Action workflows. With workflows, you can easily automate the migration process by defining a set of actions to be executed automatically, based on specified events, such as a commit or a pull request. By automating the migration process using workflows, you can ensure that every new version of your database schema is deployed consistently across all your environments. This eliminates the risk of manual errors and streamlines the deployment process. Additionally, workflows can be configured to run on a specific schedule, ensuring that the migration process is executed at the desired frequency.

GitHub Action

We will be utilizing the workflow_dispatch event, which allows for the manual triggering of a workflow directly from the GitHub Actions tab. This provides a convenient way to run workflows without needing to push any changes to the repository.

After adding the migration script to the sql directory in this repository, you can easily run the GitHub Action to execute those migrations in any location you specify.

The workflow allows the user to specify environment variables, such as the environment name, database host address, port number, and type of database (MySQL or PostgreSQL) as input parameters (You can customize these inputs as per your specific requirements.).

Example workflow:

name: Flyway Migrations

on:
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:
    inputs:
      env:
        description: 'Environment name'
        required: true
        default: 'dev'
        type: choice
        options:
        - dev
        - stage
        - prod
      dbHost:
        type: string
        description: 'Database host address'
        required: true
        default: 'localhost'
      dbPort:
        type: string
        description: 'Database host port number, i.e. MySQL 3306, PostgreSQL 5432'
        required: true
      dbType:
        description: 'Database type (MySQL/PostgreSQL)'
        required: true
        default: 'postgresql'
        type: choice
        options:
        - postgresql
        - mysql

The env section declares environment variables used throughout the job. The FLYWAY_VERSION specifies the version of Flyway to download, and DB_NAME specifies the name of the database to connect to.

name: Flyway Migrations

....
....
.
env:
  FLYWAY_VERSION: 9.16.3
  DB_NAME: postgres

The build job in our GitHub Action workflow consists of three steps.

First, the "Setup Flyway" step downloads and sets up Flyway on the system.
The next step, "Run Flyway Migrations", executes the Flyway migration command with the database connection details provided as environment variables.
Lastly, the "Flyway info and write result to file" step runs the Flyway info command to check the status of the migration and writes the result to a file in the migration-status directory. The filename includes the current date and time to make it unique. These steps can be modified to suit specific needs, such as adding additional migration steps or changing the format of the output file.

Here is an example of a complete GitHub Actions workflow file for automating Flyway database migrations:

name: Flyway Migrations

on:
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:
    inputs:
      env:
        description: 'Environment name'
        required: true
        default: 'dev'
        type: choice
        options:
        - dev
        - stage
        - prod
      dbHost:
        type: string
        description: 'Database host address'
        required: true
        default: 'localhost'
      dbPort:
        type: string
        description: 'Database host port number, i.e. MySQL 3306, PostgreSQL 5432'
        required: true
      dbType:
        description: 'Database type (MySQL/PostgreSQL)'
        required: true
        default: 'postgresql'
        type: choice
        options:
        - postgresql
        - mysql

env:
  FLYWAY_VERSION: 9.16.3
  DB_NAME: postgres

jobs:
  build:
    runs-on: ubuntu-latest

    # Service containers to run with `container-job`
    services:
      # Label used to access the service container
      postgres:
        # Docker Hub image
        image: postgres
        # Provide the password for postgres
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: postgres
        # Set health checks to wait until postgres has started
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          # Maps tcp port 5432 on service container to the host
          - 5432:5432

    steps:
    - uses: actions/checkout@v3

    - name: Setup Flyway
      run: |
        wget -qO- https://download.red-gate.com/maven/release/org/flywaydb/enterprise/flyway-commandline/${{ env.FLYWAY_VERSION }}/flyway-commandline-${{ env.FLYWAY_VERSION }}-linux-x64.tar.gz | tar -xvz && sudo ln -s `pwd`/flyway-${{ env.FLYWAY_VERSION }}/flyway /usr/local/bin 

    - name: Run Flyway Migrations
      env:
        FLYWAY_URL: jdbc:${{ inputs.dbType }}://${{ inputs.dbHost }}:${{ inputs.dbPort }}/${{ env.DB_NAME }}
        FLYWAY_USER: postgres
        FLYWAY_PASSWORD: postgres # ${{ secrets.DB_PASSWORD }}
        FLYWAY_SCHEMAS: flyway,public
      run: |
        flyway migrate

    - name: Flyway info and write result to file
      env:
        FLYWAY_URL: jdbc:${{ inputs.dbType }}://${{ inputs.dbHost }}:${{ inputs.dbPort }}/${{ env.DB_NAME }}
        FLYWAY_USER: postgres
        FLYWAY_PASSWORD: postgres # ${{ secrets.DB_PASSWORD }}
        FLYWAY_SCHEMAS: flyway,public
      run: |
        flyway info > ./migration-status/$(date -u +"%Y-%m-%d|%H:%M:%S")-${{ inputs.env }}.txt

    - name: Commit changes
      uses: EndBug/add-and-commit@v9
      with:
        add: './migration-status/*-${{ inputs.env }}.txt'

Finally, the Commit changes step commits the migration status file to the repository.

Here is the generated migration status file:

Flyway Community Edition 9.16.3 by Redgate
See release notes here: https://rd.gt/416ObMi

Database: jdbc:postgresql://localhost:5432/postgres (PostgreSQL 15.2)
Schema version: 1

+-----------+---------+------------------------------+--------+---------------------+---------+----------+
| Category  | Version | Description                  | Type   | Installed On        | State   | Undoable |
+-----------+---------+------------------------------+--------+---------------------+---------+----------+
|           |         | << Flyway Schema Creation >> | SCHEMA | 2023-05-01 11:35:23 | Success |          |
| Versioned | 1       | create person table          | SQL    | 2023-05-01 11:35:23 | Success | No       |
+-----------+---------+------------------------------+--------+---------------------+---------+----------+

WARNING: A Flyway License was not provided; fell back to Community Edition. Please contact sales at sales@flywaydb.org for license information.

In addition to the explanation provided here, you can find a complete working example of automating Flyway migrations using GitHub workflows in this GitHub repository.

By exploring this repository, you can gain a deeper understanding of how to automate your database migrations using Flyway and GitHub Actions, and how to customize the workflow to fit your specific needs.

Final words

In conclusion, automating database migrations using GitHub Actions can save a lot of time and effort in managing database changes. With this approach, developers can focus on writing code instead of manually running migrations. In addition, it provides an auditable trail of all migrations and ensures consistency across all environments.

This tutorial has provided a basic understanding of how to automate database migrations using Flyway and GitHub Actions. The provided example workflow can be used as a starting point and modified according to the specific needs of your project.

As a final note, it is important to remember that database migrations are a critical component of any project and should be handled with care. Always ensure that proper backups and testing are in place before running any migration.