DEV Community: Abeinemukama Vicent

Machine Learning in NodeJS || Part 1: TensorflowJS Basics

Abeinemukama Vicent — Sun, 28 Jan 2024 07:57:23 +0000

Machine learning is a subfield of artificial intelligence (AI) that focuses on the development of algorithms and statistical models that enable computers to perform tasks without being explicitly programmed. The primary goal of machine learning is to enable computers to learn and improve from experience.

There are three main types of machine learning:

Supervised Learning:

In supervised learning, the algorithm is trained on a labeled dataset, which means that the input data is paired with the corresponding correct output. The algorithm learns to map inputs to outputs, making predictions or classifications based on the patterns it identifies in the training data.

Unsupervised Learning:

Unsupervised learning deals with unlabeled data. The algorithm explores the data to find patterns, relationships, or structures without explicit guidance on what to look for. Clustering and dimensionality reduction are common tasks in unsupervised learning.

Reinforcement Learning:

Reinforcement learning involves an agent that learns to make decisions by interacting with an environment. The agent receives feedback in the form of rewards or punishments based on its actions, and it adjusts its strategy to maximize cumulative rewards over time.

Machine learning is applied in various domains, such as natural language processing, image recognition, speech recognition, recommendation systems, autonomous vehicles, and more. Common algorithms used in machine learning include linear regression, decision trees, support vector machines, neural networks, and deep learning models.
Deep learning, a subset of machine learning, involves the use of neural networks with multiple layers (deep neural networks). This approach has been particularly successful in tasks such as image and speech recognition.

In this article, we will discuss the basics of machine learning with TensorflowJS.
It is an open-source library developed by Google that allows machine learning models, particularly those built with TensorFlow, to run in web browsers or on Node.js. It enables developers to bring machine learning capabilities directly to the browser and NodeJS environments, allowing for client-side inference and real-time interactions. In this guide, We will concentrate on NodeJS not the browser.

This article is an intergral part of 3 a phase publication and in the second part we will train a large real world recommendation model for ecommerce products using TensorflowJS and in the third part, we will further discuss how to host and consume machine learning models including those trained with other technologies like Python's Tensorflow, PyTorch among others in a NodeJS API.

Terminologies Used in Machine Learning with TensorflowJS

Tensor:

A tensor is a multi-dimensional array, a fundamental data structure in TensorFlow.js. Tensors can be scalars (0D), vectors (1D), matrices (2D), or have higher dimensions. They are the building blocks for data representation in machine learning models.

Model:

In machine learning, a model is a representation of a system or a pattern learned from data. In TensorFlow.js, models can be created using various architectures, such as neural networks.

Neural Network:

A neural network is a computational model inspired by the human brain. It consists of layers of interconnected nodes (neurons) that process input data to produce an output. TensorFlow.js supports the implementation of neural networks for various machine learning tasks.

Layers:

In the context of neural networks, layers are the building blocks that organize and process data. TensorFlow.js provides a layer API for constructing neural network architectures.

Training:

Training is the process of adjusting the parameters of a machine learning model using labeled data to minimize the difference between predicted and actual outputs. TensorFlow.js facilitates the training of models using optimization algorithms.

Inference:

Inference is the process of using a trained model to make predictions on new, unseen data. TensorFlow.js allows for efficient inference in real-time within web browsers or Node.js environments.

Loss Function:

The loss function measures the difference between the predicted output and the actual target values during training. TensorFlow.js provides various loss functions that can be used for different types of machine learning problems.

Optimizer:

An optimizer is an algorithm used during the training phase to minimize the loss function by adjusting the model's parameters. TensorFlow.js offers a variety of optimizers, including stochastic gradient descent (SGD) and Adam.

Epoch:

An epoch is one complete pass through the entire training dataset during the training phase. Training typically involves multiple epochs to improve the model's performance.

Batch Size:

During training, data is processed in batches rather than the entire dataset at once. Batch size refers to the number of data samples used in each iteration of the training process. TensorFlow.js allows developers to set batch sizes when training models.

Validation Data:

A separate set of data used during training to evaluate the model's performance. TensorFlow.js enables the use of validation data to monitor and prevent overfitting.

Regression:

Regression is a type of supervised learning where the goal is to predict a continuous numerical output. In TensorFlow.js, regression models can be created using neural networks or other algorithms, and the training process involves minimizing the difference between predicted and actual numerical values.

Classification:

Classification is another type of supervised learning where the goal is to categorize input data into predefined classes or labels. TensorFlow.js supports classification tasks, and neural networks are often used for this purpose.

Testing Data (Test Set):

Testing data, also known as a test set, is a separate portion of the dataset that is not used during the training phase. It is reserved for evaluating the model's performance after training. TensorFlow.js allows developers to assess how well a model generalizes to new, unseen data using a test set.

Validation Data:

Validation data is a subset of the dataset used during training to evaluate the model's performance and tune hyperparameters. It is distinct from testing data, which is reserved for final evaluation after training.

Overfitting:

Overfitting occurs when a machine learning model performs well on the training data but fails to generalize to new, unseen data. TensorFlow.js provides techniques, such as regularization and dropout, to help mitigate overfitting.

Underfitting:

Underfitting happens when a model is too simple to capture the underlying patterns in the data. It leads to poor performance on both the training and testing datasets. Adjusting the model's complexity or using more advanced architectures in TensorFlow.js can help address underfitting.

Hyperparameters:

Hyperparameters are configuration settings that are not learned from the data but are set before the training process. Examples include learning rate, batch size, and the number of layers in a neural network. Tuning hyperparameters is an important aspect of optimizing model performance in TensorFlow.js.

Activation Function:

Activation functions introduce non-linearity to the neural network, allowing it to learn complex patterns. TensorFlow.js provides various activation functions, such as ReLU (Rectified Linear Unit), Sigmoid, and Tanh, that can be applied to the layers of a neural network.

Data Preprocessing:

Data preprocessing involves preparing and cleaning the data before feeding it into a machine learning model. TensorFlow.js provides tools for standardizing, normalizing, and transforming data to improve model performance.

Callback:

In TensorFlow.js, a callback is a function that is executed at specific points during the training process. Callbacks can be used to perform tasks like saving model checkpoints, adjusting learning rates, or logging training metrics.

Prerequisites

Before we embark on the journey of learning TensorflowJS basics, it's essential to ensure that your development environment is properly configured. This section outlines the prerequisites that you need to have in place before further diving into article.

Node.js and npm:

Have Node.js and npm (Node Package Manager) installed on your system. These are essential for training our models in a NodeJS environment. You can download Node.js from the official website or use a version manager like nvm for better control over Node.js versions.

Text Editor or IDE:

Choose a text editor or integrated development environment (IDE) for writing your TypeScript/JavaScript code. Popular choices include Visual Studio Code, Atom, or any editor of your preference.

JavaScript Basics

You should have basic understanding of JavaScript programming language, especially its most common data structures, arrays and objects, and their methods.

Basic Knowledge of Algorithm Scripting

Additionally, you should have some basic knowledge of algorithm scripting with JavaScript and familiar with optimising algorithms to run efficiently with a justifiable time and space complexity.

Environment setup for Machine Learning with TensorflowJS in NodeJS

In this article, we will discuss the basics of TensorflowJS by training basic and beginner friendly machine learning models.

Create a new folder in your desired location and open it with your desired code editor, in my case, I called it: nodejs_with_tensorflowjs.
Initialise a new nodejs project with the following command:



npm init -y

Step 2: Install TensorflowJS

Install the tensorflowjs with the following command:



npm install @tensorflow/tfjs

All set, create a new file: src/index.js in the root directory.
Before we start training our first AI model, lets first further discuss one of the previously discussed terminologies as its the basement for our first and second examples, by names of regression.
Like we ealier discussed, it is a type of supervised machine learning task where the goal is to predict a continuous numerical output based on input features. In other words, regression models are designed to establish a relationship between the input variables and a continuous target variable. The predicted output is a real-valued quantity, and the model aims to capture the underlying patterns in the data to make accurate predictions.

Key Concepts Related to Regression in ML with TensorflowJS:

Target Variable (Dependent Variable):

The variable that the model aims to predict. In regression, the target variable is continuous and numerical.

Features (Independent Variables):

The input variables or features that are used to make predictions for the target variable. These can be one or more variables that contribute to the prediction.

Linear Regression:

Linear regression is a common regression technique where the relationship between the input features and the target variable is modeled as a linear equation. The goal is to find the best-fitting line that minimizes the difference between the predicted values and the actual values.

Multiple Linear Regression:

In multiple linear regression, there are multiple independent variables influencing the target variable. The relationship is modeled as a linear combination of these variables.

Polynomial Regression:

Polynomial regression extends linear regression by allowing the relationship between variables to be modeled as a polynomial. This is useful when the underlying relationship is not linear.

Mean Squared Error (MSE):

MSE is a common metric used to evaluate the performance of regression models. It measures the average squared difference between the predicted values and the actual values. The goal during training is to minimize the MSE.

Training and Testing Data:

The dataset is typically divided into training and testing sets. The model is trained on the training set, and its performance is evaluated on the testing set to assess how well it generalizes to new, unseen data.

Residuals:

Residuals are the differences between the predicted values and the actual values in the testing set. Analyzing residuals helps in understanding how well the model captures the patterns in the data.

Coefficient and Intercept:

In linear regression, the coefficients represent the weights assigned to each feature, indicating their impact on the target variable. The intercept is the y-intercept of the regression line.

Regularization (L1 and L2):

Regularization techniques, such as L1 (Lasso) and L2 (Ridge), can be applied to regression models to prevent overfitting. They add penalty terms to the loss function, discouraging overly complex models.

Linear Regression Example

Scenario:

Imagine you have data points representing the number of hours students spend studying (x) and their corresponding exam scores (y).

Objective:

Build a model that can predict the exam score (y) based on the number of hours studied (x).

Model:

A simple linear regression model assumes that the relationship between hours studied and exam score is linear. It tries to fit a line to the data.

Equation:

The linear regression equation is typically represented as y = mx + b, where m is the slope of the line, and b is the y-intercept.

Training:

During training, the model adjusts the values of m and b to best fit the training data.

Prediction:

Once trained, the model can predict the exam score for a given number of hours studied.

Now that we have a basic understanding of linear regression, lets have our scenario practical by training the described model.

In the created file: src/index.js, place the following code:



const tf = require("@tensorflow/tfjs");

// Define your model and training logic here
const model = tf.sequential();
model.add(tf.layers.dense({ units: 1, inputShape: [1] }));
model.compile({ optimizer: "sgd", loss: "meanSquaredError" });

// Example data 
const xs = tf.tensor2d([1, 2, 3, 4], [4, 1]); // Features (hours of study)
const ys = tf.tensor2d([2, 4, 6, 8], [4, 1]); // Labels (exam scores)


// Train the model
model.fit(xs, ys, { epochs: 100 }).then(() => {
  // Model is trained
  model.predict(tf.tensor2d([5], [1, 1])).print();
});

Explanation

We have data representing the number of hours students spend studying (x) and their corresponding exam scores (y).
The input data, often referred to as features (x), represents the independent variable, in this case, the number of hours students spend studying.



const xs = tf.tensor2d([1, 2, 3, 4], [4, 1]);

The xs tensor represents a 2D array with 4 rows (samples) and 1 column. Each row corresponds to a different student, and the column contains the number of hours that student spent studying.

1 hour for the first student
2 hours for the second student
3 hours for the third student
4 hours for the fourth student

On the other hand, the output data, often referred to as labels (y), represents the dependent variable, in this case, the exam scores corresponding to the hours of study.



const ys = tf.tensor2d([2, 4, 6, 8], [4, 1]);

The ys tensor represents a 2D array with 4 rows (samples) and 1 column. Each row corresponds to a different student, and the column contains the corresponding exam score.

Exam score 2 for the first student (who studied 1 hour)
Exam score 4 for the second student (who studied 2 hours)
Exam score 6 for the third student (who studied 3 hours)
Exam score 8 for the fourth student (who studied 4 hours)

Relationship and Objective:

The goal of our linear regression model is to learn the relationship between the number of hours studied (x) and the corresponding exam scores (y).

During training, the model adjusts its parameters (slope and y-intercept of the line) to minimize the difference between its predictions and the actual exam scores.

Once trained, the model can predict the exam score for a new student based on the number of hours that student spent studying.

Linear Regression Equation:

In the context of linear regression, the relationship is often represented by the equation y = mx + b, where:
y is the predicted output (exam score),
x is the input feature (hours of study),
m is the slope of the line, and
b is the y-intercept.
The model's goal during training is to find the values of m and b that best fit the training data.

Training and Prediction:

During training, the model adjusts m and b to minimize the difference (error) between its predictions and the actual exam scores.

After training, the model can make predictions for new students. For example, given a new student who studied 5 hours:



const testInput = tf.tensor2d([5], [1, 1]);
const prediction = model.predict(testInput);
prediction.print();

The output of this prediction would be the model's estimate of the exam score for a student who studied 5 hours.

Lets now run our code:
Add the following script to the scripts section in your package.json:



"start": "node src/index.js"

On the terminal, run:



npm start

You should see the following in terminal:

Indicating that our model predicted the value: 9.7176924 to be the exam score for our test student who studied for 5 hours. Value is close to 10 so our model has surely tried since from our test data, the exam score was always double the hours studied.
Note that the larger the training data, the more accurate the model and the longer the training duration.

Example 2: Linear Regression for House Prices

Imagine a dataset with three features: size of the house, number of bedrooms, and distance to the city center.
We'll generate a larger synthetic dataset using random values and train a model for predicting house prices basing on the 3 features.

In the previous example, we used @tensorflow.js/tfjs, however, this library uses pure JavaScript implementations for operations, running on the CPU and is slower due to JavaScript-based execution. However, it runs both in web browsers and Node.js environments.
Since our concentration in this article is NodeJS, we will use a faster TensorflowJS library optimized to run only in NodeJS environments by names of @tensorflow/tfjs-node .
It leverages native TensorFlow C++ bindings for faster execution and is significantly faster than @tensorflow/tfjs due to native code execution and can utilize GPU acceleration with CUDA (or Compute Unified Device Architecture), a proprietary and closed-source parallel computing platform and application programming interface (API) that allows software to use certain types of graphics processing units (GPUs) for general-purpose processing), (if available).
Note that,if youre into high-performance ML tasks in Node.js, its recommended you use @tensorflow/tfjs-node-gpu as it is significantly faster (with GPU) compared to both @tensorflow/tfjs-node and @tensorflow/tfjs, and provides exceptional GPU support.
However, it requires NVIDIA GPU with CUDA, is not officially supported on macOS and also has many experimental features on windows OS despite the fact that its already stable on Linux OS.
In this guide, we will be proceeding with @tensorflow/tfjs-node

On the terminal,run the following command:



npm install @tensorflow/tfjs-node

Create a new file: src/train.js and place the following code:



const tf = require("@tensorflow/tfjs-node");

// Generate a synthetic dataset
const generateDataset = (numSamples) => {
  const sizes = Array.from(
    { length: numSamples },
    () => Math.random() * 3000 + 500
  );
  const bedrooms = Array.from(
    { length: numSamples },
    () => Math.floor(Math.random() * 5) + 1
  );
  const distances = Array.from(
    { length: numSamples },
    () => Math.random() * 20
  );

  const prices = sizes.map(
    (size, i) =>
      1000 * size +
      20000 * bedrooms[i] -
      1000 * distances[i] +
      Math.random() * 10000
  );

  return { sizes, bedrooms, distances, prices };
};

const numSamples = 1000;
const { sizes, bedrooms, distances, prices } = generateDataset(numSamples);

// Store the generated dataset for inspection
const datasetForInspection = { sizes, bedrooms, distances, prices };

// Create tensors from the data
const xs = tf
  .tensor2d([sizes, bedrooms, distances], [3, numSamples])
  .transpose();
const ys = tf.tensor2d(prices, [numSamples, 1]);

// Model architecture
const model = tf.sequential();
model.add(tf.layers.dense({ units: 64, inputShape: [3], activation: "relu" }));
model.add(tf.layers.dense({ units: 1 }));

model.compile({ optimizer: "adam", loss: "meanSquaredError" });

// Training
model.fit(xs, ys, { epochs: 50 }).then(() => {
  // Model is trained
  // You can use the trained model to make predictions

  const newHouseFeatures = tf.tensor2d([[1500, 3, 10]]);
  const prediction = model.predict(newHouseFeatures);
  console.log("Predicted Price:", prediction.dataSync()[0]);
});

Explanation

Before we run our code,lets first understand it line by line:

Step 1: Import TensorFlow.js:



const tf = require('@tensorflow/tfjs-node');

Step 2: Generate Synthetic Dataset:

Define a function generateDataset to create a synthetic dataset with random values for sizes, bedrooms, distances, and corresponding prices.
Generate a dataset with 1000 samples and store it in variables.



const generateDataset = (numSamples) => {
  // ... (Randomly generate sizes, bedrooms, distances, and prices)
};
const numSamples = 1000;
const { sizes, bedrooms, distances, prices } = generateDataset(numSamples);

This dataset will be replaced with actual training data and is randomly generated for now since we donot have a prepared dataset for training our model.

Step 3: Store Dataset for Inspection:

Store the generated dataset in a variable for inspection. You can log this variable to see the generated data.



const datasetForInspection = { sizes, bedrooms, distances, prices };

Step 4: Create Tensors from Data



const xs = tf.tensor2d([sizes, bedrooms, distances], [3, numSamples]).transpose();
const ys = tf.tensor2d(prices, [numSamples, 1]);

Create TensorFlow tensors from the features (xs) and labels (ys). The transpose() is used to reshape the data.

Creating Input Tensor (xs):

tf.tensor2d: This creates a 2D tensor. The first argument is an array containing the data, and the second argument is the shape of the tensor.
The input data for the features (sizes, bedrooms, distances) is provided as an array [sizes, bedrooms, distances].
The shape of the tensor is specified as [3, numSamples] because there are three features (size, bedrooms, distances) for each of the numSamples houses.
.transpose(): This transposes the tensor, swapping its dimensions. The original shape was [3, numSamples], and after transposing, it becomes [numSamples, 3]. This is a very common step to make the data compatible with the model's input shape.

Creating Output Tensor (ys):

tf.tensor2d: Similar to the input tensor creation, this creates a 2D tensor for the output (prices).
The first argument is an array containing the output data (prices).
The shape of the tensor is [numSamples, 1], indicating that there is one output value (price) for each of the numSamples houses.

Step 5: Model Architecture:



const model = tf.sequential();
model.add(tf.layers.dense({ units: 64, inputShape: [3], activation: 'relu' }));
model.add(tf.layers.dense({ units: 1 }));

We define a sequential model with two dense layers. The first layer has 64 units and uses the ReLU activation function. The second layer has 1 unit.

Step 6: Compile the Model



model.compile({ optimizer: 'adam', loss: 'meanSquaredError' });

Compile the model with the Adam optimizer and mean squared error loss function

Step 7: Use Trained Model to Make Prediction

Once the model is trained, you can use it to make predictions. For example, predict the price for a new house with features [1500, 3, 10]:



model.fit(xs, ys, { epochs: 50 }).then(() => {
  // Model is trained
  // You can use the trained model to make predictions
  const newHouseFeatures = tf.tensor2d([[1500, 3, 10]]);
  const prediction = model.predict(newHouseFeatures);
  console.log('Predicted Price:', prediction.dataSync()[0]);

});

This code creates a tensor for new house features, passes it to the trained model, and logs the predicted price.
In the terminal, run:



npm start

and you should have a terminal with the following ending:

Indicating that, our model predicts 200036.6875 to be the price for a new house with features [1500, 3, 10].

Time Complexity of generateDataset() Function

The time complexity of the generateDataset function, which creates a synthetic dataset with random values for sizes, bedrooms, distances, and prices, is primarily determined by the number of samples (numSamples). Let's analyze the time complexity:



const generateDataset = (numSamples) => {
  const sizes = Array.from({ length: numSamples }, () => Math.random() * 3000 + 500);
  const bedrooms = Array.from({ length: numSamples }, () => Math.floor(Math.random() * 5) + 1);
  const distances = Array.from({ length: numSamples }, () => Math.random() * 20);

  const prices = sizes.map((size, i) => 1000 * size + 20000 * bedrooms[i] - 1000 * distances[i] + Math.random() * 10000);

  return { sizes, bedrooms, distances, prices };
};

Generating Random Values:

The sizes, bedrooms, and distances arrays are generated using Array.from() with a length of numSamples.
The time complexity for each of these arrays is O(numSamples).

Generating Prices:

The prices array is generated using the map function. The time complexity of the map function is O(numSamples).
Inside the map function, each element is calculated using basic arithmetic operations, which are constant time operations.

The overall time complexity of the generateDataset function is dominated by the generation of sizes, bedrooms, and distances, making it O(numSamples), more clearly, 0(n) that is linear time complexity.
Pretty good, but keep in mind that our analysis assumes that basic arithmetic operations are considered constant time, which is a reasonable assumption for typical use cases. If you have a specific requirement for precision or if you are dealing with extremely large datasets, further considerations may be necessary. Additionally, the random number generation itself may have a time complexity that depends on the specific implementation of the random number generator.
Possibly, the time complexity can be linearly proportional to the number of samples (numSamples). If you have a large dataset, the generation time will scale linearly with the size of the dataset.

Training ML Model on Text Data (Sentiment Analysis Example)

Our previous 2 examples have all been using a dataset with numeric data, however, at times, you will have text data or a mix of both to train your AI model. Lets take a last example using text based data.

We'll build a simple model to classify text into two categories: positive and negative. This is often referred to as sentiment analysis. However, in our example, tone of the message/text will either be positive or negative.
Often in sentiment analysis, digital text is analyzed to determine if the emotional tone of the message is positive, negative, or neutral.
Create a new file: src/textdata.js and place the following code:



const tf = require("@tensorflow/tfjs-node");

// Given data
const textData = ["good", "bad", "happy", "sad", "positive", "negative"];
const labels = [1, 0, 1, 0, 1, 0]; // 1: Positive, 0: Negative

// Preprocess data
const tokenize = (text) => text.toLowerCase().split(" ");

const maxSeqLength = textData.reduce(
  (max, text) => Math.max(max, tokenize(text).length),
  0
);

const padSequence = (sequence, maxLength) => {
  return Array.from({ length: maxLength - sequence.length }, () => 0).concat(
    sequence
  );
};

const xData = textData.map((text) => {
  const tokens = tokenize(text);
  const sequence = tokens.map(
    (token) => token.charCodeAt(0) - "a".charCodeAt(0) + 1
  ); // Simple character encoding
  return padSequence(sequence, maxSeqLength);
});

const yData = tf.tensor1d(labels);

// Define the model
const model = tf.sequential();
model.add(
  tf.layers.embedding({ inputDim: 27, outputDim: 8, inputLength: maxSeqLength })
);
model.add(tf.layers.flatten());
model.add(tf.layers.dense({ units: 1, activation: "sigmoid" }));

model.compile({
  optimizer: tf.train.adam(),
  loss: "binaryCrossentropy",
  metrics: ["accuracy"],
});

// Convert data to tensors
const xTrain = tf.tensor2d(xData);
const yTrain = yData;

// Train the model
model.fit(xTrain, yTrain, { epochs: 100 }).then((history) => {
  console.log("Training complete.");

  // Test the model with custom data
  const customTestData = ["awesome", "terrible", "excited", "disappointed"];
  const xCustomTestData = customTestData.map((text) => {
    const tokens = tokenize(text);
    const sequence = tokens.map(
      (token) => token.charCodeAt(0) - "a".charCodeAt(0) + 1
    );
    return padSequence(sequence, maxSeqLength);
  });

  const xCustomTest = tf.tensor2d(xCustomTestData);

  // Make predictions
  const predictions = model.predict(xCustomTest);
  const results = predictions.arraySync();

  console.log("Custom Test Data Predictions:", results);
});

Explanation:

Import the TensorFlow.js Node.js bindings.



const tf = require("@tensorflow/tfjs-node")

Define an array of text data and corresponding labels indicating positive (1) or negative (0) sentiment.



const textData = ["good", "bad", "happy", "sad", "positive", "negative"];
const labels = [1, 0, 1, 0, 1, 0]; // 1: Positive, 0: Negative

Define a function (tokenize) to convert text to lowercase and split it into tokens. Determine the maximum sequence length in the dataset.



const tokenize = (text) => text.toLowerCase().split(" ");
const maxSeqLength = textData.reduce(
  (max, text) => Math.max(max, tokenize(text).length),
  0
);

Define a function (padSequence) to pad sequences with zeros to ensure uniform length.



const padSequence = (sequence, maxLength) => {
  return Array.from({ length: maxLength - sequence.length }, () => 0).concat(
    sequence
  );
};

Convert text data to numerical sequences by mapping each text to a numerical sequence using a simple character encoding and pad the sequences to the maximum length.



const xData = textData.map((text) => {
  const tokens = tokenize(text);
  const sequence = tokens.map(
    (token) => token.charCodeAt(0) - "a".charCodeAt(0) + 1
  ); // Simple character encoding
  return padSequence(sequence, maxSeqLength);
});

Create labels tensor by converting the labels array into a TensorFlow tensor.



const yData = tf.tensor1d(labels);

Create a sequential model with an embedding layer, a flatten layer, and a dense layer with a sigmoid activation function.



const model = tf.sequential();
model.add(
  tf.layers.embedding({ inputDim: 27, outputDim: 8, inputLength: maxSeqLength })
);
model.add(tf.layers.flatten());
model.add(tf.layers.dense({ units: 1, activation: "sigmoid" }));

Compile the model
Specify the optimizer, loss function, and metrics for model training.



model.compile({
  optimizer: tf.train.adam(),
  loss: "binaryCrossentropy",
  metrics: ["accuracy"],
});

Convert input data (xData) and labels (yData) to TensorFlow tensors.



const xTrain = tf.tensor2d(xData);
const yTrain = yData;

Train the model using the training data (xTrain and yTrain) for 100 epochs.



model.fit(xTrain, yTrain, { epochs: 100 }).then((history) => {
  console.log("Training complete.");
});

Test the model with custom data
Create sequences for custom test data, convert them to a tensor, make predictions using the trained model, and log the results.



const customTestData = ["awesome", "terrible", "excited", "disappointed"];
const xCustomTestData = customTestData.map((text) => {
  const tokens = tokenize(text);
  const sequence = tokens.map(
    (token) => token.charCodeAt(0) - "a".charCodeAt(0) + 1
  );
  return padSequence(sequence, maxSeqLength);
});

const xCustomTest = tf.tensor2d(xCustomTestData);

const predictions = model.predict(xCustomTest);
const results = predictions.arraySync();

console.log("Custom Test Data Predictions:", results);

Update the index file to point to the our new file and run:



npm start

You should have the following terminal:

In a binary classification problem (such as sentiment analysis with positive/negative classes), a common threshold is 0.5. You can interpret the predictions by considering values above 0.5 as predicting positive sentiment and values below 0.5 as predicting negative sentiment. In our case, the model's predictions are close to the threshold, indicating some uncertainty in the classification.
This is because our training data/dataset was too small for it to confidently predict.

Here is summary explanation:
[
0.4822129011154175,
0.5223260521888733,
0.5149989724159241,
0.4970185160636902
] === ["awesome", "terrible", "excited", "disappointed"]
Given the little dataset we trained our model on,it has tried.
As earlier discussed, the larger the training data/dataset, the more accurate the model and the longer the training period.

Conclusion

In conclusion, we explored the fundamentals of machine learning using TensorFlowJS, showcasing its versatility through three distinct examples. We delved into the basics with two numerical datasets, demonstrating the process of training models to make predictions. Additionally, we ventured into the realm of natural language processing by employing TensorFlow.js to create a sentiment analysis model based on text data.

Through these examples, we've witnessed how TensorFlowJS empowers developers and data scientists to implement machine learning solutions across various domains. From numerical predictions to text-based sentiment analysis, TensorFlowJS's flexibility and ease of use make it a powerful tool for anyone seeking to leverage machine learning in their NodeJS projects. As the field of machine learning continues to evolve, TensorFlowJS remains at the forefront, providing a robust framework for both beginners and seasoned practitioners to explore and implement cutting-edge solutions. Whether you're working with numerical data or engaging in the complexities of natural language processing, TensorFlowJS stands as a reliable companion in the exciting journey of machine learning in NodeJS environments.

Until next time, happy coding!

Helpful Links:

How to Build & Deploy Scalable Microservices with NodeJS, TypeScript and Docker || A Comprehesive Guide

Abeinemukama Vicent — Thu, 25 Jan 2024 02:23:07 +0000

In the ever-evolving landscape of software development, the paradigm of microservices has emerged as a game-changer, offering unparalleled scalability, flexibility, and resilience. As organizations strive to build and deploy applications that can seamlessly adapt to the dynamic demands of the digital era, the combination of Node.js, TypeScript, Docker, and other cutting-edge technologies has become a beacon of innovation. In this technological odyssey, we embark on a journey to unravel the intricacies of microservices architecture, exploring the potent synergy of Node.js's asynchronous prowess, the type safety of TypeScript, and the containerization magic of Docker. This article delves deep into the art of crafting microservices that not only elevate the performance and efficiency of applications but also empower developers to navigate the complexities of modern software development with finesse. We will navigate through the realms of next-level architecture, where Node.js, TypeScript, and Docker converge to redefine the way we conceptualize and construct scalable and resilient software systems by building and deploying 4 microservices for an ecommerce application(Auth, Products, Orders and Notifications) using various technologies in the NodeJS ecosystem.

What Exactly are Microservices

Microservices is an architectural style that structures an application as a collection of small, independent services, each focused on a specific business capability. These services can be developed, deployed, and scaled independently, allowing for greater flexibility and agility in software development.
Big companies like Netflix, Amazon, Ebay, Spotify among others have embraced and use microservice architecture, which enables rapid scaling, seamless updates, and the introduction of new features with ease among others.
In comparison with monolithic architecture, a monolithic application is built as a single unified unit using a single code base and framework.
the choice between monolithic and microservices approaches represents a fundamental decision that significantly influences the development, scalability, and maintainability of a software application. The monolithic architecture, characterized by a single, tightly integrated codebase, contrasts with the microservices architecture, where applications are decomposed into small, independent services. Each approach brings its own set of advantages and challenges, impacting factors such as scalability, development speed, and team autonomy. In this comparison, we delve into the key characteristics, pros, and cons of both monolithic and microservices architectures to help illuminate the considerations guiding the architectural decisions made by software development teams.

Microservices Vs Monolithic Architecture(Key Technicalities)

Communication Between Microservices

In microservices architecture, communication between services is a critical aspect. There are various ways to facilitate communication between microservices in NodeJS.
In this artcle we will discuss the 7 most common ones and use the first 2 on a real world project(ecommerce application) that we will build.

HTTP/RESTful APIs:

This is one of the most common and widely used methods of communication between microservices. Each microservice exposes a set of HTTP endpoints (RESTful APIs) that other services can call to request or send data.

Pros:

Simple, widely adopted, and works well in a stateless environment.

Cons:

Synchronous and can introduce latency.
Message Brokers:

Here, microservices communicate by sending messages to a central message broker (e.g., RabbitMQ, Apache Kafka). Services can subscribe to specific topics or queues to receive messages and react accordingly.

Mode of Operation:

Message brokers are designed for decoupling components in a distributed system by allowing them to communicate through messages.
Producers publish messages to a queue or topic, and consumers subscribe to receive these messages.
Brokers manage the routing, delivery, and storage of messages, ensuring that messages are reliably processed even if consumers are temporarily unavailable.

RabbitMQ comes with administrative tools to manage user permissions and broker security and is perfect for low latency message delivery and complex routing.
In comparison, Apache Kafka architecture provides secure event streams with Transport Layer Security(TLS) and is best suited for big data use cases requiring the best throughput.

Pros:

Asynchronous, decouples services, supports event-driven architectures.

Cons:

Complexity, potential for message loss (depending on the broker).

gRPC (Remote Procedure Call):

gRPC is a high-performance, open-source RPC (Remote Procedure Call) framework initially developed by Google. It uses Protocol Buffers for serialization and supports bidirectional streaming.

Pros:

Efficient, supports multiple programming languages, bi-directional communication.

Cons:

More complex setup compared to REST, may not be as widely adopted in all scenarios.
GraphQL:

A query language for APIs that allows clients to request only the data they need. It provides a more flexible and efficient alternative to RESTful APIs.
GraphQL provides a single endpoint for multiple data sources, making it efficient for clients to retrieve only the necessary information forexample if a Node.js microservice uses GraphQL, a client can send a query to request specific data, and the GraphQL service will fetch the required information from the underlying data sources and respond accordingly.

Pros:

Reduces over-fetching of data, allows clients to define the shape of the response.

Cons:

Complexity in setting up, may not be suitable for all use cases.
WebSocket:

Websockets provide full-duplex communication channels over a single, long-lived connection, allowing real-time communication between microservices.
It is an advanced technology and makes it possible to open a two-way interactive communication session between the user's browser and a server. You can send messages to a server and receive event-driven responses without having to poll the server for a reply.
WebSockets are suitable for scenarios where low-latency, bidirectional communication is crucial, such as chat applications, real-time updates, or collaborative editing tools.

Pros:

Real-time communication, low-latency updates.

Cons:

May not be suitable for all use cases, increased complexity.
Service Mesh (e.g., Istio):

It is a dedicated infrastructure layer that manages service-to-service communication, providing features like load balancing, encryption, authentication, and monitoring.
Istio deploys sidecar proxies alongside each microservice instance. These proxies handle communication, providing features like load balancing, service discovery, encryption, monitoring and authentication.

Pros:

Centralized control over communication, observability.

Cons:

Adds complexity to the infrastructure.
Distributed Databases:

Services communicate indirectly through a shared database. Each microservice has its own database, and they interact by reading and writing to the shared database.

Pros:

Simple, easy to implement.

Cons:

Tight coupling, potential for data inconsistency, may not scale well.

The choice of communication method depends on various factors such as the nature of the application, scalability requirements, latency considerations, and the team's expertise. Often, a combination of these methods is used within a microservices architecture to meet different communication needs.
Also, depending on the team's expertise, and other core influential factors, you can decide to use only one method of communicating microservices say HTTP/RESTful API, its totally fine regardless of the technologies/programming languages used to build the various services.
In this article, we will use a combination of HTTP/RESTful apis and RabbitMQ(message broker) to communicate the 4 microservices for our ecommerce application.

Project Synopsis

This article presents a hands-on exploration of architecting a robust and scalable e-commerce application using microservices architecture.
The four meticulously crafted microservices include an Authentication service (using MongoDB and mongoose), a Product Management service (using PostgreSQL and Prisma), an Order Processing service (using MySQL and Sequelize), and a real-time Notifications service (using GraphQL, MongoDB and mongoose). The Authentication microservice ensures secure user management through MongoDB and Mongoose, while the Product Management service adopts PostgreSQL and Prisma for efficient data handling. The Order Processing microservice, powered by MySQL and Sequelize, orchestrates seamless order fulfillment. To enable real-time notifications, the Notifications microservice capitalizes on the power of GraphQL, exploiting its built-in mechanisms for efficient and scalable real-time interactions. Each microservice will be built using the Express framework, and represents a unique technical challenge and a testament to the flexibility of Node.js and TypeScript for developing distributed, scalable, and maintainable microservices. The use of TypeScript adds an extra layer of type safety, making the entire development process smoother and more reliable. This article provides a deep dive into the implementation details, communication strategies, and best practices, offering valuable insights for developers venturing into microservices architecture with Node.js and TypeScript.
Additionally, we will dockerise every microservice for a seamless deployment.

Prerequisites

Before we embark on the journey of building our microservices, it's essential to ensure that your development environment is properly configured. This section outlines the prerequisites that you need to have in place before diving into building the services.

Node.js and npm:

Have Node.js and npm (Node Package Manager) installed on your system. These are essential for developing and running our microservices in a NodeJS environment. You can download Node.js from the official website or use a version manager like nvm for better control over Node.js versions.

Text Editor or IDE:

Choose a text editor or integrated development environment (IDE) for writing your TypeScript/JavaScript code. Popular choices include Visual Studio Code, Atom, or any editor of your preference.

MongoDB Atlas Account:

We will be using MongoDB as a database on both the Auth microservice and notifications microservice, sign up for a MongoDB Atlas account here incase you donot have one and donot have its desktop application(mongodb campass) installed and would like to use mongodb atlas. This cloud-based database service offers a free tier and simplifies the process of managing MongoDB databases.

GitHub Account (Optional):

For version control and implementing CI/CD pipelines in later sections, we will be using Github, ensure you have an account set up. This is optional but highly recommended for a streamlined development workflow.
We will also store our overall source code in a single github organisation with each microservice in its own public github repository.

Basic Knowledge of TypeScript, Express, Docker and NodeJS ORMs:

Familiarize yourself with TypeScript and Express.js, as they form the foundation of our microservices. If you're new to these technologies, consider exploring this article and this to get comfortable with the basics.
In addition, you need some basic knowledge of docker and ORMs in nodejs ecosystem especially prisma and sequelize.
You can checkout this article to level up your docker skills where I explained everything from environment setup to deployment with a CI/CD pipeline using nodejs and typescript.

Communicating Our 4 Micro-services

In this article, we will use a simple yet effective communication architecture for our 4 microservices.

The Authentication microservice, responsible for user management and authentication, will expose a set of HTTP endpoints (RESTful APIs). These endpoints will include user registration and login. We would have handled user profile management here or in an independent microservice but its not a big deal as there is nothing much we need from it in this guide. Other microservices, such as Products and Orders, can make HTTP requests to these endpoints to authenticate users and obtain necessary user information.

The Product Management microservice will handle for us the CRUD operations for products. It will expose HTTP/RESTful APIs for retrieving product information, adding new products, updating product details, and deleting products. Additionally, when changes occur in the product catalog, such as the addition or modification of products, the microservice will publish events to a RabbitMQ exchange. Other microservices, like the Orders microservice, will subscribe to this exchange to receive real-time updates about changes in the product catalog.

Our Order Processing microservice will focus on managing and fulfilling customer orders. It will subscribe to the RabbitMQ exchange where product-related events are published. This will allow the Order Processing microservice to receive real-time updates about changes in the product catalog and adjust order processing accordingly.

Lastly, We will expose a GraphQL API for handling notifications functionality.
When events are published to RabbitMQ that need a user to get notified, the notifications microservice will subscribe to them and deliver a realtime notification to the destined user.

Sharing Code/Functions Between Microservices

Since our application involves user authentication and authorization, we will have methods/functions we may need across all the 4 micro-services for example jwt middlewares for user authorisation, among others and we need a strategy to reuse them all through.
Various options exist to help us achieve this of course depending on the tech stack/programming languages used to build the various microservices but since for we are using nodejs to build all 4 services, we can write a simple shared library and publish it to npm, then install it in any microservice we need to consume the functions.

Now that we have a flow for our microservices, lets get our hands dirty and start building, commencing with the shared library.

Building a Shared Library for All the Microservices and Why we Need it.

Before we start on our Auth Service, from what we have architected so far, we need jwt middlewares across almost all our microservices to help us verify a user trying to access any resource on our application.
Conside a user trying to hit the endpoint products/updatebyid/:productId exposed to the client from the products microservice.
This endpoint should be hit/accessed by a loggedin/authenticated user who in addition has admin previlages(isAdmin set to true). However, the jwt middleware for verifying a user token and checking user roles, from our former monolithic architecture has always been tied to the authentication along side the one that generates jwt upon successfull login, this creates a gap for us to verify jwt from the client(in the headers) in other microservices with protected endpoints necessitating us to craft a way of verifying the token received trying to access that resource/endpoint giving birth to need for a shared library.

Steps:

Step 1: Project Setup

Create a new folder in your desired location and open it with your desired code editor, in my case, I called it: nodejs_ms_shared_library.
Initialise a new nodejsproject with the following command:

npm init -y

Step 2: Install Dependencies

Install the following dependencies with the following command:

npm install express jsonwebtoken

and development dependencies with the following command:

npm install typescript

Step 3: Configure Typescript

Inside the home directory, create a new file: tsconfig.json and place the following code:

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "esModuleInterop": true,
    "skipLibCheck": true,
    "declaration": true,
    "declarationMap": true,
    "forceConsistentCasingInFileNames": true,
    "outDir": "./dist",
    "rootDir": "./src",
    "lib": ["ES2020", "DOM", "DOM.Iterable", "ScriptHost"]
  },
  "include": ["src/**/*.ts"],
  "exclude": ["node_modules", "dist"]
}

Step 4: Write JWT Middleware Functions

Create another file: src/jwtUtils.ts and place the following code:

// shared-library/src/jwtUtils.ts
import { Request, Response, NextFunction } from "express";
import jwt, { VerifyErrors, Secret } from "jsonwebtoken";

// Define the JWT payload type
export type JWTPayload = {
  id: string;
  username: string;
  email: string;
  isAdmin: boolean;
};

// Custom Request type with 'user' property
export interface CustomRequest extends Request {
  user: JWTPayload;
  params: {
    id: string;
  };
  headers: {
    token?: string;
  };
}

// Export types used in the library
export type { Secret, VerifyErrors, NextFunction };

// Generate JWT token
export const generateToken = (
  payload: JWTPayload,
  secret: Secret,
  expiresIn: string
): string => {
  const token = jwt.sign(payload, secret, { expiresIn });
  return token;
};

// Verify JWT token middleware
export const verifyToken = (
  req: CustomRequest,
  res: Response,
  next: NextFunction,
  secret: Secret
): void => {
  const authHeader = req.headers.token;

  if (authHeader) {
    const token = Array.isArray(authHeader)
      ? authHeader[0].split(" ")[1]
      : authHeader.split(" ")[1];

    jwt.verify(token, secret, (err: VerifyErrors | null, user: any) => {
      if (err) {
        // Use the res object with a status function
        res.status(403).json("Token is not valid!");
      } else {
        req.user = user;
        next();
      }
    });
  } else {
    // Use the res object with a status function
    res.status(401).json("You are not authenticated!");
  }
};

// Authorize account owner middleware
export const verifyTokenAndAuthorization = (
  req: CustomRequest,
  res: Response,
  next: NextFunction,
  secret: Secret
): void => {
  verifyToken(
    req,
    res,
    () => {
      if (req.user.id === req.params.id || req.user.isAdmin) {
        next();
      } else {
        res.status(403).json("You are not allowed to do that!");
      }
    },
    secret
  );
};

// Authorize admin middleware
export const verifyTokenAndAdmin = (
  req: CustomRequest,
  res: Response,
  next: NextFunction,
  secret: Secret
): void => {
  verifyToken(
    req,
    res,
    () => {
      if (req.user.isAdmin) {
        next();
      } else {
        return res.status(403).json("You are not allowed to do that!");
      }
    },
    secret
  );
};

We export four functions, one for generating jwt token, other for verifying the token, other for for authorizing user who owns a resource and other for authorizing admin.
These functions act as a bark bone for our authentication and authorization system.

Lets now create an index fie and export our functions. Create a new file: src/index.ts and place the following code:

export {
  generateToken,
  verifyTokenAndAdmin,
  verifyTokenAndAuthorization,
  CustomRequest,
  JWTPayload,
  VerifyErrors,
  verifyToken,
  Secret,
} from "./jwtUtils";

step 5: Transpile TypeScript

We also need a way of converting our ts to a format that our nodejs runtime can run.
Before we run the command for transpiling our ts to js, we need to adjust our package.json and add lacking scripts and other information to help us in publishing our library.
Update package.json to look like below:

{
  "name": "nodejs_ms_shared_library",
  "version": "1.0.5",
  "description": "",
  "main": "dist/index.js",
  "module": "module",
  "types": "dist/index.d.ts",
  "scripts": {
    "build": "tsc"
  },
  "engines": {
    "node": ">=16.0.0"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "typescript": "^5.3.3"
  },
  "dependencies": {
    "express": "^4.18.2",
    "jsonwebtoken": "^9.0.2"
  }
}

We added a build script for tanspiling typescript to javascript that can be run by our nodejs runtime.
Now run the following command on terminal:

npm run build

If all is good, you should have the following folder structure including dist with the compiled typescript and declaration files:

Step 5: Publish to NPM

Before we publish our shared library to npm, we need an npm account. Head over to npm website and login or create a new account.
After that, get back to terminal and run the following command:

npm login

Follow the onscreen instructions to complete login process from the terminal too.

Before publishing, we need to update the version number in our package.json. You can manually change the version number or use the npm version command:

npm version patch  # Or "minor" or "major"

Considering 1.0.0 as our initial version number, patch increments it to 1.0.1, 1.0.2 etc,on the other hand, minor would increment it to 1.1.0, 1.2.0 etc and then major would increment to 2.0.0, 3.0.0 etc.
In our case we will be updating our library version number with patch but it all depends on the chages you have made to your library for example, patch is suited for small bug fixes, minor for some backward compatible changes and major for major changes with backward incompatible updates.

Now run the following command to publish:

npm publish --access public

Note that the --access public flag is vital since if not included, npm will assume youre library is a private one and its not free to publish a library to a private npm repository so you would need to first check out their pricing
Check your npm account and you should have your library published:

Our library is ready for use. We could have added unit tests and a ci/cd pipeline to automate deployment to our npm repository to make it better but that's outside the scope of the article and that's not why we are here 😄. However, a future article may concentrate on this subject as its way helpful in many scenarios.

Before we proceed, lets first discuss about other approaches if say we were using various programming languages, some not in nodejs ecosystem, say python(django, flask, etc), php(laravel, etc), java(spring boot, etc) among others to build various microservices, our approach of a shared nodejs library may not work for us and we have various other methods that could come to rescue.

1. Centralized Authentication Service:

You can create a dedicated authentication service that handles JWT generation and verification and then expose endpoints for microservices to validate tokens regardless of their technology stack.

2. API Gateway with Authentication:

You can also implement an API gateway that sits in front of microservices and then handle authentication and authorization at the gateway level, forwarding only validated requests to downstream services.

3. Language-Specific JWT Libraries:

Most common way of handling this situation while using various technologies/programming languages to build microservices is using language-specific JWT libraries in each microservice to verify tokens independently.
This ensures consistent secret keys and algorithms across libraries.

4. Shared Verification Logic as a Service:

Lastly, you can isolate JWT verification logic into a separate, technology-agnostic microservice and expose endpoints for other microservices to call for verification, regardless of their language.

All good, we can now start on writing our microservices and we will have our shared library installed with following command: npm i nodejs_ms_shared_library just like any other nodejs library in any of the microservice that will need our shared functions.

Building the Auth Microservice

Our authentication microservice is abit more straight forward just like we plotted, and is concerned with user authentication that is registration and login. We will be using MongoDB with mongoose ODM, together with jest and supertest for unit testing and we will dockerise it at the end:

Steps

Step 1: Initialise Project

Create a new folder in your desired location and name it auth_service.
With the folder open in your favourite code editor, open terminal and run the following command to initialise a NodeJS project:

npm init -y

After initialising the project, install the following dependencies with the following command:

npm install express cors helmet nodejs_ms_shared_library dotenv bcrypt jsonwebtoken mongoose morgan

and development dependencies with the following command:

npm install -D typescript ts-node nodemon jest ts-jest @types/jest supertest @types/supertest @types/cors @types/express @types/bcrypt @types/morgan

Folder Structure

Following will be our overall folder structure for our auth microservice:

Setp 2: COnfigure TypeScript and Nodemon

Create a file named tsconfig.json in the root directory and add the following configuration:

{
  "compilerOptions": {
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "target": "ES2020",
    "baseUrl": "src",
    "noImplicitAny": true,
    "sourceMap": true,
    "esModuleInterop": true,
    "outDir": "dist"
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules"]
}

Create another file named nodemon.json in the root directory and place the following code:

{
  "watch": ["src"],
  "ext": ".ts,.js",
  "exec": "ts-node ./src/index"
}

Inside package.json, add the following scripts in the scripts section:

"scripts": {
    "build": "tsc",
    "start": "npm run build && node dist/src/index.js",
    "dev": "nodemon",
    "test": "jest --watchAll  --detectOpenHandles"
  },

The build command helps us transpile TypeScript to JavaScript in production where npm start command is used to start or restart our server. The dist folder is the destination for our resultant JavaScript code as specified in tsconfig.json. In development, nodemon restarts our server automatically without a need for transpilation first.

Also, add the line:

"module": "module",

to your package.json to specify that we are using EsModules not NodeJS's default CommonJS pattern.

Step 3: Write Database Model

In our auth service, we will have only one model that is User model that will define our User schema. This model will handle all the users of our ecommerce application that is admin and customer.
Create a new file in the root diretory: src/models/User.ts and place the following code:

import mongoose, { Document, Schema } from "mongoose";

export interface IUser extends Document {
  email: string;
  username: string;
  password: string;
  isAdmin: boolean;
  profileImage: string;
}

const userSchema = new Schema<IUser>({
  email: { type: String, required: true, unique: true },
  username: { type: String, required: true, unique: true },
  password: { type: String, required: true },
  isAdmin: { type: Boolean, default: false },
  profileImage: { type: String },
});

const User = mongoose.model<IUser>("User", userSchema);

export default User;

Setp 4: Writing User Service and Controller

Our logic for creating and logging in a user is also straight forward but we will tear it down to have a service and a controller for easy understanding.
Create a new file in the root diretory: src/services/userService.ts and place the following code:

// Import necessary modules
import { generateToken } from "nodejs_ms_shared_library";
import User, { IUser } from "../models/User";
import { comparePassword, hashPassword } from "../utils/passwordUtils";

// Create a new user
export const createUser = async (userInput: IUser): Promise<IUser> => {
  try {
    // Hash the user's password before storing it
    const hashedPassword = await hashPassword(userInput.password);

    // Create the user with the hashed password
    const newUser = await User.create({
      ...userInput,
      password: hashedPassword,
    });

    return newUser;
  } catch (error) {
    throw new Error(`Error creating user: ${error.message}`);
  }
};

// Login user
export const loginUser = async (
  email: string,
  password: string
): Promise<{ user: Omit<IUser, "password">; token: string }> => {
  try {
    // Find user by email
    const user = await User.findOne({ email });
    if (!user) {
      throw new Error("User not found");
    }

    // Compare the provided password with the stored hashed password
    const isPasswordValid = await comparePassword(password, user.password);
    if (!isPasswordValid) {
      throw new Error("Invalid password");
    }

    // Generate JWT token
    const token = generateToken(
      {
        id: user._id,
        username: user.username,
        email: user.email,
        isAdmin: user.isAdmin,
      },
      process.env.JWT_SEC,
      process.env.JWT_EXPIRY_PERIOD
    );

    // Destructure password from the data returned
    const { password: _password, ...userData } = user.toObject();

    return { user: userData as Omit<IUser, "password">, token };
  } catch (error) {
    throw new Error(`Error logging in: ${error.message}`);
  }
};

and src/controllers/userController and place the following code:

import { Request, Response } from "express";
import * as UserService from "../services/userService";

// Create a new user
export const createUser = async (
  req: Request,
  res: Response
): Promise<void> => {
  try {
    const newUser = await UserService.createUser(req.body);
    res.status(201).json(newUser);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

// Login user
export const loginUser = async (req: Request, res: Response): Promise<void> => {
  const { email, password } = req.body;
  try {
    const { user, token } = await UserService.loginUser(email, password);
    res.status(200).json({ user, token });
  } catch (error) {
    console.log(error);
    res.status(500).json({ error: error.message });
  }
};

Our service uses separately defined functions for encrypting and decrypting password using bcrypt library for security purposes, defined at src/utils/passwordUtils.ts as shown below:

// Import necessary modules
import bcrypt from "bcrypt";

// Hash a password
export const hashPassword = async (password: string): Promise<string> => {
  const saltRounds = 10;
  const hashedPassword = await bcrypt.hash(password, saltRounds);
  return hashedPassword;
};

// Compare a password with its hash
export const comparePassword = async (
  password: string,
  hashedPassword: string
): Promise<boolean> => {
  return bcrypt.compare(password, hashedPassword);
};

Additionally, our service uses generateToken() function from our custom shared library: nodejs_ms_shared_library and takes all the three parameters, just like we defined while building the library.
Create a file in the root directory, name it .env and place the following code:

JWT_SEC=YourJWTSecret
JWT_EXPIRY_PERIOD=YourJWTExpiryTime

Step 5: Writing/Exposing Auth Routes

After writing our authentication logic, we now need to expose our routes for login and register such that the client can use them.
Create a new file in the root directory: src/routes/authRoute.ts andplace the following code:

// src/auth.route.ts
import { Router } from "express";
import * as AuthController from "../controllers/userController";

const router = Router();
// Register new user
router.post("/register", AuthController.createUser);

// Login user
router.post("/login", AuthController.loginUser);

export default router;

Inside the root directory, create a index file at src/index.ts and place the following code:

import express from "express";
const app = express();
import cors from "cors";
import helmet from "helmet";
import dotenv from "dotenv";
import mongoose from "mongoose";
import authRoute from "./routes/authRoute";
import morgan from "morgan";

dotenv.config();
app.use(morgan("common"));

// USE HELMET AND CORS MIDDLEWARES
app.use(
  cors({
    origin: ["*"], // Comma separated list of your urls to access your api. * means allow everything
    credentials: true, // Allow cookies to be sent with requests
  })
);
app.use(helmet());

app.use(express.json());

app.get("/", async (req: express.Request, res: express.Response) => {
  try {
    res.send(
      "Welcome to unit testing guide for nodejs, typescript and express!"
    );
  } catch (err) {
    console.log(err);
  }
});

// DB CONNECTION

if (!process.env.MONGODB_URL) {
  throw new Error("MONGO_URI environment variable is not defined");
}

mongoose
  .connect(process.env.MONGODB_URL)
  .then(() => {
    console.log("MongoDB connected to the backend successfully");
  })
  .catch((err) => console.log(err));

app.get("/", async (req: express.Request, res: express.Response) => {
  try {
    res.send(
      "Welcome to unit testing guide for nodejs, typescript and express"
    );
  } catch (err) {
    console.log(err);
  }
});

// Serve other routes
app.use("/api/v1/auth/", authRoute);

// Start backend server
const PORT = process.env.PORT || 8900;

app.listen(PORT, () => {
  console.log(`Backend server is running at port ${PORT}`);
});

export default app;

Our index file includes all connections to other resources like our mongodb using mongoose among others and our backend server will be listening at port 8900 on our local machine.

Lets add our database connection url from mongodb atlas into our .env file. Add the following line:

MONGODB_URL=YourDatabaseConnectionString

Step 6: Writing Unit Tests

In this service, we will write a simple unit test only to verify our unit testing environment and be sure our tests would progress given that we had them for all our code.
Create a new file in the root: src/__tests__/app.test.ts and place the folllowing code:

// Unit test for testing initial route ("/")
describe("GET /", () => {
  it("Tests initial route '/'", async () => {
    expect(true).toBe(true);
  });
});

If you would like to write unit tests for the entire service, its totally fine and you can check out the following article for assistance:

Step 7: Dockerising the Auth Service

To dockerise our auth microservice, we will need a Dockerfile to hold our instructions for building a Docker image.
If youre not familiar with docker, check out this article for a step by step guide.
Create a new file in the root: Dockerfile and place the following code:

# Use an official Node.js runtime as a parent image
FROM node:latest as builder

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install app dependencies
RUN npm install

# Copy all files from the current directory to the working directory
COPY . .

# Development stage
FROM builder as development
# Set NODE_ENV to development
ENV NODE_ENV=development

# Expose the port the app runs on
EXPOSE 8900

# Command to run the application(in development)
CMD ["npm", "run", "dev"]

# Production stage
FROM builder as production
# Set NODE_ENV to production
ENV NODE_ENV=production

# Run any production-specific build steps if needed here

# Run the production command
CMD ["npm", "start"]

and another file, still in the root: .dockerignore and place the following code:

/node_modules
npm-debug.log
.DS_Store
/*.env
./idea

Our Dockerfile contains instructions for building the docker image both in production and development according to the NODE_ENV environment variable.
Also, .dockerignore helps us ignore some files and directories while building the docker image just like you see .gitignore while using git.

All set, lets now build and run the docker image with the following commands:

docker build -t auth_service:development --target development .

and

docker run -p 8900:8900 -v $(pwd):/usr/src/app -e PORT=8900 auth_service:development

respectively.
If youre using mac or linux and didn't add your user to the docker group while installing docker, to run Docker commands without sudo, donot forget to use super user do(sudo) at the beginning of these commands for building and running docker image.

If everything is well, you should have the following in terminal:

You can spin up/split another terminal instance and run unit tests(inside the docker container), with the following command:

sudo docker exec -it your_docker_container_id  npm test

Before we start on our products service, you can first create atleast 2 users one admin and one customer(without admin previlages) using any means of your choice, say postman, insomnia or any other means.

Building Products Microservice

Our products microservice is also straight forward just like how the auth has been. As previously plotted, we will be using different technologies on each service and we are using PostgreSQL as a database and prisma orm(Object Relational Mapper) for querying our DB.
ORMs are used to translate between the data representations used by databases and those used in object-oriented programming, and in this service, we will be using one of the most common ones in the nodejs ecosystem, Prisma.
It is the only fully type-safe ORM in the TypeScript ecosystem. The generated Prisma Client ensures typed query results even for partial queries and relations.

On the other hand, PostreSQL is a free and open-source relational database management system emphasizing extensibility and SQL compliance.
In this product service, it will be our database.

Steps:

Our steps on this service are nearly the same with the auth service except some small differences. However we will redo everything since this is an entirely independent microservice that will even be hosted independently.

Step 1: Project Setup

Create a new folder in your desired location and name it products_service.
With the folder open in your favorite code editor, open terminal and run the following command to initialize a NodeJS project:

npm init -y

After initializing the project, install the following dependencies with the following command:

npm install express cors helmet nodejs_ms_shared_library dotenv bcrypt jsonwebtoken morgan

and development dependencies with the following command:

npm install -D typescript ts-node nodemon jest ts-jest @types/jest supertest @types/supertest @types/cors @types/express @types/bcrypt @types/morgan

Folder Structure

Following will be our overall folder structure for our product microservice:

Setp 2: Configure TypeScript and Nodemon

Create a file named tsconfig.json in the root directory and add the following configuration:

{
  "compilerOptions": {
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "target": "ES2020",
    "baseUrl": "src",
    "noImplicitAny": true,
    "sourceMap": true,
    "esModuleInterop": true,
    "outDir": "dist"
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules"]
}

Create another file named nodemon.json in the root directory and place the following code:

{
  "watch": ["src"],
  "ext": ".ts,.js",
  "exec": "ts-node ./src/index"
}

Inside package.json, add the following scripts in the scripts section:

"scripts": {
    "build": "tsc",
    "start": "npm run build && node dist/src/index.js",
    "dev": "nodemon",
    "test": "jest --watchAll  --detectOpenHandles"
  },

Also, add the line:

"module": "module",

to your package.json to specify that we are using EsModules not NodeJS's default CommonJS pattern.

Step 3: Install PostgreSQL

If you donot have PostgreSQL database installed, head over to their download website and get the one for your operating system.
As always, if youre using linux and are not a fan of graphical user interfaces, I will provide the commands to help you install everything without leaving your terminal:

Update Package Lists:

sudo apt update

Install the PostgreSQL server:

sudo apt install postgresql postgresql-contrib

Start and enable the PostgreSQL service to start on boot:

sudo systemctl start postgresql
sudo systemctl enable postgresql

Access the PostgreSQL shell

Access the PostgreSQL shell by switching to the postgres user and running the psql command:

sudo -u postgres psql

You should have the following terminal:

Step 4: Building the Database Schema/Model

Since we are using prisma orm, it will help us on lifting all the heavy work in setting up the database and its schema.
If you dont have it installed, run the following command:

npm install -g prisma

You can also install it in our project as a dependency not globally with the following command:

npm install prisma

After installing Prisma, we need to initialize the Prisma configuration by running the following command:

npx prisma init

You should have prisma.schema file in the location: prisma/prisma.schema. Update it to the following code:

// prisma/schema.prisma
generator client {
  provider = "prisma-client-js"
  output   = "../src/prisma/client"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model Product {
  id        Int      @id @default(autoincrement())
  title     String
  desc      String
  img       String
  categories String[]
  size      String[]
  color     String[]
  price     Float
}

Our Product schema has 8 fields as shown above.
The npx prisma init command updated our .env file by adding DATABASE_URL variable holding the databse connection url but we need to update it with our actual db name, username and password.

# This was inserted by `prisma init`:
# Environment variables declared in this file are automatically made available to Prisma.
# See the documentation for more detail: https://pris.ly/d/prisma-schema#accessing-environment-variables-from-the-schema

# Prisma supports the native connection string format for PostgreSQL, MySQL, SQLite, SQL Server, MongoDB and CockroachDB.
# See the documentation for all the connection string options: https://pris.ly/d/connection-strings

DATABASE_URL="postgresql://your_db_root_user:your_db_root_user_password@localhost:5432/your_db_name?schema=public"

Replace the password, username and db name with what you used when setting up PostgreSQl after installation.
In my case, my db name is product_service.

After initialization, we need to generate the Prisma client by running:

npx prisma generate

This command generates the Prisma client based on your schema and creates the necessary files for database access.

Now, we have Prisma installed and configured in our Node.js project, we can start using the Prisma client in your application code to interact with our database.

Step 6: Writing Product Service and Controller.

Create a new file: src/services/productService.ts and place the following code:

import { PrismaClient } from "../prisma/client";

const prisma = new PrismaClient();

// Create new product
const createProduct = async (data: any) => {
  return prisma.product.create({
    data,
  });
};

// Get all products
const getAllProducts = async () => {
  return prisma.product.findMany();
};

// Get product by id
const getProductById = async (productId: number) => {
  return prisma.product.findUnique({
    where: {
      id: productId,
    },
  });
};

// Update product
const updateProduct = async (productId: number, data: any) => {
  return prisma.product.update({
    where: {
      id: productId,
    },
    data,
  });
};

// Delete product
const deleteProduct = async (productId: number) => {
  return prisma.product.delete({
    where: {
      id: productId,
    },
  });
};

export {
  createProduct,
  getAllProducts,
  getProductById,
  updateProduct,
  deleteProduct,
};

and src/controllers/productController.ts and place the following code:

// src/controllers/productController.ts
import { Request, Response } from "express";
import * as productService from "../services/productService";
import { CustomRequest } from "nodejs_ms_shared_library";

const createProduct = async (req: CustomRequest, res: Response) => {
  try {
    const product = await productService.createProduct(req.body);
    res.status(201).json({
      message: "Product created successfully!",
      user: req.user,
      product,
    });
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

const getAllProducts = async (req: CustomRequest, res: Response) => {
  try {
    const products = await productService.getAllProducts();
    res.status(200).json({ products, user: req.user });
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

const getProductById = async (req: CustomRequest, res: Response) => {
  const productId = parseInt(req.params.id, 10);
  try {
    const product = await productService.getProductById(productId);
    if (!product) {
      res.status(404).json({ error: "Product not found" });
    } else {
      res.status(200).json({ product, user: req.user });
    }
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

const updateProduct = async (req: CustomRequest, res: Response) => {
  const productId = parseInt(req.params.id, 10);
  try {
    const product = await productService.updateProduct(productId, req.body);
    res.status(200).json({ product, user: req.user });
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

const deleteProduct = async (req: CustomRequest, res: Response) => {
  const productId = parseInt(req.params.id, 10);
  try {
    await productService.deleteProduct(productId);
    res.status(200).json({
      message: "Product deleted successfully!",
      user: req.user,
    });
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

export {
  createProduct,
  getAllProducts,
  getProductById,
  updateProduct,
  deleteProduct,
};

Our crud operations on the product are straight to the point creating, updating, deleting and retrieving products or product by id with the help of PrismaClient from prisma client.
We are sending the response from the backend server alongside with the user obtained from req.user with details we formerly stored in the jwt.
This how we are achieving communication between auth and products, when the request is received from client, the token in headers has all the user details we stored there in the auth microservice when we called generateToken and passed a payload.

Step 7: Writing/Exposing Product Routes

Before we write our product routes, we already know they will be protected that is an authenticated user and with some previlages should be the one to access some of them. we however already written the logic in our shared library and installed it too.
Lets create another file: src/middlewares/jwtMiddlewares and get them ready for use in our routes:

import { NextFunction, Response } from "express";
import {
  CustomRequest,
  verifyToken,
  verifyTokenAndAdmin,
  verifyTokenAndAuthorization,
} from "nodejs_ms_shared_library";

// Wrap the middleware functions with their parameters

// Verify token from the client
export const verifyTokenMiddleware = (
  req: CustomRequest,
  res: Response,
  next: NextFunction
) => {
  verifyToken(req, res, next, process.env.JWT_SEC);
};

// Verify token and authorise account owner
export const verifyTokenAndAuthoriationMiddleware = (
  req: CustomRequest,
  res: Response,
  next: NextFunction
) => {
  verifyTokenAndAuthorization(req, res, next, process.env.JWT_SEC);
};

// Verify token and authorise admin
export const verifyTokenAndAdminMiddleware = (
  req: CustomRequest,
  res: Response,
  next: NextFunction
) => {
  verifyTokenAndAdmin(req, res, next, process.env.JWT_SEC);
};

We do this because our functions in the custom library receive many parameters which we need to first supply when we invoke them.
Add the same JWT_SECRET and JWT_EXPIRY_PERIOD as you did in the auth microservice inside .env file.

Create a new file: src/routes/productRoutes.ts and place the following code:

import express from "express";
import * as productController from "../controller/productController";
import {
  verifyTokenAndAdminMiddleware,
  verifyTokenMiddleware,
} from "../middlewares/jwtMiddlewares";

const router = express.Router();

// Create new product
router.post(
  "/",
  verifyTokenAndAdminMiddleware,
  productController.createProduct
);

// Get all product
router.get("/", verifyTokenMiddleware, productController.getAllProducts);

// Get product by id
router.get("/:id", verifyTokenMiddleware, productController.getProductById);

// Update product by id
router.put(
  "/:id",
  verifyTokenAndAdminMiddleware,
  productController.updateProduct
);

// Delete product by id
router.delete(
  "/:id",
  verifyTokenAndAdminMiddleware,
  productController.deleteProduct
);

export default router;

Finally, create an index file for the project at src/index.ts and place the following code:

import express from "express";
const app = express();
import cors from "cors";
import helmet from "helmet";
import dotenv from "dotenv";
import productRoute from "./routes/productRoute";
import morgan from "morgan";

dotenv.config();
app.use(morgan("common"));

// USE HELMET AND CORS MIDDLEWARES
app.use(
  cors({
    origin: ["*"], // Comma separated list of your urls to access your api. * means allow everything
    credentials: true, // Allow cookies to be sent with requests
  })
);
app.use(helmet());

app.use(express.json());

// Serve other routes
app.use("/api/v1/products/", productRoute);

// Start backend server
const PORT = process.env.PORT || 8000;

app.listen(PORT, () => {
  console.log(`Backend server is running at port ${PORT}`);
});

export default app;

On the client, when a request is made to any of the product endpoints, a token header will be supplied and when the product service receives it, it checks to see if the user thats making a request has all the previlages he is supposed to be with depending on the jwt middleware invoked on that route handler and req.user will contain all the data we stored in the token.
The following should be obtained when you make a request to create a product without admin previlages:

and the following for some one with admin previlages:

All set lets now dockerise our product microservice too.

Step 8: Dockerising the Product Microservice

Create a new file in the root: Dockerfile and place the following code:

# Use an official Node.js runtime as a parent image
FROM node:latest as builder

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install app dependencies
RUN npm install

# Copy all files from the current directory to the working directory
COPY . .

# Development stage
FROM builder as development
# Set NODE_ENV to development
ENV NODE_ENV=development

# Expose the port the app runs on
EXPOSE 8900

# Command to run the application(in development)
CMD ["npm", "run", "dev"]

# Production stage
FROM builder as production
# Set NODE_ENV to production
ENV NODE_ENV=production

# Run any production-specific build steps if needed here

# Run the production command
CMD ["npm", "start"]

and another file, still in the root: .dockerignore and place the following code:

/node_modules
npm-debug.log
.DS_Store
/*.env
./idea

Also, .dockerignore helps us ignore some files and directories while building the docker image just like you see .gitignore while using git.

All set, lets now build and run the docker image with the following commands:

docker build -t product_service:development --target development .

and

docker run -p 8700:8700 -v $(pwd):/usr/src/app -e PORT=8700 auth_service:development

respectively.

If everything is well, you should have the following in terminal:

You can also spin up/split another terminal instance and run unit tests(inside the docker container) just like we did on the auth service, with the following command:

sudo docker exec -it your_docker_container_id  npm test

Building the Orders Microservice

Our orders microservice will have its own set of teachnologies just like we earlier plotted that is mysql database and sequelize orm.
MySQL is an open-source relational database management system (RDBMS) that is widely used for building web applications and managing data. It is a popular choice for many developers and organizations due to its performance, reliability, and ease of use.
Sequelize is a popular Object-Relational Mapping (ORM) library for Node.js. It provides a way to interact with relational databases like MySQL, PostgreSQL, SQLite, and MSSQL using JavaScript or TypeScript. It simplifies database operations by allowing developers to use JavaScript objects to represent database tables and records, instead of writing raw SQL queries.
In this microservice, we will use it to query our MySQL database.

Steps

Step 1: Project Setup

Create a new folder in your desired location and name it order_service.
With the folder open in your favorite code editor, open terminal and run the following command to initialize a NodeJS project:

npm init -y

After initializing the project, install the following dependencies with the following command:

npm install express cors helmet nodejs_ms_shared_library dotenv bcrypt jsonwebtoken morgan

and development dependencies with the following command:

npm install -D nodemon jest supertest

Folder Structure

Following will be our overall folder structure for our orders microservice:

Inside package.json, add the following scripts in the scripts section:

"scripts": {
    "start": "node index.js",
    "dev": "nodemon index.js",
    "test": "jest --watchAll --detectOpenHandles"
  },

Step 2: Install MySQL

If youre using windows or mac, I still recommend a desktop application for mysql(MySQL workbench).
Visit this site to download the one for your operating system.
After downloading, follow the on screen instructions to install it as well

If you're using linux, you can use the following commands to get everything sorted right at the comfort of your terminal:

Update Package Lists:

sudo apt update

Install MySQL Server:

sudo apt install mysql-server

Start MySQL Service:

sudo service mysql start

Secure MySQL Installation (Optional but recommended):

sudo mysql_secure_installation

This command will guide you through securing your MySQL installation by setting a root password, removing anonymous users, and other security-related configurations.
Access MySQL Shell:

mysql -u root -p

You will be prompted to enter the password you set during the secure installation or the default password if you didn't set one.

Now, you are in the MySQL shell, and you can interact with the MySQL database entirely through the terminal.
However, just like we said, we willbe using sequelize to help us on the heavy lifting like writing sql queries.

Step 3: Setup and Initialize Sequelize

Inside the root directory, open terminal and install the following extra dependencies:

npm install mysql2 sequelize

We also need to install sequelize-cli, a powerful tool that helps you manage database migrations, models, and configurations in a Sequelize project.
Open terminal and run following command:

npm install -g sequelize-cli

and initialise sequelize with the following command:

sequelize init

This command initializes a basic Sequelize project structure in your current directory. It creates a config folder for database configurations, a models folder for your Sequelize models, and a migrations folder for handling database schema changes.
Update the generated config/config.json with your database credentials for development like username, password, database name, etc. I recommend you place all the credentials in a .env file and reference them instead of placing them directly in the config file.

Step 4: Writing Database Models

In sequelize, the database models(model names) you write will translate to tables in the database with all the fields you include with their data types.
Create a new file: models/Order.js and place the following code:

module.exports = (sequelize, DataTypes) => {
  const Orders = sequelize.define("Orders", {
    userId: {
      type: DataTypes.STRING,
    },

    orderId: {
      type: DataTypes.STRING,
      primaryKey: true,
    },

    products: {
      type: DataTypes.JSON,
    },

    amount: {
      type: DataTypes.INTEGER,
    },

    status: {
      type: DataTypes.STRING,
      enum: ["Pending", "Delivering", "Delivered"],
      defaultValue: "Pending",
    },
  });

  return Orders;
};

Our Order model has 5 fields as shown including userId, the id of the user trying to make an order. This id will be supplied from the auth microservice. On the client, when a user logs in, his id is part of the data returned alongside the token(jwt). Inside the application, when a user tries to make an order you will send a req.body including the fields specified in the schema including the userId, from localStorage or any where you desired to keep it at login time.
Optionally, you can pick it from the jwt sent in headers as we are soon to discuss.

Step 5: Writing Controller Logic

Create a new file: controllers/orderController.ts and place the following code:

const { Orders } = require("../models"); // Adjust the path accordingly
const { generateOrderId } = require("../utils/orderIdGenerator");
// Create a new order
const createOrder = async (req, res) => {
  try {
    const order = await Orders.create({
      orderId: generateOrderId(),
      ...req.body,
    });
    res.status(201).json({
      order,
      user: req.user,
    });
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

// Get all orders
const getAllOrders = async (req, res) => {
  try {
    console.log(req.user, "u");
    const orders = await Orders.findAll();
    const ordersWithParsedProducts = orders.map((order) => {
      return {
        ...order.toJSON(),
        products: JSON.parse(order.products),
      };
    });

    res.status(200).json({
      user: req.user,
      orders: ordersWithParsedProducts,
    });
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

// Get a specific order by orderId
const getOrderById = async (req, res) => {
  const { orderId } = req.params;
  try {
    const order = await Orders.findOne({
      where: { orderId },
    });
    if (order) {
      res.status(200).json({
        user: req.user,
        order,
      });
    } else {
      res.status(404).json({ error: "Order not found" });
    }
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

// Update an order by orderId
const updateOrderById = async (req, res) => {
  const { orderId } = req.params;
  try {
    const [updatedCount, updatedOrders] = await Orders.update(req.body, {
      where: { orderId },
      returning: true,
    });
    if (updatedCount > 0) {
      res.status(200).json({ user: req.user, orders: updatedOrders[0] });
    } else {
      res.status(404).json({ error: "Order not found" });
    }
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

// Delete an order by orderId
const deleteOrderById = async (req, res) => {
  const { orderId } = req.params;
  try {
    const deletedCount = await Orders.destroy({
      where: { orderId },
    });
    if (deletedCount > 0) {
      res.status(200).json({ message: "Order deleted successfully" });
    } else {
      res.status(404).json({ error: "Order not found" });
    }
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

module.exports = {
  createOrder,
  getAllOrders,
  getOrderById,
  updateOrderById,
  deleteOrderById,
};

We define a set of functions for handling CRUD (Create, Read, Update, Delete) operations on our orders controller. The createOrder function creates a new order by generating an order ID and saving the order details to the database. getAllOrders retrieves all orders, parsing the product information from JSON format. getOrderById retrieves a specific order based on the provided order ID. The updateOrderById function updates an existing order by its order ID, and deleteOrderById deletes an order based on the order ID. Each function handles potential errors, responding with appropriate HTTP status codes and error messages. We also utilize a utility function (generateOrderId) for order ID generation. Additionally, the functions include user information in the response to provide context about the user interacting with the orders.

The utility function at utils/orderIDGenerator.js is as shown:

const generateOrderId = () => {
  let dt = new Date().getTime();
  let orderID = "xxxxxxxxxxxx4xxxyxxxxxxxxxxxxxxx".replace(/[xy]/g, (c) => {
    let r = (dt + Math.random() * 16) % 16 | 0;
    dt = Math.floor(dt / 16);
    return (c == "x" ? r : (r & 0x3) | 0x8).toString(16);
  });
  return orderID;
};

module.exports = { generateOrderId };

Step 6: Exposing our Order Routes

Before we expose our Order routes, we also need to prepare our jwt middleware functions from our custom library as we will also use them in our routes that we will expose to the client.

Lets create another file: middlewares/jwtMiddlewares and place the following code:

const {
  verifyToken,
  verifyTokenAndAdmin,
  verifyTokenAndAuthorization,
} = require("nodejs_ms_shared_library");

// Wrap the middleware functions with their parameters
// Verify token from the client
const verifyTokenMiddleware = (req, res, next) => {
  verifyToken(req, res, next, process.env.JWT_SEC);
};

// Verify token and authorise account owner
const verifyTokenAndAuthoriationMiddleware = (req, res, next) => {
  verifyTokenAndAuthorization(req, res, next, process.env.JWT_SEC);
};

// Verify token and authorise admin
const verifyTokenAndAdminMiddleware = (req, res, next) => {
  verifyTokenAndAdmin(req, res, next, process.env.JWT_SEC);
};

module.exports = {
  verifyTokenAndAdminMiddleware,
  verifyTokenAndAuthoriationMiddleware,
  verifyTokenMiddleware,
};

Now create a new file: routes/orderRoutes.js and place the following code:

const {
  createOrder,
  getAllOrders,
  getOrderById,
  updateOrderById,
  deleteOrderById,
} = require("../controllers/orderController");
const {
  verifyTokenMiddleware,
  verifyTokenAndAuthoriationMiddleware,
  verifyTokenAndAdminMiddleware,
} = require("../middlewares/jwtMiddlewares");
const router = require("express").Router();

// Create new order
router.post("/", verifyTokenMiddleware, createOrder);

// Get all orders
router.get("/", verifyTokenAndAdminMiddleware, getAllOrders);

// Get order by id
router.get("/:orderId", verifyTokenMiddleware, getOrderById);

// Update order by id
router.put("/:orderId", verifyTokenAndAuthoriationMiddleware, updateOrderById);

// Delete order by id
router.delete("/:orderId", verifyTokenAndAdminMiddleware, deleteOrderById);

module.exports = router;

Finally, place the following code in index file inside root directory:

const express = require("express");
const app = express();
const dotenv = require("dotenv").config();
const cors = require("cors");
const helmet = require("helmet");
const morgan = require("morgan");
const cors = require("cors");
const database = require("./models");
const ordersRoute = require("./routes/ordersRoute");

// Middleware
app.use(express.json());
app.use(cors());
app.use(morgan("common"));
app.use(helmet());
app.use("/api/v1/orders", ordersRoute);

// Configure sequelize to sync all models and create corresponding tables accordingly
database.sequelize.sync().then(() => {
  console.log("Db connection successful");
  const PORT = process.env.PORT || 8300;
  app.listen(PORT, () => {
    console.log(`Backend server is listening at port ${PORT}`);
  });
});

Here, we are using various middleware modules such as dotenv, cors, helmet, and morgan to enhance security, handle environment variables, enable Cross-Origin Resource Sharing (CORS), and log HTTP requests, respectively.Our application defines a route for handling orders ("/api/v1/orders") using the "ordersRoute" module. we also configured sequelize to synchronize with the database, ensuring that tables corresponding to the defined models are created.
Finally, we set our server to listen on a specified port (retrieved from the environment variable or defaulting to 8300), and a log message is displayed upon successful database connection and server initialization.

Step 7: Dockerising our Orders Service

Unlike other services we have so far, we will dockerise this service and configure an additional docker-compose to help us run multiple containers at ago.
we will have our nodejs order service in a container(docker) and use docker-compose to run it together with its mysql database.
Create a new file: Dockerfile in the root directory and place the following code:

# Use an official Node.js runtime as a parent image
FROM node:latest as builder

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install app dependencies
RUN npm install

# Copy all files from the current directory to the working directory
COPY . .

# Development stage
FROM builder as development
# Set NODE_ENV to development
ENV NODE_ENV=development

# Expose the port the app runs on
EXPOSE 8300

# Command to run the application(in development)
CMD ["npm", "run", "dev"]

# Production stage
FROM builder as production
# Set NODE_ENV to production
ENV NODE_ENV=production

# Run any production-specific build steps if needed here

# Run the production command
CMD ["npm", "start"]

and .dockerignore and place the following code:

node_modules
dist
.git
.dockerignore

Lets now create docker-compose.yml file in the root directory and place the following code:

version: "3.8"

services:
  node-api:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8300:8300"
    depends_on:
      - mysql
    env_file:
      - .env # Use the same .env file for both services
    working_dir: /usr/src/app
    volumes:
      - /home/abeine/eoyprojects/nodejsapis/containerisedprojects/microservices_with_nodejs/orders_service:/usr/src/app
    command: npm run dev

  mysql:
    image: mysql:latest
    env_file:
      - .env # Use the same .env file for both services
    command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --innodb_force_recovery=1
    environment:
      MYSQL_ROOT_PASSWORD: ${DB_DEV_PASSWORD}
    ports:
      - "3306:3306"
    networks:
      - my-network
    volumes:
      - mysql-data:/var/lib/mysql

volumes:
  mysql-data:

networks:
  my-network:

We defined a multi-container environment for our Node.js API service and our MySQL database service. The Node.js API service, named "node-api," is built from the specified Dockerfile, exposes its application on port 8300, and depends on the MySQL service. It uses an environment file (.env) for configuration and mounts the local project directory as a volume into the container. The MySQL service uses the latest MySQL Docker image, sets up the necessary environment variables, including the root password from the .env file, and exposes MySQL on port 3306. It utilizes a custom network called "my-network" to enable communication between the services and employs a volume named "mysql-data" to persistently store MySQL data.
Our overall configuration facilitates the development environment for a Node.js API interacting with our MySQL database, ensuring ease of deployment and reproducibility across different environments.

Building Notifications Microservice

We have so far completed 3 microservices and dockerised them all as well. However, in an ecommerce application, we always need to notify our users/customers when various events happen forexample, when new products are added, their previous orders get fulfilled, when their goods get shipped and many other scenarios. we will handle all the notification related logic in an independent microservice.
In this service, we will be using graphql, mongodb and mongoose.
For illustration in this article, we will use express-graphql.It is a piece of middleware, to quickly setup a GraphQL Server, either with Express, or any web-framework that supports middleware.
In sophisticated implementations of graphql, I recommend using Apollo-Server in the place of express-graphql as its more feature rich than the later, with poweful features like subscriptions(allows a server to send data to its clients when a specific event happens in realtime) and support for nearly all Apollo Client libraries.
A future article may concentrate on this subject too as its also helpful in many scenarios.
We will use mongoose ODM to interact with our MongoDB database.

Steps:

Step 1: Project Setup

Create a new folder in your desired location and name it notifications_service.
With the folder open in your favorite code editor, open terminal and run the following command to initialize a NodeJS project:

npm init -y

After initializing the project, install the following dependencies with the following command:

npm install express cors helmet nodejs_ms_shared_library dotenv bcrypt jsonwebtoken morgan mongoose

and development dependencies with the following command:

npm install -D typescript ts-node nodemon jest ts-jest @types/jest supertest @types/supertest @types/cors @types/express @types/bcrypt @types/morgan

Folder Structure

Following will be our overall folder structure for our notifications microservice:

Setp 2: Configure TypeScript and Nodemon

Create a file named tsconfig.json in the root directory and add the following configuration:

{
  "compilerOptions": {
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "target": "ES2020",
    "baseUrl": "src",
    "noImplicitAny": true,
    "sourceMap": true,
    "esModuleInterop": true,
    "outDir": "dist"
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules"]
}

Create another file named nodemon.json in the root directory and place the following code:

{
  "watch": ["src"],
  "ext": ".ts,.js",
  "exec": "ts-node ./src/index"
}

Inside package.json, add the following scripts in the scripts section:

"scripts": {
    "build": "tsc",
    "start": "npm run build && node dist/src/index.js",
    "dev": "nodemon",
    "test": "jest --watchAll  --detectOpenHandles"
  },

Also, add the line:

"module": "module",

to your package.json to specify that we are using EsModules not NodeJS's default CommonJS pattern.

Step 3: Writing Notification Model

Create a new folder: src/models/Notification.ts and place the following code:

// Import necessary modules
import mongoose, { Schema, Document } from "mongoose";

// Define the interface for User document
export interface INotification extends Document {
  title: string;
  text: string;
  userId: string;
}

// Create a schema for the User model
const notificationSchema: Schema<INotification> = new Schema(
  {
    title: { type: String, required: true },
    text: { type: String, required: true },
    userId: { type: String, required: true },
  },
  { timestamps: true }
);

// Create and export the User model
export default mongoose.model<INotification>(
  "Notification",
  notificationSchema
);

Our model describes 3 fields we will have in the notifications schema as shown above.

Step 4: Creating Notification Service

In our notification service, we isolate the business logic for handling notification related functionality.
Create a new file: src/services/notificationService.ts and place the following code.

import Notification from "../models/Notification";

export const getAllNotifications = async () => {
  try {
    return await Notification.find();
  } catch (error) {
    throw new Error(error.message);
  }
};

export const getNotificationById = async (id: string) => {
  try {
    return await Notification.findById(id);
  } catch (error) {
    throw new Error(error.message);
  }
};

export const addNotification = async (args: {
  title: string;
  text: string;
  userId: string;
}) => {
  try {
    const notification = new Notification(
      args as {
        title: string;
        text: string;
        userId: string;
      }
    );
    return await notification.save();
  } catch (error) {
    throw new Error(error.message);
  }
};

export const updateNotification = async (args: {
  id: string;
  title: string;
  text: string;
}) => {
  try {
    return await Notification.findByIdAndUpdate(
      args.id,
      { title: args.title, text: args.text },
      { new: true }
    );
  } catch (error) {
    throw new Error(error.message);
  }
};

export const deleteNotification = async (id: string) => {
  try {
    return await Notification.findByIdAndDelete(id);
  } catch (error) {
    throw new Error(error.message);
  }
};

Lets now write our graphql schema and then consume our services:
Create a new file: src/schema/Notification.ts and place the following code:

import { GraphQLObjectType, GraphQLID, GraphQLString } from "graphql";

const NotificationType = new GraphQLObjectType({
  name: "Notification",
  fields: () => ({
    id: { type: GraphQLID },
    userId: { type: GraphQLID },
    title: { type: GraphQLString },
    text: { type: GraphQLString },
  }),
});

export default NotificationType;

and src/schema/index.ts and place the following code:

import {
  GraphQLSchema,
  GraphQLObjectType,
  GraphQLList,
  GraphQLNonNull,
  GraphQLString,
} from "graphql";
import NotificationType from "./Notification";
import * as NotificationService from "../services/notificationService";

// Queries
const RootQuery = new GraphQLObjectType({
  name: "RootQueryType",
  fields: {
    // Query to get all notifications
    notifications: {
      type: GraphQLList(NotificationType),
      resolve: async () => NotificationService.getAllNotifications(),
    },

    // Query to get a notification by ID
    notification: {
      type: NotificationType,
      args: { id: { type: GraphQLNonNull(GraphQLString) } },
      resolve: async (_, args) =>
        NotificationService.getNotificationById(args.id),
    },
  },
});

// Mutations
const Mutation = new GraphQLObjectType({
  name: "Mutation",
  fields: {
    // Mutation to add a new notification
    addNotification: {
      type: NotificationType,
      args: {
        title: { type: GraphQLNonNull(GraphQLString) },
        text: { type: GraphQLNonNull(GraphQLString) },
        userId: { type: GraphQLNonNull(GraphQLString) },
      },
      resolve: async (
        _,
        args: { title: string; text: string; userId: string }
      ) => NotificationService.addNotification(args),
    },

    updateNotification: {
      type: NotificationType,
      args: {
        id: { type: GraphQLNonNull(GraphQLString) },
        title: { type: GraphQLNonNull(GraphQLString) },
        text: { type: GraphQLNonNull(GraphQLString) },
      },
      resolve: async (_, args: { id: string; title: string; text: string }) =>
        NotificationService.updateNotification(args),
    },

    deleteNotification: {
      type: NotificationType,
      args: { id: { type: GraphQLNonNull(GraphQLString) } },
      resolve: async (_, args) =>
        NotificationService.deleteNotification(args.id),
    },
  },
});

export default new GraphQLSchema({
  query: RootQuery,
  mutation: Mutation,
});

Our GraphQL schema is defined to handle notifications, including queries and mutations. The RootQueryType contains two queries: notifications for retrieving all notifications and notification for fetching a specific notification by its ID. The Mutation type encompasses three mutations: addNotification for creating a new notification with mandatory title, text, and userId parameters, updateNotification for modifying an existing notification's title and text by ID, and deleteNotification for removing a notification by its ID. Each resolver function within the schema delegates its logic to corresponding functions in the NotificationService module, which interacts with the database and performs CRUD operations on notifications. The schema is constructed using GraphQLObjectType instances, and it exports a GraphQLSchema containing the defined queries and mutations. This structure adheres to GraphQL conventions, providing a clear and organized way to handle notifications within our GraphQL-powered notifications service.

Lastly, setup index file: src/index.ts with the following code:

import express from "express";
const app = express();
import cors from "cors";
import helmet from "helmet";
import dotenv from "dotenv";
import mongoose from "mongoose";
import morgan from "morgan";
import { graphqlHTTP } from "express-graphql";
import schema from "./schema";

dotenv.config();
app.use(morgan("common"));

// USE HELMET AND CORS MIDDLEWARES
app.use(
  cors({
    origin: ["*"], // Comma separated list of your urls to access your api. * means allow everything
    credentials: true, // Allow cookies to be sent with requests
  })
);
// app.use(helmet());
app.use(
  helmet({
    contentSecurityPolicy:
      process.env.NODE_ENV === "production" ? undefined : false,
  })
);

app.use(express.json());

// DB CONNECTION

if (!process.env.MONGODB_URL) {
  throw new Error("MONGO_URI environment variable is not defined");
}

mongoose
  .connect(process.env.MONGODB_URL)
  .then(() => {
    console.log("MongoDB connected to the backend successfully");
  })
  .catch((err: Error) => console.log(err));

app.use(
  "/graphql",
  graphqlHTTP({
    schema,
    graphiql: true,
  })
);

// Start backend server
const PORT = process.env.PORT || 8500;

// Check if it's not a test environment before starting the server

app.listen(PORT, () => {
  console.log(`Backend server is running at port ${PORT}`);
});

export default app;

Step 6: Dockerising our Notifications Service

Create a new file in the root: Dockerfile and place the following code:

# Use an official Node.js runtime as a parent image
FROM node:latest as builder

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install app dependencies
RUN npm install

# Copy all files from the current directory to the working directory
COPY . .

# Development stage
FROM builder as development
# Set NODE_ENV to development
ENV NODE_ENV=development

# Expose the port the app runs on
EXPOSE 8900

# Command to run the application(in development)
CMD ["npm", "run", "dev"]

# Production stage
FROM builder as production
# Set NODE_ENV to production
ENV NODE_ENV=production

# Run any production-specific build steps if needed here

# Run the production command
CMD ["npm", "start"]

and another file, still in the root: .dockerignore and place the following code:

/node_modules
npm-debug.log
.DS_Store
/*.env
./idea

Also, .dockerignore helps us ignore some files and directories while building the docker image just like you see .gitignore while using git.

All set, lets now build and run the docker image with the following commands:

docker build -t notification_service:development --target development .

and

docker run -p 8500:8500 -v $(pwd):/usr/src/app -e PORT=8700 notification_service:development

respectively.

If everything is well, you should have the following in terminal:

You can also spin up/split another terminal instance and run unit tests(inside the docker container) just like we did on other services, with the following command:

sudo docker exec -it your_docker_container_id  npm test

RabbitMQ Setup and Implementation

RabbitMQ, with its decoupling mechanism, will provide us several benefits:

Asynchronous Communication:

It will enable asynchronous communication, allowing our microservices to continue processing requests independently without waiting for a response from other services.
Forexample when placing an order, the order service can publish a message to update inventory or trigger other processes without blocking the user's request.

Resilience and Scalability:

Our microservices will be able to handle requests more resiliently. If a service is temporarily unavailable, the message broker will hold the message until the service is back online.
Scalability will be improved, as microservices can scale independently without direct dependencies on one another.

Loose Coupling:

Our microservices will evolve independently without tight dependencies. Changes in one service won't necessarily affect others, as long as the message format remains consistent.
This is particularly useful in our ecommerce application, where you might add new features or services without affecting the existing ones.

Event-Driven Architecture:

Message brokers support event-driven architecture, allowing microservices to react to events or updates in real-time.
In our ecommerce app, we will be able to trigger events for promotions, inventory updates, or order status changes among others.

consider the order placement scenario:

Without Message Broker:

User places an order.
Order service processes the order, updates inventory, and notifies other services directly via HTTP.
If one service is slow or unavailable, it might affect the entire user experience.

With Message Broker:

User places an order.
Order service publishes an "Order Placed" event to the message broker.
Inventory service and other relevant microservices subscribe to the "Order Placed" event and perform their actions asynchronously.
The user receives a response immediately, and the order processing continues in the background.

While it's possible to build microservices without a message broker just like our current system at the moment, using one provides a more flexible, scalable, and resilient architecture, especially in complex scenarios like ecommerce where various microservices need to collaborate without introducing tight coupling.
That being said, lets commence with our previously chosen RabbitMQ setup.

RabbitMQ Setup

In development, I highly recommend you setup a local RabbitMQ instance and as always its setup may vary depending on your operating system.
Because RabbitMQ primarily operates as a server and doesn't have a native desktop application for Windows, macOS, or Linux, I will provide all the commands you need to thrive setting it up on your local machine if youre using linux. If youre using windows or mac, I recommend you checkout this guide to have your environment setup.

RabbitMQ Docker Image with Management UI:

You can also run RabbitMQ in a Docker container with the management plugin enabled, which allows you to access the management UI. This is more suitable for development and testing purposes.
Run the following command on your terminal:

docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:management

and then access the management UI at http://localhost:15672/.

If youre using linux, follow the following instructions to get RabbitMQ setup on your local machine:

sudo apt-get update
sudo apt-get install rabbitmq-server

Start the server with the following command:

sudo service rabbitmq-server start

enable RabbitMQ management plugin with the following command:

# Enable the plugin
rabbitmq-plugins enable rabbitmq_management

After that, create a user and virtual host for your application

# Create a user
rabbitmqctl add_user your_user your_password

# Give the user administrative privileges
rabbitmqctl set_user_tags your_user administrator

# Create a virtual host
rabbitmqctl add_vhost your_virtual_host

# Set permissions for the user on the virtual host
rabbitmqctl set_permissions -p your_virtual_host your_user ".*" ".*" ".*"

Rabbitmq in Prodcution Environments.

Before we start implementing RabbitMQ in our microservice, I prefer we fist discuss about how you go about it in production and its what I will actually use in this guide, not a local server.
Also note that, this is all about environment setup and we havent approached its actual implementation, thats where we will discuss client libraries for various technologies and we will be moving on with most commonly used one with nodejs.

In production, we have various options for setting up our RabbitMQ depending on the hosting service/cloud provider.
Lets discuss 3 most commonly used ones and we will be moving with only one in this guide.

Cloud-Neutral Option: External RabbitMQ Service:

You can host RabbitMQ on a cloud provider or service that is cloud-neutral and can be accessed from any hosting provider.
Examples include services like RabbitMQ on AWS (using Amazon MQ), RabbitMQ on Azure, or a RabbitMQ instance hosted on a platform like Heroku.
This approach allows your microservices hosted on different cloud providers to connect to the same RabbitMQ instance.

Host RabbitMQ on One of the Cloud Providers:

You can also choose one of your cloud providers (e.g., AWS) to host RabbitMQ.
Ensure that the RabbitMQ instance is accessible from other cloud providers. This might involve setting up networking configurations, security groups, or VPC peering, depending on the specific cloud providers you are using.
Microservices hosted on different cloud providers would then connect to the RabbitMQ instance on the chosen cloud provider.

Hybrid Approach: Multiple RabbitMQ Instances:

Lastly, you can host separate RabbitMQ instances on each cloud provider (AWS, Render, Railway, etc) but configure them to communicate with each other.
Use RabbitMQ's federation or shovel features to link different RabbitMQ instances together, allowing messages to be exchanged across instances.
This approach provides a degree of isolation but still allows for communication between microservices on different cloud providers.

In this article we will setup our RabbitMQ on AWS and configure it to communicate with other microservices wherever they will be hosted, using Amazon MQ. It is a managed message broker service for Apache ActiveMQ and RabbitMQ that simplifies setup and operation of open-source message brokers on AWS.

Setup

Step 1: Create AWS Account

Head over to AWS and login or create a new account. Note that, AWS has a 12 month free tier and allows customers to use the product for free up to specified limits. Additionally, creating a new account involves submitting your credit card details.

Step 2: Create Broker

On the AWS console, search for Amazon MQ and proceed to https://console.aws.amazon.com/amazon-mq.
Click the get started button, select RabbitMQ as your broker engine and follow on screen instructions to create a new message broker selecting options that apply to your use case.
Configure the broker instance details, including the instance type, storage, and authentication method.
Choose the VPC and subnet settings for your RabbitMQ broker.
Set up security groups and define rules to control inbound and outbound traffic.

Step 3: Complete Broker Setup

After reviewing your configuration, click create broker button and the provisioning may take to around 20 minutes.
If all is well, you should have the following on your created broker's dashboard.

In our case, we will be using this RabbitMQ setup for both development and production, so after provisioning the broker, we will pick some credentials and take them to our .env file and start implementation.

RabbitMQ Implementation in NodeJS Microservices

From what we have architected so far, we need only 2 functions to implement RabbitMQ and use it in communicating our microservices.

The first function is to publish an event to RabbitMQ server. In our case, we are meaning the one we just created on AWS Amazon MQ. If you did setup a local RabbitMQ instance, you will also have credentials from your local setup.

The second function is for subscribing to these events on RabbitMQ.
with these 2 functions, you will find RabbitMQ implementation easier than you ever imaged. Since we need the same functions across nearly all our microservices, we are going to add them to our shared library instead of hard coding them in every service. We will pass all the credentials and everything else we need as a parameter and we will be good to go.

Writing Shared RabbitMQ Functions

Inside the nodejs_ms_shared_library's root directory, create a new file: src/rabbitMQUtils.ts and place the following code:

import * as amqp from "amqplib";

// Subscribe to rabbitmq
export async function subscribeToRabbitMQ(
  exchange: string,
  routingKey: string,
  handleMessage: (message: Buffer) => void,
  rabbitMQConfig: {
    host: string;
    port: number;
    username: string;
    password: string;
  }
): Promise<void> {
  const connection = await amqp.connect({
    protocol: "amqp",
    hostname: rabbitMQConfig.host,
    port: rabbitMQConfig.port,
    username: rabbitMQConfig.username,
    password: rabbitMQConfig.password,
  });

  const channel = await connection.createChannel();
  await channel.assertExchange(exchange, "direct", { durable: false });

  const queue = await channel.assertQueue("", { exclusive: true });
  channel.bindQueue(queue.queue, exchange, routingKey);

  channel.consume(
    queue.queue,
    (msg) => {
      if (msg !== null && msg.content) {
        handleMessage(msg.content as Buffer);
      }
    },
    { noAck: true }
  );
}

// Publish event to rabbitmq
export async function publishToRabbitMQ(
  exchange: string,
  routingKey: string,
  message: string,
  rabbitMQConfig: {
    host: string;
    port: number;
    username: string;
    password: string;
  }
): Promise<void> {
  const connection = await amqp.connect({
    protocol: "amqp",
    hostname: rabbitMQConfig.host,
    port: rabbitMQConfig.port,
    username: rabbitMQConfig.username,
    password: rabbitMQConfig.password,
  });

  const channel = await connection.createChannel();
  await channel.assertExchange(exchange, "direct", { durable: false });

  // Publish the message to the exchange with the specified routing key
  channel.publish(exchange, routingKey, Buffer.from(message));

  // Close the connection
  await channel.close();
  await connection.close();
}

amqplib is a library for making AMQP 0-9-1 clients for Node.JS, and an AMQP 0-9-1 client for Node.JS v10+. It implements the machinery needed to make clients for AMQP 0-9-1, and includes such a client.

The exchange is the routing mechanism for RabbitMQ. When you publish a message, you specify an exchange and a routing key.
The routingKey is a key that the exchange uses to route the message to the correct queue.
handleMessage: This is a callback function that will be invoked when a message is received from the RabbitMQ queue. You define the logic for handling the message inside this function.
rabbitMQConfig:
host: You can find the RabbitMQ host in the AWS Management Console under the details of your RabbitMQ broker. It's usually in the format of your-broker-name.YourRegion.amazonaws.com.
port: The default port for RabbitMQ is 5672. This is the port your RabbitMQ broker is listening on.
username and password: These are the credentials you set up when configuring your RabbitMQ broker. You can find these in the AWS Management Console as well.

All set, before we publish a new version of our library, we need to pass through all the steps we passed through and they are pretty straight forward.
First, include the new functions in the index file:

export {
  generateToken,
  verifyTokenAndAdmin,
  verifyTokenAndAuthorization,
  CustomRequest,
  JWTPayload,
  VerifyErrors,
  verifyToken,
  Secret,
  Response,
} from "./jwtUtils";

export { subscribeToRabbitMQ, publishToRabbitMQ } from "./rabbitMQUtils";

Then update library version number. Like we agreed, we will be using patch. Run:

npm version patch

Now we need to transpile typescript once again to generate fresh JavaScript files in dist that includes our changes. Run:

npm run build

All set, lets finalise with the command that pushes to npm repository(Assuming we are still logged in both on terminal and in the browser).

npm publish --access public

Check your npm repository and you should have an update with the version number corresponsing to the one you generated when your ran: npm version patch.
For now, since all our work towards npm is still manual, you will uninstall the shared library and install it afresh in all the microservices that you had already installed it.

We have quite many places already where we may possibly send events to RabbitMQ with our deployed function for doing so, such that other services that are concerned about the event can subscribe to it, lets however start with product creation.
When a new product is created, we will publish a ProductCreated event to rabbitmq and the notifications service will subscribe to the event.

Update the controller for creating a new product to look like this:

import { Request, Response } from "express";
import * as productService from "../services/productService";
import { CustomRequest } from "nodejs_ms_shared_library";
import { publishToRabbitMQ } from "nodejs_ms_shared_library";

const createProduct = async (req: CustomRequest, res: Response) => {
  try {
    const product = await productService.createProduct(req.body);

    // After successfully creating a product, publish a "ProductCreated" event
    const productCreatedEvent = {
      productId: product.id, // Assuming you have the product's ID in the model
      productName: product.title,
      // add any other relevant product data ...
    };

    // Use the RabbitMQ publishing function
    await publishToRabbitMQ(
      "your_exchange",
      "ProductCreated",
      JSON.stringify(productCreatedEvent),
      {
        host: process.env.RABBIT_MQ_HOST,
        port: Number(process.env.RABBIT_MQ_PORT),
        username: process.env.RABBIT_MQ_USERNAME,
        password: process.env.RABBIT_MQ_HOST,
      }
    );

    res.status(201).json({
      message: "Product created successfully!",
      user: req.user,
      product,
    });
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Internal Server Error" });
  }
};

In the notifications_service, head over to the index file, just after mongodb connection section and add the following code for subscribing to RabbitMQ event:

// RabbitMQ subscription for "ProductCreated" event
subscribeToRabbitMQ(
  "your_exchange",
  "ProductCreated",
  async (message) => {
    try {
      // Parse the incoming message
      const productCreatedEvent = JSON.parse(message.toString());

      // Handle the event in your NotificationService e.g send a notification to user that created
      // the product and tell him a new product was created under his account for transprency
      // You can destructure this event to pick up only information you need
      // key value pairs in the new notification
      await NotificationService.addNotification(productCreatedEvent);

    } catch (error) {
      console.error("Error processing ProductCreated event:", error);
    }
  },
  {
    host: process.env.RABBIT_MQ_HOST,
    port: Number(process.env.RABBIT_MQ_PORT),
    username: process.env.RABBIT_MQ_USERNAME,
    password: process.env.RABBIT_MQ_HOST,
  }
);

Donot forget to import the function from our library on top in the imports section:

import { subscribeToRabbitMQ } from "nodejs_ms_shared_library";

Congulatulations, you have successfully implemented RabbitMQ message broker in nodejs microservice architecture.
Note that, RabbitMQ message exchange doesnot happen in realtime by default, if you need realtime message exchange you implement it on your own using technologies like socket.io , vanilla websockets or graphql subscriptions among others.
Nothing changes from the actual implemenation, so I would suggest you use a technology youre already familiar with implementing real time features in nodejs in case need for realtime RabbitMQ message exchange arises.

We have just implemented a small portion of the overall usage in an ecommerce application but the main goal is to get the concept as we may have various scenarios we need both to publish events to RabbitMQ and have them subscribed to by other services.
Another potential usecase is when an order is placed, assuming we have the shipping and inventory microservices, they would all need to subscribe to that event published by order microservice and keep getting updates about order status as well. In the same way, like we have seen publishng a new event upon successfully creting a new product, you can publish as many events as you need to in as many microservices as you need to.

Hosting Microservices

Hosting microservices involves deploying and managing individual services that make up the application architecture. There are various ways to host microservices, and the choice depends on your specific requirements, technology stack, and preferences.
In this article, we will discuss the most common approaches in hosting microservices:

Cloud Platforms:

Amazon Web Services (AWS): Services like Elastic Container Service (ECS), Elastic Kubernetes Service (EKS), and AWS Lambda can be used to deploy and manage microservices.
Microsoft Azure: Azure Kubernetes Service (AKS), Azure Container Instances (ACI), and Azure Functions are popular options for hosting microservices.
Google Cloud Platform (GCP): Google Kubernetes Engine (GKE), Google Cloud Run, and Cloud Functions are suitable for hosting microservices.

Container Orchestration:

Kubernetes is a powerful container orchestration platform widely used for deploying, managing, and scaling containerized applications, including microservices.
Docker Swarm: Docker Swarm is Docker's native clustering and orchestration solution that allows you to deploy and manage containers at scale.

Serverless Computing:

AWS Lambda:
Ideal for functions and small microservices that can be triggered by events.
Azure Functions: Serverless computing offering on Azure, supporting multiple programming languages.
Google Cloud Functions: Allows you to deploy and run event-driven functions.

Platform-as-a-Service (PaaS):

Services like Heroku, Cloud Foundry, and others provide a platform for deploying and managing applications without worrying about the underlying infrastructure.

Containerization:

Use containerization platforms like Docker to package microservices along with their dependencies into containers. You can then deploy these containers to various environments, including Kubernetes, Docker Swarm, or other container orchestration solutions.

Self-Managed Infrastructure:

You can host microservices on your own infrastructure, either on-premises or using virtual machines in a cloud environment.

In our case, since we used Docker to containerise each microservice, with some requiring docker-compose to run various services at ago, you can look through options that involve docker/containerisation.

Until next time, happy coding!

Helpful Links:

How to Dockerise a NodeJS - TypeScript API || A Comprehensive Guide from Environment Setup to Deployment with a CI/CD Pipeline

Abeinemukama Vicent — Thu, 18 Jan 2024 10:11:00 +0000

In the dynamic world of software development, streamlining the deployment process and ensuring consistency across different environments are crucial aspects of building robust and scalable applications. Docker, a powerful containerization platform, has become an indispensable tool for developers looking to achieve these goals. This article will guide you through the process of Dockerizing a Node.js TypeScript API, from setting up the development environment to deploying a dockerised application with an efficient CI/CD pipeline.

What Exactly is Docker

Docker is a containerization platform designed to simplify the packaging, distribution, and deployment of applications.
It uses OS-level virtualization to deliver software in packages called containers.
At its core, a Docker container is a lightweight, standalone executable package that includes everything needed to run a piece of software, including the code, runtime, libraries, and system tools. This article delves into the fundamental concepts of Docker, explaining how containers differ from traditional virtualization and highlighting the key components that make Docker a versatile and powerful tool for developers. Gain insights into the Docker image, container, and Dockerfile, and comprehend the significance of Docker in creating reproducible and isolated environments for Node.js TypeScript APIs.

Pros and Cons of Docker

Docker brings a myriad of advantages to the table, transforming the way we develop, deploy, and manage applications. In this section, we'll delve into the benefits that make Docker a go-to solution for developers. From consistent environments across different stages of development to improved scalability and resource utilization, Docker streamlines the development lifecycle.

However, no technology is without its considerations. We'll also explore the potential disadvantages and challenges that developers may encounter when adopting Docker. Understanding both the strengths and limitations will empower you to make informed decisions as you Dockerize a Node.js TypeScript API. Let's navigate through the advantages and disadvantages of Docker, ensuring a comprehensive view of its role in modern application development.

Pros:

Portability:

Containers encapsulate the application and its dependencies, making it highly portable across different environments. The same container can run consistently on a developer's laptop, testing servers, and production systems.

Isolation:

Containers provide a level of isolation, ensuring that an application and its dependencies are isolated from the host system and other containers. This helps in avoiding conflicts between different applications.

Resource Efficiency:

Docker containers share the host OS kernel, making them more lightweight than traditional virtual machines. This results in better resource utilization and faster startup times.

Scalability:

Docker simplifies the process of scaling applications. Multiple instances of a containerized application can be easily deployed and managed, either manually or through orchestration tools like Docker Swarm or Kubernetes.

Consistency:

Docker ensures consistency across different environments. Developers can work in the same environment as the production system, reducing the "it works on my machine" problem.

Versioning and Rollback:

Docker allows versioning of images, making it easy to roll back to previous versions in case of issues. This is particularly useful for deployments and updates.

Community and Ecosystem:

Docker has a large and active community, leading to a vast ecosystem of pre-built images, plugins, and tools. This community support can be valuable for problem-solving and knowledge sharing.

Continuous Integration and Deployment (CI/CD):

Docker containers integrate well with CI/CD pipelines, allowing for automated testing, building, and deployment. This promotes a more streamlined and efficient development process.

Cons:

Learning Curve:

Docker has a learning curve, especially for those who are new to containerization concepts. Understanding container orchestration tools and Dockerfile syntax may take time.

Security Concerns:

While Docker provides isolation, misconfigurations or vulnerabilities in the host kernel could potentially pose security risks. It's crucial to follow best practices and keep containers and images up to date.

Resource Overhead:

While containers are more lightweight than virtual machines, there is still some overhead associated with running multiple containers on a host system.

Limited Windows Support:

Docker was initially designed for Linux, and although it has improved its support for Windows, it may not be as seamless or feature-rich on Windows as it is on Linux.

Persistence and Stateful Applications:

Docker containers are often designed to be stateless, which can be challenging for applications that require persistent storage. Handling data persistence and managing stateful applications may require additional configurations.

Orchestration Complexity:

While Docker simplifies containerization, managing large-scale container deployments with orchestration tools like Kubernetes can introduce complexity. Understanding and configuring these tools can be challenging for some users.

Networking Challenges:

Configuring and managing container networks can be complex, especially when dealing with multiple containers that need to communicate with each other. Understanding Docker networking modes and creating custom networks may require additional expertise.

Image Size:

Docker images can become large, especially if they include unnecessary dependencies. This can impact storage requirements and increase the time it takes to transfer images over the network.

While docker offers many advantages in terms of portability, isolation, and resource efficiency, it's important to be aware of potential challenges such as security considerations, the learning curve, and the complexities associated with managing large-scale containerized applications in order to make an informed decision adopting docker.

Terminologies Used in Docker (Most Common)

Dockerfile:

A Dockerfile is a text file that contains a set of instructions for building a Docker image. These instructions define the base image, set up environment variables, copy files into the image, and execute commands during the image build process. Understanding the Dockerfile is crucial for customizing and optimizing the containerized environment for a Node.js TypeScript API.

docker-compose:

Docker Compose is a tool for defining and running multi-container Docker applications. It allows you to specify the services, networks, and volumes for your application in a single YAML file (docker-compose.yml). This simplifies the process of orchestrating multiple containers, making it easier to manage complex configurations and dependencies.

.dockerignore:

Similar to .gitignore, the .dockerignore file specifies patterns for files and directories that should be excluded when building a Docker image. This helps reduce the size of the image by excluding unnecessary files and directories like node_modules, making the image more efficient and lightweight.

Docker Image:

A Docker image is a lightweight, standalone, and executable package that includes everything needed to run a software application, including the code, runtime, libraries, and dependencies. Images are the building blocks used to create Docker containers.

Docker Container:

A Docker container is a running instance of a Docker image. It encapsulates the application and its dependencies in an isolated environment, ensuring consistency across different environments. Containers are portable and can run on any system that supports Docker.

Docker Registry:

A Docker registry is a repository for storing and sharing Docker images. Popular public registries include Docker Hub, where you can find a vast collection of pre-built images. Organizations often use private registries for hosting proprietary images.
Other common docker registries include Amazon Elastic Container Registry (Amazon ECR) for AWS, Google Container Registry (GCR) for Google Cloud and Azure Container Registry (ACR) for Microsoft Azure.
In this comprehensive guide, we will be deploying our image to dockerhub both manually and using a custom ci/cd pipeline with github actions. A future article may specialise on deploying docker on any of the other registries discussed but the steps are generally the same except minimal platform specific configurations.

When you deploy a Docker image to Docker Hub, it means that your containerized application is available for others to pull and run on their local systems or servers. However, Docker Hub itself is primarily a registry for Docker images and doesn't provide a runtime environment for hosting your application.
If you want to run your backend application in a production environment, you'll still need to deploy it to a hosting platform. Platforms like Heroku, AWS, Google Cloud, Azure, and others provide infrastructure and services to run and manage containerized applications.

Volume:

A Docker volume is a mechanism for persisting data generated by and used by Docker containers. Volumes enable data to persist beyond the lifecycle of a container, making them essential for scenarios like database storage or sharing data between containers.

Docker Network:

Docker networks provide communication between containers. By default, containers on the same network can communicate with each other using container names as hostnames. Docker networks facilitate seamless communication between different services within a Dockerized application.

Understanding these common Docker terminologies is fundamental to mastering the art of containerization and will serve as a solid foundation as we proceed with Dockerizing a Node.js TypeScript API.

Prerequisites

Before we embark on the journey of Dockerizing a Node.js TypeScript API, it's essential to ensure that your development environment is properly configured. This section outlines the prerequisites that you need to have in place before diving into the Dockerization process.

Node.js and npm:

Have Node.js and npm (Node Package Manager) installed on your system. These are essential for developing and running our Node.js TypeScript API. You can download Node.js from the official website or use a version manager like nvm for better control over Node.js versions.

Text Editor or IDE:

Choose a text editor or integrated development environment (IDE) for writing your TypeScript code. Popular choices include Visual Studio Code, Atom, or any editor of your preference.

MongoDB Atlas Account:

If you plan to use MongoDB as a database, sign up for a MongoDB Atlas account. This cloud-based database service offers a free tier and simplifies the process of managing MongoDB databases for our Dockerized Node.js TypeScript API.

GitHub Account (Optional):

Basic Knowledge of TypeScript and Express:

Familiarize yourself with TypeScript and Express.js, as they form the foundation of our Node.js API. If you're new to these technologies, consider exploring this article and this to get comfortable with the basics.

Understanding of RESTful APIs:

A basic understanding of RESTful API concepts is beneficial. This knowledge will help you structure the API endpoints effectively as we Dockerize our Node.js TypeScript API.
By ensuring that these prerequisites are met, you'll be well-prepared to follow the subsequent steps in this comprehensive guide. Now, let's move forward and start building our Dockerized Node.js TypeScript API.

Folder Structure

Following will be the overall folder structure for our dockerised nodejs-typescript api:

we will understand what each file contains step by step.

Steps to Dockerize our Application

Step 1: Install Docker and Docker Compose

Depending on your operating system, installation of docker may differ, if you're using windows or mac, I highly recommend installing docker desktop, follow this guide for windows and this for mac.
As of this writing, docker has separate desktop applications for mac with apple silicon and mac with intel chip, so if you're using mac, be careful installing one that aligns with your processor.

If youre using windows, do not forget to enable virtualization in the BIOS/UEFI settings. After that, also enable Hyper-V(Hardware Virtualisation).It is Microsoft's hypervisor technology, and it provides virtualization capabilities for running virtual machines (VMs) on Windows.
When you install Docker on Windows, it uses Hyper-V to create a lightweight virtual machine called MobyLinuxVM to host the containers. Hyper-V is responsible for managing these containers and ensuring their isolation from the host system.

If youre using linux, instead of docker desktop, I recommend installing docker engine as it comes bundled with Docker Desktop for Linux. This is the easiest and quickest way to get started on linux. You can follow this guide to complete its setup.

If youre using any Debian-based Linux distribution, I will provide the commands to help you install everything required without leaving your command line:

First, update the package index:



sudo apt update

Install necessary dependencies:



sudo apt install -y apt-transport-https ca-certificates curl software-properties-common

Add the Docker GPG key:



curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Add the Docker repository:



echo "deb [signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Update the package index again:



sudo apt update

Install Docker:



sudo apt install -y docker-ce docker-ce-cli containerd.io

Add your user to the docker group to run Docker commands without sudo:



sudo usermod -aG docker $USER

Log out and log back in or restart your system to apply the group changes.

Install Docker Compose:

Download the Docker Compose binary:



sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Apply execute permissions to the binary:



sudo chmod +x /usr/local/bin/docker-compose

Test the installation:
Run:



docker-compose --version

to confirm docker compose installation and:



docker --version

to confirm docker installation.
These commands should display the installed Docker Compose and docker version respectively.

Step 2: Initialize a NodeJS-TypeScript Project

Instead of initialising a new NodeJS app, we will clone our previous project for the artcle that was about unit testing with node and typescript using jest and supertest in order to concentrate on docker in today's article.
It was a 3 part article and here is it:

To clone the repository, first create a new folder in your desired location and open it with your desired code editor e.g vscode.
I called mine: node_ts_api_with_docker but feel free to call any other name of your convenience.
On the terminal, run:



git clone https://github.com/Abeinevincent/nodejs_unit_testing_guide .

to clone the repository to your created folder.
Install dependencies:



npm install

Step 3: Dockerize the Application with Dockerfile

Create a Dockerfile in the project root. This file defines the steps for building your Docker image. Use the following as a starting point:



# Use an official Node.js runtime as a parent image
FROM node:latest as builder

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install app dependencies
RUN npm install

# Copy all files from the current directory to the working directory
COPY . .

# Development stage
FROM builder as development
# Set NODE_ENV to development
ENV NODE_ENV=development

# Expose the port the app runs on
EXPOSE 8800

# Command to run the application(in development)
CMD ["npm", "run", "dev"]

# Production stage
FROM builder as production
# Set NODE_ENV to production
ENV NODE_ENV=production

# Run any production-specific build steps if needed here

# Run the production command
CMD ["npm", "start"]

Lets understand each line of code in our Dockerfile one at time because this is why we are here 😄

FROM node:latest as builder

This line specifies the base image for your Docker image. In this case, it uses the official Node.js image from Docker Hub with the latest tag. The as builder is an optional alias for this stage, allowing you to reference this stage later in the Dockerfile.

WORKDIR /usr/src/app

Sets the working directory inside the container to /usr/src/app. This is the directory where subsequent commands will be executed, and it's a common practice to organize your application code.

COPY package*.json ./

Copies the package.json and package-lock.json files from the host machine (where your Dockerfile is located) to the working directory inside the container.

RUN npm install

Installs the Node.js application dependencies specified in package.json using the npm package manager. This step is crucial to ensure that all required dependencies are available in the container.

COPY . .

Copies all files from the current directory on the host machine to the working directory inside the container. This includes the application code, configuration files, and any other files required for the application.

FROM builder as development

Starts a new stage in the Dockerfile named development, using the alias builder from the previous stage. This allows you to reference the previous stage's files and dependencies.

ENV NODE_ENV=development

Sets the NODE_ENV environment variable to development. This helps us configure different behaviors based on the environment.

EXPOSE 8800

Informs Docker that the application inside the container will use port 8800. Note that this does not actually publish the port; it's more of a documentation feature.

CMD ["npm", "run", "dev"]

Specifies the default command to run when the container starts in development mode. In this case, it runs the npm run dev script, assuming that it's defined in your package.json.

FROM builder as production

Starts a new stage in the Dockerfile named production, using the alias builder from the previous stage.

ENV NODE_ENV=production

Sets the NODE_ENV environment variable to production. This is common for production environments to ensure optimized settings.

Run any production-specific build steps if needed here

This line is a comment indicating that you can add any production-specific build steps or configurations if needed. This is a placeholder for actions specific to the production environment.

CMD ["npm", "start"]

Specifies the default command to run when the container starts in production mode. In this case, it runs the npm start script, assuming that it's defined in your package.json.

Collectively, we have defined a multi-stage Dockerfile for a Node.js application. It allows us to build the application with different settings for development and production environments. we have structured it to optimize caching during the build process and separate concerns between development and production stages.

Create a .dockerignore File:

Create a .dockerignore file in the project root. This file specifies patterns for files and directories that should be excluded during the Docker image build process. Place the following code:



node_modules
dist
.git
.dockerignore
npm-debug.log

Exclude unnecessary files such as node_modules, dist (transpiled TypeScript files), and .git from being copied into the Docker image.

All set, lets first try to build our docker image and run it before we start on the docker compose journey.
To build the image in development,run the following command:



sudo docker build -t node_ts_api_with_docker:development --target development .

The command builds a Docker image named node_ts_api_with_docker with the tag development. It specifically targets the development stage in our multi-stage Dockerfile. This allows us to build the development stage independently of the production stage, which can be useful for testing and development workflows.
The . at the end of the command specifies the build context. It represents the current directory where the Dockerfile is located. All files and directories in this context will be sent to the Docker daemon during the build process.

If all is good, you should have this in your terminal:

Before we run our built image,lets first check for the image availability. To do that, run:



docker images

and check if your built image exists in the list of images you have.
If it does, run the following command to run the image:



sudo docker run -p 8800:8800 -v $(pwd):/usr/src/app -e PORT=8800 node_ts_api_with_docker:development

The sudo docker run command is used to launch a Docker container based on the specified image. In our case, the -p 8800:8800 flag maps port 8800 on the host machine to port 8800 within the running container, allowing external access to the application. The -v $(pwd):/usr/src/app flag mounts the current working directory ($(pwd)) on the host to the /usr/src/app directory inside the container, ensuring that any changes in the code are reflected in the container. The -e PORT=8800 flag sets the environment variable PORT to the value 8800 inside the container, allowing the Node.js application to use the specified port. Finally, node_ts_api_with_docker:development specifies the Docker image to be used, and development indicates that the container should be started in the development stage as defined in the multi-stage Dockerfile. Our command orchestrates the containerization of a Node.js application, exposing it on port 8800, and providing a seamless synchronization of code changes between the host and the container during development.

If all is well you should see the following in the terminal:

Congratulations, you have successfully run a docker image in development mode.
Now on wards you can start developing in the docker container. Try making some changes in the code and the development server inside the container should restart using nodemon upon saving.
To discover other docker commands that might help you in various scenarios like listing running containers, killing/stopping containers(you may need this to free some ports or related) among others, run the following command:



docker --help

The command lists all the possible docker commands alongside what they can help you achieve.

Because we are in development, you can spin up/split another terminal instance and run your unit tests(inside the docker container). To do that, you can use the following command:



sudo docker exec -it 0f9380148bb5  npm test

0f9380148bb5 is the docker container id for the image whose unit tests youre running.
If all is good, you should seethe following terminal:

Docker Compose Setup for MongoDB

Docker Compose is a tool for defining and running multi-container Docker applications. Utilizing Docker Compose for MongoDB in our Node.js API brings numerous benefits. It ensures environment consistency by defining the entire development environment, including MongoDB and the Node.js API, in a single configuration file, facilitating collaboration among developers. With a simplified setup, developers can start the entire stack with a single command, reducing configuration issues and ensuring proper dependency management. Docker Compose also offers isolation, running MongoDB and the Node.js API in separate containers for easier troubleshooting and independent scaling. This approach simplifies testing environments, as configurations can be tailored for testing scenarios. Furthermore, the same Docker Compose configuration used for development and testing can serve as a foundation for production deployment, promoting consistency across environments and minimizing deployment issues.

Since we are using MongoDB Atlas for our database, we don't have any specific requirements that necessitate Docker Compose (such as running multiple services locally).However,for illustration, we will create a simple service for running our database independent from the api.
If you use MongoDB campass or any RDBMS like PostgreSQL, MySQL among others, this is a must do, you need docker-compose truly to run the database service with the api service.
In our case, however much we are using MongoDB Atlas that gives us a fully-managed cloud database that handles all the complexity of deploying, managing, and healing your deployments on the cloud service, I will still demonstrate how docker-compose can help you achieve running more than one container at ago.

Create a new file in the root of the project, name it: docker-compose.yml and place the following code:



version: "3.8"

services:
  node-api:
    build:
      context: . #. means this docker-compose is inside the node api home directory, in this scenario, you can even abandon this option.
      dockerfile: Dockerfile
    ports:
      - "8800:8800"
    depends_on:
      - mongodb
    env_file:
      - .env # Use the same .env file for both services
    working_dir: /usr/src/app
    volumes:
      - /path/to/your/api:/usr/src/app
    command: npm run dev

  mongodb:
    image: mongo
    env_file:
      - .env # Use the same .env file for both services
    volumes:
      - mongodb-data:/data/db

volumes:
  mongodb-data:

Lets also understand each line of code in our docker-compose.yml file before we proceed.
Our Docker Compose file orchestrates two services (node-api and mongodb) and uses named volumes to ensure data persistence for the MongoDB service. The shared .env file contains environment variables used by both services.

version: "3.8": Specifies the version of the Docker Compose file format being used. In this case, it's version 3.8.
services: Begins the section where you define your services.

node-api: This is the name of the service. It represents your Node.js API service.

build: This section is used to specify the build context and Dockerfile for building the image.

context: Specifies the path to the directory containing your Node.js API code. In this case, it's the current directory (.) where the docker-compose.yml file is located.

dockerfile: Dockerfile: Specifies the name of the Dockerfile to use for building the image. In this case, it's Dockerfile in the same directory.
ports: Maps port 8800 on the host to port 8800 in the container. This allows you to access your Node.js API at http://localhost:8800 on your host machine.
depends_on: Specifies that this service depends on the mongodb service, ensuring that it starts after mongodb.

env_file: Specifies the file or files from which to read environment variables. In this case, it's .env, which is shared between both services.
working_dir: /usr/src/app: Sets the working directory inside the container to /usr/src/app.

volumes: Mounts the local /home/abeine/eoyprojects/nodejsapis/containerisedprojects/node_ts_api_with_docker directory into the container at /usr/src/app. This facilitates live code reloading during development.

command: npm run dev: Specifies the command to run when the container starts, which is npm run dev for your Node.js API.
mongodb: This is the name of the MongoDB service.

image: mongo: Specifies the Docker image to use for the MongoDB service. In this case, it's the official MongoDB image from Docker Hub.

env_file: Specifies the file or files from which to read environment variables. In this case, it's also .env, which is shared between both services.
volumes: Creates a named volume named mongodb-data and mounts it to /data/db inside the MongoDB container. This ensures data persistence.

mongodb-data: This is the name of the volume created for MongoDB data persistence.

To execute the docker-compose.yml configuration, run the following command:



docker-compose up

inside the directory where docker-compose.yml is.

If everything is well, you should see the success messages in the terminal like we saw before that Backend server is running at port 8800 and MongoDB connected to the backend successfully.

Deploying Docker Images

The easiest, most enjoyable, yet trickiest part of docker is deployment. In this article we will deploy our docker image first at a public docker registry called docker-hub and we will see how other team members can clone the image and start working right away after running the image.
For large teams or for hosting images on private repository on docker hub, charges may incur. You can visit the pricing page for a detailed billing structure.

Deploying Docker Images on Docker Hub

To deploy our previously built docker image on docker hub, head over to Docker Hub official website and login or create an account if you dont have one.
After logging in to docker hub, you also need to login on your terminal. Use the following command:



docker login

and follow on screen instructions to complete login process.
After that, create a new repository on docker hub and grab the repository name. In my case, I used same name as local image(node_ts_api_with_docker)

Back to terminal, run the following command to build the image for deployment:



docker build -t abeinevicenthome/node_ts_api_with_docker:latest .

Me I used latest an example for a generic tag. You might want to use a version number or any other tag that suits your versioning strategy.
Also, replace abeinevicenthome with your dockerhub username.

Lets now push our image to dockerhub.Use the following command:



docker push abeinevicenthome/node_ts_api_with_docker:latest

If the push is successful, you should have some thing like this in terminal:

Open docker hub once again and confirm your image:

Congulatulations once again, you successfully pushed a docker image to docker hub registry.

As you can see on the bottom right, automated builds are only available with Pro, Team and Business subscriptions.
You also have access to webhooks that can help you automate your builds if you conform to the pricing requirements.

However, on our free tier, we still have an option of using our custom CI/CD pipeline using any platform of our choice and in this article we will be using github actions as a CI/CD platform.
This will help us build and push new images when we make say a push to the main branch of our github repository.

The process is as easy as setting up any other pipeline with github actions.
in the project root, create a new file: .github/workflows/docker_deploy.yml and place the following code:



name: Node Docker Deploy

on:
  push:
    branches:
      - master

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Install Dependencies
        run: npm install

      - name: Run Unit Tests
        env:
          MONGODB_URL: ${{ secrets.MONGODB_URL }}
          JWT_EXPIRY_PERIOD: ${{ secrets.JWT_EXPIRY_PERIOD }}
          JWT_SEC: ${{ secrets.JWT_SEC }}
        run: npm test

  build:
    runs-on: ubuntu-latest

    needs: test

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Build Docker Image
        run: docker build -t ${{ secrets.DOCKER_HUB_USERNAME }}/node_ts_api_with_docker:latest .

      - name: Push Docker Image
        run: |
          echo ${{ secrets.DOCKER_HUB_TOKEN }} | docker login -u ${{ secrets.DOCKER_HUB_USERNAME }} --password-stdin
          docker push ${{ secrets.DOCKER_HUB_USERNAME }}/node_ts_api_with_docker:latest

Head over to your github and create a new repository, then include all the secrets used in the pipeline such as DOCKER_HUB_USERNAME, DOCKER_HUB_TOKEN, MONGODB_URL among others in the repository secrets. Follow the following path: settings -> Secrets and Variables -> Actions -> Repository Secrets -> New Repository Secret.
DOCKER_HUB_TOKEN can be gotten from your docker hub account settings. Follow the following path to get there: Profile Image -> My Account -> Security -> Access Tokens -> New Access Token and give it atleast read and write permissions.
The other secrets used are self explanatory like MongoDB URL and JWT secrets are already available in your local .env file. Copy and include them in your repository secrets too since we are not moving with .env on github and we need them in the process of unit testing.

Lets further understand our pipeline:
Our GitHub Actions workflow is designed to automate the testing, building, and deployment of our Node.js application with Docker.
The workflow consists of two jobs, "test" and "build," triggered on a push to the master branch. In the "test" job, the repository is checked out, dependencies are installed using npm, and unit tests are executed with environment variables such as MONGODB_URL, JWT_EXPIRY_PERIOD, and JWT_SEC being set from GitHub Secrets. The "build" job, dependent on the "test" job, involves checking out the repository again, building a Docker image with the application code, and pushing the image to Docker Hub using the provided secrets for Docker Hub authentication.
Our workflow streamlines the process of testing and deploying a Node.js application with Docker, ensuring a seamless integration into a CI/CD pipeline.

All set, lets push our code to github and observe our github actions workflow. To do that run the following commands



git add .
git commit -m "YourCommitMessage"
git push -u origin master

If all is good,you should see the following in github actions tab:

Pulling Docker Images from DockerHub

Next step is to understand how your team mates will clone and run the image. To pull image from docker hub, run the following command inside your directory housing the source code:



docker pull abeinevicenthome/node_ts_api_with_docker:latest

After pulling the image, the next step is to build and run it as previously discussed.

Sammary:

The Docker image crafted in this guide serves as a versatile and portable encapsulation of the Node.js API developed with TypeScript. This image, containing all the necessary dependencies and configurations, becomes a self-contained unit that can be effortlessly deployed on various hosting services. Whether it's Heroku, AWS, Google Cloud, Microsoft Azure or any other platform supporting Docker, the consistent environment provided by the Docker image ensures seamless deployment. The image acts as a standardized package, abstracting away the underlying infrastructure intricacies, and enabling developers to deploy their application across diverse hosting services with minimal effort. This level of abstraction not only simplifies deployment but also promotes compatibility, making the Dockerized Node.js API an adaptable solution for different hosting environments.

Conclusion:

In conclusion, the integration of Docker with a Node.js API written in TypeScript provides a robust and efficient solution for modern software development practices. The GitHub Actions workflow showcased here not only automates the testing and deployment processes but also encapsulates the application in a portable and reproducible Docker image. This containerization approach enhances the application's scalability, simplifies deployment across various environments, and facilitates seamless collaboration among developers. By harnessing the power of Docker and TypeScript in tandem, developers can achieve greater consistency, reliability, and ease of maintenance for their Node.js applications, paving the way for a more streamlined and agile development workflow. As technology continues to evolve, embracing containerization with Docker remains a valuable strategy for building resilient and scalable Node.js APIs.
Till next time, happy coding!

Important Links:

Build a GraphQL API with NodeJS and TypeScript || A Comprehensive Guide

Abeinemukama Vicent — Fri, 12 Jan 2024 05:11:34 +0000

GraphQL is an open-source data query and manipulation language for APIs and a query runtime engine. GraphQL enables declarative data fetching where a client can specify exactly what data it needs from an API.
GraphQL offers several benefits, including reducing over-fetching or under-fetching of data, enabling precise queries, and providing a more efficient and flexible API. It has gained widespread adoption in various domains due to its developer-friendly approach and ability to handle complex data requirements.
In this article, we will build a real world GraphQL movie directory API using NodeJS, TypeScript, MongoDB and Express.

GrapqL VS REST

Rest stands for Representational State Transfer and is the most commonly used architecure while developing and interacting with APIs (Application Programming Interfaces). In the ever-evolving realm of software development, choosing the right API architecture is a pivotal decision. Two prominent contenders, GraphQL and REST, have emerged as the go-to options, each with its own set of principles and methodologies. As developers navigate the intricate terrain of data retrieval and manipulation, understanding the distinctions between GraphQL and REST becomes imperative.

The two differ in their architectural principles and how they handle data retrieval.

REST (Representational State Transfer):

Endpoint-Centric:

In REST, each resource (e.g., user, post) is represented by a specific URL (endpoint).
Different endpoints are used for different operations (GET for retrieval, POST for creation, PUT/PATCH for updates, DELETE for deletion).

Fixed Structure:

REST APIs typically have a fixed structure, and the data returned from an endpoint is predetermined by the server.
Over-fetching (receiving more data than needed) or under-fetching (not getting enough data) can be common issues.

Stateless:

REST is stateless, meaning each request from a client contains all the information needed for the server to fulfill the request.

Multiple Endpoints:

Clients may need to make multiple requests to different endpoints to fetch all the required data.

Versioning:

Versioning is often used to handle changes in the API.

GraphQL:

Query Language:

GraphQL uses a query language to request only the data needed. Clients specify the shape and structure of the response.

Single Endpoint:

GraphQL typically uses a single endpoint, and clients can request exactly the data they need, avoiding over-fetching or under-fetching.

Real-time Data:

GraphQL supports real-time data with subscriptions, allowing clients to receive updates when data changes.

Introspection:

Clients can introspect the schema to discover what data can be queried, reducing the need for extensive documentation.

Flexibility:

Clients have more flexibility in defining the shape and structure of the data they want to receive.

GraphQL Concepts/Terminologies

Schema

The foundation of a GraphQL API is its schema, which defines the types and relationships within the data.

Types: GraphQL allows you to define custom types, such as Object, Scalar, Enum, Interface, Union, and Input.
Fields: Each type can have fields, and each field can return another type.



type User {
  id: ID!
  name: String!
  email: String!
  posts: [Post!]!
}

type Post {
  id: ID!
  title: String!
  content: String!
  author: User!
}

The exclamation mark (!) at the end of a type definition denotes that a particular field is non-nullable. It means that the field must always have a value and cannot be null.

Query

Clients request the specific data they need using queries.
The query structure mirrors the shape of the response.



query {
  user(id: "123") {
    name
    email
    posts {
      title
    }
  }
}

Mutation

Mutations are used for modifying or creating data on the server. ```javascript

mutation {
createUser(name: "John Doe", email: "john@example.com") {
id
name
}
}


###Subscription 
GraphQL supports real-time data with subscriptions, allowing clients to receive updates when specific events occur without relying on external libraries or additional technologies like socket.io.
The subscription is defined in the GraphQL schema, similar to queries and mutations.
Under the hood, GraphQL subscriptions typically use WebSocket connections to establish a persistent connection between the client and server.
When an event occurs (e.g., new data is available), the server can push updates to all subscribed clients over the WebSocket connection.
```javascript


subscription {
  newPost {
    title
    content
  }
}

Resolver

Resolvers are functions that define how to retrieve or manipulate data for each field in the schema.
Resolvers are specific to each field and type.

Introspection

GraphQL provides introspection, allowing clients to query the schema itself to discover available types, fields, and their types.



query {
  __schema {
    types {
      name
    }
  }
}

Project Setup

In this article,we will be using the following folder structure:

This structure is minimal but will help us understand the main concepts while using GraphQL to develop APIs in a NodeJS environment, after which you can come up with your own folder structure. A future article may concentrate on folder structure for a production grade GraphQL API, which may also include unit testing, user authentication and authorisation say using json web token and much more separation of concerns. For now, the above will work for us.

Install Dependencies

Create a new folder in your desired location and name it qraphql_api_guide.
With the folder open in your favourite code editor, open terminal and run the following command to initialise a NodeJS project:



npm init -y

After initialising the project, install the following dependencies with the following command:



npm install express cors helmet bcrypt dotenv mongoose express-graphql graphql graphql-http morgan

and



npm install -D @types/cors @types/express @types/bcrypt @types/morgan nodemon ts-node typescript

to install development dependencies.

In Node.js applications, we enhance functionality and security by incorporating several essential npm packages. Begin by installing express, a versatile web framework, along with cors for managing cross-origin resource sharing and helmet to fortify the app's security with HTTP header configuration. Ensure password security using bcrypt for hashing, and manage environment variables effortlessly with dotenv. For seamless interaction with MongoDB, we will use mongoose, an Object Data Modeling (ODM) library. To integrate GraphQL into our app, use express-graphql middleware, while the graphql package enables efficient query processing. Lastly, we will facilitate HTTP request logging for debugging purposes using morgan.

Before we start writing code, lets first complete setup of our development environment:
In the home directory, create a new file: nodemon.json and place the following code:



{
  "watch": ["src"],
  "ext": ".ts,.js",
  "exec": "ts-node ./src/index"
}

, tsconfig.json and place the following:



{
  "compilerOptions": {
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "target": "ES2020",
    "baseUrl": "src",
    "noImplicitAny": true,
    "sourceMap": true,
    "esModuleInterop": true,
    "outDir": "dist"
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules"]
}

and update package.jsons scripts section to add scripts for starting our server for both development and production mode:



"scripts": {
    "build": "tsc",
    "start": "npm run build && node dist/src/index.js",
    "dev": "nodemon"
  },

Also, add the line:



"module": "module",

to your package.json to specify that we are using EsModules not NodeJS's default CommonJS pattern.

All set, lets create our main file: src/index.ts and place the following:



import express from "express";
const app = express();
import cors from "cors";
import helmet from "helmet";
import dotenv from "dotenv";
import mongoose from "mongoose";
import morgan from "morgan";
import { graphqlHTTP } from "express-graphql";
import schema from "./schema";

dotenv.config();
app.use(morgan("common"));

// USE HELMET AND CORS MIDDLEWARES
app.use(
  cors({
    origin: ["*"], // Comma separated list of your urls to access your api. * means allow everything
    credentials: true, // Allow cookies to be sent with requests
  })
);
app.use(
  helmet({
    contentSecurityPolicy:
      process.env.NODE_ENV === "production" ? undefined : false,
  })
);

app.use(express.json());

// DB CONNECTION

if (!process.env.MONGODB_URL) {
  throw new Error("MONGODB_URL environment variable is not defined");
}

mongoose
  .connect(process.env.MONGODB_URL)
  .then(() => {
    console.log("MongoDB connected to the backend successfully");
  })
  .catch((err: Error) => console.log(err));

app.use(
  "/graphql",
  graphqlHTTP({
    schema,
    graphiql: true,
  })
);

// Start backend server
const PORT = process.env.PORT || 8500;
app.listen(PORT, () => {
  console.log(`Backend server is running at port ${PORT}`);
});

export default app;

we have initialized an Express server, configured with middleware to enhance security and functionality. We use the "cors" package for handling cross-origin resource sharing, allowing specified origins to access the API, and "helmet" to set secure HTTP headers, with conditional content security policy based on the environment. Environment variables are managed using "dotenv", and request logging is facilitated by the "morgan" middleware. We also have established a connection to a MongoDB database using the "mongoose" library. The GraphQL schema is defined in a separate file and integrated with the Express server using the "express-graphql" middleware. The server is set to run on a specified port, with console logs indicating successful connections and server initiation.

Lets start with completing database connection by creating a new file in home directory: .env and place the following code:



MONGODB_URL="Your MongoDB URL"

Head over to MongoDB and create an account or login to grab your connection string.

Writing Database Models

In our API, we will have only 2 models, User and Movie.
Create a new file: src/models/User.ts and place the following code:



// Import necessary modules
import mongoose, { Schema, Document, Types } from "mongoose";

// Define the interface for User document
export interface IUser extends Document {
  email: string;
  username: string;
  password: string;
  isAdmin: boolean;
  createdAt: Date;
  updatedAt: Date;
}

// Create a schema for the User model
const userSchema: Schema<IUser> = new Schema(
  {
    email: { type: String, required: true, unique: true },
    username: { type: String, required: true, unique: true },
    password: { type: String, required: true },
    isAdmin: { type: Boolean, default: false },
  },
  { timestamps: true }
);

// Create and export the User model
export default mongoose.model<IUser>("User", userSchema);

andsrc/models/Movie.ts and place the following code:



// Import necessary modules
import mongoose, { Schema, Document } from "mongoose";

// Define the interface for User document
export interface IMovie extends Document {
  title: string;
  genre: string;
  rating: number;
  duration: string; // e.g 2 hours
}

// Create a schema for the User model
const movieSchema: Schema<IMovie> = new Schema(
  {
    title: { type: String, required: true, unique: true },
    genre: { type: String, required: true },
    rating: { type: Number, required: true },
    duration: { type: String, required: true },
  },
  { timestamps: true }
);

// Create and export the User model
export default mongoose.model<IMovie>("Movie", movieSchema);

The User model is represented by an interface IUser, extending the Mongoose Document interface, specifying the structure of a user document. The user schema is created with Mongoose's Schema class, specifying fields such as email, username, password, and isAdmin (indicating whether the user has admin privileges). The schema includes additional options, such as setting timestamps for createdAt and updatedAt. These timestamps automatically update whenever a document is created or modified. The user model is then exported using mongoose.model, making it available for use throughout the application.

The Movie model is defined through an interface named IMovie, extending the Mongoose Document interface to outline the structure of a movie document. The schema is created using Mongoose's Schema class, specifying fields such as title, genre, rating, and duration. The schema also includes validations for required fields, and uniqueness is enforced for the movie titles. Additionally, timestamps for createdAt and updatedAt are automatically managed by Mongoose. The movie model is exported using mongoose.model, allowing it to be utilized throughout the application.

GraphQL Schema Design

All good, lets move to designing graphql schema.
Create a new file: src/schema/User.ts and place the following code:



import {
  GraphQLObjectType,
  GraphQLID,
  GraphQLString,
  GraphQLBoolean,
} from "graphql";

const UserType = new GraphQLObjectType({
  name: "User",
  fields: () => ({
    id: { type: GraphQLID },
    email: { type: GraphQLString },
    username: { type: GraphQLString },
    password: { type: GraphQLString },
    isAdmin: { type: GraphQLBoolean },
    createdAt: { type: GraphQLString },
    updatedAt: { type: GraphQLString },
  }),
});

export default UserType;

and src/schema/Movie.ts and place the following code:




import {
  GraphQLObjectType,
  GraphQLID,
  GraphQLString,
  GraphQLInt,
} from "graphql";

const MovieType = new GraphQLObjectType({
  name: "Movie",
  fields: () => ({
    id: { type: GraphQLID },
    title: { type: GraphQLString },
    genre: { type: GraphQLString },
    rating: { type: GraphQLInt },
    duration: { type: GraphQLString },
  }),
});

export default MovieType;

In the Movie schema, we define a GraphQL object type called MovieType using the GraphQLObjectType from the "graphql" library. This type is designed to represent movie-related data in a GraphQL schema. The MovieType has fields such as id, title, genre, rating, and duration, each specifying the type of data it holds (e.g., GraphQLID, GraphQLString, and GraphQLInt). The fields property is a function that returns an object containing the field definitions. This setup enables us to structure and standardize how movie information is queried and returned in GraphQL operations. We export MovieType for use throughout the application, facilitating a clear and consistent representation of movie entities in the GraphQL schema.

In the User schema, we create a GraphQL object type called UserType using the GraphQLObjectType from the "graphql" library. This type serves to define the structure of user-related data within a GraphQL schema. The UserType is composed of fields such as id, email, username, password, isAdmin, createdAt, and updatedAt, each specifying the type of data it represents (e.g., GraphQLID, GraphQLString, GraphQLBoolean). These fields collectively represent essential information about a user, including identifiers, authentication credentials, role permissions, and timestamps for creation and updates.

The use of UserType facilitates a standardized and coherent representation of user entities in GraphQL operations, promoting clarity and consistency throughout the application. By exporting UserType, we can easily integrate it into various parts of our GraphQL schema to define and interact with user-related data in a uniform manner.

Lets now create our GraphQL index file:src/schema/index.ts and put the following code:



import {
  GraphQLSchema,
  GraphQLObjectType,
  GraphQLList,
  GraphQLNonNull,
  GraphQLString,
  GraphQLInt,
  GraphQLBoolean,
} from "graphql";
import UserType from "./User";
import MovieType from "./Movie";
import Movie from "../models/Movie";
import User from "../models/User";
import { hashPassword } from "../utils/passwordUtils";

// Queries
const RootQuery = new GraphQLObjectType({
  name: "RootQueryType",
  fields: {
    // Query to get all users
    users: {
      type: GraphQLList(UserType),
      resolve: async () => {
        try {
          const users = await User.find();
          return users.map((user) => ({
            ...user.toObject(),
            id: user._id,
            createdAt: user.createdAt.toISOString(), // Format createdAt as ISO 8601
            updatedAt: user.updatedAt.toISOString(), // Format createdAt as ISO 8601
          }));
        } catch (error) {
          throw new Error(error.message);
        }
      },
    },

    // Query to get a user by ID
    user: {
      type: UserType,
      args: { id: { type: GraphQLNonNull(GraphQLString) } },
      resolve: async (_, args) => {
        try {
          const user = await User.findById(args.id);
          return {
            ...user.toObject(),
            id: user._id,
            createdAt: user.createdAt.toISOString(),
            updatedAt: user.updatedAt.toISOString(),
          };
        } catch (error) {
          throw new Error(error.message);
        }
      },
    },

    // Query to get all movies
    movies: {
      type: GraphQLList(MovieType),
      resolve: async () => {
        try {
          return await Movie.find();
        } catch (error) {
          throw new Error(error.message);
        }
      },
    },

    // Query to get a movie by ID
    movie: {
      type: MovieType,
      args: { id: { type: GraphQLNonNull(GraphQLString) } },
      resolve: async (_, args) => {
        try {
          return await Movie.findById(args.id);
        } catch (error) {
          throw new Error(error.message);
        }
      },
    },
  },
});

// Mutations
const Mutation = new GraphQLObjectType({
  name: "Mutation",
  fields: {
    // Mutation to add a new user
    addUser: {
      type: UserType,
      args: {
        email: { type: GraphQLNonNull(GraphQLString) },
        username: { type: GraphQLNonNull(GraphQLString) },
        password: { type: GraphQLNonNull(GraphQLString) },
        isAdmin: { type: GraphQLNonNull(GraphQLBoolean) },
      },

      resolve: async (_, args) => {
        try {
          // Destructure password
          const { password, ...others } = args;

          //   Send a hashed password
          const hashedPassword = await hashPassword(password);

          console.log(hashPassword);

          const user = new User({
            password: hashedPassword,
            ...others,
          });
          return await user.save();
        } catch (error) {
          throw new Error(error.message);
        }
      },
    },

    // Mutation to update a user by ID
    updateUser: {
      type: UserType,
      args: {
        id: { type: GraphQLNonNull(GraphQLString) },
        email: { type: GraphQLNonNull(GraphQLString) },
        username: { type: GraphQLNonNull(GraphQLString) },
        password: { type: GraphQLNonNull(GraphQLString) },
        isAdmin: { type: GraphQLNonNull(GraphQLString) },
      },
      resolve: async (_, args) => {
        try {
          return await User.findByIdAndUpdate(args.id, args, { new: true });
        } catch (error) {
          throw new Error(error.message);
        }
      },
    },

    // Mutation to delete a user by ID
    deleteUser: {
      type: UserType,
      args: { id: { type: GraphQLNonNull(GraphQLString) } },
      resolve: async (_, args) => {
        try {
          return await User.findByIdAndDelete(args.id);
        } catch (error) {
          throw new Error(error.message);
        }
      },
    },

    // Mutation to add a new movie
    addMovie: {
      type: MovieType,
      args: {
        title: { type: GraphQLNonNull(GraphQLString) },
        genre: { type: GraphQLNonNull(GraphQLString) },
        rating: { type: GraphQLNonNull(GraphQLInt) },
        duration: { type: GraphQLNonNull(GraphQLString) },
      },
      resolve: async (_, args) => {
        try {
          const movie = new Movie(args);
          return await movie.save();
        } catch (error) {
          throw new Error(error.message);
        }
      },
    },

    // Mutation to update a movie by ID
    updateMovie: {
      type: MovieType,
      args: {
        id: { type: GraphQLNonNull(GraphQLString) },
        title: { type: GraphQLNonNull(GraphQLString) },
        genre: { type: GraphQLNonNull(GraphQLString) },
        rating: { type: GraphQLNonNull(GraphQLInt) },
        duration: { type: GraphQLNonNull(GraphQLString) },
      },
      resolve: async (_, args) => {
        try {
          return await Movie.findByIdAndUpdate(args.id, args, { new: true });
        } catch (error) {
          throw new Error(error.message);
        }
      },
    },

    // Mutation to delete a movie by ID
    deleteMovie: {
      type: MovieType,
      args: { id: { type: GraphQLNonNull(GraphQLString) } },
      resolve: async (_, args) => {
        try {
          return await Movie.findByIdAndDelete(args.id);
        } catch (error) {
          throw new Error(error.message);
        }
      },
    },
  },
});

export default new GraphQLSchema({
  query: RootQuery,
  mutation: Mutation,
});

In our index file, we create a RootQuery and Mutation using the GraphQLObjectType from the "graphql" library to handle queries and mutations, respectively. The schema includes two main types: UserType for representing user-related data and MovieType for movie-related data. These types define the structure of the data that can be queried or mutated in the GraphQL API.

The RootQuery includes queries to retrieve users (users and user) and movies (movies and movie). For example, the users query retrieves a list of users, while the user query fetches a specific user by their ID. Similarly, the movies query retrieves all movies, and the movie query fetches a specific movie by its ID.

The Mutation type defines operations to modify data. For users, there are mutations to add a new user (addUser), update a user by ID (updateUser), and delete a user by ID (deleteUser). For movies, there are mutations to add a new movie (addMovie), update a movie by ID (updateMovie), and delete a movie by ID (deleteMovie). The mutations handle necessary logic, such as hashing passwords before saving a new user.

Finally, the GraphQLSchema is created, incorporating the defined RootQuery and Mutation. This schema serves as the entry point for the GraphQL API, providing a structured and standardized way to interact with user and movie data. We can now use the exported schema main index file: src/index.ts as previously illustrated.

Lets finalise with our password hashing function in src/utils/passwordUtils.ts:



// Import necessary modules
import bcrypt from "bcrypt";

// Hash a password
export const hashPassword = async (password: string): Promise<string> => {
  const saltRounds = 10;
  const hashedPassword = await bcrypt.hash(password, saltRounds);
  return hashedPassword;
};

At this point, we can open terminal and run:



npm run dev

and should see the following output in the console:

In your browser, visit localhost:8500/graphql and you should see the following interface:

This is the graphql playground allows developers to interact with a GraphQL API, test queries and mutations, and explore the available schema. It provides a convenient and visual way to understand the structure of the API, test different queries, and experiment with the data.

Lets make a sample mutation request for creating a new user.
In the left pane of the playground,place:



// Mutation to add a new user
mutation {
  addUser(email: "example@email.com", username: "exampleUser", password: "password123", isAdmin: false) {
    id
    username
    email
    isAdmin
  }
}

and hit cmd+enter or control+enter or the play button in the navigation panel.
The following output should be seen on the right hand side of the playground:

As you can observe, only requested data is returned from the server and nothing less, nothing more, a case in point, createdAt and updatedAt were not requested because we probably didnt need them, which is why they were not returned. This architecture is way more convenient because of this.
You can test all the crud operations on both user and movie using the same pattern. For all get requests, we use query not mutation. Mutation is for put, post and delete.

Conclusion

In conclusion, building a GraphQL API with Node.js and TypeScript offers a powerful and flexible solution for designing efficient, scalable, and strongly typed APIs. Leveraging the expressive syntax of GraphQL, you can precisely define data structures, reducing over-fetching and under-fetching issues. The combination of Node.js's event-driven architecture and TypeScript's static typing enhances code maintainability and catches potential errors during development. With seamless integration of popular libraries and tools, such as Express, Mongoose, among others constructing robust and feature-rich APIs becomes both intuitive and efficient. GraphQL's introspective nature, coupled with the convenience of TypeScript, fosters a streamlined development process and facilitates collaboration among team members. Ultimately, adopting GraphQL in a Node.js and TypeScript environment empowers creation of APIs that align closely with client needs, fostering a more efficient and enjoyable development experience.

Important Links

Github Repo
Twitter

Unit Testing in NodeJS with Express, TypeScript, Jest and Supertest || Part Three: Writing Unit Tests || A Comprehensive Guide

Abeinemukama Vicent — Tue, 02 Jan 2024 14:17:55 +0000

Welcome back to the third installment of our series on building a powerful API with Node.js, Express, and MongoDB and writing unit tests for each line of code! In the previous 2 parts: part one and part two, we set up the environment and meticulously crafted our application, ensuring robust user authentication and seamless database interactions. Now, it's time to fortify our codebase and elevate our development practices through the implementation of thorough unit tests.
In this part, we'll embark on a step-by-step journey through the process of writing unit tests for each line of code in our API. From utility functions, routes, controllers to services and models, we'll ensure that every aspect of our application is rigorously tested.

Why Unit Testing Matters

Unit testing plays a pivotal role in the development lifecycle, offering numerous benefits, including:

Early Issue Detection:

Identify and address bugs and issues early in the development process, reducing the likelihood of encountering them in later stages.

Code Maintainability:

Establish a safety net for code changes by ensuring that existing functionalities remain intact during modifications.

Documentation:

Unit tests serve as living documentation, showcasing the expected behavior of each component, function, or module.

Enhanced Collaboration:

Facilitate collaboration among team members by providing a clear understanding of the intended functionality and expected outcomes.

Folder Structure for Unit Tests

Organizing your unit tests is crucial for clarity and ease of maintenance. Consider adopting a structure similar to your application's folder layout. For our API, I recommend the following structure:

Just like I promised, we will be writing unit tests for each line of code. In the first part of this guide, after setting up the environment, we wrote some tests in the app.test.ts but by then we didn't have all the code in the index file like we do now, lets first update that file and make sure everything we have in the src/index.ts is working as expected including datababase connection to MongoDB Atlas, among others before starting with models.
Replace code inside app.test.ts with the following code:

import request from "supertest";
import app from "../index";
import mongoose from "mongoose";

beforeAll(async () => {
  // Set up: Establish the MongoDB connection before running tests
  if (!process.env.MONGODB_URL) {
    throw new Error("MONGO_URI environment variable is not defined/set");
  }

  await mongoose.connect(process.env.MONGODB_URL);
});

afterAll(async () => {
  // Teardown: Close the MongoDB connection after all tests have completed
  await mongoose.connection.close();
});

// Unit test for testing initial route ("/")
describe("GET /", () => {
  it('responds with "Welcome to unit testing guide for nodejs, typescript and express!"', async () => {
    const response = await request(app).get("/");

    expect(response.status).toBe(200);
    expect(response.text).toBe(
      "Welcome to unit testing guide for nodejs, typescript and express!"
    );
  });
});

Our app.test.ts file orchestrates the setup and teardown of the testing environment, establishing a MongoDB connection before tests begin and closing it afterward. The primary test case verifies the behavior of the root route ("/"), ensuring it responds with a welcome message and a 200 status. The code includes error handling to prompt the developer if the MongoDB connection string (MONGODB_URL) is not defined. This file serves as a foundational template for testing the core functionalities of the application, paving the way for more extensive unit testing across various directories and components.

Open your terminal and run:

npm run test

The output should be as shown below:

All set, now ready to write unit tests for our code in other directories.

Unit Tests for Database Models

Lets start with the data layer by writing unit tests for our database models.
Testing models allows us to validate the structure, functionality, and interactions with the database. By establishing robust tests for our models, we create a solid base upon which the rest of our unit tests can thrive.
The beforeAll hook orchestrates the setup process before running any tests, ensuring a valid MongoDB connection is established based on the MONGODB_URL environment variable. It throws an error if the variable is not defined, alerting the developer to set the MongoDB connection string. On the other hand, the afterAll hook handles cleanup after all tests are completed, closing the MongoDB connection to prevent resource leaks and maintain a clean testing environment. These hooks collectively manage the initialization and teardown phases, ensuring a consistent and isolated testing environment for the unit tests.

Create a new file: src/tests/models/userModel.test.ts and place the following code:

import mongoose from "mongoose";
import User, { IUser } from "../../models/User";
import dotenv from "dotenv";
dotenv.config();

describe("User Model Tests", () => {
  let createdUser: IUser;

  beforeAll(async () => {
    // Set up: Establish the MongoDB connection before running tests
    if (!process.env.MONGODB_URL) {
      throw new Error("MONGODB_URL environment variable is not defined/set");
    }

    await mongoose.connect(process.env.MONGODB_URL);
  });

  afterAll(async () => {
    // Remove the created user
    // await User.deleteMany();

    // Teardown: Close the MongoDB connection after all tests have completed
    await mongoose.connection.close();
  });

  // Test Case: Create a new user
  it("should create a new user", async () => {
    const userData: Partial<Omit<IUser, "_id">> = {
      email: "test@example.com",
      username: "testuser",
      password: "testpassword",
      isAdmin: false,
      savedProducts: [],
    };

    createdUser = await User.create(userData);

    expect(createdUser.email).toBe(userData.email);
    expect(createdUser.username).toBe(userData.username);
    expect(createdUser.isAdmin).toBe(userData.isAdmin);
  }, 10000); // Increase timeout to 10 seconds

  // Test Case: Ensure email and username are unique
  it("should fail to create a user with duplicate email or username", async () => {
    const userData: Partial<Omit<IUser, "_id">> = {
      email: "test@example.com",
      username: "testuser",
      password: "testpassword",
      isAdmin: false,
      savedProducts: [],
    };

    try {
      // Attempt to create a user with the same email and username
      await User.create(userData);
      // If the above line doesn't throw an error, the test should fail
      expect(true).toBe(false);
    } catch (error) {
      // Expect a MongoDB duplicate key error (code 11000)
      expect(error.code).toBe(11000);
    }
  }, 10000); // Increase timeout to 10 seconds

  // Test Case: Get all users
  it("should get all users", async () => {
    // Fetch all users from the database
    const allUsers = await User.find();

    // Expectations
    const userWithoutTimestamps = {
      _id: createdUser._id,
      email: createdUser.email,
      username: createdUser.username,
      isAdmin: createdUser.isAdmin,
      savedProducts: createdUser.savedProducts,
    };

    expect(allUsers).toContainEqual(
      expect.objectContaining(userWithoutTimestamps)
    );
  });

  const removeMongoProps = (user: any) => {
    const { __v, _id, createdAt, updatedAt, ...cleanedUser } = user.toObject();
    return cleanedUser;
  };

  // Test Case: Get all users
  it("should get all users", async () => {
    const allUsers = await User.find();

    // If there is a created user, expect the array to contain an object
    // that partially matches the properties of the createdUser
    if (createdUser) {
      const cleanedCreatedUser = removeMongoProps(createdUser);

      expect(allUsers).toEqual(
        expect.arrayContaining([expect.objectContaining(cleanedCreatedUser)])
      );
    }
  });

  // Test Case: Update an existing user
  it("should update an existing user", async () => {
    // Check if there is a created user to update
    if (createdUser) {
      // Define updated data
      const updatedUserData: Partial<IUser> = {
        username: "testuser",
        isAdmin: true,
      };

      // Update the user and get the updated user
      const updatedUser = await User.findByIdAndUpdate(
        createdUser._id,
        updatedUserData,
        { new: true }
      );

      // Expectations
      expect(updatedUser?.username).toBe(updatedUserData.username);
      expect(updatedUser?.isAdmin).toBe(updatedUserData.isAdmin);
    }
  });

  // Test Case: Get user by ID
  it("should get user by ID", async () => {
    // Get the created user by ID
    const retrievedUser = await User.findById(createdUser._id);

    // Expectations
    expect(retrievedUser?.email).toBe(createdUser.email);
    expect(retrievedUser?.username).toBe(createdUser.username);
    // Add other properties that you want to compare

    // For example, if updatedAt is expected to be different, you can ignore it:
    // expect(retrievedUser?.updatedAt).toBeDefined();
  });

  // Test Case: Delete an existing user
  it("should delete an existing user", async () => {
    // Delete the created user
    await User.findByIdAndDelete(createdUser._id);

    // Attempt to find the deleted user
    const deletedUser = await User.findById(createdUser._id);

    // Expectations
    expect(deletedUser).toBeNull();
  });
});

Our userModel.test.ts script contains a series of Jest test cases for the user model. Before running the tests, it establishes a connection to a MongoDB database specified in the environment variable MONGODB_URL. The test suite covers creating a new user, ensuring the uniqueness of email and username, retrieving all users, updating an existing user, fetching a user by ID, and finally, deleting an existing user. The tests use the User model from the src/models/User file and leverage various assertions to verify the expected behavior of these operations. Additionally, a utility function removeMongoProps is used to clean MongoDB-specific properties from user objects for precise comparisons. The script performs setup and teardown procedures to connect to and disconnect from the database, making it a comprehensive set of tests for the user model.
I always prefer to test all crud operations on the model at this stage to be sure but the main effort should be on the data since we are on the data layer.
Feel free to add more test cases on the model forxample the one that tests the data type of data received on each field and compare it with what the model has among others.
All good for our user model because all our tests passed under different test data. Its time to move to the product model.

Create a new file: src/__tests__/models/productModel.test.ts and place the following code:

import mongoose from "mongoose";
import Product, { IProduct } from "../../models/Product";
import dotenv from "dotenv";
dotenv.config();

describe("Product Model Tests", () => {
  let createdProduct: IProduct;

  beforeAll(async () => {
    // Set up: Establish the MongoDB connection before running tests
    if (!process.env.MONGODB_URL) {
      throw new Error("MONGODB_URL environment variable is not defined/set");
    }

    await mongoose.connect(process.env.MONGODB_URL);
  });

  afterAll(async () => {
    // Remove the created product
    // await Product.deleteMany();

    // Teardown: Close the MongoDB connection after all tests have completed
    await mongoose.connection.close();
  });

  // Test Case: Create a new product
  it("should create a new product", async () => {
    const productData: Partial<IProduct> = {
      title: "Test Product",
      description: "Product description",
      image: "https://testimage.png",
      category: "test category",
      quantity: "20 kgs",
      // You can add other fields
    };

    createdProduct = await Product.create(productData);

    expect(createdProduct.title).toBe(productData.title);
    expect(createdProduct.description).toBe(productData.description);
    // Add other expectations for additional fields
  }, 10000); // Increase timeout to 10 seconds

  // Test Case: Fail to create a product with missing required fields
  it("should fail to create a product with missing required fields", async () => {
    const productData: Partial<IProduct> = {
      // Omitting required fields
    };

    try {
      // Attempt to create a product with missing required fields
      await Product.create(productData);
      // If the above line doesn't throw an error, the test should fail
      expect(true).toBe(false);
    } catch (error) {
      // Expect a MongoDB validation error
      expect(error.name).toBe("ValidationError");
    }
  }, 10000); // Increase timeout to 10 seconds

  // Test Case: Get all products
  it("should get all products", async () => {
    // Fetch all products from the database
    const allProducts = await Product.find();

    // Expectations
    const productWithoutTimestamps = {
      //   _id: createdProduct._id,
      title: createdProduct.title,
      description: createdProduct.description,
      // Add other necessary fields
    };

    expect(allProducts).toContainEqual(
      expect.objectContaining(productWithoutTimestamps)
    );
  });

  const removeMongoProps = (product: any) => {
    const { __v, _id, createdAt, updatedAt, ...cleanedProduct } =
      product.toObject();
    return cleanedProduct;
  };

  // Test Case: Get all products
  it("should get all products", async () => {
    const allProducts = await Product.find();

    // If there is a created product, expect the array to contain an object
    // that partially matches the properties of the createdProduct
    if (createdProduct) {
      const cleanedCreatedProduct = removeMongoProps(createdProduct);

      expect(allProducts).toEqual(
        expect.arrayContaining([expect.objectContaining(cleanedCreatedProduct)])
      );
    }
  });

  // Test Case: Update an existing product
  it("should update an existing product", async () => {
    // Check if there is a created product to update
    if (createdProduct) {
      // Define updated data
      const updatedProductData: Partial<IProduct> = {
        title: "Test Product", // replace hre with your updated title
        // Update other necessary fields
      };

      // Update the product and get the updated product
      const updatedProduct = await Product.findByIdAndUpdate(
        createdProduct._id,
        updatedProductData,
        { new: true }
      );

      // Expectations
      expect(updatedProduct?.title).toBe(updatedProductData.title);
      // Add expectations for other updated fields
    }
  });

  // Test Case: Get product by ID
  it("should get product by ID", async () => {
    // Get the created product by ID
    const retrievedProduct = await Product.findById(createdProduct._id);

    // Expectations
    expect(retrievedProduct?.title).toBe(createdProduct.title);
    // Add other expectations for properties you want to compare
  });

  // Test Case: Delete an existing product
  it("should delete an existing product", async () => {
    // Delete the created product
    await Product.findByIdAndDelete(createdProduct._id);

    // Attempt to find the deleted product
    const deletedProduct = await Product.findById(createdProduct._id);

    // Expectations
    expect(deletedProduct).toBeNull();
  });
});

Our Product Model Tests are well-structured and cover essential CRUD operations on our data layer. Once again, the beforeAll and afterAll hooks ensure a connection to the MongoDB database is established before the tests and closed afterward. The "Create a new product" test validates the creation of a product with specified data, while the "Fail to create a product with missing required fields" test checks that the model correctly enforces required fields. The "Get all products" tests ensure that products retrieved from the database match the expected properties, and the utility function removeMongoProps helps clean unnecessary MongoDB-specific properties. The "Update an existing product" and "Get product by ID" tests validate the update and retrieval of products by ID. Lastly, the "Delete an existing product" test confirms the successful deletion of a product.
Overall, our tests provide comprehensive coverage for your Product model, ensuring its correctness and reliability in a MongoDB environment.
All set, lets proceed to services:

Writing UnitTests for Services

Create a new file: src/__tests__/services/userService.test.ts and place the following code:

import * as userService from "../../services/userService";
import * as passwordUtils from "../../utils/passwordUtils";
import * as jwtUtils from "../../utils/jwtUtils";
import User, { IUser } from "../../models/User";
import mongoose from "mongoose";
import dotenv from "dotenv";
import * as productModel from "../../models/Product"; // Import the Product model
dotenv.config();

// Mock the Product model
jest.mock("../../models/Product", () => ({
  __esModule: true,
  default: {
    findById: jest.fn(),
  },
}));

// Mock the populateUser function
jest
  .spyOn(userService, "populateUser")
  .mockImplementation(async (user: IUser) => {
    // Mock the behavior of populateUser function
    return user; // You can replace this with your desired mock value
  });

describe("User Service Tests", () => {
  let createdUser: IUser;

  // Clean up after tests
  beforeAll(async () => {
    // Set up: Establish the MongoDB connection before running tests
    if (!process.env.MONGODB_URL) {
      throw new Error("MONGODB_URL environment variable is not defined/set");
    }

    await mongoose.connect(process.env.MONGODB_URL);
  });

  afterAll(async () => {
    // Remove the created user
    await User.deleteMany();

    // Teardown: Close the MongoDB connection after all tests have completed
    await mongoose.connection.close();

    // Clear all jest mocks
    jest.clearAllMocks();
  });

  // Mock the hashPassword function
  jest
    .spyOn(passwordUtils, "hashPassword")
    .mockImplementation(async (password) => {
      // Mocked hash implementation
      return password + "_hashed";
    });
  // Mock the jwtUtils' generateToken function
  jest
    .spyOn(jwtUtils, "generateToken")
    .mockImplementation(() => "mocked_token");

  // Test Case: Create a new user
  it("should create a new user", async () => {
    const userData: Partial<IUser> = {
      email: "test@example.com",
      username: "testuser",
      password: "testpassword",
      isAdmin: false,
      savedProducts: [],
    };

    createdUser = await userService.createUser(userData as IUser);

    // Expectations
    expect(createdUser.email).toBe(userData.email);
    expect(createdUser.username).toBe(userData.username);
    expect(createdUser.isAdmin).toBe(userData.isAdmin);
    expect(passwordUtils.hashPassword).toHaveBeenCalledWith(userData.password);
  }, 30000);

  // Test Case: Login user
  it("should login a user and generate a token", async () => {
    // Mock user data for login
    const loginEmail = "test@example.com";
    const loginPassword = "testpassword";

    // Mock the comparePassword function
    jest
      .spyOn(passwordUtils, "comparePassword")
      .mockImplementation(async (inputPassword, hashedPassword) => {
        return inputPassword === hashedPassword.replace("_hashed", "");
      });

    const { user, token } = await userService.loginUser(
      loginEmail,
      loginPassword
    );

    // Expectations
    expect(user.email).toBe(createdUser.email);
    expect(user.username).toBe(createdUser.username);
    expect(user.isAdmin).toBe(createdUser.isAdmin);
    expect(jwtUtils.generateToken).toHaveBeenCalledWith({
      id: createdUser._id,
      username: createdUser.username,
      email: createdUser.email,
      isAdmin: createdUser.isAdmin,
    });
    expect(token).toBe("mocked_token");
  }, 20000);

  const removeMongoProps = (user: any) => {
    const { __v, _id, createdAt, updatedAt, ...cleanedUser } = user.toObject();
    return cleanedUser;
  };

  // Test Case: Get all users
  it("should get all users", async () => {
    // Fetch all users from the database
    const allUsers = await userService.getAllUsers();

    // If there is a created user, expect the array to contain an object
    // that partially matches the properties of the createdUser
    if (createdUser) {
      const cleanedCreatedUser = removeMongoProps(createdUser);

      expect(allUsers).toEqual(
        expect.arrayContaining([expect.objectContaining(cleanedCreatedUser)])
      );
    }
  }, 20000);

  // Test Case: Delete an existing user
  it("should delete an existing user", async () => {
    // Delete the created user
    await User.findByIdAndDelete(createdUser._id);

    // Attempt to find the deleted user
    const deletedUser = await User.findById(createdUser._id);

    // Expectations
    expect(deletedUser).toBeNull();
  });
});

and src/__tests__/services/producService.test.ts and place the following code:

import * as productService from "../../services/productService";
import Product, { IProduct } from "../../models/Product";
import mongoose from "mongoose";
import dotenv from "dotenv";
dotenv.config();

// Mock the Product model
jest.mock("../../models/Product", () => ({
  __esModule: true,
  default: {
    create: jest.fn(),
    find: jest.fn(),
    findById: jest.fn(),
    findByIdAndUpdate: jest.fn(),
    findByIdAndDelete: jest.fn(),
  },
}));

// Mock the product data
const productId = "mockedProductId";
const mockProduct: IProduct = {
  _id: productId,
  title: "Mocked Product",
  description: "A description for the mocked product",
  image: "mocked_image.jpg",
  category: "Mocked Category",
  quantity: "10",
  inStock: true,
} as IProduct;

// Use the toObject method to include additional properties
const mockProductWithMethods = {
  ...mockProduct,
  toObject: jest.fn(() => mockProduct),
};

// Mock the product retrieval by ID
(Product.findById as jest.Mock).mockResolvedValueOnce(mockProductWithMethods);

describe("Product Service Tests", () => {
  // Clean up after tests
  beforeAll(async () => {
    // Set up: Establish the MongoDB connection before running tests
    if (!process.env.MONGODB_URL) {
      throw new Error("MONGODB_URL environment variable is not defined/set");
    }

    await mongoose.connect(process.env.MONGODB_URL);
  });

  afterAll(async () => {
    // Teardown: Close the MongoDB connection after all tests have completed
    await mongoose.connection.close();

    // Clear all jest mocks
    jest.clearAllMocks();
  });

  // Test Case: Create a new product
  it("should create a new product", async () => {
    // Mock the product data
    const productData: Partial<IProduct> = {
      title: "Test Product",
      // Add other product properties based on your schema
    };

    // Mock the product creation
    (Product.create as jest.Mock).mockResolvedValueOnce({
      ...productData,
      _id: "mockedProductId", // Mocked product ID
    });

    // Create the product
    const createdProduct = await productService.createProduct(
      productData as IProduct
    );

    // Expectations
    expect(createdProduct.title).toBe(productData.title);
    // You can add more expectations based on your schema and business logic
  }, 20000);

  // Test Case: Get product by ID
  it("should get product by ID", async () => {
    // Mock product data
    const productId = "mockedProductId";
    const mockProduct: IProduct = {
      _id: productId,
      title: "Mocked Product",
      description: "A description for the mocked product",
      image: "mocked_image.jpg",
      category: "Mocked Category",
      quantity: "10",
      inStock: true,
    } as IProduct;

    // Mock the findById method of the Product model
    (Product.findById as jest.Mock).mockResolvedValueOnce(mockProduct);

    // Call the service
    const retrievedProduct = await productService.getProductById(productId);

    // Expectations
    expect(retrievedProduct).toEqual(
      expect.objectContaining({
        _id: mockProduct._id,
        title: mockProduct.title,
        description: mockProduct.description,
        image: mockProduct.image,
        category: mockProduct.category,
        quantity: mockProduct.quantity,
        inStock: mockProduct.inStock,
      })
    );
    expect(Product.findById).toHaveBeenCalledWith(productId);
  }, 20000);

  // Test Case: Update product by ID
  it("should update product by ID", async () => {
    // Mock product data
    const productId = "mockedProductId";
    const mockProduct: IProduct = {
      _id: productId,
      title: "Mocked Product",
      description: "A description for the mocked product",
      image: "mocked_image.jpg",
      category: "Mocked Category",
      quantity: "10",
      inStock: true,
    } as IProduct;

    // Mock the findByIdAndUpdate method of the Product model
    (Product.findByIdAndUpdate as jest.Mock).mockResolvedValueOnce(mockProduct);

    // Mock updated product data
    const updatedProductData: Partial<IProduct> = {
      title: "Mocked Product", // update some fields
      quantity: "10",
    };

    // Call the service
    const updatedProduct = await productService.updateProduct(
      productId,
      updatedProductData
    );

    // Expectations
    expect(updatedProduct?._id).toBe(mockProduct._id);
    expect(updatedProduct?.title).toBe(updatedProductData.title);
    expect(updatedProduct?.quantity).toBe(updatedProductData.quantity);
    // Add similar expectations for other properties you want to compare

    expect(Product.findByIdAndUpdate).toHaveBeenCalledWith(
      productId,
      updatedProductData,
      { new: true }
    );
  }, 20000);

  // Test Case: Delete product by ID
  it("should delete product by ID", async () => {
    // Mock product data
    const productId = "mockedProductId";
    const mockProduct: IProduct = {
      _id: productId,
      title: "Mocked Product",
      description: "A description for the mocked product",
      image: "mocked_image.jpg",
      category: "Mocked Category",
      quantity: "10",
      inStock: true,
    } as IProduct;

    // Mock the findByIdAndDelete method of the Product model
    (Product.findByIdAndDelete as jest.Mock).mockResolvedValueOnce(mockProduct);

    // Call the service
    await productService.deleteProduct(productId);

    // Expectations
    expect(Product.findByIdAndDelete).toHaveBeenCalledWith(productId);
  }, 20000);

  // Test Case: Get all products
  it("should get all products", async () => {
    // Mock product data
    const mockProducts: IProduct[] = [
      {
        _id: "product1",
        title: "Product 1",
        description: "Description for Product 1",
        image: "product1_image.jpg",
        category: "Category 1",
        quantity: "5",
        inStock: true,
      },
      {
        _id: "product2",
        title: "Product 2",
        description: "Description for Product 2",
        image: "product2_image.jpg",
        category: "Category 2",
        quantity: "10",
        inStock: false,
      },
    ] as IProduct[];

    // Mock the find method of the Product model
    (Product.find as jest.Mock).mockResolvedValueOnce(mockProducts);

    // Call the service
    const retrievedProducts = await productService.getAllProducts();

    // Expectations
    expect(Product.find).toHaveBeenCalled();
    expect(retrievedProducts).toEqual(mockProducts);
  }, 20000);
});

Our createUser service uses two external functions; hashPassword utility and generateToken middleware, all in src/utils. Lets first diagonise these utility functions and discuss their unit tests that this service is mocking.
Create a new file: src/__tests__/utils/passwordUtils.test.ts and place the following code:

import * as passwordUtils from "../../utils/passwordUtils";

describe("Password Utilities Tests", () => {
  // Test Case: Hash Password
  it("should hash a password", async () => {
    const password = "testpassword";
    const hashedPassword = await passwordUtils.hashPassword(password);
    expect(hashedPassword).toBeDefined();
    expect(typeof hashedPassword).toBe("string");
  });

  // Test Case: Compare Password
  it("should compare a valid password", async () => {
    const password = "testpassword";
    const hashedPassword = await passwordUtils.hashPassword(password);
    const isPasswordValid = await passwordUtils.comparePassword(
      password,
      hashedPassword
    );
    expect(isPasswordValid).toBe(true);
  });

  // Test Case: Compare Invalid Password
  it("should compare an invalid password", async () => {
    const password = "testpassword";
    const hashedPassword = await passwordUtils.hashPassword(password);
    const isPasswordValid = await passwordUtils.comparePassword(
      "wrongpassword",
      hashedPassword
    );
    expect(isPasswordValid).toBe(false);
  });
});

and src/__tests__/utils/jwtUtils.test.ts and place the following code:

import {
  JWTPayload,
  generateToken,
  verifyToken,
  verifyTokenAndAuthorization,
} from "../../utils/jwtUtils";

import dotenv from "dotenv";
dotenv.config();

describe("JWT Utils Tests", () => {
  const mockPayload: JWTPayload = {
    id: "mockUserId",
    username: "mockUsername",
    email: "mock@example.com",
    isAdmin: false,
  };

  const mockToken = "mockToken";

  process.env.JWT_SEC = "mockSecret";
  process.env.JWT_EXPIRY_PERIOD = "1h";

  // Test Case: Generate Token
  it("should generate a JWT token", () => {
    const token = generateToken(mockPayload);
    expect(token).toBeDefined();
  });


  it("should verify a valid JWT token", (done) => {
    const req: any = {
      headers: {
        token: `Bearer ${mockToken}`,
      },
    };

    const res: any = {
      status: (status: number) => {
        expect(status).toBe(403);
        return {
          json: (message: string) => {
            expect(message).toBe("Token is not valid!");
            done();
          },
        };
      },
    };

    const next = () => {
      // Should not reach here
      done.fail("Should not reach next middleware on invalid token");
    };

    verifyToken(req, res, next);
  });

  // Test Case: Verify Token (invalid token)
  it("should handle an invalid JWT token", (done) => {
    const req: any = {
      headers: {
        token: "InvalidToken",
      },
    };

    const res: any = {
      status: (status: number) => {
        expect(status).toBe(403);
        return {
          json: (message: string) => {
            expect(message).toBe("Token is not valid!");
            done();
          },
        };
      },
    };

    const next = () => {
      // Should not reach here
      done.fail("Should not reach next middleware on invalid token");
    };

    verifyToken(req, res, next);
  });

  // Cleanup: Reset environment variables after tests
  afterAll(() => {
    delete process.env.JWT_SEC;
    delete process.env.JWT_EXPIRY_PERIOD;
  });
});

In our suite for hashing the password, we examine the core functionalities integral to safeguarding sensitive credentials. The initial litmus test involves the seamless hashing of passwords, ensuring a robust defense against unauthorized access. A keen eye is cast upon the compare password mechanism, where the suite deftly validates the correctness of a given password against its hashed counterpart. In cases of both valid and invalid password comparisons, the suite orchestrates a meticulous evaluation, fortifying our application's resilience against potential security threats. This comprehensive scrutiny instills confidence in the reliability and effectiveness of our Password Utilities, as they stand guard to fortify the fortress of our user authentication system.

Also, in the suite for token generation and verification, we scrutinize the fundamental functionalities that underpin the secure handling of JSON Web Tokens (JWTs). The first checkpoint ensures the seamless generation of JWT tokens, a cornerstone in our approach to secure authentication. Delving into token verification, the suite rigorously examines scenarios of both valid and invalid tokens, intricately assessing the resilience of our verification mechanism. In cases where a valid token is presented, the verification process is expected to seamlessly proceed, ensuring that the user details are defined. Conversely, when confronted with an invalid token, our suite orchestrates a precise response, safeguarding our application against unauthorized access. As the final act of diligence, the suite gracefully resets environment variables, leaving the testing landscape pristine.

In the User Service Tests suite, we meticulously verify the functionality of our user service within a Node.js backend. Leveraging the power of mocking, we intentionally isolate the service from external dependencies, ensuring a controlled testing environment. Our suite encompasses crucial aspects, such as creating a new user, logging in a user, retrieving all users, and deleting an existing user. Each test case meticulously orchestrates mock data and expectations, guaranteeing the service operates seamlessly. Our commitment to best practices shines through as we meticulously clean up by removing the created user post-testing. To facilitate more accurate comparisons of user objects, we employ the removeMongoProps function, eliminating MongoDB-specific properties. This suite is a testament to our dedication to thorough and isolated testing, laying a robust foundation for a resilient backend service.

On the otherhand, in the Product Service Tests suite, we scrutinize the functionality of our backend API's product-related operations. Employing the power of mocking, we deliberately isolate the service from external dependencies, ensuring a controlled testing environment. The suite comprehensively tests key functionalities, including creating a new product, fetching a product by ID, updating a product, deleting a product, and fetching all products. Each test case is meticulously crafted, incorporating mock data, precise expectations, and detailed property comparisons, thereby validating the robustness and reliability of our ProductService. The suite adheres to best practices, ensuring that the testing environment is thoroughly cleaned up post-execution by closing the MongoDB connection and clearing all Jest mocks. This comprehensive and systematic testing approach reinforces our commitment to delivering a resilient and dependable backend service.

Writing Unit Tests for Controllers

Create a new file: src/__tests__/controllers/userController.test.ts and place the following code:

import { Request, Response } from "express";
import * as UserController from "../../controllers/userController";
import * as UserService from "../../services/userService";
import { IProduct } from "models/Product";
import { IUser } from "models/User";

// Mock the UserService
jest.mock("../../services/userService");

describe("User Controller Tests", () => {
  // Test Case: Create a new user
  it("should create a new user", async () => {
    // Mock user data from the request body
    const mockUserData: IUser = {
      email: "test@example.com",
      username: "testuser",
      password: "testpassword",
      isAdmin: false,
      savedProducts: [],
    } as IUser;

    // Mock the create user response from the UserService
    const mockCreatedUser = {
      _id: "mockUserId",
      ...mockUserData,
    };

    // Mock the request and response objects
    const mockRequest = {
      body: mockUserData,
    } as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the createUser function of the UserService
    (UserService.createUser as jest.Mock).mockResolvedValueOnce(
      mockCreatedUser
    );

    // Call the createUser controller
    await UserController.createUser(mockRequest, mockResponse);

    // Expectations
    expect(UserService.createUser).toHaveBeenCalledWith(mockUserData);
    expect(mockResponse.status).toHaveBeenCalledWith(201);
    expect(mockResponse.json).toHaveBeenCalledWith(mockCreatedUser);
  }, 20000);

  // Test Case: Login user
  it("should login a user and return a token", async () => {
    // Mock user credentials from the request body
    const mockUserCredentials = {
      email: "test@example.com",
      password: "testpassword",
    };

    type IMockCreatedUser = {
      user: Partial<IUser>;
      token: string;
    };
    // Mock the login response from the UserService
    const mockLoginResponse: IMockCreatedUser = {
      user: {
        _id: "mockUserId",
        email: mockUserCredentials.email,
        username: "testuser",
        isAdmin: false,
        savedProducts: [],
      },
      token: "mocked_token",
    };

    // Mock the request and response objects
    const mockRequest = {
      body: mockUserCredentials,
    } as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the loginUser function of the UserService
    (UserService.loginUser as jest.Mock).mockResolvedValueOnce(
      mockLoginResponse
    );

    // Call the loginUser controller
    await UserController.loginUser(mockRequest, mockResponse);

    // Expectations
    expect(UserService.loginUser).toHaveBeenCalledWith(
      mockUserCredentials.email,
      mockUserCredentials.password
    );
    expect(mockResponse.status).toHaveBeenCalledWith(200);
    expect(mockResponse.json).toHaveBeenCalledWith(mockLoginResponse);
  }, 20000);

  // Test Case: Get all users
  it("should get all users", async () => {
    // Mock the array of users from the UserService
    const mockUsers: IUser[] = [
      {
        _id: "mockUserId1",
        email: "user1@example.com",
        username: "user1",
        isAdmin: false,
        savedProducts: [],
      },
      {
        _id: "mockUserId2",
        email: "user2@example.com",
        username: "user2",
        isAdmin: true,
        savedProducts: [],
      },
    ] as IUser[];

    // Mock the request and response objects
    const mockRequest = {} as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the getAllUsers function of the UserService
    (UserService.getAllUsers as jest.Mock).mockResolvedValueOnce(mockUsers);

    // Call the getAllUsers controller
    await UserController.getAllUsers(mockRequest, mockResponse);

    // Expectations
    expect(UserService.getAllUsers).toHaveBeenCalled();
    expect(mockResponse.status).toHaveBeenCalledWith(200);
    expect(mockResponse.json).toHaveBeenCalledWith(mockUsers);
  }, 20000);

  // Test Case: Error fetching users
  it("should handle error when fetching users", async () => {
    // Mock the error response from the UserService
    const mockErrorResponse = {
      message: "Error fetching users",
    };

    // Mock the request and response objects
    const mockRequest = {} as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the getAllUsers function of the UserService to throw an error
    (UserService.getAllUsers as jest.Mock).mockRejectedValueOnce(
      mockErrorResponse
    );

    // Call the getAllUsers controller
    await UserController.getAllUsers(mockRequest, mockResponse);

    // Expectations
    expect(UserService.getAllUsers).toHaveBeenCalled();
    expect(mockResponse.status).toHaveBeenCalledWith(500);
    expect(mockResponse.json).toHaveBeenCalledWith({
      error: mockErrorResponse.message,
    });
  }, 20000);

  // Test Case: Get user by ID
  it("should get user by ID", async () => {
    // Mock the user from the UserService
    const mockUser: IUser = {
      _id: "mockUserId",
      email: "mock@example.com",
      username: "mockUser",
      isAdmin: false,
      savedProducts: [],
    } as IUser;

    // Mock the request and response objects
    const mockRequest: any = {
      params: {
        userId: "mockUserId",
      },
    };

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the getUserById function of the UserService
    (UserService.getUserById as jest.Mock).mockResolvedValueOnce(mockUser);

    // Call the getUserById controller
    await UserController.getUserById(mockRequest, mockResponse);

    // Expectations
    expect(UserService.getUserById).toHaveBeenCalledWith(
      mockRequest.params.userId
    );
    expect(mockResponse.status).toHaveBeenCalledWith(200);
    expect(mockResponse.json).toHaveBeenCalledWith(mockUser);
  }, 20000);

  // Test Case: User not found
  it("should handle case where user is not found", async () => {
    // Mock the request and response objects
    const mockRequest: any = {
      params: {
        userId: "nonexistentUserId",
      },
    };

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the getUserById function of the UserService to return null
    (UserService.getUserById as jest.Mock).mockResolvedValueOnce(null);

    // Call the getUserById controller
    await UserController.getUserById(mockRequest, mockResponse);

    // Expectations
    expect(UserService.getUserById).toHaveBeenCalledWith(
      mockRequest.params.userId
    );
    expect(mockResponse.status).toHaveBeenCalledWith(404);
    expect(mockResponse.json).toHaveBeenCalledWith({ error: "User not found" });
  }, 20000);

  // Test Case: Error fetching user
  it("should handle error when fetching user by ID", async () => {
    // Mock the error response from the UserService
    const mockErrorResponse = {
      message: "Error fetching user",
    };

    // Mock the request and response objects
    const mockRequest: any = {
      params: {
        userId: "errorUserId",
      },
    };

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the getUserById function of the UserService to throw an error
    (UserService.getUserById as jest.Mock).mockRejectedValueOnce(
      mockErrorResponse
    );

    // Call the getUserById controller
    await UserController.getUserById(mockRequest, mockResponse);

    // Expectations
    expect(UserService.getUserById).toHaveBeenCalledWith(
      mockRequest.params.userId
    );
    expect(mockResponse.status).toHaveBeenCalledWith(500);
    expect(mockResponse.json).toHaveBeenCalledWith({
      error: mockErrorResponse.message,
    });
  }, 20000);

  // Test Case: Delete user by ID
  it("should delete user by ID", async () => {
    // Mock the request and response objects
    const mockRequest: Request<
      { userId: string },
      any,
      any,
      any,
      Record<string, any>
    > = {
      params: {
        userId: "mockUserId",
      },
    } as Request<{ userId: string }, any, any, any, Record<string, any>>;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the deleteUser function of the UserService
    (UserService.deleteUser as jest.Mock).mockResolvedValueOnce(null);

    // Call the deleteUser controller
    await UserController.deleteUser(mockRequest, mockResponse);

    // Expectations
    expect(UserService.deleteUser).toHaveBeenCalledWith(
      mockRequest.params.userId
    );
    expect(mockResponse.status).toHaveBeenCalledWith(204);
    expect(mockResponse.send).toHaveBeenCalled();
    expect(mockResponse.json).not.toHaveBeenCalled(); // Ensure json is not called for a 204 status
  }, 20000);
});

and src/__tests__/controllers/productController.test.ts and place the following code:

import { Request, Response } from "express";
import * as ProductController from "../../controllers/productController";
import * as ProductService from "../../services/productService";

// Mock the ProductService
jest.mock("../../services/productService");

describe("Product Controller Tests", () => {
  // Test Case: Create a new product
  it("should create a new product", async () => {
    // Mock the request and response objects
    const mockRequest: Request<{}, any, any, any, Record<string, any>> = {
      body: {
        title: "Mocked Product",
        description: "A description for the mocked product",
        image: "mocked_image.jpg",
        category: "Mocked Category",
        quantity: "10",
        inStock: true,
      },
    } as Request<{}, any, any, any, Record<string, any>>;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the createProduct function of the ProductService
    (ProductService.createProduct as jest.Mock).mockResolvedValueOnce({
      _id: "mockedProductId",
      title: "Mocked Product",
      description: "A description for the mocked product",
      image: "mocked_image.jpg",
      category: "Mocked Category",
      quantity: "10",
      inStock: true,
    });

    // Call the createProduct controller
    await ProductController.createProduct(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.createProduct).toHaveBeenCalledWith(mockRequest.body);
    expect(mockResponse.status).toHaveBeenCalledWith(201);
    expect(mockResponse.json).toHaveBeenCalledWith({
      _id: "mockedProductId",
      title: "Mocked Product",
      description: "A description for the mocked product",
      image: "mocked_image.jpg",
      category: "Mocked Category",
      quantity: "10",
      inStock: true,
    });
    expect(mockResponse.send).not.toHaveBeenCalled(); // Ensure send is not called for a 201 status
  });

  // Test Case: Get all products - Success
  it("should get all products successfully", async () => {
    // Mock the request and response objects
    const mockRequest: any = {};

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the getAllProducts function of the ProductService
    (ProductService.getAllProducts as jest.Mock).mockResolvedValueOnce([
      {
        _id: "mockedProductId1",
        title: "Mocked Product 1",
        description: "Description for mocked product 1",
        image: "image1.jpg",
        category: "Category 1",
        quantity: "5",
        inStock: true,
      },
      {
        _id: "mockedProductId2",
        title: "Mocked Product 2",
        description: "Description for mocked product 2",
        image: "image2.jpg",
        category: "Category 2",
        quantity: "10",
        inStock: false,
      },
    ]);

    // Call the getAllProducts controller
    await ProductController.getAllProducts(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.getAllProducts).toHaveBeenCalled();
    expect(mockResponse.status).toHaveBeenCalledWith(200);
    expect(mockResponse.json).toHaveBeenCalledWith([
      {
        _id: "mockedProductId1",
        title: "Mocked Product 1",
        description: "Description for mocked product 1",
        image: "image1.jpg",
        category: "Category 1",
        quantity: "5",
        inStock: true,
      },
      {
        _id: "mockedProductId2",
        title: "Mocked Product 2",
        description: "Description for mocked product 2",
        image: "image2.jpg",
        category: "Category 2",
        quantity: "10",
        inStock: false,
      },
    ]);
    expect(mockResponse.send).not.toHaveBeenCalled(); // Ensure send is not called for a 200 status
  });

  // Test Case: Get all products - Error
  it("should handle errors when getting all products", async () => {
    // Mock the request and response objects
    const mockRequest: any = {};

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the getAllProducts function of the ProductService to throw an error
    (ProductService.getAllProducts as jest.Mock).mockRejectedValueOnce(
      new Error("Error getting products")
    );

    // Call the getAllProducts controller
    await ProductController.getAllProducts(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.getAllProducts).toHaveBeenCalled();
    expect(mockResponse.status).toHaveBeenCalledWith(500);
    expect(mockResponse.json).toHaveBeenCalledWith({
      error: "Error getting products",
    });
    expect(mockResponse.send).not.toHaveBeenCalled(); // Ensure send is not called for a 500 status
  });

  // Test Case: Get product by ID - Success
  it("should get product by ID successfully", async () => {
    // Mock the request and response objects
    const mockRequest: Request = {
      params: { productId: "mockedProductId" },
    } as unknown as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the getProductById function of the ProductService
    (ProductService.getProductById as jest.Mock).mockResolvedValueOnce({
      _id: "mockedProductId",
      title: "Mocked Product",
      description: "Description for mocked product",
      image: "mocked_image.jpg",
      category: "Mocked Category",
      quantity: "10",
      inStock: true,
    });

    // Call the getProductById controller
    await ProductController.getProductById(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.getProductById).toHaveBeenCalledWith(
      "mockedProductId"
    );
    expect(mockResponse.status).toHaveBeenCalledWith(200);
    expect(mockResponse.json).toHaveBeenCalledWith({
      _id: "mockedProductId",
      title: "Mocked Product",
      description: "Description for mocked product",
      image: "mocked_image.jpg",
      category: "Mocked Category",
      quantity: "10",
      inStock: true,
    });
    expect(mockResponse.send).not.toHaveBeenCalled(); // Ensure send is not called for a 200 status
  });

  // Test Case: Get product by ID - Not Found
  it("should handle product not found when getting by ID", async () => {
    // Mock the request and response objects
    const mockRequest: Request = {
      params: { productId: "nonExistentProductId" },
    } as unknown as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the getProductById function of the ProductService to return null
    (ProductService.getProductById as jest.Mock).mockResolvedValueOnce(null);

    // Call the getProductById controller
    await ProductController.getProductById(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.getProductById).toHaveBeenCalledWith(
      "nonExistentProductId"
    );
    expect(mockResponse.status).toHaveBeenCalledWith(404);
    expect(mockResponse.json).toHaveBeenCalledWith({
      error: "Product not found",
    });
    expect(mockResponse.send).not.toHaveBeenCalled(); // Ensure send is not called for a 404 status
  });

  // Test Case: Get product by ID - Error
  it("should handle errors when getting product by ID", async () => {
    // Mock the request and response objects
    const mockRequest: Request = {
      params: { productId: "mockedProductId" },
    } as unknown as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the getProductById function of the ProductService to throw an error
    (ProductService.getProductById as jest.Mock).mockRejectedValueOnce(
      new Error("Error getting product by ID")
    );

    // Call the getProductById controller
    await ProductController.getProductById(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.getProductById).toHaveBeenCalledWith(
      "mockedProductId"
    );
    expect(mockResponse.status).toHaveBeenCalledWith(500);
    expect(mockResponse.json).toHaveBeenCalledWith({
      error: "Error getting product by ID",
    });
    expect(mockResponse.send).not.toHaveBeenCalled(); // Ensure send is not called for a 500 status
  });

  // Test Case: Update product by ID - Success
  it("should update product by ID successfully", async () => {
    // Mock the request and response objects
    const mockRequest: Request = {
      params: { productId: "mockedProductId" },
      body: {
        title: "Updated Product",
        description: "Updated description",
        image: "updated_image.jpg",
        category: "Updated Category",
        quantity: "15",
        inStock: false,
      },
    } as unknown as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the updateProduct function of the ProductService
    (ProductService.updateProduct as jest.Mock).mockResolvedValueOnce({
      _id: "mockedProductId",
      title: "Updated Product",
      description: "Updated description",
      image: "updated_image.jpg",
      category: "Updated Category",
      quantity: "15",
      inStock: false,
    });

    // Call the updateProduct controller
    await ProductController.updateProduct(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.updateProduct).toHaveBeenCalledWith(
      "mockedProductId",
      mockRequest.body
    );
    expect(mockResponse.status).toHaveBeenCalledWith(200);
    expect(mockResponse.json).toHaveBeenCalledWith({
      _id: "mockedProductId",
      title: "Updated Product",
      description: "Updated description",
      image: "updated_image.jpg",
      category: "Updated Category",
      quantity: "15",
      inStock: false,
    });
    expect(mockResponse.send).not.toHaveBeenCalled(); // Ensure send is not called for a 200 status
  });

  // Test Case: Update product by ID - Not Found
  it("should handle product not found when updating by ID", async () => {
    // Mock the request and response objects
    const mockRequest: Request = {
      params: { productId: "nonExistentProductId" },
      body: {
        title: "Updated Product",
        description: "Updated description",
        image: "updated_image.jpg",
        category: "Updated Category",
        quantity: "15",
        inStock: false,
      },
    } as unknown as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the updateProduct function of the ProductService to return null
    (ProductService.updateProduct as jest.Mock).mockResolvedValueOnce(null);

    // Call the updateProduct controller
    await ProductController.updateProduct(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.updateProduct).toHaveBeenCalledWith(
      "nonExistentProductId",
      mockRequest.body
    );
    expect(mockResponse.status).toHaveBeenCalledWith(404);
    expect(mockResponse.json).toHaveBeenCalledWith({
      error: "Product not found",
    });
    expect(mockResponse.send).not.toHaveBeenCalled(); // Ensure send is not called for a 404 status
  });

  // Test Case: Update product by ID - Error
  it("should handle errors when updating product by ID", async () => {
    // Mock the request and response objects
    const mockRequest: Request = {
      params: { productId: "mockedProductId" },
      body: {
        title: "Updated Product",
        description: "Updated description",
        image: "updated_image.jpg",
        category: "Updated Category",
        quantity: "15",
        inStock: false,
      },
    } as unknown as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the updateProduct function of the ProductService to throw an error
    (ProductService.updateProduct as jest.Mock).mockRejectedValueOnce(
      new Error("Error updating product by ID")
    );

    // Call the updateProduct controller
    await ProductController.updateProduct(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.updateProduct).toHaveBeenCalledWith(
      "mockedProductId",
      mockRequest.body
    );
    expect(mockResponse.status).toHaveBeenCalledWith(500);
    expect(mockResponse.json).toHaveBeenCalledWith({
      error: "Error updating product by ID",
    });
    expect(mockResponse.send).not.toHaveBeenCalled(); // Ensure send is not called for a 500 status
  });

  // Test Case: Delete product by ID - Success
  it("should delete product by ID successfully", async () => {
    // Mock the request and response objects
    const mockRequest: Request = {
      params: { productId: "mockedProductId" },
    } as unknown as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
    } as unknown as Response;

    // Call the deleteProduct function of the ProductService
    await ProductController.deleteProduct(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.deleteProduct).toHaveBeenCalledWith(
      "mockedProductId"
    );
    expect(mockResponse.status).toHaveBeenCalledWith(204);
    expect(mockResponse.send).toHaveBeenCalled();
  });

  // Test Case: Delete product by ID - Not Found
  it("should handle product not found when deleting by ID", async () => {
    // Mock the request and response objects
    const mockRequest: Request = {
      params: { productId: "nonExistentProductId" },
    } as unknown as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the deleteProduct function of the ProductService to return null
    (ProductService.deleteProduct as jest.Mock).mockResolvedValueOnce(null);

    // Call the deleteProduct controller
    await ProductController.deleteProduct(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.deleteProduct).toHaveBeenCalledWith(
      "nonExistentProductId"
    );
    expect(mockResponse.status).toHaveBeenCalledWith(204);
    expect(mockResponse.send).toHaveBeenCalled();
    expect(mockResponse.json).not.toHaveBeenCalled(); // Ensure json is not called for a 204 status
  });

  // Test Case: Delete product by ID - Error
  it("should handle errors when deleting product by ID", async () => {
    // Mock the request and response objects
    const mockRequest: Request = {
      params: { productId: "mockedProductId" },
    } as unknown as Request;

    const mockResponse = {
      status: jest.fn().mockReturnThis(),
      send: jest.fn(),
      json: jest.fn(),
    } as unknown as Response;

    // Mock the deleteProduct function of the ProductService to throw an error
    (ProductService.deleteProduct as jest.Mock).mockRejectedValueOnce(
      new Error("Error deleting product by ID")
    );

    // Call the deleteProduct controller
    await ProductController.deleteProduct(mockRequest, mockResponse);

    // Expectations
    expect(ProductService.deleteProduct).toHaveBeenCalledWith(
      "mockedProductId"
    );
    expect(mockResponse.status).toHaveBeenCalledWith(500);
    expect(mockResponse.json).toHaveBeenCalledWith({
      error: "Error deleting product by ID",
    });
    expect(mockResponse.send).not.toHaveBeenCalled(); // Ensure send is not called for a 500 status
  });
});

In the test suite for the User Controller, we employ thorough mocking using jest.mock to isolate the UserService, ensuring controlled testing of the controller's functionalities. Each test case is meticulously crafted to cover key aspects of user management, such as creating a new user, user login, fetching all users, retrieving a user by ID, handling scenarios where the user is not found, and deleting a user by ID. The suite demonstrates the usage of precise mock data and expectations, validating the User Controller's reliability. Notably, the jest.mock line is crucial, replacing the actual UserService with a mock implementation to create a controlled environment for testing. This approach ensures that the tests focus solely on the User Controller's behavior without interference from external dependencies. This testing strategy aims to deliver a secure, efficient, and error-resilient user management system in our backend API.

In the Product Controller Tests test suite for the Product Controller, we orchestrate a comprehensive set of scenarios to validate the functionality of the controller methods. The tests cover creating a new product, fetching all products (both successful and error cases), retrieving a product by ID (with successful, not found, and error scenarios), updating a product by ID (again, covering success, not found, and error cases), and finally, deleting a product by ID (encompassing successful, not found, and error scenarios). Each test is structured with meticulous mock data, expectations, and error handling, ensuring a robust examination of the Product Controller's behavior. The jest.mock("../../services/productService") helps us to isolate the ProductService for controlled testing.
All set for our controllers, lets also discuss how we write tests for our routes.

Writing Unit Tests for Routes

Before we start to write our unit tests for all our routes, we need to first examine our setup and make some adjustments.
First we need to isolate a testing server to avoid conflicts testing various suites for different routes.
Create a new file in the root directory, call it: test_setup.ts and place the following code:

import app from "./src/index"; // Import your main app from the index file

const port = 9000; // Choose a suitable port

const test_server = app.listen(port, () => {
  console.log(`Server is running on port ${port}`);
});

export default test_server;

This helps us interact with the testing server instead of using the actual server of ours running at port 8800.
Lets also add the following line to our .env file:

TEST_ENV=true

and start our main server conditionally in our src/index.ts according to TEST_ENV environment variable.
Replace:

 app.listen(PORT, () => {
    console.log(`Backend server is running at port ${PORT}`);
  });

with

// Check if it's not a test environment before starting the server
if (!process.env.TEST_ENV) {
  app.listen(PORT, () => {
    console.log(`Backend server is running at port ${PORT}`);
  });
}

to start our main server only if we are not testing mode.
Don't forget to alter the TEST_ENV by setting it to false when youre done running unit tests or when you get in production.

Create a new file: src/__tests__/routes/userRoutes.test.ts and place the following code:

// userRoutes.test.ts
import request from "supertest";
import test_server from "../../../test_setup";

import User from "../../models/User";
import supertest from "supertest";

// You can either test with a real token like this:
const adminToken = "YourAdminToken";
// Alternatively:
const nonAdminToken = "YourNonAdminToken";
// OR
// Write a function that simulates an actual login process and extract the token from there
// And then store it in a variable for use in various test cases e.g:
// Assuming you have admin credentials for testing

// const adminCredentials = {
//   email: "admin@example.com",
//   password: "adminpassword",
// };

// Assuming you have a function to generate an authentication token
// const getAuthToken = async () => {
//   const response = await supertest(test_server)
//     .post("/api/v1/users/login")
//     .send(adminCredentials);
//   return response.body.token;
// };

// Get the admin authentication token (inside an async function or code block)
// const authToken = await getAuthToken();

// ------------------> do the same for non admin user and store the token in
// its own variable

describe("User Routes", () => {
  beforeAll(async () => {});

  // Clean up after tests
  afterAll(async () => {
    // Remove the created user
    await User.deleteOne({
      username: "testuser",
    });

    // Clear all jest mocks
    jest.clearAllMocks();
    test_server.close();
  });

  // Test case for creating a new user
  it("should create a new user", async () => {
    const response = await request(test_server)
      .post("/api/v1/users/create") // Update the route path accordingly
      .send({
        // Your user data for testing
        email: "admin@example.com",
        password: "adminpassword",
        username: "testuser",
        isAdmin: false,
        savedProducts: [],
      });

    // Expectations
    expect(response.status).toBe(201);
    expect(response.body).toHaveProperty("email", "admin@example.com");
    expect(response.body).toHaveProperty("username", "testuser");
    // Add more expectations based on your user data

    // Optionally, you can store the created user for future tests
    const createdUser = response.body;
  }, 20000);

  // Test case for logging in a user
  it("should login a user and return a token", async () => {
    // Assuming you have a test user created previously
    const testUser = {
      email: "admin@example.com",
      password: "adminpassword",
    };

    const response = await request(test_server)
      .post("/api/v1/users/login") // Update the route path accordingly
      .send({
        email: testUser.email,
        password: testUser.password,
      });

    // Expectations
    expect(response.status).toBe(200);
    expect(response.body).toHaveProperty("user");
    expect(response.body).toHaveProperty("token");
    // Add more expectations based on your login data

    // Optionally, you can store the token for future authenticated requests
    const authToken = response.body.token;
  }, 20000);

  // Assuming you have admin credentials for testing
  const adminCredentials = {
    email: "admin@example.com",
    password: "adminpassword",
  };

  // Assuming you have a function to generate an authentication token
  const getAuthToken = async (credentials: object) => {
    const response = await supertest(test_server)
      .post("/api/v1/users/login")
      .send(credentials);
    return response.body.token;
  };

  // Test case for getting all users (admin access)
  it("should get all users with admin access", async () => {
    // Assuming you have admin credentials for testing
    const adminCredentials = {
      email: "admin@example.com",
      password: "adminpassword",
    };

    // Log in as admin to get the admin token
    // const adminToken = await getAuthToken(adminCredentials);

    // Send a request to the route with the admin token
    const response = await request(test_server)
      .get("/api/v1/users/all")
      .set("token", `Bearer ${adminToken}`);

    // Expectations
    expect(response.status).toBe(200);
    // Add more expectations based on your user data

    // Optionally, you can store the users for further assertions
    const allUsers = response.body;
  }, 20000);

  // Test case for getting all users without admin access
  it("should return 403 Forbidden when accessing all users without admin access", async () => {
    // Assuming you have non-admin credentials for testing
    const nonAdminCredentials = {
      email: "nonadmin@example.com",
      password: "nonadminpassword",
    };

    // Log in as non-admin to get the non-admin token
    // const nonAdminToken = await getAuthToken(nonAdminCredentials);

    // Send a request to the route with the non-admin token
    const response = await request(test_server)
      .get("/api/v1/users/all")
      .set("token", `Bearer ${nonAdminToken}`);

    // Expectations
    expect(response.status).toBe(403);
    // Add more expectations based on how you handle non-admin access
  }, 20000);

  // Test case: Should successfully update a user with valid credentials (admin or account owner)
  it("should successfully update a user with valid credentials (admin or account owner)", async () => {
    // Assuming you have a test user created previously
    // If you dont have you can create one programattically like we did writing
    // Tests for create new user

    const testUser = {
      email: "testuser@example.com",
      password: "testuserpassword",
      username: "testuser",
    };

    // Update the user using the hardcoded token
    const updateResponse = await request(test_server)
      .put(`/api/v1/users/update/6593ca275db905747ea085aa`)
      .set("token", `Bearer ${adminToken}`)
      .send({
        // Your updated user data
        username: "updateduser",
      });

    // Expectations
    expect(updateResponse.status).toBe(200);
    expect(updateResponse.body).toHaveProperty("username", "updateduser");
    // Add more expectations based on your updated user data
  }, 20000);

  // Test case: Should return 403 Forbidden when updating a user without valid credentials
  it("should return 403 Forbidden when updating a user without valid credentials", async () => {
    // Assuming you have a different user for testing
    const otherUserCredentials = {
      email: "otheruser@example.com",
      password: "otheruserpassword",
      username: "otheruser",
    };

    // Create a different user
    const createResponse = await request(test_server)
      .post("/api/v1/users/create")
      .send(otherUserCredentials);

    // Get the ID of the created user
    const otherUserId = createResponse.body._id;

    // Attempt to update the user without valid credentials (not admin or account owner)
    const updateResponse = await request(test_server)
      .put(`/api/v1/users/update/${otherUserId}`)
      .set("token", `Bearer ${nonAdminToken}`)
      .send({
        // Your updated user data
        username: "updateduser",
      });

    // Expectations
    expect(updateResponse.status).toBe(403);
    // Add more expectations based on how you handle non-authorized updates
  }, 20000);

  // Test case: Should successfully delete a user with valid credentials (admin or account owner)
  it("should successfully delete a user with valid credentials (admin or account owner)", async () => {
    // Assuming you have a test user created previously
    const testUser = {
      email: "testuser@example.com",
      password: "testuserpassword",
      username: "testuser",
    };

    // Create a test user
    const createResponse = await request(test_server)
      .post("/api/v1/users/create")
      .send(testUser);

    // Get the ID of the created user
    const userId = createResponse.body._id;

    // Log the ID to check if it's correct
    console.log("User ID:", userId);

    // Delete the user using the hardcoded token
    const deleteResponse = await request(test_server)
      .delete(`/api/v1/users/delete/${userId}`)
      .set("token", `Bearer ${adminToken}`);

    // Expectations
    expect(deleteResponse.status).toBe(204);
  }, 20000);
});

and another file:
src/__tests__/routes/productRoutes.test.ts and place the following code:

import request from "supertest";
import test_server from "../../../test_setup";
import supertest from "supertest";

// You can either test with a real token like this:
const adminToken = "YourAdminToken";
// Alternatively:
const nonAdminToken = "YourNonAdminToken";

// OR
// Write a function that simulates an actual login process and extract the token from there
// And then store it in a variable for use in various test cases e.g:
// Assuming you have admin credentials for testing

// const adminCredentials = {
//   email: "admin@example.com",
//   password: "adminpassword",
// };

// Assuming you have a function to generate an authentication token
// const getAuthToken = async () => {
//   const response = await supertest(test_server)
//     .post("/api/v1/users/login")
//     .send(adminCredentials);
//   return response.body.token;
// };

// Get the admin authentication token (inside an async function or code block)
// const authToken = await getAuthToken();

// ------------------> do the same for non admin user and store the token in
// its own variable

// Assuming you have a test product data
const testProduct = {
  title: "Test Product",
  description: "This is a test product",
  image: "test-image.jpg",
  category: "Test Category",
  quantity: "10",
  inStock: true,
};
describe("User Routes", () => {
  // Test case: Should successfully create a new product with valid admin credentials
  it("should successfully create a new product with valid admin credentials", async () => {
    // Create a new product using the admin token
    const createResponse = await request(test_server)
      .post("/api/v1/products/create")
      .set("token", `Bearer ${adminToken}`)
      .send(testProduct);

    // Expectations
    expect(createResponse.status).toBe(201);
    expect(createResponse.body).toHaveProperty("title", "Test Product");
    // Add more expectations based on your product data
  }, 20000);

  // Test case: Should successfully get all products without authentication
  it("should successfully get all products without authentication", async () => {
    // Make a request to the endpoint without providing an authentication token
    const response = await request(test_server).get("/api/v1/products/all");

    // Expectations
    expect(response.status).toBe(200);
    // Add more expectations based on your implementation
  }, 20000);

  // Test case: Should successfully get a product by ID with a valid token
  it("should successfully get a product by ID with a valid token", async () => {
    // Assuming you have a product ID for testing
    const productId = "6593e048731070abb0939faf";

    // Make a request to the endpoint with a valid token and product ID
    const response = await request(test_server)
      .get(`/api/v1/products/${productId}`)
      .set("token", `Bearer ${adminToken}`);

    // Expectations
    expect(response.status).toBe(200);
    // Add more expectations based on your implementation
  }, 20000);

  // Test case: Should successfully update a product by ID with admin access
  it("should successfully update a product by ID with admin access", async () => {
    // Assuming you have a product ID for testing
    const productId = "6593e048731070abb0939faf";

    // Replace 'yourUpdatedProductData' with the data you want to update the product with
    const updatedProductData = {
      title: "Updated Product",
      description: "Updated Product Description",
      // Add more fields as needed
    };

    // Make a request to the endpoint with an admin token and product ID
    const response = await request(test_server)
      .put(`/api/v1/products/update/${productId}`)
      .set("token", `Bearer ${adminToken}`)
      .send(updatedProductData);

    // Expectations
    expect(response.status).toBe(200);
    // Add more expectations based on your implementation
  }, 20000);

  // Test case: Should successfully delete a product by ID with admin access
  it("should successfully delete a product by ID with admin access", async () => {
    // Assuming you have a product ID for testing
    const productId = "6593e0a5451f5e47ce363e00";

    // Make a request to the endpoint with an admin token and product ID
    const response = await request(test_server)
      .delete(`/api/v1/products/delete/${productId}`)
      .set("token", `Bearer ${adminToken}`);

    // Expectations
    expect(response.status).toBe(204);
    // Add more expectations based on your implementation
  }, 20000);
});

afterAll(() => {
  test_server.close();
});

In our routes files like src/routes/userRoutes.ts, our routes donot include the prefix /api/v1 as its only included in the src/index.ts file where the src/routes/index.ts is imported and included. However,dont forget to add it while writing unit tests for all the routes as they will all return a 404 status code in its absence.

Our tests involve usage of our utility functions, JWT middlewares in src/utils/jwtUtils.ts so lets first complete their unit tests in a separate file: src/__tests__/utils/jwtAuth.test.ts with the following code:

import {
  verifyTokenAndAdmin,
  verifyTokenAndAuthorization,
} from "../../utils/jwtUtils";
import dotenv from "dotenv";
dotenv.config();

// Mock token for testing purposes
const mockTokenForNonAdmin = "YourMockTokenForNonAdmin";

const mockTokenForAdmin = "YourMockTokenForAdmin";
// You can also simulate an actual login process,
// extract the token and store it in a variable like this:

// const adminCredentials = {
//   email: "admin@example.com",
//   password: "adminpassword",
// };

// Assuming you have a function to generate an authentication token
// const getAuthToken = async () => {
//   const response = await supertest(test_server)
//     .post("/api/v1/users/login")
//     .send(adminCredentials);
//   return response.body.token;
// };

// Get the admin authentication token (inside an async function or code block)
// const authToken = await getAuthToken();

// Authorization Test Cases
describe("Authorization Tests", () => {
  // Test Case: Verify Token and Authorization (valid token and authorization)
  it("should verify a valid JWT token and authorization", (done) => {
    const req: any = {
      headers: {
        token: `Bearer ${mockTokenForNonAdmin}`,
      },
      user: {
        id: "mockUserId",
        isAdmin: false,
      },
      params: {
        id: "mockUserId",
      },
    };

    const res: any = {
      status: () => res, // Mock status function
      json: (message: string) => {
        // Assert the message or other expectations
        expect(req.user).toBeDefined();
        done();
      },
    };

    const next = () => {
      // Should not reach here
      done.fail("Should not reach next middleware on valid token");
    };

    verifyTokenAndAuthorization(req, res, next);
  });

  // Test Case: Verify Token and Authorization (unauthorized)
  it("should handle unauthorized access for authorization", (done) => {
    const req: any = {
      headers: {
        token: `Bearer ${mockTokenForNonAdmin}`,
      },
      user: {
        id: "otherUserId",
        isAdmin: false,
      },
      params: {
        id: "mockUserId",
      },
    };

    const res: any = {
      status: () => res, // Mock status function
      json: (message: string) => {
        // Assert the message or other expectations
        expect(message).toBe("You are not allowed to do that!");
        done();
      },
    };

    const next = () => {
      // Should not reach here
      done.fail("Should not reach next middleware on unauthorized access");
    };

    verifyTokenAndAuthorization(req, res, next);
  });
});

// Admin Access Test Cases
describe("Admin Access Tests", () => {
  // Test Case: Verify Token and Admin Access (valid token and admin)
  it("should verify a valid JWT token and admin access", (done) => {
    const req: any = {
      headers: {
        token: `Bearer ${mockTokenForNonAdmin}`,
      },
      user: {
        id: "mockUserId",
        isAdmin: true,
      },
      params: {
        id: "mockUserId",
      },
    };

    const res: any = {
      status: () => res, // Mock status function
      json: (message: string) => {
        // Assert the message or other expectations
        expect(req.user).toBeDefined();
        done();
      },
    };

    const next = () => {
      // Should not reach here
      done.fail(
        "Should not reach next middleware on valid token and admin access"
      );
    };

    verifyTokenAndAdmin(req, res, next);
  });

  // Test Case: Verify Token and Admin Access (non-admin)
  it("should handle non-admin access for admin authorization", (done) => {
    const req: any = {
      headers: {
        token: `Bearer ${mockTokenForNonAdmin}`,
      },
      user: {
        id: "mockUserId",
        isAdmin: false,
      },
      params: {
        id: "mockUserId",
      },
    };

    const res: any = {
      status: () => res, // Mock status function
      json: (message: string) => {
        // Assert the message or other expectations
        expect(message).toBe("You are not allowed to do that!");
        done();
      },
    };

    const next = () => {
      // Should not reach here
      done.fail("Should not reach next middleware on non-admin access");
    };

    verifyTokenAndAdmin(req, res, next);
  });
});

This suite has robust test cases to verify the functionality of two crucial middleware functions, verifyTokenAndAuthorization and verifyTokenAndAdmin, and is an intergral part of our previous tests written in src/__tests__/utils/jwtUtils.test.ts.

In the "Authorization Tests" section, we meticulously validate the proper functioning of token validation and authorization. We take pride in ensuring that our application accurately handles scenarios where a valid JWT token is presented along with appropriate authorization, as well as scenarios where unauthorized access is attempted, with clear error messages to guide the user.

Moving on to the "Admin Access Tests" section, we continue to demonstrate our dedication to a secure user access control system. Our tests encompass scenarios where valid tokens with admin privileges are successfully processed, and where non-admin users encounter appropriate error messages when attempting admin-protected actions.

By employing mocked request (req), response (res), and the next function, we simulate the middleware execution flow during testing. The use of the done callback underscores our commitment to thorough and reliable asynchronous testing methodologies.

In essence, our test suite for these middleware functions serves as a testament to our unwavering commitment to delivering a secure and robust Express.js application, ensuring that token verification, authorization, and admin access control are implemented with the utmost precision and reliability.

In our User Routes test suite, we ensure the robust functionality of various endpoints within our Express.js application. Leveraging the Supertest library, we validate the creation, login, retrieval, update, and deletion of user data. By employing both admin and non-admin tokens, we meticulously simulate scenarios that cover a spectrum of user access levels. Our tests underscore the meticulousness with which we handle user authentication, authorization, and access control, providing a solid foundation for the secure operation of our application. Furthermore, we prioritize cleanliness by incorporating cleanup procedures, inside afterAll method, to remove test users after execution, demonstrating our commitment to maintaining a pristine testing environment. With these tests, we confidently assure the reliability and security of our User Routes, fostering a robust user management system within our application.

On the otherhand, our Product Routes suite orchestrates a meticulous examination of various product endpoints to guarantee the seamless operation of our Express.js application. The tests encompass the creation, retrieval, update, and deletion of products, ensuring their secure management. The initiation of a new product with valid admin credentials is rigorously validated, emphasizing our commitment to robust functionality. Furthermore, we ensure that product retrieval, whether by ID or not, transpires smoothly, even in the absence of authentication. For administrative tasks like updating and deleting products, our suite validates the integrity of these operations with precision. Each test is meticulously crafted to assess the application's response under different scenarios, solidifying our confidence in the resilience and reliability of our Product Routes. Following the suite's execution, we gracefully close the server, maintaining an orderly testing environment.

By now you should have an output in the terminal with this ending:

Indicating that all the tests succeeded.
The Snapshots: 0 total message indicates that no Jest snapshots were created or updated during the test run. Snapshots in Jest are a way to automatically capture the output of a component or data structure and compare it against future changes to detect unintended changes.
Jest snapshots are often used in front-end testing, especially for UI components to capture the rendered output of components and help identify unintended changes. For our backend API, we were mostly dealing with server-side logic, database interactions, and APIs, snapshots were less critical which is why we dont have any.
Test Suites: 12 passed, 12 total mean we have 12 test suites and they all passed and Tests: 66 passed, 66 total means we have 66 tests and they all passed.
As you can see, opposed to rumours from most developers that unit testing is hard, its truly not hard given time but time consuming instead though worth it as previously discussed.

Overall Sammary

Embarking on a three-phase voyage, our article delved into the realm of Node.js and TypeScript development, sculpting a masterpiece of efficiency and reliability. In the inaugural phase, we meticulously established a robust environment, configuring the stage with precision using tools like Jest, Supertest among others. With the foundation laid, our journey traversed the expansive landscape of building a dynamic API with Express, navigating the intricacies of routing, controllers, and middleware. Finally, as our opus neared completion, we explored the symphony of unit testing, employing Jest and Supertest to ensure our code's fortitude. This trilogy encapsulates not just the process but the artistry behind crafting resilient, scalable, and well-tested Node.js applications in TypeScript, leaving developers equipped and inspired for their own epic odyssey in the world of software development.

Conclusion

In the symphony of Node.js and TypeScript development, orchestrating harmony through unit testing with Jest and Supertest, coupled with the formidable MongoDB and Mongoose ORM in the grandeur of Express, transforms the software development journey into an enlightened expedition. Through meticulous testing, we've navigated the intricacies of our codebase, ensuring resilience and reliability. With Jest as our virtuoso conductor, and Supertest as the keen-eared observer, the seamless integration of MongoDB and Mongoose has rendered our Express applications not just functional, but robust and secure. As we conclude this odyssey through the landscape of unit testing, we embrace a future where our code stands as a testament to its own quality, a melody composed with precision and played with confidence in the ever-evolving symphony of software development. May your code remain harmonious, your tests unwavering, and your development journey a continual crescendo of success.

Useful Links

Github Repo Twitter

Unit Testing in NodeJS with Express & TypeScript || Part Two: Building the API

Abeinemukama Vicent — Thu, 28 Dec 2023 08:01:28 +0000

Welcome back, In the previous article, we discussed about part one of our unit testing guide by setting up the environment for a NodeJS api with Express and TypeScript. We set up jest as a testing framework and supertest for providing high level of abstraction to test http requests.
In this second part of the guide, we are going to build the api and connect it to an online database, MongoDB Atlas and in the third and last part of the guide, we are going to write unit tests for each line of code after which we will conclude with a glimpse on life after unit testing, just like you have been thinking, intergration testing.
Here is the github repository for the first part of the guide.
If you don't have it on your local machine, run



git clone https://github.com/Abeinevincent/nodejs_unit_testing_guide.git .

inside your desired directory to get on the same track with us.

Connecting MongoDB to our NodeJS Backend

Developed by MongoDB Inc, MongoDB is a source-available, cross-platform, document-oriented database program, classified as a NoSQL database product. It utilizes JSON-like documents with optional schemas.
In this guide we will connect to a beginner friendly MongoDB atlas for simplicity. If you already have some experience with MongoDB, feel free to use any other MongoDB service such as MongoDB Campass.
In addition we will use mongoose to connect MongoDB to our NodeJS backend and query the database. It is JavaScript object-oriented programming library that creates a connection between MongoDB and the Node.js JavaScript runtime environment and provides a uniform API for accessing numerous different databases including MongoDB.

Steps

On the terminal run the following command:



npm install mongoose dotenv

Mongoose is an ODM for connecting our backend to MongoDB and querying the database, whereas dotenv is for helping us access our environment variables such as DB connection url in our NodeJS API.
I am using npm and if youre using yarn you may replace npm install with yarn add.
mongoose provides its own type definitions, so you don't need @types/mongoose installed.

Head over to MongoDB atlas and login or create an account to access the database. I recommend proceeding with a free cluster for now in development. In the databases section click on connect and grab the database url.

Inside the home directory, create a new file named .env and add the following code:



MONGODB_URL=YourConnectionUrl

To complete the connection, lets add the following code to our src/index.ts



import dotenv from "dotenv";
import mongoose from "mongoose";
dotenv.config()

in the imports section and



// DB CONNECTION

if (!process.env.MONGODB_URL) {
  throw new Error("MONGODB_URL environment variable is not defined");
}

mongoose
  .connect(process.env.MONGODB_URL)
  .then(() => {
    console.log("MongoDB connected to the backend successfully");
  })
  .catch((err) => console.log(err));

before starting the backend server

On the terminal run



npm run dev

to start the development server in development mode.
Output should be as shown below:

Overview

Before we start writing code, lets remind ourselves of our beloved folder structure which is the mostly for helping us stay on track by separating concerns.
First we have models for accomodating our database schema, then services for handling services logic forexample the userService.ts may handle user related operations like creating a new user among others, then controllers for handling specific routes or endpoints and the routes for defining routes of our application. As a plus we also have middlewares for accomodating middleware functions that can be used across multiple contollers or services. We also have helpers for accomodating utility functions or helper modules that assist in various tasks related to the services or contollers and __tests__ for holding our unit tests that we will write in the next article. Lastly we have utils for accomodating generic and reusable functions that can be used anywhere in the application. Most of the times, you will place here functions that do not belong to specific category but are instead generic.

If you are used to a different folder structure from this, dont worry, you will still understand the concepts very well, just follow along.

Writing DB Models

A model is blueprint for organizing and storing information in the database.
In this guide, our api will have only 2 models one for users other for products. We will have an ecommerce-like api in order to understand every concept in some use case and also know how to write a unit test for it.
We will write our models inside src/models.
Create a new file: src/models/User.ts and place the following code:



// Import necessary modules
import mongoose, { Schema, Document, Types } from "mongoose";

// Define the interface for User document
export interface IUser extends Document {
  email: string;
  username: string;
  password: string;
  isAdmin: boolean;
  savedProducts: Types.ObjectId[];
}

// Create a schema for the User model
const userSchema: Schema<IUser> = new Schema({
  email: { type: String, required: true },
  username: { type: String, required: true },
  password: { type: String, required: true },
  isAdmin: { type: Boolean, default: false },
  savedProducts: [{ type: mongoose.Schema.Types.ObjectId, ref: "Product" }],
}, {timestamps: true});

// Create and export the User model
export default mongoose.model<IUser>("User", userSchema);

The model has 5 fields as described.
Interface (IUser) defines the structure of a user document, specifying the types for each field.
Schema (userSchema) describes how the data should be organized for a user. Each field is defined with its type, and in the case of savedProducts, it's an array of MongoDB ObjectIds referencing the 'Product' model.
mongoose.model('User', userSchema) creates the Mongoose model named 'User' using the schema we defined.

Create another file: src/models/Product.ts and place the following code:



// Import necessary modules
import mongoose, { Schema, Document } from "mongoose";

// Define the interface for Product document
export interface IProduct extends Document {
  title: string;
  description: string;
  image: string;
  category: string;
  quantity: string;
  inStock: boolean;
}

// Create a schema for the Product model
const productSchema: Schema<IProduct> = new Schema(
  {
    title: { type: String, required: true },
    description: { type: String, required: true },
    image: { type: String, required: true },
    category: { type: String, required: true },
    quantity: { type: String, required: true },
    inStock: { type: Boolean, default: false },
  },
  { timestamps: true }
);

// Create and export the Product model
export default mongoose.model<IProduct>("Product", productSchema);

Writing Services

Create a new file: src/services/userService.ts and place the following code:



// Import necessary modules
import { generateToken } from "../utils/jwtUtils";
import User, { IUser } from "../models/User";
import { hashPassword } from "../utils/passwordUtils";

// Create a new user
export const createUser = async (userInput: IUser): Promise<IUser> => {
  try {
    // Hash the user's password before storing it
    const hashedPassword = await hashPassword(userInput.password);

    // Create the user with the hashed password
    const newUser = await User.create({
      ...userInput,
      password: hashedPassword,
    });

    return newUser;
  } catch (error) {
    throw new Error(`Error creating user: ${error.message}`);
  }
};

// Login user
export const loginUser = async (
  email: string,
  password: string
): Promise<{ user: Omit<IUser, "password">; token: string }> => {
  try {
    // Find user by email
    const user = await User.findOne({ email });
    if (!user) {
      throw new Error("User not found");
    }

    // Compare the provided password with the stored hashed password
    const isPasswordValid = await comparePassword(password, user.password);
    if (!isPasswordValid) {
      throw new Error("Invalid password");
    }

    // Generate JWT token
    const token = generateToken({
      id: user._id,
      username: user.username,
      email: user.email,
      isAdmin: user.isAdmin,
    });

    // Destructure password from the data returned
    const { password: _password, ...userData } = user.toObject();

    return { user: userData as Omit<IUser, "password">, token };
  } catch (error) {
    throw new Error(`Error logging in: ${error.message}`);
  }
};

// Get all users
export const getAllUsers = async (): Promise<IUser[]> => {
  try {
    const users = await User.find();
    return users;
  } catch (error) {
    throw new Error(`Error getting users: ${error.message}`);
  }
};

// Get user by ID with his saved products
export const getUserById = async (userId: string): Promise<IUser | null> => {
  try {
    const user = await User.findById(userId).populate("savedProducts");
    return user;
  } catch (error) {
    throw new Error(`Error getting user: ${error.message}`);
  }
};

// Update user by ID
export const updateUser = async (
  userId: string,
  updatedUser: Partial<IUser>
): Promise<IUser | null> => {
  try {
    const user = await User.findByIdAndUpdate(userId, updatedUser, {
      new: true,
    });
    return user;
  } catch (error) {
    throw new Error(`Error updating user: ${error.message}`);
  }
};

// Delete user by ID
export const deleteUser = async (userId: string): Promise<void> => {
  try {
    await User.findByIdAndDelete(userId);
  } catch (error) {
    throw new Error(`Error deleting user: ${error.message}`);
  }
};

We are handling auth in the same file containing other user services since our app is abit minimal but feel free to separate login and register/create user to a separate file say auth.ts.

In the section for creating a user, we have a function for hashing our password using bcrypt from src/utils/passwordUtils.ts whose code is as shown below:



import bcrypt from "bcrypt";

// Hash a password
export const hashPassword = async (password: string): Promise<string> => {
  const saltRounds = 10;
  const hashedPassword = await bcrypt.hash(password, saltRounds);
  return hashedPassword;
};

// Compare a password with its hash
export const comparePassword = async (
  password: string,
  hashedPassword: string
): Promise<boolean> => {
  return bcrypt.compare(password, hashedPassword);
};

Bcrypt with its type definitions for typescript can be installed using the following commands:



npm install bcrypt

and



npm install --save-dev @types/bcrypt

Lets move to the productsService,
create another file; src/services/productService.ts and put the following code:



// Import necessary modules
import Product, { IProduct } from "../models/Product";

// Create a new product
export const createProduct = async (
  productInput: IProduct
): Promise<IProduct> => {
  try {
    const newProduct = await Product.create(productInput);
    return newProduct;
  } catch (error) {
    throw new Error(`Error creating product: ${error.message}`);
  }
};

// Get all products
export const getAllProducts = async (): Promise<IProduct[]> => {
  try {
    const products = await Product.find();
    return products;
  } catch (error) {
    throw new Error(`Error getting products: ${error.message}`);
  }
};

// Get product by ID
export const getProductById = async (
  productId: string
): Promise<IProduct | null> => {
  try {
    const product = await Product.findById(productId);
    return product;
  } catch (error) {
    throw new Error(`Error getting product: ${error.message}`);
  }
};

// Update product by ID
export const updateProduct = async (
  productId: string,
  updatedProduct: Partial<IProduct>
): Promise<IProduct | null> => {
  try {
    const product = await Product.findByIdAndUpdate(productId, updatedProduct, {
      new: true,
    });
    return product;
  } catch (error) {
    throw new Error(`Error updating product: ${error.message}`);
  }
};

// Delete product by ID
export const deleteProduct = async (productId: string): Promise<void> => {
  try {
    await Product.findByIdAndDelete(productId);
  } catch (error) {
    throw new Error(`Error deleting product: ${error.message}`);
  }
};

Writing Controllers

All set for our 2 services,lets proceed to controllers, starting with userController once again.
Create a new file: src/controllers/userController.ts and place the following code:



import { Request, Response } from 'express';
import * as UserService from '../services/userService';

// Create a new user
export const createUser = async (req: Request, res: Response): Promise<void> => {
  try {
    const newUser = await UserService.createUser(req.body);
    res.status(201).json(newUser);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

// Login user
export const loginUser = async (req: Request, res: Response): Promise<void> => {
  const { email, password } = req.body;

  try {
    const { user, token } = await UserService.loginUser(email, password);
    res.status(200).json({ user, token });
  } catch (error) {
    res.status(401).json({ error: error.message });
  }
};

// Get all users
export const getAllUsers = async (_req: Request, res: Response): Promise<void> => {
  try {
    const users = await UserService.getAllUsers();
    res.status(200).json(users);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

// Get user by ID
export const getUserById = async (req: Request, res: Response): Promise<void> => {
  try {
    const user = await UserService.getUserById(req.params.userId);
    if (user) {
      res.status(200).json(user);
    } else {
      res.status(404).json({ error: 'User not found' });
    }
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

// Update user by ID
export const updateUser = async (req: Request, res: Response): Promise<void> => {
  try {
    const updatedUser = await UserService.updateUser(req.params.userId, req.body);
    if (updatedUser) {
      res.status(200).json(updatedUser);
    } else {
      res.status(404).json({ error: 'User not found' });
    }
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

// Delete user by ID
export const deleteUser = async (req: Request, res: Response): Promise<void> => {
  try {
    await UserService.deleteUser(req.params.userId);
    res.status(204).send();
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

and then to productController,
lets create another file: src/controllers/productController.ts and place the following code:



import { Request, Response } from 'express';
import * as ProductService from '../services/productService';

// Create a new product
export const createProduct = async (req: Request, res: Response): Promise<void> => {
  try {
    const newProduct = await ProductService.createProduct(req.body);
    res.status(201).json(newProduct);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

// Get all products
export const getAllProducts = async (_req: Request, res: Response): Promise<void> => {
  try {
    const products = await ProductService.getAllProducts();
    res.status(200).json(products);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

// Get product by ID
export const getProductById = async (req: Request, res: Response): Promise<void> => {
  try {
    const product = await ProductService.getProductById(req.params.productId);
    if (product) {
      res.status(200).json(product);
    } else {
      res.status(404).json({ error: 'Product not found' });
    }
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

// Update product by ID
export const updateProduct = async (req: Request, res: Response): Promise<void> => {
  try {
    const updatedProduct = await ProductService.updateProduct(req.params.productId, req.body);
    if (updatedProduct) {
      res.status(200).json(updatedProduct);
    } else {
      res.status(404).json({ error: 'Product not found' });
    }
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

// Delete product by ID
export const deleteProduct = async (req: Request, res: Response): Promise<void> => {
  try {
    await ProductService.deleteProduct(req.params.productId);
    res.status(204).send();
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
};

All set for our controllers too. We should be proceeding to the routes but we need to take a moment and think about how this api will be used permission wise, a case in point a user should not be able to delete a product, update a product, or delete his fellow users. We need a way of distributing permissions and protecting some sensitive endpoints from being hit by unauthorized or unauthenticated users. Realistically, some endpoints are not as vulnerable as others forexample, an endpoint that will be consuming the getAllProducts controller may be hit by any user, authenticated or not, because it only returns a list products, read only,and on the user interface,you may need to display these products even for guests before you prompt him to login just in case he gets interested in purchasing it.The endpoints he may need to hit after desiring to purchase may ofcourse be protected and require him to be authenticated and also have some specific previlages or permissions.
To achieve this we will use a library called jsonwebtoken.
It is a proposed Internet standard for creating data with optional signature and/or optional encryption whose payload holds JSON that asserts some number of claims. The tokens are signed either using a private secret or a public/private key. It is also a good way of securely transmitting information between parties because they can be signed, which means you can be certain that the senders are who they say they are. You can read more about json web token here

Open your terminal and run the following command



npm install jsonwebtoken

After installing the libray, we also need to install @types/jsonwebtoken for providing its typescript definitions:
On the same terminal, run



npm i --save-dev @types/jsonwebtoken

Lets get started writing some jwt logic,
Create another file: utils/jwtUtils.ts and place the following code:



// utils/jwtUtils.ts
import { Request, Response, NextFunction } from "express";
import jwt, { VerifyErrors } from "jsonwebtoken";
import { Types } from "mongoose";

// Generate jwt token
type JWTPayload = {
  id: Types.ObjectId;
  username: string;
  email: string;
  isAdmin: boolean;
};

// Custom Request type with 'user' property
interface CustomRequest extends Request {
  user: {
    id: Types.ObjectId;
    username: string;
    email: string;
    isAdmin: boolean;
  };
}

export const generateToken = (payload: JWTPayload): string => {
  if (!process.env.JWT_SEC) {
    throw new Error("JWT_SEC environment variable is not defined");
  }

  const token = jwt.sign(payload, process.env.JWT_SEC, {
    expiresIn: process.env.JWT_EXPIRY_PERIOD,
  });
  return token;
};

// Verify token
export const verifyToken = (
  req: CustomRequest,
  res: Response,
  next: NextFunction
): void => {
  const authHeader = req.headers.token;
  if (authHeader) {
    const token = Array.isArray(authHeader)
      ? authHeader[0].split(" ")[1]
      : authHeader.split(" ")[1];

    if (!process.env.JWT_SEC) {
      throw new Error("JWT_SEC environment variable is not defined");
    }

    jwt.verify(
      token,
      process.env.JWT_SEC,
      (err: VerifyErrors | null, user: any) => {
        if (err) return res.status(403).json("Token is not valid!");
        req.user = user;
        next();
      }
    );
  } else {
    res.status(401).json("You are not authenticated!");
  }
};

// Authorize account owner
export const verifyTokenAndAuthorization = (
  req: CustomRequest,
  res: Response,
  next: NextFunction
): void => {
  verifyToken(req, res, () => {
    if (req.user.id.toString() === req.params.id || req.user.isAdmin) {
      next();
    } else {
      return res.status(403).json("You are not allowed to do that!");
    }
  });
};

// Authorize admin
export const verifyTokenAndAdmin = (
  req: CustomRequest,
  res: Response,
  next: NextFunction
): void => {
  verifyToken(req, res, () => {
    if (req.user.isAdmin) {
      next();
    } else {
      return res.status(403).json("You are not allowed to do that!");
    }
  });
};

We have all this code in src/utils/jwtUtils.ts because our app is abit minimal, feel free to separate each jwt middleware function in a separate file in the middlewares folder: src/middlewares, and pass appropriate props with their typescript types.

we are exporting four functions; one for creating our jwt token, generateToken, other for verifying our jwt, verifyToken, other for authorizing a user, account owner or admin, to do hit some endpoints such as updating his account detailslike username, password, among others, verifyTokenAndAuthorization and lastly for authorising only admins to be able to hit some endpoints such as deleting or updaing products, verifyTokenAndAdmin.
This way, youre able to protect some endpoints that may have sensitive data such that you either need them to be hit by an authenticated user or by a user with specific previlages in addition to being authorised.
The endpoints you may need one to hit when logged in regardless of his previlages will only require generateToken function.

Donot forget to add the environment variable we were using in .env file like this:



JWT_SEC=YourJWTSecret
JWT_EXPIRY_PERIOD=YourJWTExpiryTime e.g 15m (15 minutes)

On the expiry time of JWT, I recommend a period of time not more than 30 minutes for security purposes. However in development, you can set whatever you want to avoid being locked out frequently while coding. On the frontend, this token is decoded and the user activity tracked such that if say the user spends YourJWTExpiryTime with out actively using your app, he is logged out automatically.
The period should be in format: 15m for 15 minutes, 15s for 15 seconds, 15h for 15 hours, 15d for 15 days, 15w for 15 weeks, 15y for 15 years and so on.

Lets now proceed to the routes and get our jwt functions consumed too accordingly:

Writing Routes

Create a new file: src/routes/userRoute.ts and place the following code:



import express from "express";
import {
  verifyTokenAndAuthorization,
  verifyTokenAndAdmin,
} from "../utils/jwtUtils";
import {
  getAllUsers,
  updateUser,
  deleteUser,
  createUser,
  loginUser,
} from "../controllers/userController";

const userRoutes = (router: express.Router) => {
  // Route for creating a new user
  router.post("/users/create", createUser);

  //   Route for logging in a user
  router.post("/users/login", loginUser);

  // Route for getting all users (only accessible by admin)
  router.get("/users/all", verifyTokenAndAdmin, getAllUsers);

  // Route for updating a user (protected, only account owner or admin)
  router.put("/users/update/:id", verifyTokenAndAuthorization, updateUser);

  // Route for deleting a user (protected, only accessible by admin)
  router.delete("/users/delete/:id", verifyTokenAndAuthorization, deleteUser);
};

export default userRoutes;

and another one: src/routes/productRoute.ts and place the following code:



import express from "express";
import {
  verifyToken,
  verifyTokenAndAuthorization,
  verifyTokenAndAdmin,
} from "../utils/jwtUtils";
import {
  getAllProducts,
  createProduct,
  getProductById,
  updateProduct,
  deleteProduct,
} from "../controllers/productController";

const productRoutes = (router: express.Router) => {
  // Route for getting all products
  router.get("/products/all", getAllProducts);

  // Route for creating a new product (protected, only accessible by admin)
  router.post("/products/create", verifyTokenAndAdmin, createProduct);

  // Route for getting a product by ID
  router.get("/products/:productId", verifyToken, getProductById);

  // Route for updating a product by ID (protected, only accessible by admin)
  router.put("/products/update/:productId", verifyTokenAndAdmin, updateProduct);

  // Route for deleting a product by ID (protected, only accessible by admin)
  router.delete(
    "/products/delete/:productId",
    verifyTokenAndAdmin,
    deleteProduct
  );
};

export default productRoutes;

Some routes are protected as we discussed while writing jwt logic.
Lets finalise our api by connecting our routes to the index file inside routes and then the main index file.
Create a new file: src/routes/index.ts and place the following code:



import express from "express";
import userRoutes from "./userRoute";
import productRoutes from "./productRoute";
const router = express.Router();

export default (): express.Router => {
  // USER
  userRoutes(router);

  //   PRODUCT
  productRoutes(router);
  return router;
};

All set, lets headover again to the main index file: src/index.ts and add the following line of code after db connection and before starting the backend server:



// Serve other routes
app.use("/api/v1/", routes());

Don't forget to import the routes in the imports section like this:



import routes from "./routes";

Our api is now steady and running at PORT:8800, of course on your local machine 😄.

Conclusion

In this second part of the unit testing guide, we embarked on a journey to build a robust and scalable API using Node.js, Express, MongoDB, and TypeScript. We covered various aspects of the development process, from setting up the project structure to implementing user and product functionalities.

Recap of Achievements:

Project Structure:

We started by establishing a well-organized project structure, with dedicated folders for tests, models, controllers, services, routes, middlewares, helpers and utility functions. We may not have used some of the folders because we only have 2 schema we were using for illustration but I explained what each folder contains and you might use them all in a real world production grade API. This modular approach enhances maintainability and readability.

User Authentication:

Security is paramount in any application, and we implemented user authentication using JSON Web Tokens (JWT). We created middleware functions to verify tokens, authorize account owners, and validate administrator privileges.

Database Interaction:

Leveraging MongoDB Atlas as our database, we seamlessly integrated Mongoose as the ODM (Object Data Modeling) library. This allowed us to define schemas, models, and perform CRUD operations effortlessly.

User and Product Functionalities:

We implemented essential user functionalities, such as user registration, login, and profile updates. Additionally, we extended our API to handle product-related operations, including creation, retrieval, updating, and deletion.

Express Routes:

Our routes were thoughtfully designed to encapsulate functionality and maintain a clear separation of concerns. JWT authentication was selectively applied to secure some routes, ensuring that sensitive operations are only accessible by authorized and authenticated users.

What's Next?

As we conclude this phase, we eagerly anticipate the next chapter. In Part 3 of this guide, we will delve into the realm of testing. Comprehensive unit tests are crucial for ensuring the reliability and correctness of our API. Stay tuned as we explore the testing landscape and employ best practices to validate the functionality of our endpoints.

Thank you for reading to this far, your commitment to understanding the intricacies of development and security is commendable. Until next time, happy coding!

Checkout these helpful Links:
Github Repo Twitter

Unit Testing in NodeJS with Express & TypeScript || Part One: Environment Setup

Abeinemukama Vicent — Sat, 23 Dec 2023 06:29:59 +0000

Node.js is a cross-platform, open-source JavaScript run time environment that can run on Windows, Linux, Unix, macOS, and other operating systems. It runs on the V8 JavaScript engine, and executes JavaScript code outside a web browser.
On the other hand, ExpressJS is a web application backend framework for building restful apis with NodeJS.
TypeScript is a free and open-source high-level programming language developed by Microsoft that adds static typing with optional type annotations to JavaScript. In this guide, we will build an express api and add type checking using TypeScript and use it to learn unit testing while building restful apis with NodeJS.

This part is an intergral part of a 3 part series and a complete guide will involve environment setup, connecting an external database like MongoDB and writing database models and writing unit tests for crud operations(get, post, delete, put).
This first part will only discuss the environment setup and overall folder structure.

What Exactly is Unit Testing?

Unit testing is a software development process in which the smallest testable parts of an application, called units, are individually scrutinized for proper operation. In other wards, each part of the application is tested in isolation before merging to the main repository.
In the fast-paced realm of software development, ensuring the reliability of your code is paramount. Unit testing plays a crucial role in maintaining code quality, catching bugs early, and facilitating smoother collaboration within a team. In this multipart series, I'll guide you through the process of setting up a robust unit testing environment for your Node.js application built with Express and TypeScript using Jest and Supertest.
Jest is a delightful JavaScript Testing Framework with a focus on simplicity. It works with projects using: Babel, TypeScript, Node, React, Angular, Vue and other JavaScript technologies. We will also use Supertest for http assertions, forming the backbone of our testing suite. It is a NodeJS library, and enables developers and testers to write automated tests for routes and endpoints and can be used as a standalone library or with testing frameworks like Jest and provides a high-level abstraction to test HTTP requests.

Project Setup

Before diving into unit testing, it's essential to have a well-organized project structure. Whether you're starting a new project or integrating testing into an existing one, maintaining a clear separation of concerns will make your life easier in the long run. Consider structuring your project with directories for source code, tests, and configuration files. This foundational step ensures that your codebase is ready to embrace a testing mindset.

Steps:

Create a new folder in your desired location and name it unit_testing_api_guide.
With the folder open in your favourite code editor, open terminal and run the following command to initialise a NodeJS project:



npm init -y

After initialising the project, install the following dependencies with the following command:



npm install express cors helmet

express is NodeJS framework we are using for routing, cors helps us define a way for accessing our api on different domains whereas helmet is collection of middleware functions that set various HTTP headers to enhance the security of your Node. I prefer starting with it in any NodeJS project because it starts by hiding vulnerable headers like 'X-Powered-By' header which tells about the technology youre using giving attackers a chance to concentrate on its vulnerabilities to penetrate your API.

All the dependencies for unit testing are only used in development so we will install them as dev dependencies(with a -D or --save-dev flag) along with other development dependencies like typescript after which I will explain.

On the same terminal, run the following command:



npm install -D typescript ts-node nodemon jest ts-jest @types/jest supertest @types/supertest @types/cors @types/express

Let me explain:
TypeScript is meant to help us in type checking our code and and transpiles to JavaScript. ts-node is a TypeScript execution engine and REPL for Node.js and will will help us transform TypeScript into JavaScript, enabling direct execution of TypeScript on Node.js without precompiling.
nodemon is meant to help us restart the server in case of changes while in development. Also note that, in newer versions of NodeJS, nodemon is being phased out and you will find yourself using --watch flag instead. Nevertheless, we will use it in this guide to restart our server automatically in case need arises.
jest is the testing framework we are using and @types/jest for adding TypeScript definitions for jest. On the other hand, ts-jest is a Jest transformer with source map support and lets you use Jest to test projects written in TypeScript and supports all features of TypeScript including type-checking.
supertest is for providing high level of abstraction to test http requests and @types/supertest for providing TypeScript definitions for it.
@types/cors and @types/express are also for providing type definitions for cors and express respectively.

These are the dependencies we need for now and note that we may require other dependencies in the next parts of this guide with their type definitions for TypeScript but we will install them when need arises.

Project Folder Structure

Here is our overall project folder structure:

I find this level of separation of concerns appropriate in most of my NodeJS projects and we will be sticking to it in this guide.

Lets create our project step by step and remember our main emphasis is on testing directories and files. However, since we are going to learn how to write unit tests for each line of code from controllers, to routes and code in other directories, we need to understand how other folders are created, what they contain and why separate them.

First, create a new folder in the root directory and call it src. Inside src, create all folders as shown in our folder structure. Worry not, we will understand what each folder consists of step by step.
However, in this first part of the guide, we will write all our code inside the index file, index.ts as its all about environment setup and will go through most of the folders in the coming parts.
Also, create a file named index.ts inside src and place the following code:



import express from "express";
import cors from "cors";
import helmet from "helmet";
const app = express();

// USE HELMET AND CORS MIDDLEWARES
app.use(
  cors({
    origin: ["*"], // Comma separated list of your urls to access your api. * means allow everything
    credentials: true, // Allow cookies to be sent with requests
  })
);
app.use(helmet());

app.use(express.json());

app.get("/", async (req: express.Request, res: express.Response) => {
  try {
    res.send(
      "Welcome to unit testing guide for nodejs, typescript and express!"
    );
  } catch (err) {
    console.log(err);
  }
});

// Start backend server
const PORT = process.env.PORT || 8800;
app.listen(PORT, () => {
  console.log(`Backend server is running at port ${PORT}`);
});

export default app;

Before running our code to start writing unit tests, lets first make some configurations:
Create a file named tsconfig.json in the root directory and add the following configuration:



{
  "compilerOptions": {
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "target": "ES2020",
    "baseUrl": "src",
    "noImplicitAny": true,
    "sourceMap": true,
    "esModuleInterop": true,
    "outDir": "dist"
  },
  "include": ["src/**/*"]
}

Create another file named nodemon.json in the root directory and place the following code:



{
  "watch": ["src"],
  "ext": ".ts,.js",
  "exec": "ts-node ./src/index"
}

Inside package.json, add the following scripts in the scripts section:



"scripts": {
    "build": "tsc",
    "start": "npm run build && node dist/src/index.js",
    "dev": "nodemon",
    "test": "jest --watchAll  --detectOpenHandles"
  },

Also, add the line:



"module": "module",

to your package.json to specify that we are using EsModules not NodeJS's default CommonJS pattern.

All set, now open the terminal and traverse to the location of your home directory and run:



npm run dev

to run the project in development mode.
Note that the command:nodemon is executed and the result should be as shown below:

Jest Setup

Create a new file in the home directory and name it jest.config.js and place the following code:



module.exports = {
  preset: "ts-jest",
  testEnvironment: "node",
  testMatch: ["**/**/*.test.ts"],
  verbose: true,
  forceExit: false,
};

This is a basic configuration for jest and we will be adding some additional config code when need arises in the next parts of this series.

Writing Sample Unit Tests

To confirm that our project is setup correctly for unit testing, lets write some tests.
Inside the tests folder, create a new file named app.test.ts and place the following code:



import app from "../index";
import request from "supertest";

describe("GET /", () => {
  it('responds with "Welcome to unit testing guide for nodejs, typescript and express!', async () => {
    const response = await request(app).get("/");
    expect(response.status).toBe(200);
    expect(response.text).toBe(
      "Welcome to unit testing guide for nodejs, typescript and express!"
    );
  });
});

Let me explain:
We started by importing our Express app (app) and the supertest library, which is used for making HTTP requests in our tests.
We then have a simple Jest test case, It describes and tests the behavior of your Express app when a GET request is made to the root ("/") endpoint. It expects the HTTP response status to be 200 and the response text to match the expected message.

On your terminal, run the following command:



npm run test

or just



npm test

and you should see the following output:

Congulatulations, you just setup a unit testing environment for a NodeJS, Express app using TypeScript.

Interested in the source code? Checkout the repository for the article here
If you liked the article, consider following me on Twitter
Stay tuned for the next 2 parts in this series and furture articles.

Data Fetching from an API using React Query - useQuery Hook Explained in Plain English

Abeinemukama Vicent — Tue, 20 Dec 2022 19:39:58 +0000

Often described as the missing data fetching library for react, react query makes fetching, caching, updating and synchronizing server state in react applications easier than ever imagined.
React query provides a custom hook, useQuery for all this without touching any global state.

Why React Query

If you ever used default options in react such as useEffect hook to fetch data from an api end point, you in one way or the other scratched your head trying to figure out how some of these aspects react query brings to the table could be achieved with less effort. In this article, we are going to diagonise everything you need to know to get confident using react query as your next data fetching library.
The article is specific to data fetching that is getting data from an api endpoint to efficiently understand the concept. Other capabilities of react query such as updating server data, deleting server data and posting data to the server will be discussed in the coming articles.

Prerequisites

Before you read this article, you should have fundamental knowledge of JavaScript such as arrow functions, array and object destructuring, asynchronous and synchronous behavior among others. If youre not confident enough with these concepts, you can reminisce your self here. Additionally, you need intermediate knowledge of react 16.8 or higher. I assume zero knowledge of data fetching in react so we are going to diagonise this concept from scratch.
However, if you ever used other data fetching libraries such as redux toolkit or used hooks provided by react itself, its a big plus reading this article.

What are some issues with data fetching in Effects?

If you write your data fetching code in the classic style, call fetch and set state inside useEffect, there are a few problems you will most likely encounter:

No content in initial HTML. If you fetch all data in effects, this likely means that you're not using server rendering, so the initial HTML is an empty shell, and the user doesn't see anything until all JS loads which can significantly slow down your application. If you do use server rendering or try to adopt it later, you'll notice that your effects don't run on the server so you're not able to render any useful content on the server. By the point you realize this, it might be late to change your app's architecture.
Race conditions. If you don't write a cleanup function that ignores stale responses, you'll likely introduce bugs when responses arrive in a different order.
No instant Back button. If the user navigates to another page and then clicks Back, by default, you don't have the data to show to them. This is because, by default, there is no cache, and the previous page component will have already unmounted. This means that when the user presses Back, they see a spinner instead of the place where they left off. This kind of user experience makes it easy to tell if something is written as a client-side single-page app, and not in a good way.
Slow navigations between screens. If you have parent and child components both doing fetching in effects, then the child component can't even start fetching until the parent component finishes fetching. These types of performance problems are very common in single-page apps and cause a lot more slowness than "excessive re-rendering" that people tend to focus on.

In this article, I will be using react without any template such as typescript but the upcoming aticles on this subject will be with a typescript template.
Phew! enough, lets get hands dirty by diagonising a code sample for further understanding how react query comes in handy to the rescue.

I am starting with creating a new react app using vite.

npx create-vite react-query-fetch

In my case, I called the project react-query-fetch
Select react from the drop-down and then JavaScript.
Lets now install project dependencies with the following command:

npm install

We can now run our project with the command:

npm run dev

We need two extra dependencies in our project; react-query for data fetching and axios for making HTTP requests. Lets install them with the following command:

npm install react-query axios

We need to wrap the entry point of our application with the QueryClientProvider to give our app access to the awesome hooks provided by react-query. To achieve this, lets replace the code in src/main.jsx with the following code:

import React from 'react'
import ReactDOM from 'react-dom/client'
import { QueryClient, QueryClientProvider } from 'react-query'
import App from './App'
import './index.css'

ReactDOM.createRoot(document.getElementById('root')).render(
  <QueryClientProvider client={new QueryClient}>
    <App />
  </QueryClientProvider>
)

We shall do our data fetching from an online API, jsonplaceholder, inside src/app.jsx.
Lets first clean up the default code inside src/app.jsx to leave it look this:

import './App.css'

function App() {
  // Data fetching will take place here..
  return (
    <div className="App">
      {/* JSX here.. */}
    </div>
  )
}

export default App

Lets start with a basic fetch from users endpoint and display their phone numbers in the browser.

import axios from 'axios'
import { useQuery } from 'react-query'
import './App.css'

function App() {
  const getUsers =  async () => {
    const {data} =  await axios.get('https://jsonplaceholder.typicode.com/users')
    return  data;
  }

  const { isLoading, isError, error, data } = useQuery('users', getUsers)
  if (isLoading) {
    return <div>Loading..</div>
  }
  if (isError) {
    return <div>Errror, {error.message}</div>
  }
  return (
    <div className="App">
      {data &&  data.map((user, index) => (
        <div  key={index}>{user.phone}</div>
      ))}
    </div>
  )
}

export default App

Explanation

We import the useQuery hook from react-query to eliminate need for useEffect hook from React. If you are familiar with data fetching in React, you know that the useEffect can be used to fetch data and reload the page. The difference between react-query and the useEffect is that react-query will initially return the previously fetched data and then re-fetch.
We import axios from axios library for use in making HTTP requests. In our case, we are making a get request.
'users' is called query key. react-query manages caching for us using query keys. This argument is required and can be a plain string, a variable or an array. In our case, its a plain string.
The getUsers function is an asynchronous one and returns data fetched from the end point. This argument is also required if no default query function has been defined in the entry point of the application. Incase of an error, useQuery hook will take care of it and in the next section of this article we will detail how this hook handles possible errors that may occur during data fetching. For this reason we didn't wrap our function in try-catch block.

In mycase, I wrote this function in the same file with the useQuery hook but we are not locked to this approach, you can have the function in a different file or folder and import it like you would for other files. This comes in handy for bigger projects.
In this basic example, we only have error, isFetching, isLoading and data.
Our, useQuery hook takes more arguments but in our example, we are taking two arguments that is a unique key to differentiate what is being fetched from the API and an asynchronous function where the fetching is done. The unique key is users and the function is getUsers.

The isLoading displays while the async-function runs, and an error is thrown when there is one. In our case there is no error and here is what we have after fetching: The hook returns an object with many properties but in our basic usage example we are returning error, isLoading, data and isError.

Advanced Usage of useQuery hook

In this section, we are going to diagonise advanced concepts including but not limited to error handling, refetching on window focus, refetching on connect and background fetching while fetching data with useQuery hook provided by react-query.
As ealier highlighted, useQuery hook takes quite a number of options, most of them being optional and returns an object with quite a number of properties, most of which are also optional but pick their react-query defaults if not specified. Refer to this page from the official TanStack Query documentation for background reading about all the options taken as arguments and the object properties returned by this hook.
In this article, we diagonise the most commonly passed arguments and object properties with a practical use case.

Error Handling

Handling errors is an integral part of working with data from asynchronous api calls, especially data fetching. We have to face it: Not all requests will be successful, and not all Promises will be fulfilled. This indicates that, not thinking about how we are going to handle our errors might negatively affect user experience of our applications.
Important points to note concerning error handling in react-query:

React query needs a rejected promise to handle errors for us correctly. Fortunately, this is what we get while working with axios(which we are using in our case). If you're using the fetch API or some other library that doesn't give a rejected Promise when an error is encountered during data fetching, I will recommend some background reading from the official react-query docs on this concept.
By default, react-query will try to refetch 3 times before it returns an error. In this illustration, I intentionally altered the api endpoint to fetch from a non-existent url and you will observe that there are three retries before the error is returned

In the basic example above, we handled error situations by checking for the isError boolean flag, which is derived from the status enum provided by react query and in our basic usage, it worked for us, but has some limitations in some scenarios. To mention but afew; it doesnot handle background errors(errors arising from background fetches) very well and also, it introduces more boilerplate code if you have to do this for every component that wants to use a query. The question in mind right now is; how do we go about this in react-query?

Before, lets first take a look at what react encrouges us to do regarding this error handling.
To catch run time errors and display a fallback ui, react has a concept called error boundaries which involves wrapping our components in Error Boundaries at any granularity we want, so that the rest of the UI will be unaffected by that error. This is nice but doesn't catch asynchronous errors. This is because, these errors do not occur during rendering.

To use error boundaries in react-query, the library internally catches the error for you and re-throws it in the next render cycle so that the error boundary can pick it up.
Pretty genius, right! and all we have to do is to add useErrorBoundary flag to our query and set it to true.

const usersObject = useQuery('users', getUsers, {
    useErrorBoundary: true
  })

Additional tip: As you can see, destructing the object returned by react query is optional and we can possibly access properties of this object for example data by writing usersObject.data and all others in a similar way as shown:

const usersObject = useQuery("users", getUsers, { useErrorBoundary: true })

if(usersObject.isLoading){
 return <div>Loading...</div>
}

if(usersObject.data){
 return (
    {data.map((user, index) => (
        <p>{user.name}</p>
    ))}
)
}

You can as well possibly customize which errors should go towards an error boundary and which errors should be handled locally. This can be achieved by specifying a function to useErrorBoundary. Lets take an example of a common situation where all server errors, 5xx are handled by error boundaries and other errors are handled locally:

const usersObject = useQuery('users', fetchUsers, {
  useErrorBoundary: error => error.response?.status >= 500
})

This concept also works for mutations using useMutation hook and is quite helpful while doing form submissions. We shall diagonise this concept in the coming articles.

We can as well show error notifications to the user to create a better user experience. Lets use an imperative api, react-hot-toast to understand this concept.
To install this library, run the following command:

npm install react-hot-toast

react-query optionally takes another callback function: onError to help us customise error handing the more as illustrated bellow:

import toast from 'react-hot-toast'
// Custom error handling function
 const handleError = (error: unknown, query: Query) => {
 // Do anything with the error here e.g conditionally render modals with error.message as body to improve your app's user experience
    toast.error(`Something went wrong: ${err.message}`)
    console.log(`An error occured, ${err.message}`)
    return<p>Error..</p>
  }

  const { data, error, isError, isLoading } = useQuery('users', getUsers, {
    onError: handleError,
    useErrorBoundary: error => error.response?.status >= 500
  })

The onError function takes two arguments that is the error and the query itself so the function has access to the query's details including the queryKey just in case need for such information arises inside the function body, it's available at your disposal.
This function is called if an error occurs while executing the query to fetch a list of users from the API. The error object is passed as an argument to the onError function, which can be used to handle the error in some way, such as logging it in the console or displaying an error message to the user.

This is nice, but every observer will get to know that our fetch failed. In some scenarios, we may need to only notify the user once that our underlying fetch failed. In this this case, we will utilise react-query's *global callbacks.
When we created a new QueryClient that we passed in the QueryClientProvider in the entry file of our app, we implicitly created a QueryCache.
Global callbacks need to be provided when create a QueryCache. The implicit creation of the QueryCache can be customized too.

// main.jsx
const queryClient = new QueryClient({
  queryCache: new QueryCache({
    onError: error => toast.error(`Something went wrong ${error.message}`)
  })
})

Phew! we nailed it, the user will only see an error toast once for each query and this is exactly what we intended to achieve.
This is the best place to track errors as it will only run once per API call and won't be overidden including defaultOptions.

Wrapping up this section, we have discussed the three ways to track and monitor errors in react query that is;

The error property returned in the object by useQuery hook
Using error boundaries provided by react by setting useErrorBoundary to true in the object passed as a parameter to useQuery hook to enable react-query re-throw the catched error in the next render cycle for the Error Boundary to pick it.
Using the onError callback on the query itself or on the global QueryCache.

Refetching on window focus and on reconnect

By default, react-query will refetch new data in the background if there is some when you focus on the window or reconnect internet. This default behavior can however be customized by specifying refetchOnWindowFocus and refetchOnReconnect options in the object passed as an argument.

const { isLoading, isError, error, data } = useQuery({ 
    queryKey: 'users', 
    queryFn: getUsers, 
    refetchOnWindowFocus: false, //Defaults to true
    refetchOnReconnect: false //Defaults to true
})

These two options can as well be disabled globally in the entry point of the application.

import React from 'react'
import ReactDOM from 'react-dom/client'
import { QueryClient, QueryClientProvider } from 'react-query'
import App from './App'
import './index.css'
const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      refetchOnWindowFocus: true, 
      refetchOnReconnect: true 
    }
  }
}) 
ReactDOM.createRoot(document.getElementById('root')).render(
  <QueryClientProvider client={queryClient}>
    <App />
  </QueryClientProvider>
)

Additional tip: refetchOnWindowFocus should be disabled on all the queries involving file uploads because react-query will try to refetch in the background upon closing the file upload modal. This abstracts the onChange function of this input whose type is file which may possibly create unexpected behaviour. However, in some situations, you might need to refetchOnWindowFocus on particular query except for this scenario of file upload. In this case you can use react-query's focusManager and customise it to your needs or reference this issue on github for background reading. You can find github repo for this article here

Conclusion

In conclusion, React Query is a powerful tool for managing server state and is configurable to each observer instance of a query with knobs and options to fit every use-case. This makes it exceptionally effective.
Thanks for reading.
Leave a comment below.
Lets Connect:
Github LinkedIn twitter BuyMeCoffee