DEV Community: Rishabh Mishra

NodeJS Microservice with gRPC and TypeScript

Rishabh Mishra — Tue, 18 Apr 2023 20:47:31 +0000

It is very difficult for an engineering team to manage a very large application. Thus instead of building a large monolith application, companies are writing multiple microservices which can be managed by multiple small teams. These microservices are loosely coupled to each other and can be developed, deployed, and scaled independently. However, in this microservice architecture, the microservices often need to communicate with each other via a lightweight communication protocol. For this inter-process communication, gRPC is the popular choice of developers.

gRPC is an open-source framework for Remote Procedure Calls. It is developed by Google. It is a lightweight, high-performant, and type-safe framework for inter-service communication. It uses http/2 for transport and protocol buffers for serializing structured data.

In this blog, we will write microservices with nodejs and use gRPC for communication between them.

Repo Setup

We will create a monorepo where we will keep all our microservices and protobuf files. We will have a server called product-service and one client to test the server.

Let's create our project repo and change the current working directory to it.

mkdir nodejs-microservices
cd nodejs-microservices

We will create an empty node project inside it. Here we will pass the -y flag to accept all the defaults.
We will use npm workspace for local shared dependencies resolution.

npm init -y

We will create 2 directories packages, and services. In the packages directory, we will keep all our reusable libraries like generated Typescript files from protobuf etc. And in the services directory, we will keep all our microservices.

mkdir packages services

Let's create the protos package first.

NOTE: This is not an ideal way to use protobuf files, as the version is not locked in the services. So any changes we make in the protos package will be auto-reflected in services. It is better to keep protobuf files in a separate repo and publish generated Typescript files to a registry. And use that in services as an npm package with a fixed version

npm init -w packages/protos --scope=@nodejs-microservices -y

In the protos package, we will create 2 directories src and dist. In the src directory, we will keep our protobuf files and in the dist directory, we will keep the generated Typescript files from the protobuf files.

cd packages/protos
mkdir src dist

After this, the project structure should look something like this.

.
├── package.json
├── packages
│   └── protos
│       ├── dist
│       ├── package.json
│       └── src
└── services

Now we will create protobuf files in the src directory. This protos package will contain multiple protobuf files. So, we will create subdirectories inside the src directory as per the proto package name.

Create Protobuf file

For the product-service, we will create product.proto file inside the product subdirectory. We will define the Product message here. Read more about the protobuf message here

src/product/product.proto

syntax = "proto3";

package product;

import "google/protobuf/timestamp.proto";

message Product {
  int32 id = 1;
  string name = 2;
  string description = 3;
  string image = 4;
  repeated string tags = 5;
  google.protobuf.Timestamp created_at = 6;
  google.protobuf.Timestamp updated_at = 7;
}

We will create a service called ProductService and define the schemas of Request and Response for the rpc methods. The ProductService will have 3 rpc methods CreateProduct, GetProduct, and ListProducts.

syntax = "proto3";

package product;

import "google/protobuf/timestamp.proto";

message Product {
  int32 id = 1;
  string name = 2;
  string description = 3;
  string image = 4;
  repeated string tags = 5;
  google.protobuf.Timestamp created_at = 6;
  google.protobuf.Timestamp updated_at = 7;
}

message CreateProductRequest {
  string name = 1;
  string description = 2;
  string image = 3;
  repeated string tags = 4;
}

message CreateProductResponse {
  Product product = 1;
}

message GetProductRequest {
  int32 id = 1;
}

message GetProductResponse {
  Product product = 1;
}

message ListProductsRequest {

}

message ListProductsResponse {
  repeated Product products = 1;
}

service ProductService {
  rpc CreateProduct(CreateProductRequest) returns (CreateProductResponse);
  rpc GetProduct(GetProductRequest) returns (GetProductResponse);
  rpc ListProducts(ListProductsRequest) returns (ListProductsResponse);
}

Generating Typescript files from Protobuf

To Generate the TypeScript files, we need to first install the protobuf compiler. The protobuf compiler will be installed globally in the system. Here I am installing in macos via brew. It will different for other operating systems. check the github repo for installation instructions.

brew install protobuf

We also need to install the ts-proto npm package to use it as a plugin for protoc. We will create a build script to generate typescript files from protbuf. In the build script, we will set dist as the output directory and src as the source directory for protobuf files.

npm i -D ts-proto typescript
touch build.sh

build.sh

#!/bin/bash
protoc --plugin=$(npm root)/.bin/protoc-gen-ts_proto \
 --ts_proto_out=dist \
 --ts_proto_opt=outputServices=grpc-js \
 --ts_proto_opt=esModuleInterop=true \
 -I=src/ src/**/*.proto

In the build script, we are using $(npm root) instead of ./node_modules because due to the npm workspace, the binary is installed in the root node_modules. Read more about npm root here.

We will make the build script executable and run it.

chmod +x build.sh
./build.sh

After executing the build script, the typescript files will be generated in the dist folder, and folder structures will look like this.

.
├── dist
│   ├── google
│   │   └── protobuf
│   │       └── timestamp.ts
│   └── product
│       └── product.ts
├── package.json
└── src
    └── product
        └── product.proto

Create gRPC Server

Let's create the product-service. The product-service will be a gRPC server that will store the product data in the postgres database. We will use typeorm in this service.

From the root of the repo, we will run the npm init command to scaffold the package.json file.

npm init -w services/product-service --scope=@nodejs-microservices -y

Then change the current working directory to product-service. In the product-service directory, we will create src, and dist subdirectories similar to the protos package.

cd services/product-service
mkdir src dist

Let's install the dependencies and dev dependencies for the server.

npm i -D typescript ts-node
npm i @grpc/grpc-js typeorm reflect-metadata pg

We will create the tsconfig.json file inside the server directory next to the package.json file.

tsconfig.json

{
  "compilerOptions": {
    "target": "es2016",
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "module": "commonjs",
    "rootDir": "./src",
    "outDir": "./dist",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
  }
}

Let's create a Model for the products table. The id is the primary key of the table and will be auto-generated with an auto-increment value. The createdAt & updatedAt fields will be auto-generated too and set during insert and update operations.

src/models/product.ts

import {
  Entity,
  PrimaryGeneratedColumn,
  Column,
  CreateDateColumn,
  UpdateDateColumn,
} from "typeorm";

@Entity()
export class Product {
  @PrimaryGeneratedColumn()
  id!: number;

  @Column()
  name!: string;

  @Column()
  description!: string;

  @Column()
  image!: string;

  @Column("text", { array: true })
  tags!: string[];

  @CreateDateColumn()
  createdAt!: Date;

  @UpdateDateColumn()
  updatedAt!: Date;
}

Let's configure the database connection. We will create db/index.ts file inside the src directory.
We have to import the Product model here and add it to the entities property.

src/db/index.ts

import { DataSource } from "typeorm";
import { Product } from "../models/product";

const dataSource = new DataSource({
  type: "postgres",
  host: process.env.POSTGRES_HOST || "localhost",
  port: Number(process.env.POSTGRES_PORT) || 5432,
  username: process.env.POSTGRES_USER || "postgres",
  password: process.env.POSTGRES_PASSWORD || "postgres",
  database: process.env.POSTGRES_DB || "postgres",
  entities: [Product],
  logging: true,
  synchronize: true,
});

export default dataSource;

Let's setup the gRPC server. We will create a main.ts file inside the src directory. Here we will import the dataSource. After initializing the dataSource, we will get the db instance. We will start the server only after the database is connected.

src/main.ts

import "reflect-metadata";
import dataSource from "./db";
import { Server, ServerCredentials } from "@grpc/grpc-js";

const server = new Server();

const HOST = process.env.HOST || "0.0.0.0";
const PORT = Number(process.env.PORT) || 50051;

const address = `${HOST}:${PORT}`;

dataSource
  .initialize()
  .then((db) => {
    server.bindAsync(
      address,
      ServerCredentials.createInsecure(),
      (error, port) => {
        if (error) {
          throw error;
        }
        console.log("server is running on", port);
        server.start();
      }
    );
  })
  .catch((error) => console.log(error));

Update the scripts in the package.json file to run and build the server.

package.json

{
  ...
  "scripts": {
    "dev": "ts-node src/main.ts",
    "build": "tsc",
    "start": "node dist/main.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  ...
}

We can test the server and database connection by running it. It should create the products table in the database.

npm run dev

or

npm run build
npm start

Let's start implementing the methods for the server. First, we need to install the protos package. Then we create a server.ts file.

npm i @nodejs-microservices/protos
touch src/server.ts

We will create a getProductServer function in the server.ts file which will return an object with createProduct, getProduct, and listProducts methods. We are using this server object instead of creating a class due to the server signature issue

We will define the business logic later, right now all the methods will return the status UNIMPLEMENTED in the response. We are doing this to test the server.

src/server.ts

import { sendUnaryData, ServerUnaryCall, status } from "@grpc/grpc-js";
import {
  CreateProductRequest,
  CreateProductResponse,
  GetProductRequest,
  GetProductResponse,
  ListProductsRequest,
  ListProductsResponse,
  ProductServiceServer,
} from "@nodejs-microservices/protos/dist/product/product";
import { DataSource } from "typeorm";

export function getProductServer(db: DataSource): ProductServiceServer {
  async function createProduct(
    call: ServerUnaryCall<CreateProductRequest, CreateProductResponse>,
    callback: sendUnaryData<CreateProductResponse>
  ) {
    callback({ code: status.UNIMPLEMENTED }, null);
  }
  async function getProduct(
    call: ServerUnaryCall<GetProductRequest, GetProductResponse>,
    callback: sendUnaryData<GetProductResponse>
  ) {
    callback({ code: status.UNIMPLEMENTED }, null);
  }
  async function listProducts(
    call: ServerUnaryCall<ListProductsRequest, ListProductsResponse>,
    callback: sendUnaryData<ListProductsResponse>
  ) {
    callback({ code: status.UNIMPLEMENTED }, null);
  }

  return {
    createProduct,
    getProduct,
    listProducts,
  };
}

We will import the getProductServer function and ProductService definition in the main.ts file and then add the product service to the gRPC server.

src/main.ts

import "reflect-metadata";
import dataSource from "./db";
import { Server, ServerCredentials } from "@grpc/grpc-js";
import { getProductServer } from "./server";
import { ProductServiceService } from "@nodejs-microservices/protos/dist/product/product";

const server = new Server();

const HOST = process.env.HOST || "0.0.0.0";
const PORT = Number(process.env.PORT) || 50051;

const address = `${HOST}:${PORT}`;

dataSource
  .initialize()
  .then((db) => {
    server.addService(ProductServiceService, getProductServer(db));
    server.bindAsync(
      address,
      ServerCredentials.createInsecure(),
      (error, port) => {
        if (error) {
          throw error;
        }
        console.log("server is running on", port);
        server.start();
      }
    );
  })
  .catch((error) => console.log(error));

After running the server, We will test the service methods with grpcurl. Read more about grpcurl here

Let's test the CreateProduct method, we need to pass the proto file to the grpcurl command. I am calling the grpcurl from the root of the monorepo, thus the path to the proto files are relatives to the root. After that, We need to pass the request body as a JSON string. Then pass the address to the server and service method name.

grpcurl -import-path packages/protos/src -proto product/product.proto -d '{ "name": "product 1", "description": "foo bar", "image": "", "tags": ["tag 1"]}' -plaintext localhost:50051 product.ProductService.CreateProduct

ERROR:
  Code: Unimplemented
  Message: Unknown Error

After running the grpcurl call, we should get an error from the server. Here we are getting the status code Unimplemented as we expected.

Let's add the controller for product model. We will create product.controller.ts file inside src/controllers directory. In this file we will write the logic for database operations. We create the createProduct function, the function accepts the database instance and request body as the arguments.

We are defining the interface for createProductReq here, We can also use CreateProductRequest from the protos package instead of this. It is just my personal preference to define the interface inside the server as we are mapping the request fields with Product model fields. So, If in the future there is a change in CreateProductRequest then we can easily transform CreateProductRequest to createProductReq

src/controllers/product.controller.ts

import { DataSource } from "typeorm";
import { Product } from "../models/product";

interface createProductReq {
  name: string;
  description: string;
  image: string;
  tags: string[];
}

export const createProduct = async (
  db: DataSource,
  req: createProductReq
): Promise<Product> => {
  const productRepository = db.getRepository(Product);
  const product = new Product();
  product.name = req.name;
  product.description = req.description;
  product.image = req.image;
  product.tags = req.tags;
  return productRepository.save(product);
};

We then import the ProductController in the server.ts file and update the createProduct method. We will transform the result from the database insert operation to the Product message by passing it to the fromJSON method. We will wrap this logic inside a try-catch block and return the status code INTERNAL in case of an error.

src/server.ts

import { sendUnaryData, ServerUnaryCall, status } from "@grpc/grpc-js";
import {
  CreateProductRequest,
  CreateProductResponse,
  GetProductRequest,
  GetProductResponse,
  ListProductsRequest,
  ListProductsResponse,
  Product,
  ProductServiceServer,
} from "@nodejs-microservices/protos/dist/product/product";
import { DataSource } from "typeorm";
import * as ProductController from "./controllers/product.controller";

export function getProductServer(db: DataSource): ProductServiceServer {
  async function createProduct(
    call: ServerUnaryCall<CreateProductRequest, CreateProductResponse>,
    callback: sendUnaryData<CreateProductResponse>
  ) {
    try {
      const product = await ProductController.createProduct(db, call.request);
      const productPB = Product.fromJSON(product);
      const response: CreateProductResponse = {
        product: productPB,
      };
      callback(null, response);
    } catch (err) {
      callback({ code: status.INTERNAL }, null);
      console.error(err);
    }
  }
  async function getProduct(
    call: ServerUnaryCall<GetProductRequest, GetProductResponse>,
    callback: sendUnaryData<GetProductResponse>
  ) {
    callback({ code: status.UNIMPLEMENTED }, null);
  }
  async function listProducts(
    call: ServerUnaryCall<ListProductsRequest, ListProductsResponse>,
    callback: sendUnaryData<ListProductsResponse>
  ) {
    callback({ code: status.UNIMPLEMENTED }, null);
  }

  return {
    createProduct,
    getProduct,
    listProducts,
  };
}

Let's test the CreateProduct method again with grpcurl. We should get the response back from the server now instead of the status UNIMPLEMENTED error.

grpcurl -import-path packages/protos/src -proto product/product.proto -d '{ "name": "product 1", "description": "foo bar", "image": "", "tags": ["tag 1"]}' -plaintext localhost:50051 product.ProductService.CreateProduct

{
  "product": {
    "id": 1,
    "name": "product 1",
    "description": "foo bar",
    "tags": [
      "tag 1"
    ],
    "createdAt": "2023-03-29T00:07:23.426Z",
    "updatedAt": "2023-03-29T00:07:23.426Z"
  }
}

Let's add the listProducts and getProduct functions to the product controller. Both functions will accept database instance as an argument similar to the createProduct function

src/controllers/product.controller.ts

import { DataSource } from "typeorm";
import { Product } from "../models/product";

interface createProductReq {
  name: string;
  description: string;
  image: string;
  tags: string[];
}

export const createProduct = async (
  db: DataSource,
  req: createProductReq
): Promise<Product> => {
  const productRepository = db.getRepository(Product);
  const product = new Product();
  product.name = req.name;
  product.description = req.description;
  product.image = req.image;
  product.tags = req.tags;
  return productRepository.save(product);
};

export const listProducts = async (db: DataSource): Promise<Product[]> => {
  const productRepository = db.getRepository(Product);
  return productRepository.find();
};

export const getProduct = async (
  db: DataSource,
  id: number
): Promise<Product | null> => {
  const productRepository = db.getRepository(Product);
  return productRepository.findOneBy({ id: id });
};

We will update the getProduct and listProducts methods in the server.ts file.

In the getProduct method, we will check if the database returns the product, then only we return the response. Otherwise, we will return the status code NOT_FOUND with the message.

In the listProducts method, we will get an array of product objects from the database, we will run the map function to transform each of them into the Product message.

src/server.ts

import { sendUnaryData, ServerUnaryCall, status } from "@grpc/grpc-js";
import {
  CreateProductRequest,
  CreateProductResponse,
  GetProductRequest,
  GetProductResponse,
  ListProductsRequest,
  ListProductsResponse,
  Product,
  ProductServiceServer,
} from "@nodejs-microservices/protos/dist/product/product";
import { DataSource } from "typeorm";
import * as ProductController from "./controllers/product.controller";

export function getProductServer(db: DataSource): ProductServiceServer {
  async function createProduct(
    call: ServerUnaryCall<CreateProductRequest, CreateProductResponse>,
    callback: sendUnaryData<CreateProductResponse>
  ) {
    try {
      const product = await ProductController.createProduct(db, call.request);
      const productPB = Product.fromJSON(product);
      const response: CreateProductResponse = {
        product: productPB,
      };
      callback(null, response);
    } catch (err) {
      callback({ code: status.INTERNAL }, null);
      console.error(err);
    }
  }
  async function getProduct(
    call: ServerUnaryCall<GetProductRequest, GetProductResponse>,
    callback: sendUnaryData<GetProductResponse>
  ) {
    try {
      const product = await ProductController.getProduct(db, call.request.id);
      if (product) {
        const productPB = Product.fromJSON(product);
        const response: GetProductResponse = {
          product: productPB,
        };
        callback(null, response);
      } else {
        callback(
          {
            code: status.NOT_FOUND,
            message: `Product ${call.request.id} not found`,
          },
          null
        );
      }
    } catch (err) {
      callback({ code: status.INTERNAL }, null);
      console.error(err);
    }
  }
  async function listProducts(
    call: ServerUnaryCall<ListProductsRequest, ListProductsResponse>,
    callback: sendUnaryData<ListProductsResponse>
  ) {
    try {
      const products = await ProductController.listProducts(db);
      const productsPB = products.map(Product.fromJSON);
      const response: ListProductsResponse = {
        products: productsPB,
      };
      callback(null, response);
    } catch (err) {
      callback({ code: status.INTERNAL }, null);
      console.error(err);
    }
  }

  return {
    createProduct,
    getProduct,
    listProducts,
  };
}

We will test the ListProducts and GetProduct methods with grpcurl. They shouldn't be returning an error with the status UNIMPLEMENTED.

grpcurl -import-path packages/protos/src -proto product/product.proto -plaintext localhost:50051 product.ProductService.ListProducts

{
  "products": [
    {
      "id": 1,
      "name": "product 1",
      "description": "foo bar",
      "tags": [
        "tag 1"
      ],
      "createdAt": "2023-03-29T00:07:23.426Z",
      "updatedAt": "2023-03-29T00:07:23.426Z"
    }
  ]
}

grpcurl -import-path packages/protos/src -proto product/product.proto -d '{"id": 1}' -plaintext localhost:50051 product.ProductService.GetProduct

{
  "product": {
    "id": 1,
    "name": "product 1",
    "description": "foo bar",
    "tags": [
      "tag 1"
    ],
    "createdAt": "2023-03-29T00:07:23.426Z",
    "updatedAt": "2023-03-29T00:07:23.426Z"
  }
}

Create gRPC Client

Let's create a client to test the product-service. We will create a simple program that will connect to the server, make the call and log the response in the console.

From the root of the repo, we will run the npm init command to scaffold the package.json file and change the current working directory to the test-client. Inside the test-client we will create the src and dist directories.

npm init -w services/test-client --scope=@nodejs-microservices -y
cd services/test-client/
mkdir src dist

We will install the dependencies and dev dependencies for the client. We will install the @nodejs-microservices/protos package to get the client stub.

npm i -D typescript ts-node
npm i @grpc/grpc-js @nodejs-microservices/protos

We will create the tsconfig.json file inside test-client, the tsconfig.json is the same as in the product-service

tsconfig.json

{
  "compilerOptions": {
    "target": "es2016",
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "module": "commonjs",
    "rootDir": "./src",
    "outDir": "./dist",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
  }
}

We will update the scripts in the package.json file to run and build the client.

package.json

{
  ...
  "scripts": {
    "dev": "ts-node src/main.ts",
    "build": "tsc",
    "start": "node dist/main.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  ...
}

Let's create the main.ts file inside the src directory. We will create a main function and test each service method separately.

First, We need to import the credentials object from the grpc-js package and ProductServiceClient from our protos package. Then we will create a gRPC client. While creating the client instance, we need to pass the server url and authentication credentials. The product-service doesn't have any authentication thus we are passing credentials.createInsecure() as authentication credentials.

After creating the client instance, we will call the service method name. Here we are calling the createProduct method on the client. We will create the request object for the call and then call the method with the request object and callback function.

src/main.ts

import { credentials } from "@grpc/grpc-js";
import {
  CreateProductRequest,
  ProductServiceClient,
} from "@nodejs-microservices/protos/dist/product/product";

const PRODUCT_SERVICE_URL = process.env.USER_SERVICE_URL || "0.0.0.0:50051";

function main() {
  const client = new ProductServiceClient(
    PRODUCT_SERVICE_URL,
    credentials.createInsecure()
  );

  const req: CreateProductRequest = {
    name: "test product",
    description: "foo bar",
    image: "https://example.com/image",
    tags: ["tag-1"],
  };

  client.createProduct(req, (err, resp) => {
    if (err) {
      console.error(err);
    } else {
      console.log("Response :", resp);
    }
  });
}

main();

We can similarly call and test the getProduct and listProducts methods. I am skipping that part in this blog.

All the source code for this blog is available on GitHub.

Dockerizing a React.js app

Rishabh Mishra — Wed, 31 Mar 2021 09:48:41 +0000

In this post, we will dockerize a React application. We will setup Docker with auto-reloading for development setup and optimized multistage docker build for production deployment. We can even dockerize Next.js or Gatsby Static builds with the same process.

There are many advantages of using Docker. Right now Docker is the defacto standard of containerizing applications. It is easy to build, package, share, and ship applications with Docker. As Docker images are portable it is easy to deploy the application to any modern cloud provider.

Initialize React application

Let's start by creating a React application. You can use your existing React project. For this blog post, I am creating a new React application with create-react-app.

$ npx create-react-app react-docker

Here I created a new React app named react-docker. Let's verify the app by running the npm start command inside the project directory.

$ npm start

It will start the app and we can verify by visiting http://localhost:3000 in the browser. The application should be running.

Write Dockerfile.

Now let's create a Docker image for the React application. We need a Dockerfile to create Docker images. Let's create a file named Dockerfile in the root directory of the React application.

Dockerfile

FROM node:14-alpine

WORKDIR /app

COPY package.json ./

COPY yarn.lock ./

RUN yarn install --frozen-lockfile

COPY . .

EXPOSE 3000

CMD ["npm", "start"]

Here We are using node v14 alpine as the base image to build and run the application. We are running the npm start command the default command which will run the React development server. We also need the .dockerignore file which will prevent node_modules and other unwanted files to get copied in the Docker image.

.dockerignore

node_modules
npm-debug.log
Dockerfile
.dockerignore

Let's build the Docker image from the Dockerfile by running the docker build command. Here we are tagging it with the name react-docker.

$ docker build -t react-docker .

After building the docker images we can verify the image by running the docker images command. We can see an image with the name react-docker is created.

$ docker images

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
react-docker        latest              6b782301e271        2 minutes ago       438MB

We can run the Docker image with the docker run command. Here we are mapping the system port 3000 to Docker container port 3000. We can verify if the application is running or not by visiting http://localhost:3000.

$ docker run -p 3000:3000 react-docker

Add Docker Compose

The React application is working fine inside the docker container, but we need to build and run the docker container every time we make any changes in the source files as auto reloading is not working with this setup. We need to mount the local src folder to the docker container src folder, so every time we make any change inside the src folder, it will auto-reload the page on any code changes.

We will add the docker-compose.yml file to the root of the project to mount the local src folder to the /app/src folder of the container.

docker-compose.yml

version: "3"

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    volumes:
      - ./src:/app/src
    ports:
      - "3000:3000"

Run the docker-compose up command to start the container. The React development server will be running inside the container and will be watching the src folder.

$ docker-compose up

We can't ship this docker image to the production as it is not optimized and runs a development server inside. Let's rename the Dockerfile as Dockerfile.dev and update the docker-compose.yaml file to use the Dockerfile.dev file. We will use docker-compose and the Dockerfile.dev file only for development. We will create a new Dockerfile for the production build.

$ mv Dockerfile Dockerfile.dev

docker-compose.yml

version: "3"

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - ./src:/app/src
    ports:
      - "8000:8000"

Add Production Dockerfile

Let's first verify the React application production config by running the yarn build command to build the app for production.

$ yarn build

We can verify the production build by running it locally. I am using serve to serve the build folder files.

$ npx serve -s build

After verifying the server locally we can create a new Dockerfile for the production build. We will be using multi-stage builds to create the docker image. One Stage to build the production files and the other stage to serve them.

Dockerfile

FROM node:14-alpine AS builder
WORKDIR /app
COPY package.json ./
COPY yarn.lock ./
RUN yarn install --frozen-lockfile
COPY . .
RUN yarn build

FROM nginx:1.19-alpine AS server
COPY --from=builder ./app/build /usr/share/nginx/html

The builder stage is nearly the same as the previous Dockerfile. Instead of running the npm start command here, we are running the yarn build command to build the production files.

We will use Nginx to serve the files. It will create a very lightweight image. From the builder stage, we need to copy the files of the build folder to the /usr/share/nginx/html folder. Nginx docker image uses this folder to serve the contents. Nginx docker image will use the port 80 to serve the files and auto expose that port.

Let's build the image again by running the docker build command and verify if the image is built or not by running the docker images command.

$ docker build -t react-docker .

$ docker images

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
react-docker        latest              5f885aeca09e        7 seconds ago       23.1MB

The size of the production docker image will be very less in comparison to the development one. Let's run the docker image with the docker run command. Here we are mapping the host 3000 port with the container's port 80

docker run -p 3000:80 react-docker

The application should be running fine on http://localhost:3000. Now let's verify if the client-side routing is working fine or not. For that, we need to install the react-router-dom to the application.

$ yarn add react-router-dom

We also need to add a few routes and links to verify. I just copied the example from the react-router website to test.

import React from "react";
import { BrowserRouter as Router, Switch, Route, Link } from "react-router-dom";

export default function App() {
  return (
    <Router>
      <div>
        <nav>
          <ul>
            <li>
              <Link to="/">Home</Link>
            </li>
            <li>
              <Link to="/about">About</Link>
            </li>
            <li>
              <Link to="/users">Users</Link>
            </li>
          </ul>
        </nav>
        <Switch>
          <Route path="/about">
            <About />
          </Route>
          <Route path="/users">
            <Users />
          </Route>
          <Route path="/">
            <Home />
          </Route>
        </Switch>
      </div>
    </Router>
  );
}

function Home() {
  return <h2>Home</h2>;
}

function About() {
  return <h2>About</h2>;
}

function Users() {
  return <h2>Users</h2>;
}

Let's verify the local setup by running the development server and visit the web page and click on every link and refresh the pages.

$ npm start

The application should be working fine on the local development server. Now try the same thing with docker-compose. First, we need to build the image again as auto-reload works only with the src folder as we only mount that. For changes outside the src folder, we need to build the image again with the docker-compose build command.

$ docker-compose build
$ docker-compose up

Now Let's try the same thing with the production docker build. First, we need to build the docker image and run the image again.

docker build -t react-docker .
docker run -p 3000:80 react-docker

Accessing the pages other than the index directly should throw a 404 error. The React application here is a single-page application. Thus the routing is happing on the client-side with JavaScript and when we hit any route it first hits the Nginx server and tries to find the file there and when it was unable to find the fine it throws the 404 error.

We need to pass a custom Nginx configuration to the docker image. We will create an etc folder in the project's root directory and create an nginx.conf file there.

etc/nginx.conf

server {
    listen   80;
    listen   [::]:80 default ipv6only=on;

    root /usr/share/nginx/html;
    index index.html;

    server_tokens  off;
    server_name _;

    gzip on;
    gzip_disable "msie6";

    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_min_length 0;
    gzip_types text/plain application/javascript text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/vnd.ms-fontobject application/x-font-ttf font/opentype;

    location / {
        try_files $uri /index.html;
    }
}

Here we are configuring Nginx to fallback to /index.html if it is unable to find the route. We also enable gzip compression for the contents.

We need to copy the custom Nginx configuration file to the /etc/nginx/conf.d folder. Ngnix will auto-read all the configurations from that folder.

FROM node:14-alpine AS builder
WORKDIR /app
COPY package.json ./
COPY yarn.lock ./
RUN yarn install --frozen-lockfile
COPY . .
RUN yarn build

FROM nginx:1.19-alpine AS server
COPY ./etc/nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=builder ./app/build /usr/share/nginx/html

After copying the custom Nginx configuration file we need to build and run the docker image again.

$ docker build -t react-docker .
$ docker run -p 3000:80 react-docker

Visiting all the routes and refreshing the pages should work fine.

All the source code for this tutorial is available on GitHub.

For dockerizing node backend app please read the other blog post

Building REST API with Express, TypeScript - Part 2: Docker Setup

Rishabh Mishra — Sun, 18 Oct 2020 00:00:00 +0000

In the previous post, we build a REST API server with Express and TypeScript. In this post, we will dockerize the sever.

Why Docker.

Docker helps organizations to ship and develop applications better and faster. It will be easy to set up the development environment on any new machine with docker as it abstracts out lots of complexity of setting up dependencies and environment. Docker also isolates the project from other projects in the same machine so the developer can run multiple projects without having any conflict with the required dependencies.

Docker makes it easy to configure and setup dependencies and environments for the application. As most of the companies have dedicated teams to do setup and manage infrastructure, Docker gives more power to developers to configure without depending on other teams to do the setup.

Write Dockerfile.

To Dockerize the server, we need to create a Dockerfile. A Dockerfile is just a list of instructions to create a docker image. Read more about Dockerfile here

Each line in the Dockerfile is a command and create a new image layer of its own. Docker caches the images during the build, so every rebuild will only create the new layer which got changed from the last build. Here the order of commands is very significant as it helps to reduce the build time.

Let's start writing Dockerfile for the server. Here we are taking node:12 as the base image for the server docker image. Explore dockerhub for more node image version. Here we are copying the package.json and doing npm install first, then copying the other files. Docker will cache the images of these two steps during the build and reuse them later as they change less frequently. Here we will be running the development server with the docker image, so we need to give npm run dev as the executing command.

Dockerfile

FROM node:12

WORKDIR /app

COPY package*.json ./

RUN npm install

COPY . .

EXPOSE 8000

CMD ["npm", "run", "dev"]

We need to add .dockerignore to tell docker build to ignore some files during the COPY Command.

.dockerignore

node_modules
npm-debug.log

After creating the Dockerfile, we need to run the docker build to create a docker image from the Dockerfile. Here we are naming the docker image as express-ts

docker build -t express-ts .

We can verify the docker image by running the docker images command. Here we can see the name, size, and tag of the docker images.

docker images
REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE
express-ts                        latest              d0ce1e38958b        2 minutes ago       1.11GB

We can run the docker image with the docker run command. Here we can mapping the system port 8000 to docker container port 8000. We can verify if the server is running or not by visiting http://localhost:8000/ping

docker run -p 8000:8000 express-ts

Add Docker Compose

The development server is running fine inside docker, but now we need to run the docker build command every time after making any changes to the source files to update the changes during development because the nodemon inside the docker container cannot watch the src folder on the local machine. We need to mount the local src folder to the docker container folder, so every time we make any change inside the src folder, nodemon restarts the development server inside the docker container.

We will add the docker-compose.yml file to the root of the project to mount the local src folder. Read more about docker-compose here

docker-compose.yml

version: "3"

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    volumes:
      - ./src:/app/src
    ports:
      - "8000:8000"

We need to run the command docker-compose up to start the server. Now the server is running in development mode with auto-restart on code changes. We can verify the server is restarting on code changes by making any code change in the TypeScript files.

docker-compose up

The docker setup for the development server is completed. Let's rename the Dockerfile as Dockerfile.dev and update the docker-compose.yaml file. We will use the Dockerfile for the production image, which we are going to set up in the next section.

mv Dockerfile Dockerfile.dev

docker-compose.yml

version: "3"

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - ./src:/app/src
    ports:
      - "8000:8000"

Add Production Dockerfile

Let's start building a docker image for the production server. We need to create a new Dockerfile and add the following commands. Here after copying the files, we need to build the JavaSript files and execute the npm start command.

FROM node:12

WORKDIR /app

COPY package*.json ./

RUN npm install

COPY . .

RUN npm run build

EXPOSE 8000

CMD ["node", "start"]

After running the docker build command, we can see the docker image is created for the production server.

docker images
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
express-ts                      latest              d0ce1e38958b        2 minutes ago       1.11GB

Here the image size is 1.11GB, which is not optimized. Let's optimize the docker image and reduce the size.

First, instead of taking node:12 as the base image, we will be taking its alpine variant. Alpine Linux is very lightweight. Read more about alpine-docker here.

FROM node:12-alpine

Let's build the docker image with the updated Dockerfile. Here we are tagging the docker image as alpine so we can compare the image size with the previous build.

docker build -t express-ts/alpine .

docker images
REPOSITORY                       TAG                 IMAGE ID            CREATED             SIZE
express-ts                       alpine              2b06fcba880e        46 seconds ago      280MB
express-ts                       latest              d0ce1e38958b        2 minutes ago       1.11GB

After running the docker images command we can see the difference in the sizes of docker images. The docker image is much leaner than the previous build.

There are still some issues with our docker image as development dependencies are there in production build and TypeScript code is there, which is not required while running the server in production. So let's optimize the docker image further with a multi-stage build.

Here we create two stages, one for building the server and the other for running the server. In the builder stage, we generate Javascript code from the Typescript files. Then in the server stage, we copy the generated files from the builder stage to the server stage. In the Server stage, we need only production dependencies, that's why we will pass the --production flag to the npm install command.

FROM node:12-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

FROM node:12-alpine AS server
WORKDIR /app
COPY package* ./
RUN npm install --production
COPY --from=builder ./app/public ./public
COPY --from=builder ./app/build ./build
EXPOSE 8000
CMD ["npm", "start"]

Let's build the docker image with the updated multi-staged Dockerfile. Here we are tagging the docker image as ms so we can compare the image sizes with the previous builds.

docker build -t express-ts/ms .

docker images

REPOSITORY                       TAG                 IMAGE ID            CREATED             SIZE
express-ts                       alpine              2b06fcba880e        46 seconds ago      280MB
express-ts                       latest              d0ce1e38958b        2 minutes ago       1.11GB
express-ts                       ms                  26b67bfe45b0        9 minutes ago       194MB

After running the docker images command we can see the difference in the sizes of docker images. The multi staged image is the leanest among all images.

We have dockerized the development and production version of the Express and TypeScript REST API server.

All the source code for this tutorial is available on GitHub.

Building REST API with Express, TypeScript and Swagger

Rishabh Mishra — Sat, 03 Oct 2020 00:00:00 +0000

I’ve started working with JS in 2017, since then I am writing frontend and backend code with it. It is easy to write web-server with NodeJS and I never found any serious performance issue in using NodeJS. According to Stack Overflow 2020 survey, NodeJS is the most popular technology. I prefer using Express with NodeJS. It is one of the most popular Node.js web application frameworks. There are multiple frameworks, and you can choose whichever you want according to the need.

After working with TypeScript, it became my preferred language of choice between JS and TS. TypeScript is the superset of JavaScript, means all valid JS is valid TypeScript. So it is easy to learn Typescript if you already knew JavaScript. TypeScript is 2nd most loved language according to the Stack Overflow 2020 survey. TypeScript helps you to add static types to the Javascript code. It is very helpful in writing, maintaining, and debugging code.

What you will build

You will build REST API server with Express and TypeScript. It will generate production JavaScript code on build command. It will auto restart server on any code change during development, and it will auto generate OpenAPI documentation with Swagger.

Bootstrap project

Let's create a directory with your preferred application name and set up an empty node project inside it. You can choose to customize package.json or accepts all of the default options by passing -y flag to init command.

mkdir express-typescript
cd express-typescript
npm init -y

Install Typescript as development dependency

npm i -D typescript

Add tsconfig.json in the root of the project directory. Here we define outDir as ./build to put generated JavaScript files. You can put your preferred directory name. You can customize the config file more as per your need. Check TypeScript Handbook for more details.

tsconfig.json

{
  "compilerOptions": {
    "target": "es6",
    "module": "commonjs",
    "outDir": "./build",
    "strict": true,
    "esModuleInterop": true
  }
}

Install Express as dependency and type definitions of node and express as development dependencies.

npm i -S express
npm i -D @types/express @types/node

Write server code

Let's add minimal code to make server up and running. Create a folder src inside the root folder. We will be going to put all the Typescript code inside it. It depends on personal choice. You can keep the code anywhere in the project.

This code will run the express server, listening to port 8000. It will add /ping route, which will reply JSON response on the GET call.

src/index.ts

import express, { Application } from "express";

const PORT = process.env.PORT || 8000;

const app: Application = express();

app.get("/ping", async (_req, res) => {
  res.send({
    message: "pong",
  });
});

app.listen(PORT, () => {
  console.log("Server is running on port", PORT);
});

Let's add the build command. it will transpile the TypeScript code into JavaScript and put the generated code in the output directory as mentioned in tsconfig.json.

package.json

"scripts": {
  "build": "tsc",
}

Now let's build the JavaScript code with the build command.

npm run build

After running the above command we can see the JS code generated in the build folder. Now with node, we can run the server. We can visit http://localhost:8000/ping to see the JSON response.

node build/index.js

Server is running on port 8000

Add development setup

The server is up and running. But still, development is difficult due to building and running the server manually after every code changes. It is better to automate this task. For this, we will use ts-node to run the typescript code directly, so then we don't have to run the typescript compiler during development. And to restart the ts-node on every code change, we will use nodemon which will watch the code and re-run the command on any changes.

Lets add ts-node nodemon as development dependencies in the project.

npm i -D ts-node nodemon

Now add the dev script to package.json, which will run the nodemon command. Add nodemon config to package.json. We can keep the config as a separate file. But I prefer to add it to package.json to keep the root of the project clean. Here we are configuring nodemon to watch all the .ts files inside the src folder and execute ts-node src/index.ts on any code change.

package.json

  "scripts": {
    "build": "tsc",
    "dev": "nodemon",
  },

  "nodemonConfig": {
    "watch": [
      "src"
    ],
    "ext": "ts",
    "exec": "ts-node src/index.ts"
  }

After running the dev command, we can see the nodemon is running. And the server is up and running as well.

npm run dev

[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): src/**/*
[nodemon] watching extensions: ts
[nodemon] starting `ts-node src/index.ts`
Server is running on port 8000

Add middlewares

Let's extend the server by adding some middlewares. We are going to add three middleware to the server. express.json is built-in middleware to parse the request body, express.static is also built-in middleware used to serve the static files, and morgan is used to logs the requests. Let's install them as dependencies and their type definitions as development dependencies in the project.

npm i -S morgan
npm i -D @types/morgan

After installing the middleware, we can use them in the code. We will add them to the server with app.use() function. Here we make the public folder to serve the static files.

src/index.ts

import express, { Application } from "express";
import morgan from "morgan";

const PORT = process.env.PORT || 8000;

const app: Application = express();

app.use(express.json());
app.use(morgan("tiny"));
app.use(express.static("public"));

Now after running the server, open http://localhost:8000/ping in the browser. We can see the request gets logged in the terminal.

Server is running on port 8000
GET /ping 304 - - 2.224 ms

Refactor

Till now the server is one single file. It is okay for small servers, but it is difficult to extend the server if it is one file. So we will create multiple files.

Let's create a controller for the ping request in src/controllers/ping.ts path. Here we add a class called PingController with method getMessage, we define the response interface with a property message as a string.

src/controllers/ping.ts

interface PingResponse {
  message: string;
}

export default class PingController {
  public async getMessage(): Promise<PingResponse> {
    return {
      message: "pong",
    };
  }
}

Now create a sub router in src/routes/index.ts file and move all the routing login there. In the server, we will add this sub router as a middleware.

src/routes/index.ts

import express from "express";
import PingController from "../controllers/ping";

const router = express.Router();

router.get("/ping", async (_req, res) => {
  const controller = new PingController();
  const response = await controller.getMessage();
  return res.send(response);
});

export default router;

src/index.ts

import express, { Application } from "express";
import morgan from "morgan";
import Router from "./routes";

const PORT = process.env.PORT || 8000;

const app: Application = express();

app.use(express.json());
app.use(morgan("tiny"));
app.use(express.static("public"));

app.use(Router);

app.listen(PORT, () => {
  console.log("Server is running on port", PORT);
});

Swagger integration

Let's add OpenAPI documentation with the Swagger. We need to add tsoa to generates a JSON file with OpenAPI Specifications for all the APIs. We also need swagger-ui-express to host the Swagger JSON with Swagger UI.

npm i -S tsoa swagger-ui-express
npm i -D @types/swagger-ui-express concurrently

We need to add support for Decorators in the tsconfig.json file.

tsconfig.json

{
  "compilerOptions": {
    ...
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true
  }
}

We need to create the config file for tsoa. Add tsoa.json at the root of the directory. Add entryFile and outputDirectory in the config. Here we are setting public as the output folder for the generated JSON file.

tsoa.json

{
  "entryFile": "src/index.ts",
  "noImplicitAdditionalProperties": "throw-on-extras",
  "spec": {
    "outputDirectory": "public",
    "specVersion": 3
  }
}

We update the dev and build command to generate Swagger docs. We add tsoa spec to generate Swagger docs. We will be running the swagger command before build and dev command with prebuild and predev Respectively. We add concurrently to the dev command, which will run the nodemon and tsoa spec on parallel. The Swagger docs will get auto-updated on every code change during development.

package.json

  "scripts": {
    "start": "node build/index.js",
    "predev": "npm run swagger",
    "prebuild": "npm run swagger",
    "build": "tsc",
    "dev": "concurrently \"nodemon\" \"nodemon -x tsoa spec\"",
    "swagger": "tsoa spec",
  },

Let's update the server file to serve the Swagger UI. We add swagger-ui-express to serve the Swagger UI for the hosted swagger JSON file.

src/index.ts

import express, { Application, Request, Response } from "express";
import morgan from "morgan";
import swaggerUi from "swagger-ui-express";

import Router from "./routes";

const PORT = process.env.PORT || 8000;

const app: Application = express();

app.use(express.json());
app.use(morgan("tiny"));
app.use(express.static("public"));

app.use(
  "/docs",
  swaggerUi.serve,
  swaggerUi.setup(undefined, {
    swaggerOptions: {
      url: "/swagger.json",
    },
  })
);

app.use(Router);

Now let's update the controller and add decorators to the class and methods to define the path and route for the API documentation. tsoa will pick the return type PingResponse as the response type for the /ping route.

src/controllers/ping.ts

import { Get, Route } from "tsoa";

interface PingResponse {
  message: string;
}

@Route("ping")
export default class PingController {
  @Get("/")
  public async getMessage(): Promise<PingResponse> {
    return {
      message: "pong",
    };
  }
}

After making all the changes and running the server, visit http://localhost:8000/docs/ to access the APIs documentation.

All the source code for this tutorial is available on GitHub.

Additional Resources

Building a Node.js/TypeScript REST API, Part 1: Express.js