Modernizing Express, Heroku tech stack with NestJS and AWS - PoC

TL;DR

We will create a working PoC which proves that migration from Express based app hosted in Heroku to NestJS app hosted in AWS with serverless architecture based on API Gateway, Fargate and Aurora is not so complicated. We will also use the IaC approach based on AWS CDK.

MVP Solution

When I joined moojo as a Head of Engineering September 2022 - beta tests phase was almost done and we were preparing for the official launch of our MVP (Minimum Viable Product) with invoicing and instant payment features.

Backend was a monolithic, Node.js app using Express framework and Postgress database, both hosted on Heroku. Solution was using a functional programming approach, TypeScript and modern, next-generation ORM (Object Relational Mapper) like Prisma, which provided us with type-safety and automated migration.

It was simple approach following KISS (Keep It Simple Stupid) design principle, which was perfect at the time for the MVP solution. This was also a conscious decision the interim tech lead, who started the project and wanted to create unopinioned solution and leave bigger decisions for the successor.

Challenges

This setup was quite good, but when the application started to grow - we noticed a few challenges.

Framework

It was hard to manage dependencies without modern design patterns known as Dependency Injection, OpenAPI/Swagger documentation was often out of sync with actual implementation, freedom of doing things in unopinionated framework was not optimal for less experienced team members (they needed some guidance and documentation).

We could of course improve existing solution, but on the other hand - there are already frameworks, which provide those things out-of-the-box, so it would be like reinventing the wheel. NestJS seemed like a perfect fit for us, question was - how hard will it be to migrate.

Cloud

Heroku's advantage is that it's super easy to setup and deploy your app. A few clicks in the Heroku Dashboard is enough to setup a compute or data store. Some disadvantage is lack of flexibility - it's always about trade-offs. Heroku is built on top of AWS, but it doesn't give you access to the full richness of 200+ AWS services. Configuration is sometimes limited e.g. Bucketeer Heroku add-on vs S3 (Simple Storage Service).

We potentially needed something more advanced (in the long term). AWS (Amazon Web Services), which I used in other projects seemed like a natural migration step.

PoC (Proof of Concept)

As every startup - we are busy with delivering new features e.g. Insurance for IT freelancers, so rewriting everything was not an option. I decided to create PoC (Proof of Concept) to check what is possible and answer some questions:

1. Is it possible to implement new features using NestJS, but leave already implemented features in Express unchanged in our monolith (but rewrite them one-by-one using The Boy Scout Rule when we have time)?
2. Is it easy to Dockerize our app, take advantage of modern approach such as serverless, IaC (Infrastructure as Code) in AWS?

NestJS

Hello World!

I started with creating a new Nest project, by following First steps in NestJS Documentation.

npm i -g @nestjs/cli
nest new moojo

This created the working "Hello World!" NestJS app:

Source code: NestJS new project | GitHub

Express app in NestJS

Next thing I wanted to verify is using the existing Express app in NestJS. I found a solution in the Stack Overflow question - Adding NestJS as express module results in Nest being restarted.

All you have to do is to modify bootstrap function in main.ts and use ExpressAdapter:

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  const adapter = new ExpressAdapter(expressApp);
  const app = await NestFactory.create(AppModule, adapter);

  await app.listen(3000);
}

At this point I had two working endpoints, one for NestJS and one from Express:

Source code: Express app in NestJS | GitHub

Prisma

I wanted to create some more realistic example for the migration to AWS, so I decided to add a Postgres database and Prisma (ORM we already use at moojo).

There is a great example for that (with users and blog posts) in NestJS documentation - Prisma | NestJS, so just follow the recipe.

The only thing missing in the recipe steps was updating app.module.ts with a newly created services

import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { PostService } from './post.service';
import { PrismaService } from './prisma.service';
import { UserService } from './user.service';

@Module({
  imports: [],
  controllers: [AppController],
  providers: [AppService],
  providers: [AppService, UserService, PostService, PrismaService],
})
export class AppModule {}

Source code: Prisma recipe | GitHub

OpenAPI (Swagger)

In order to easily test this solution I decided to setup OpenAPI/Swagger documentation.

Everything you need to know is again in NestJS documentation - OpenAPI (Swagger) | NestJS, so you can take a look.

You just have to initialize Swagger using SwaggerModule in main.ts and add use @ApiProperty decorators in a few places.

At this point OpenAPI / Swagger UI was working

and I was able to create user, post and publish it

Source code: OpenAPI/Swagger | GitHub

AWS

Architecture

It would be great to start migration to AWS without bigger refactoring of our code. Later we can gravitate to more cloud-native solution, but lift-and-shift is a great first step.

I really like the serverless-first approach and used it a lot in my previous projects. With lift-and-shift approach - possibilities are limited, but we still can leverage benefits of serverless computing such as no servers to manage, automatic scaling, built-in high availability or pay-for-use billing which can be especially beneficial for staging and developer environments.

All of this can be achieved with the following high level architecture with API Gateway, Fargate and Aurora Serverless.

In order to integrate API Gateway with Fargate in a secure and reliable way - we also need ALB (Application Load Balancer) and VPC Link.

Fargate and Aurora will run within a private subnet... actually two private subnets in two different Availability Zones to achieve high availability.

To enable Fragate tasks to download Docker image from ECR (Elastic Container Registry), we will have to deploy two NAT Gatways in a public subnets.

Security groups can be used to control the traffic that is allowed between all resources.

You can find out more about this architecture on AWS blog e.g. Access Private applications on AWS Fargate using Amazon API Gateway PrivateLink

Infrastructure as Code

There are many benefits of IaC (Infrastructure as Code), including speed of provisioning new environments, lower risk of human errors, improved consistency or a way to version your infrastructure.

There are also many great solutions which can help with that - Terraform, Pulumi, CloudFormation, AWS CDK (Cloud Development Kit) - just to name a few. For me - using the same programming language e.g. TypeScript for all aspects of the technology stack is a powerful concept. I also like to use native technologies in the given environment and I already have experience with AWS CDK, so I decided to go with it.

AWS CDK app

Before we create a new AWS CDK app, we need to install and setup a few things. There is a good tutorial Getting started with the AWS CDK in the official AWS CDK v2 Developer Guide.

When all prerequisites done and AWS CDK installed

npm install -g aws-cdk

we can finally create our First AWS CDK app in moojo folder, which we rename later to cloud.

cdk init app --language typescript

Source code: AWS CDK app | GitHub

VPC (Virtual Private Cloud)

Let's start building our infrastructure with VPC (Virtual Private Cloud) with private and public subnet configurations, Availability Zones and NAT Gateways.

vpc.cdk.ts

import { Construct } from "constructs";
import * as ec2 from "aws-cdk-lib/aws-ec2";

export class VpcCdkConstruct extends Construct {
  public readonly vpc: ec2.Vpc;

  constructor(scope: Construct, id: string) {
    super(scope, id);

    this.vpc = new ec2.Vpc(this, "moojo-vpc", {
      vpcName: "moojo-vpc",
      cidr: "10.0.0.0/16",
      maxAzs: 2,
      natGateways: 2,
      natGatewayProvider: ec2.NatProvider.gateway(),
      subnetConfiguration: [
        {
          name: "moojo-public-subnet",
          subnetType: ec2.SubnetType.PUBLIC,
          cidrMask: 24,
        },
        {
          name: "moojo-private-subnet",
          subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
          cidrMask: 24,
        },
      ],
    });
  }
}

moojo-stack.ts

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";

import { VpcCdkConstruct } from "./vpc.cdk";

export class MoojoStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    cdk.Tags.of(this).add("app", "Moojo");

    const vpc = new VpcCdkConstruct(this, "vpc").vpc;
  }
}

After those changes we can deploy our app

cdk deploy

and watch our stack being created in CloudFormation

You can also verify if VPC was created in a VPC dashboard

Source code: VPC | GitHub

RDS (Relational Database Service)

Let's create Aurora Serverless (part of RDS) next. We will create a Serverless Cluster with Aurora PostgreSQL engine in our VPC. We will setup things like removal policy to destroy to easily cleanup after we are done with PoC, scaling configuration to some bare minimum to save some costs. We also need to enable the Data API to be able to use Query Editor in AWS Console.

As you can see - some of the configuration makes sense only for PoC and it's not production ready. Point of PoC is to check and verify possible solutions, answer some questions NOT become production ready solution.

rds.cdk.ts

import { Construct } from "constructs";
import * as cdk from "aws-cdk-lib";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as rds from "aws-cdk-lib/aws-rds";

interface RdsProps {
  vpc: ec2.Vpc;
}

export class RdsCdkConstruct extends Construct {
  public readonly cluster: rds.ServerlessCluster;

  constructor(scope: Construct, id: string, { vpc }: RdsProps) {
    super(scope, id);

    const securityGroup = new ec2.SecurityGroup(this, "moojo-rds-sg", {
      vpc,
      securityGroupName: "moojo-rds-sg",
      description: "Security group for Moojo RDS (Amazon Relational Databases)",
    });

    this.cluster = new rds.ServerlessCluster(this, "moojo-dbcluster", {
      clusterIdentifier: "moojo-dbcluster",
      defaultDatabaseName: "moojo",
      engine: rds.DatabaseClusterEngine.AURORA_POSTGRESQL,
      parameterGroup: new rds.ParameterGroup(this, "ParameterGroup", {
        engine: rds.DatabaseClusterEngine.auroraPostgres({
          version: rds.AuroraPostgresEngineVersion.VER_10_21, // TODO: adjust supported version
        }),
      }),
      enableDataApi: true, // TODO: needed for Query Editor (AWS console)
      removalPolicy: cdk.RemovalPolicy.DESTROY, // TODO: only for PoC
      vpc,
      scaling: {
        // TODO: adjust it
        autoPause: cdk.Duration.hours(1),
        minCapacity: rds.AuroraCapacityUnit.ACU_2,
        maxCapacity: rds.AuroraCapacityUnit.ACU_2,
      },
      securityGroups: [securityGroup],
    });
  }
}

moojo-stack.ts

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";

import { RdsCdkConstruct } from "./rds.cdk";
import { VpcCdkConstruct } from "./vpc.cdk";

export class MoojoStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    cdk.Tags.of(this).add("app", "Moojo");

    const vpc = new VpcCdkConstruct(this, "vpc").vpc;
    const rds = new RdsCdkConstruct(this, "rds", { vpc });
  }
}

After deploying our stack again, we can check if our database works. Let's go to Query Editor and connect to our database

You copy Secrets manger ARN in Secrets Manager

When connected - we can copy SQL from the migration.sql file generated by Prisma

and execute it to create our tables.

Source code: RDS | GitHub

ELB (Elastic Load Balancer)

As a next step - let's create ALB (Application Load Balancer) with a target group, listener and security group.

elb.cdk.ts

import { Construct } from "constructs";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as elbv2 from "aws-cdk-lib/aws-elasticloadbalancingv2";

interface ElbProps {
  vpc: ec2.Vpc;
}

export class ElbCdkConstruct extends Construct {
  public readonly alb: elbv2.ApplicationLoadBalancer;
  public readonly listener: elbv2.ApplicationListener;
  public readonly securityGroup: ec2.SecurityGroup;
  public readonly targetGroup: elbv2.ApplicationTargetGroup;

  constructor(scope: Construct, id: string, { vpc }: ElbProps) {
    super(scope, id);

    this.securityGroup = new ec2.SecurityGroup(this, "moojo-elb-sg", {
      vpc,
      securityGroupName: "moojo-elb-sg",
      description: "Security group for Moojo ELB (Elastic Load Balancer)",
    });

    this.securityGroup.addIngressRule(
      ec2.Peer.anyIpv4(), // TODO: restrict?
      ec2.Port.tcp(80),
      "Allow from anyone on port 80"
    );

    this.alb = new elbv2.ApplicationLoadBalancer(this, "moojo-alb", {
      loadBalancerName: "moojo-alb",
      vpc,
      securityGroup: this.securityGroup,
      deletionProtection: false,
    });

    this.targetGroup = new elbv2.ApplicationTargetGroup(this, "moojo-alb-tg", {
      vpc,
      targetType: elbv2.TargetType.IP,
      protocol: elbv2.ApplicationProtocol.HTTP,
      port: 80,
      targetGroupName: "moojo-alb-tg",
    });

    this.listener = new elbv2.ApplicationListener(this, "moojo-alb-listener", {
      loadBalancer: this.alb,
      port: 80,
      protocol: elbv2.ApplicationProtocol.HTTP,
      defaultAction: elbv2.ListenerAction.forward([this.targetGroup]),
    });
  }
}

moojo-stack.ts

import { RdsCdkConstruct } from "./rds.cdk";
import { VpcCdkConstruct } from "./vpc.cdk";
import { ElbCdkConstruct } from "./elb.cdk";

export class MoojoStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    cdk.Tags.of(this).add("app", "Moojo");

    const vpc = new VpcCdkConstruct(this, "vpc").vpc;
    const rds = new RdsCdkConstruct(this, "rds", { vpc });
    const elb = new ElbCdkConstruct(this, "elb", { vpc });
  }
}

Source code: ELB | GitHub

API Gateway

Let's create API Gateway (v2) with HTTP API and integrate it with our ALB (Application Load Balancer) using VPC Link.

gw.cdk.ts

import { Construct } from "constructs";
import * as apigwv2 from "@aws-cdk/aws-apigatewayv2-alpha";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as integrations from "@aws-cdk/aws-apigatewayv2-integrations-alpha";

import { ElbCdkConstruct } from "./elb.cdk";

interface ApiGatewayProps {
  vpc: ec2.Vpc;
  elb: ElbCdkConstruct;
}

export class ApiGatewayCdkConstruct extends Construct {
  constructor(scope: Construct, id: string, { vpc, elb }: ApiGatewayProps) {
    super(scope, id);

    const httpApi = new apigwv2.HttpApi(this, "moojo-http-api", {
      apiName: "moojo-http-api",
    });

    const vpcLink = new apigwv2.VpcLink(this, "moojo-vpc-link", {
      vpc,
      vpcLinkName: "moojo-vpc-link",
    });

    const integration = new integrations.HttpAlbIntegration(
      "moojo-http-alb-integration",
      elb.listener,
      {
        method: apigwv2.HttpMethod.ANY, // TODO: restrict?
        vpcLink,
      }
    );

    const route = new apigwv2.HttpRoute(this, "moojo-http-route", {
      httpApi,
      routeKey: apigwv2.HttpRouteKey.with("/{proxy+}", apigwv2.HttpMethod.ANY), // TODO: restrict?
      integration,
    });
  }
}

moojo-stack.ts

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";

import { ApiGatewayCdkConstruct } from "./gw.cdk";
import { ElbCdkConstruct } from "./elb.cdk";
import { RdsCdkConstruct } from "./rds.cdk";
import { VpcCdkConstruct } from "./vpc.cdk";
import { ElbCdkConstruct } from "./elb.cdk";

export class MoojoStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    cdk.Tags.of(this).add("app", "Moojo");

    const vpc = new VpcCdkConstruct(this, "vpc").vpc;
    const rds = new RdsCdkConstruct(this, "rds", { vpc });
    const elb = new ElbCdkConstruct(this, "elb", { vpc });
    const gw = new ApiGatewayCdkConstruct(this, "gw", { vpc, elb });
  }
}

Source code: GW | GitHub

ECS (Elastic Container Service)

It's time for the last part - the heart of our backend. We will create ECS (Elastic Container Service) Cluster, Fargate task definition and service. I will show you how I Dockerized our app and pushed the docker image to ECR (Elastic Container Registry).

Dockerization is super simple. All you have to do is to grab Dockerfile from Tom Ray's article How to write a NestJS Dockerfile optimized for production and save it in our NestJS moojo folder.

Image will be created and pushed to ECR with this simple code

const image = new DockerImageAsset(this, "moojo-ecr-image", {
  directory: path.join(__dirname, "../../", "moojo"),
  platform: Platform.LINUX_AMD64,
});

Keep in mind that this ECR configuration is for PoC only, it has to be adjusted for the production workload. There is also a security issue - I get the database URL and pass it as an environment variable in an unsafe way (this will have to be fixed in a production ready solution, but it's good enough for the PoC).

ecs.cdk.ts

import { Construct } from "constructs";
import { DockerImageAsset, Platform } from "aws-cdk-lib/aws-ecr-assets";
import * as cdk from "aws-cdk-lib";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as ecs from "aws-cdk-lib/aws-ecs";
import * as elbv2 from "aws-cdk-lib/aws-elasticloadbalancingv2";
import * as sm from "aws-cdk-lib/aws-secretsmanager";
import * as path from "path";

import { ElbCdkConstruct } from "./elb.cdk";
import { RdsCdkConstruct } from "./rds.cdk";

interface EcsProps {
  vpc: ec2.Vpc;
  elb: ElbCdkConstruct;
  rds: RdsCdkConstruct;
}

export class EcsCdkConstruct extends Construct {
  constructor(scope: Construct, id: string, { vpc, elb, rds }: EcsProps) {
    super(scope, id);

    const cluster = new ecs.Cluster(this, "moojo-ecs-cluster", {
      clusterName: "moojo-ecs-cluster",
      vpc,
    });

    const securityGroup = new ec2.SecurityGroup(this, "moojo-ecs-sg", {
      vpc,
      securityGroupName: "moojo-ecs-sg",
      description: "Security group for Moojo ECS (Elastic Container Service)",
    });

    const taskDefinition = new ecs.FargateTaskDefinition(
      this,
      "moojo-ecs-task-definition",
      {
        // TODO: adjust it
        memoryLimitMiB: 1024,
        cpu: 512,
      }
    );

    const image = new DockerImageAsset(this, "moojo-ecr-image", {
      directory: path.join(__dirname, "../../", "moojo"),
      platform: Platform.LINUX_AMD64,
    });

    const logging = new ecs.AwsLogDriver({
      streamPrefix: "moojo",
    });

    taskDefinition.addContainer("moojo-ecs-container", {
      containerName: "moojo-ecs-container",
      logging,
      image: ecs.ContainerImage.fromDockerImageAsset(image),
      portMappings: [
        {
          containerPort: 3000,
          protocol: ecs.Protocol.TCP,
        },
      ],
      environment: {
        NODE_ENV: "production",
        DATABASE_URL: this.getDatabaseUrlUnsafe(rds.cluster.secret!), // TODO: do it in a secure way
      },
    });

    const service = new ecs.FargateService(this, `${id}`, {
      serviceName: "moojo-ecs-service",
      taskDefinition,
      securityGroups: [securityGroup],
      cluster,
      // TODO: adjust it
      desiredCount: 1,
      enableExecuteCommand: true,
      maxHealthyPercent: 200,
      minHealthyPercent: 100,
    });

    // TODO: adjust it
    const scaling = service.autoScaleTaskCount({
      minCapacity: 1,
      maxCapacity: 2,
    });

    // TODO: adjust it
    scaling.scaleOnCpuUtilization(`${id}-cpuscaling`, {
      targetUtilizationPercent: 85,
      scaleInCooldown: cdk.Duration.seconds(120),
      scaleOutCooldown: cdk.Duration.seconds(30),
    });

    rds.cluster.connections.allowDefaultPortFrom(
      service,
      "Fargate access to Aurora"
    );

    service.connections.allowFrom(
      elb.alb,
      ec2.Port.tcp(80),
      "Allow traffic from ELB"
    );

    elb.listener.addTargets("moojo-ecs-targets", {
      protocol: elbv2.ApplicationProtocol.HTTP,
      targets: [service],
      healthCheck: {
        // TODO: adjust it
        healthyHttpCodes: "200",
        healthyThresholdCount: 3,
        interval: cdk.Duration.seconds(30),
      },
    });
  }

  // TODO: do it in a secure way!!!
  private getDatabaseUrlUnsafe(secret: sm.ISecret): string {
    const host = secret.secretValueFromJson("host").unsafeUnwrap();
    const port = secret.secretValueFromJson("port").unsafeUnwrap();
    const username = secret.secretValueFromJson("username").unsafeUnwrap();
    const password = secret.secretValueFromJson("password").unsafeUnwrap();
    const database = secret.secretValueFromJson("dbname").unsafeUnwrap();

    return `postgresql://${username}:${password}@${host}:${port}/${database}`;
  }
}

moojo-stack.ts

import { Construct } from "constructs";

import { ApiGatewayCdkConstruct } from "./gw.cdk";
import { EcsCdkConstruct } from "./ecs.cdk";
import { ElbCdkConstruct } from "./elb.cdk";
import { RdsCdkConstruct } from "./rds.cdk";
import { VpcCdkConstruct } from "./vpc.cdk";
export class MoojoStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    cdk.Tags.of(this).add("app", "Moojo");

    const vpc = new VpcCdkConstruct(this, "vpc").vpc;
    const rds = new RdsCdkConstruct(this, "rds", { vpc });
    const elb = new ElbCdkConstruct(this, "elb", { vpc });
    const gw = new ApiGatewayCdkConstruct(this, "gw", { vpc, elb });
    const ecs = new EcsCdkConstruct(this, "ecs", { vpc, elb, rds });
  }
}

When deployed - you can grab API Gateway URL from the AWS Console

and give it a try using Swagger UI.

As you can see everything worked like in our local environment.

Source code: ECS | GitHub