DEV Community: Awaji-mitop N. Gilbert

Circular dependency in Node.js and Nest.js

Awaji-mitop N. Gilbert — Mon, 10 Apr 2023 11:56:55 +0000

Circular dependency is a situation that arises when modules depend directly or indirectly on one another in a program.

In Nest.js, with dependency injection (DI), it can be easily identified as the affected module would not be created during application startup until this dependency issue is fixed. However, if this is a Node.js circular dependency, typical of incomplete module loading somewhere else in the application not using DI, it may go unnoticed resulting in runtime errors.

In this article, we will create, discuss and fix circular dependency in Node.js and Nest.js.

Circular dependency in Node.js

This is a case where 2 or more modules depend on each other by sharing program elements. For example, a module A imports from B and at same time, B imports from A.
We can summarise the dependency like so (A -> B -> A).

The problem here, based on how module loading works is that Node.js while loading A will find B and while loading B, will discover B requires again module A. In other to avoid an infinite loop, Node.js will return an incompletely loaded module A in B which will result in undefined imports in module B.

The weird undefined

As pointed out earlier, incomplete module loading can result in unexpected errors during runtime as certain imports will be undefined. To drive this point, we will create a small Node example script to show how this can arise, how to troubleshoot it and refactor to verify our solution.

Consider a small Node.js project containing 3 files, A.js, B.js and index.js where A.js depends on B.js and vice versa.

Say the content of A.js is as follows:



const { utilB } = require('./B')

const nameObject = {
    name: 'whatever'
}

const utilA = () => {
    console.log('inside A')
    console.log(`name is '${nameObject.name}' in A`)
    utilB()
}

module.exports = {
    utilA,
    nameObject
}

B.js is as follows:



const { nameObject } = require('./A')

const utilB = () => {
    console.log('inside B')
    console.log(`name is '${nameObject.name}' in B`)
}

module.exports = {
    utilB,
}

index.js is as follows:



const { utilA } = require('./A')

utilA()

As we can see, A.js depends on B.js and B.js depends on A.js. As a result of this, imports from A.js in B.js will be undefined as A.js is not fully loaded at this time.
Run node index.js in this project directory and observe that the output in the terminal throws an error Cannot read properties of undefined which we know for a fact is defined. This is simply because of an incompletely loaded module A.js in B.js.



profg@AwajimitopsMBP circular-node (main) $ node index.js
inside A
name is 'whatever' in A
inside B
/Users/profg/Desktop/github/circular-deps/circular-node/B.js:5
    console.log(`name is '${nameObject.name}' in B`)
                                    ^

TypeError: Cannot read properties of undefined (reading 'name')
    at utilB (/Users/profg/Desktop/github/circular-deps/circular-node/B.js:5:37)
    at utilA (/Users/profg/Desktop/github/circular-deps/circular-node/A.js:10:5)
    at Object.<anonymous> (/Users/profg/Desktop/github/circular-deps/circular-node/index.js:3:1)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
    at node:internal/main/run_main_module:17:47

Troubleshoot circular dependency with Madge

Identifying this in a real project may be hectic and that is where madge comes in.

Madge is a tool for identifying circular dependencies in Node.js projects. I use it in my Nest.js projects too.

To do this, we need to install it as a development dependency and create a script for it.

Add madge to the project



npm install madge -D

My package.json looks like this now



{
  "name": "circular-node",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "find-circular-deps": "madge . -c"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "madge": "^6.0.0"
  }
}

Now, check for circular dependencies with madge by running the script find-circular-deps in the package.json file. Note that madge . -c simply tells madge we are checking in this same directory. emphasis on the dot . used.



 npm run find-circular-deps

Notice it correctly detects we have a circular dependency.

Avoid circular dependency in Node.js by refactor

We will fix this issue by a simple refactor. The refactor will depend on your project. However, for this dummy example used above, we can solve this by combining A.js and B.js into one file or move their relationship to a new file from where they can independently import their dependency. I will go with the latter here for sake of demonstration. Any option works.

We will create a new module C.js and move into it the object nameObject and refactor A.js and B.js to use C.js thus, breaking the circular dependency.

C.js will look like so



const nameObject = {
    name: 'whatever'
}


module.exports = {
    nameObject
}

A.js will now look like so



const { utilB } = require('./B')
const { nameObject } = require('./C')

const utilA = () => {
    console.log('inside A')
    console.log(`name is '${nameObject.name}' in A`)
    utilB()
}

module.exports = {
    utilA,
}

B.js will now look like so



const { nameObject } = require('./C')

const utilB = () => {
    console.log('inside B')
    console.log(`name is '${nameObject.name}' in B`)
}

module.exports = {
    utilB,
}

Run madge with npm run find-circular-deps and notice the circular dependency is now gone.

Run the script index.js with node index.js and notice it now runs as expected to completion with no errors.

Circular dependency in Nest.js

Circular dependency in Nest.js occurs when two classes depend on each other. This can be easily identified as Nest.js will throw an error and will only build after you resolve the circular dependency.

To show this, we need to quickly bootstrap a nest application and create 2 module classes that depend on each other. I will be leveraging the Nest.js cli tool to quickly bootstrap one and modify to achieve this.

Note that though we chose to demonstrate this with module classes, same can be applied in exactly same way with service/provider classes.

The Nest.js circular dependency problem

For this example, we will consider 2 module classes with names AppModule and FileModule which depends on each other.

The app.module.ts module file



import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { FileModule } from './file.module';

@Module({
  imports: [FileModule],
  controllers: [AppController],
  providers: [AppService],
  exports: [AppService],
})
export class AppModule {}

The file.module.ts module file



import { Module } from '@nestjs/common';
import { AppController as FileController } from './app.controller';
import { AppService } from './app.service';
import { FileService } from './file.service';
import { AppModule } from './app.module';

@Module({
  imports: [AppModule],
  controllers: [FileController],
  providers: [FileService],
})
export class FileModule {
  constructor(
    private readonly appService: AppService,
  ) {}

  onApplicationBootstrap() {
    console.log(this.appService.getHello());
  }
}

What we have done above is to forcefully create a circular dependency by importing also the AppModule in the FileModule just to access our appService. On attempt to start the app, you should get this error in the terminal.

Fix Nest.js circular dependency by refactor

A refactor most times, is what we need to get around this issues.

In other cases, one may need to create a new module to hold the shared dependencies and have that module imported in our 2 new modules to break the circular dependency but it depends on what one intends to achieve and the project architecture at hand. However, in this simple example, we created the problem and so it is easy to fix as we know all we need to do is not import the entire AppModule module and just inject our appService as a provider in FileModule.

Refactor file.module.ts like so



import { Module } from '@nestjs/common';
import { AppController as FileController } from './app.controller';
import { AppService } from './app.service';
import { FileService } from './file.service';

@Module({
  controllers: [FileController],
  providers: [FileService, AppService],
})
export class FileModule {
  constructor(
    private readonly appService: AppService,
  ) {}

  onApplicationBootstrap() {
    console.log(this.appService.getHello());
  }
}

Fix Nest.js circular dependency with forwardRef

When a refactor is not an option for your use case, then Nest.js forwardRef utility function can be used to overcome this issue.

Back to our original failing FileModule version that imports the entire AppModule, we just need to use forwardRef with the import of AppModule.

The file file.module.ts will now look like so



import { Module, forwardRef } from '@nestjs/common';
import { AppController as FileController } from './app.controller';
import { AppService } from './app.service';
import { FileService } from './file.service';
import { AppModule } from './app.module';

@Module({
  imports: [forwardRef(() => AppModule)],
  controllers: [FileController],
  providers: [FileService],
})
export class FileModule {
  constructor(
    private readonly appService: AppService,
  ) {}

  onApplicationBootstrap() {
    console.log(this.appService.getHello());
  }
}

Run the app and notice it works fine.

Conclusion:

Circular dependency in Nest.js and Node.js are simply the same things. However, Nest.js detects this during dependency injection and prevents it by throwing a circular dependency error. The main problem is just an incompletely loaded module which can still happen in other parts of a Nest.js application not using dependency injection and go unnoticed until something begins to break.

We can find these issues using the npm package madge and fix them by a simple refactor. Where a refactor is not an option, we can always use the Nest.js utility function forwardRef which works for both modules and providers.

The code used in this article can be found on github. The initial circular dependency code problems are in the branch circular-dependency-problem

Work around to Deploy Large Lambda Functions with Claudia.js 🕵🏻‍♂️

Awaji-mitop N. Gilbert — Tue, 10 Jan 2023 22:01:45 +0000

A common error encountered while deploying to AWS Lambda is the error associated with the size of the function's deployment package which is as a result of the 50 MB (zipped, for direct upload) and 250 MB (unzipped) quota limitation applied to all the files you upload by AWS.

Photo from Lambda quotas.

This Error associated with the size of the deployment package comes in two forms; the RequestEntityTooLargeException and InvalidParameterValueException error for 50 MB and 250 MB limits respectively.

This article shares a work around for these limitations with deploying large functions by leveraging AWS S3 bucket for storage and downloading of the deployment package when the Lambda function loads.

Note: this will increase startup time and may introduce significant latency for initial request processing.

This article, though uses Claudia.js for deployments, shares a general approach that can be applied irrespective of how you deploy your Lambda function.

Requirements and setup

AWS account
Claudia.js

For this article, the deployment tool of choice is Claudia.js which is simply a tool to deploy Node.js projects to AWS Lambda and API Gateway. It abstracts the deployment configuration, sets everything required out of the box and makes it easy to get started with Lambdas, thus, focusing on the business value.

Follow this quick guide to setup your machine with Claudia.js and AWS.

Create the problem

To demonstrate the solution, we need to first create the problem and this section will focus on that. You can jump to the solution if you already have the problem 😉.

Here, we create a Lambda function with deployment package limit for direct upload by installing random npm packages to increase the zip size for direct upload.
Also, we will create a large file in the project so as to increase the unzipped size to simulate the 250MB limit.

With these two setups, we will encounter the errors for size limit and work to solve them as we finally deploy our lambda function.

step 1:
Create a directory for our lambda function and initialize it as a node.js project.

mkdir claudia-large && cd claudia-large && npm init

step 2:
Setup package.json.

Install a few random dependencies to increase our deployment package size to beat the direct upload limit (50MB).
Add a deploy:create script. lambda.handler in the script means lambda is the module/file name and handler is the function to be used in the module.

After this, our package.json file should look like this

{
  "name": "claudia-large",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "deploy:create": "claudia create --region us-east-1 --handler lambda.handler"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "aws-sdk": "^2.814.0",
    "babel-eslint": "^10.1.0",
    "eslint-plugin-prettier": "^3.4.0",
    "faker": "^5.5.3",
    "jest": "^27.3.1",
    "sinon": "^11.1.1",
    "typescript": "^4.2.4"
  }
}

step 3:
Create a large file in the directory of a 100MB.

This will ensure we simulate the second problem by increasing the unzipped size beyond the 250MB limit after solving the first problem.

mkfile -n 100m largefile1.txt

You should see a file largefile1.txt in the project directory.

step 4:
Create a simple Lambda function that reads a file and reports it's size.

First, we create a file lambda.js and add the contents below to it.

const fs = require('fs');

exports.handler = function (_, context) {
  const info = fs.statSync('largefile1.txt');

  context.succeed(`file size is: ${info.size}`);
};

step 5:
Inspect the size of deployment package by running claudia pack which will package the deployment without deploying it.

A file claudia-large-1.0.0.zip should now be created in your directory.

Also, add ls -l which will list the size of the different files in the directory.

claudia pack && ls -l

You should see that the size of the generated zip is 53411983 bytes which is 53.411983 MB and goes beyond the limit for direct upload.

Note: ensure to remove the zipped file created by claudia pack so it does not add to the lambda when we intend to deploy.

Attempt a deploy:
Attempt to deploy the function with the deploy:create script.

npm run deploy:create

The deployment should fail because the zip size is beyond the limit for direct upload. You should see a message containing RequestEntityTooLargeException: Request must be smaller than 69905067 bytes

Solution #1 (RequestEntityTooLargeException)

When the size of the zip package created exceeds the limit for direct upload, we can use an S3 bucket to upload the function code before installing in Lambda.

As such, we will create an S3 bucket on AWS console which will be used to upload our function code from where it will be installed in AWS Lambda using the Claudia.js option --use-s3-bucket.

step 1:
Create an S3 bucket with name claudia-large accepting all defaults (depends on your specific needs) with same region as your Lambda function. In our case, our region will be us-east-1.

step 2:
update the deploy:create script to use the bucket created in previous step with the option --use-s3-bucket.

{
...,
    "deploy:create": "claudia create --region us-east-1 --handler lambda.handler --use-s3-bucket claudia-large",
...
}

step 3:
Deploy again.

Note: Since the first deployment failed, we need to delete the default role created by AWS for our Lambda.

Claudia creates a default role for your Lambda in AWS IAM, this role can be customised by passing a policy file.
In our case, the role is claudia-large-executor. If you do not delete it before trying again, you will get an EntityAlreadyExists error in the terminal.
This is the issue on Claudia.js

npm run deploy:create

Congratulations 🚀, we have gotten a different error 🥲😀, the InvalidParameterValueException: Unzipped size must be smaller than 262144000 byteserror.

Basically, what this means is that our size limit is no longer 50MB since we do not use direct upload anymore but now 250MB for unzipped limit.

That is, if we never created the largefile1.txt file of a 100MB in the directory, we would have had a successful function upload by now.

Also, you should see the deployment package in the S3 bucket claudia-large we created.

Solution #2 (InvalidParameterValueException)

In the case where the Lambda function's unzipped size goes beyond the 250MB limit, one thing to do would be to ensure all external dependencies/files are kept in a separate storage and downloaded during execution time.

In other words, to solve this issue, we will upload our large file largefile1.txt to AWS S3 and only download it when the function runs.

step 1:
Upload the file largefile1.txt manually to the S3 bucket we created claudia-large.

step 2:
Refactor our function code to not load the file from package directory but from AWS S3.

The file lambda.js should be refactored to now have the content below.

const AWS = require("aws-sdk");
const s3 = new AWS.S3();

async function getMetaInfoFromS3(key, bucket) {
  const metaInfo = await s3
    .headObject({ Key: key, Bucket: bucket })
    .promise()

  return metaInfo;
}

exports.handler = async function (_, context) {
  const metaInfo = await getMetaInfoFromS3("largefile1.txt", "claudia-large");

  context.succeed(`file size is: ${metaInfo.ContentLength}`);
};

step 3:
Delete the file largefile1.txt from the project since it is now on S3.

rm largefile1.txt

step 4:
Add a policy folder and json file to the directory to allow our Lambda function access to read from S3. This policy would be used to add additional permissions to our lambda executor role claudia-large-executor created by Claudia.

mkdir policy && touch policy/policy.json

Add the content below to the policy.json file.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Folder structure should now look like this

step 5:
update the deploy:create script to use the policy.json created in previous step by adding --policies policy.

{
...,
    "deploy:create": "claudia create --region us-east-1 --policies policy --handler lambda.handler --use-s3-bucket claudia-large",
...
}

step 6:
Deploy again. Remember to delete the role if existing before attempting this else you would get an error.

npm run deploy:create

Success!.

You should see a claudia.json file created in the root of the project with the contents

{
  "lambda": {
    "role": "claudia-large-executor",
    "name": "claudia-large",
    "region": "us-east-1"
  }
}

Test the Lambda we just created:
Finally, to test the lambda's response, run the command below.

claudia test-lambda --config claudia.json

you should see a response in the terminal containing the file size as expected in the function code.

Conclusion:

Generally, we have been able to work around the 50MB zipped and 250MB unzipped lambda limits by use of S3 buckets to upload functions larger than 50MB over a slower network and moving large files which might increase the size of the unzipped package to S3 (or any other storage).

The techniques shared in this article is simply to keep large dependencies/files outside the function package and is a general approach irrespective of the deployment tool used.

However, the limit of 250MB still exists and cannot be exceeded. All we did was apply storage techniques to make our function smaller.

If the Lambda function cannot be reduced, maybe consider checking container image support for lambda which can provide you with Lambda functions as container images of up to 10 GB in size.

The final code used for this article can be found on my Github here

Deploy your static generated files to AWS S3 from GitLab CI/CD using git tags

Awaji-mitop N. Gilbert — Mon, 10 Oct 2022 09:42:28 +0000

One simple approach I often rely on to deploy my static frontend application is a storage service and with AWS, AWS S3.

In this article, I provide a step by step guide on how to automate the deployment of your static files to AWS S3 leveraging Gitlab's CI/CD pipelines, and with git tags as deploy triggers.

We start by creating an S3 bucket, after which we prepare Gitlab with AWS credentials, write the pipeline and finally, test the setup end to end by pushing a git tag which will trigger the pipeline and deploy our static files to our AWS S3 bucket.

The approach chosen in this article is a generic approach to share a concept which can be applied irrespective of your project or language domain.

Requirements

Gitlab account
AWS account

Setup AWS S3 bucket and access

Step 1:
Go to AWS S3 to create a bucket for storing the static files by clicking on the create bucket button.

Step 2:
In the form provided, enter your bucket name.
For this guide, we will use the bucket name gitlab-ci-tutorial.

Now, select a region of your choice (us-east-1), leave everything else as is and click on create bucket.

Notice that the created bucket is empty.

Step 3:
Go to AWS IAM to create a user with programatic access to S3.

Click on Add users.

Step 4:
Following the steps in the form provided, we create a user that will be used from Gitlab to access/deploy files to our S3 bucket.

Provide a name for the user, gitlab_s3_tutotrial and check the Access key - Programmatic access checkbox.

For permissions, we will attach an existing policy for full S3 access AmazonS3FullAccess in this guide. It is recommended to create a more fine grained access based on your project requirements.

Step 5:
Review and create user.

Prepare Gitlab

Step 1:
Create your project on Gitlab. I created a project gitlab-ci-tutorial1

Step 2:
Within the created project, navigate to settings > CI/CD and expand the variables section.

Add the following environment variables which will be used for deployment to AWS S3 as created in step 5 Setup AWS S3 bucket and access > step 5 above

AWS_ACCESS_KEY_ID: <Your Access key ID>
AWS_SECRET_ACCESS_KEY: <Your secret access key>
AWS_DEFAULT_REGION: us-east-1 (as chosen in steps above)
S3_BUCKET: gitlab-ci-tutorial (Name of your bucket)

Write the pipeline

Now, we are ready to create the pipeline and this is done using a .gitlab-ci.yml file at the root of the project.

Whenever a push is made to Gitlab, this file will be detected and the pipeline will run. Though, in this case we want the pipeline to only deploy when we push a new tag.

Create a .gitlab-ci.yml file and add the contents below to the file.



stages:
  - build
  - deploy

build:
  stage: build
  script:
  # We simulate a build phase by manually creating files in a build folder
  # An example would be using `yarn build` to build a react project that would create static files in a build or public folder
    - mkdir -p build && touch build/build.html build/build.txt
  artifacts:
    when: on_success
    paths:
      - build/
    expire_in: 20 mins


deploy:
  stage: deploy
  needs:
    - build
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  rules:
    - if: $CI_COMMIT_TAG                 # Run this job when a tag is created
  script:
    - echo "Running deploy"
    - aws s3 cp ./build s3://$S3_BUCKET/ --recursive
    - echo "Deployment successful"

In the pipeline above, we made use of stages and passed artifacts (outputs) from one stage to another.
The build stage was written to mimic an actual build of a project which will then create output files in a build folder. Here, we created a build folder containing 2 files with shell commands.
The image used in the deploy phase aws-base contains the aws CLI command necessary to run the aws deploy script and by default, detects the AWS environment variables with the exact names we added to the variables section of Gitlab earlier.

For a full description of the configuration options of the .gitlab-ci.yml file, check here.

Now, commit the file to the repo and push to the project we created earlier.

Notice in the images below that on the Gitlab project, only the build job was run.
This is because of the line - if: $CI_COMMIT_TAG in the rules section of the deploy phase which will ensure that the deployment is only run when we push a new tag or create it on the Gitlab UI. Also, see here Trigger pipeline with git tags.

Test the overall setup

Step 1:
Tag your commit.



git tag 'V1'

Step 2:
Push your tag.



git push origin 'V1'

Notice, we now have a new pipeline at the top that ran 2 jobs, the build and deploy jobs.

Below is the completed deploy job.

Step 3:
Check the bucket on AWS S3

Refresh and notice that the S3 bucket is no longer empty and contains the files built and deployed from our pipeline as seen below.

Congratulations, we did it! 🎉

Conclusion

There are different ways to deploy code with the Gitlab CI/CD pipeline. This approach with git tags is one common approach used with npm packages and most organisations I have worked with and brings the advantage of easy rollback amongst many.

When using git tags for deployment, it is recommended to use protected tags which will ensure your deployment is intensional. Additionally, the environment variables we set on gitlab must be set as protected which will ensure they are only exposed to protected tags or branches.

Thanks!