DEV Community

Cover image for AWS Lambda and NestJS: Communication between services
Venya Brodetskiy
Venya Brodetskiy

Posted on • Originally published at Medium

AWS Lambda and NestJS: Communication between services

Originally published on Medium on November 11, 2023.

In this guide, we’ll speak about communication in a distributed system using AWS Lambda and NestJS.

Building on our previous exploration, where we integrated NestJS with AWS Lambda and set up debugging and hot reload within Docker, this guide takes the next logical step. We’ll use the same foundational setup and enhance it to enable seamless communication between lambdas.

The approach in this guide is inspired by a past project where I used Dapr with C# microservices. Dapr simplifies the challenges of building microservices. For example, instead of manually setting up communication between services, Dapr provides straightforward methods to do so. This lets developers concentrate on writing the actual functionality without getting bogged down by the technical details of inter-service communication. This Lambda + NestJS communication strategy is deeply inspired by the simplicity and effectiveness of Dapr.

If you’re interested in learning more about Dapr and C#, check out my other repository.

This guide focuses on synchronous requests, but I’ll cover asynchronous requests with SQS later on in other guide.

You can find demo application in this repo: https://github.com/VenyaBrodetskiy/Lambda-NestJS-Demo

The demo includes:

  • An app configured to work locally as per the previous guide.
  • Implementation of communication between lambdas to function both locally and post-deployment (the focus of this guide).
  • How to add queues (SQS) to your solution

In the subsequent sections, we’ll break down the steps involved in creating communication between lambdas:

  1. Understanding the Service Design for Simplified Lambda Communication
  2. Building the Lambda Communication service
  3. Enhancing the Lambda Communication Service with Retries
  4. Implementing Lambda Factory to support Local and Cloud Environments

Let’s get started!

Part 1. Understanding the Service Design for Simplified Lambda Communication

The Concept: The goal is to create a service that encapsulates complex logic, making it as user-friendly and straightforward as Dapr. In essence, this service acts as a bridge, simplifying communication between different components of our system.

Practical Example: Let’s illustrate how this service should work with a simple NestJS controller. This controller handles an external request (e.g., from a frontend application or Postman) and communicates with another lambda function to fetch data from a database.

Here’s the code for our PlanController:

import { LambdaCommunicationService } from 'src/core/modules/communication';

@Controller('plan')
export class PlanController {
  private readonly logger = new Logger(PlanController.name);
  constructor(
    private readonly lambdaService: LambdaCommunicationService,
  ) {}

  @Get('/:id')
  public async getPlanById(@Param('id') id: string): Promise<PlanRes> {
    this.logger.log(`Inside ${this.getPlanById.name}, id: ${id}`);

    // service to call other lambda
    const result: PlanRes = await this.lambdaService.invoke<PlanRes>(
      Accessor.Plan,          // name of lambda to be called. `Accessor` is an enum representing different lambda functions
      `/planaccessor/${id}`,  // path of request
      HttpMethod.Get,         // the HTTP method for the request
    );
    return result;
  }
Enter fullscreen mode Exit fullscreen mode

signature of method lambdaService.invoke :

LambdaCommunicationService.invoke<TResponse>(
  service: string, 
  path: string, 
  httpMethod?: HttpMethod, 
  payload?: object): Promise<TResponse>
Enter fullscreen mode Exit fullscreen mode
  • service: Identifies the name of lambda function to be called.
  • path: Specifies the request's path.
  • httpMethod: The HTTP method for the request, defaulting to GET if not specified.
  • payload: Optional object containing data to be sent to the lambda.

The Benefit: With a service like this, calling other lambdas becomes straightforward. All you need is the function name, the path for the call, the method, and any payload if necessary.

This design significantly reduces development time, as there’s less need to write and maintain complex code for lambda interactions. It also minimizes errors by standardizing how services communicate, ensuring consistency and reliability. Furthermore, such a service improves code readability and maintainability, making it easier for teams to understand and modify the codebase over time.

Note about Error Handling: You might notice the absence of a try-catch block. This is because NestJS has built-in exception filters, which handle errors elegantly. You can learn more about this feature in the NestJS documentation on exception filters. Additionally, my Lambda-NestJS Demo repository contains a custom implementation of an exception filter, though it’s a topic beyond the scope of this guide.

Part 2. Building the Lambda Communication service

In this part let’s implement the service described in Part 1 :

@Injectable()
export class LambdaCommunicationService {
  private readonly logger = new Logger(LambdaCommunicationService.name);
  constructor(private lambdaFactory: LambdaFactory) {}

  public async invoke<TResponse>(
    service: string,
    path: string,
    httpMethod: HttpMethod = HttpMethod.Get,
    payload?: object,
  ): Promise<TResponse> {
    try {
      // get instance of lambda object
      const { lambda, functionName } = this.lambdaFactory.getLambda(service);

      // prepare payload
      const lambdaPayload: ICommunicationPayload = {
        httpMethod: httpMethod,
        path: path,
        body: payload ?? undefined,
        headers: {
          'Content-Type': 'application/json',
        },
      };

      const params: InvokeCommandInput = {
        FunctionName: functionName,
        Payload: JSON.stringify(lambdaPayload),
      };

      this.logger.debug(
        `Inside ${this.invoke.name}. Invoking function: ${params.FunctionName} with payload: ${params.Payload}`,
      );

      // call other lambda using aws-sdk
      const response: InvokeCommandOutput = await lambda.invoke(params);

      // handle the response
      const responsePayload = JSON.parse(Buffer.from(response.Payload).toString());
      if (RetriableStatusCodes.includes(responsePayload.statusCode)) {
        throw new CommunicationException(
          JSON.parse(responsePayload.body),
          responsePayload.statusCode,
        );
      }

      this.logger.debug(
        `Inside ${this.invoke.name}. Lambda invoke response payload: ${JSON.stringify(
          responsePayload,
          null,
          ' ',
        )}`,
      );

      // parse the response to expected type
      if (typeof responsePayload.body === 'string' && this.isJsonString(responsePayload.body)) {
        const result = JSON.parse(responsePayload.body) as TResponse;
        return result;
      }

      return responsePayload.body as TResponse;
    } catch (error: any) {
      if (error.code === 'ECONNREFUSED')
        throw new CommunicationException(
          `Failed to invoke lambda: ${service}`,
          HttpStatus.INTERNAL_SERVER_ERROR,
        );
      throw error;
    }
  }

  private isJsonString(str: string): boolean {
    try {
      JSON.parse(str);
      return true;
    } catch (e) {
      return false;
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Now I will break down the code above and explain it one by one.

  1. For the first step, we utilize the AWS SDK to instantiate a lambda object. This object is responsible for invoking other lambda functions. Alongside this, we also retrieve the specific function name that we intend to call. The creation and management of this lambda object are efficiently handled by the LambdaFactory. Understanding the workings of LambdaFactory is key to our implementation, and it will be explored in greater detail in the further sections (part 4):
const { lambda, functionName } = this.lambdaFactory.getLambda(service);
Enter fullscreen mode Exit fullscreen mode

2. Next step is to build Lambda Payload and invoke params,invoke lambda:

const lambdaPayload: ICommunicationPayload = {
  ...
};

const params: InvokeCommandInput = {
  ...
};

const response: InvokeCommandOutput = await lambda.invoke(params);
Enter fullscreen mode Exit fullscreen mode

3. Handling the Response

This is a critical step. We need to understand the internal status code in responsePayload to determine if the invocation was successful:

const responsePayload = JSON.parse(Buffer.from(response.Payload).toString());

if (RetriableStatusCodes.includes(responsePayload.statusCode)) {
  throw new CommunicationException(
    ...
  );
}
Enter fullscreen mode Exit fullscreen mode

4. Parsing the Response

The response from the lambda can be an object of type TResponse (defined by the developer) or a string. Here's how we handle it:

if (typeof responsePayload.body === 'string' && this.isJsonString(responsePayload.body)) {
  const result = JSON.parse(responsePayload.body) as TResponse;
  return result;
}

return responsePayload.body as TResponse;
Enter fullscreen mode Exit fullscreen mode

This part of the code checks the type of the body in the response and parses it accordingly. The use of TypeScript allows us to cast the response to the type TResponse specified by the developer.

Part 3. Enhancing the Lambda Communication Service with Retries

In this part, we are adding retry functionality to our Lambda Communication Service. The ability to retry failed requests is a critical feature, particularly for dealing with transient network issues or temporary service unavailability.

Note. However, it’s essential to carefully select HTTP methods due to their idempotency characteristics. GET and DELETE are primarily targeted for retries because they do not change the state in a way that leads to side effects upon repetition. PUT and PATCH can also be considered for retries as they are designed to be idempotent, ensuring that repeated requests result in the same state. However, caution is advised with POST requests, which typically modify state or create resources, could lead to unintended consequences if retried without proper idempotence handling.

Retries should be applied for error codes that indicate transient issues or server errors where a repeat request might succeed. Generally, 5xx series errors (e.g., 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout) are suitable candidates for retries, suggesting temporary server-side issues, but retrying on 501 Not Implemented is not advisable as this error indicates a permanent limitation of the server, and subsequent retries are unlikely to succeed. As for 4xx series errors (e.g., 400 Bad Request, 401 Unauthorized, 404 Not Found), they typically indicate client-side problems that are unlikely to be resolved by retrying without changes to the request. However, transient errors like 408 Request Timeout, 423 Locked, and 429 Too Many Requests can be exceptions, where retrying with a backoff strategy might be successful.

To integrate this feature, we’ll use the async-retry npm package, which provides a straightforward way to implement retry logic. The invoke method in the Lambda Communication Service is enhanced as follows:

  1. Adding a Retry Parameter: The invoke method now includes an additional parameter retries with a default value of 3. This parameter determines the maximum number of retries for a request.
public async invoke<TResponse>(
    service: string,
    path: string,
    httpMethod: HttpMethod = HttpMethod.Get,
    payload?: object,
    retries: number = 3,  // new parameter
  ): Promise<TResponse> {
Enter fullscreen mode Exit fullscreen mode

2. Conditional Retry Logic: We introduce a check to enable retries only for idempotent requests. For other HTTP methods, the effective number of retries is set to zero.

// enable retries only for Retriable Http Methods requests
let effectiveRetries;
if (RetriableHttpMethods.includes(httpMethod)) {
  effectiveRetries = retries;
} else {
  effectiveRetries = 0;
}
Enter fullscreen mode Exit fullscreen mode

3. Retry Mechanism: Using the retry function from the async-retry package, we wrap the actual lambda invocation logic. This function will automatically retry the invocation based on the provided retry conditions.

let responsePayload;
await retry(
  async () => {
    this.logger.debug(
      `Inside ${this.invoke.name}. Invoking function: ${params.FunctionName} with payload: ${params.Payload}`,
    );

    const response = await lambda.invoke(params);
    responsePayload = JSON.parse(Buffer.from(response.Payload).toString());

    if (RetriableStatusCodes.includes(responsePayload.statusCode)) {
      throw new CommunicationException(
        JSON.parse(responsePayload.body),
        responsePayload.statusCode,
      );
    }
  },
  // retry configuration
  {
    retries: effectiveRetries,
    onRetry: (error) => {
      error &&
        this.logger.warn(`Error while calling lambda: ${params.FunctionName} with payload: ${
          params.Payload
        }, retrying...
        Error: ${JSON.stringify(error, null, ' ')}`);
    },
  },
);
Enter fullscreen mode Exit fullscreen mode

Error Handling During Retries: In case of an error during an invocation attempt, an error log is generated. This helps in monitoring and debugging issues related to failed lambda invocations.

By implementing this retry mechanism, we ensure that our Lambda Communication Service is more robust and can handle transient failures more gracefully, thereby improving the overall reliability of the system.

Part 4. Implementing Lambda Factory to support Local and Cloud Environments

In the previous section, we introduced the LambdaFactory within our Lambda Communication Service.

const { lambda, functionName } = this.lambdaFactory.getLambda(service);
Enter fullscreen mode Exit fullscreen mode

Now, let’s explore why it’s important and how to implement it to ensure smooth operation both locally and in the cloud.

The LambdaFactory role is to abstract the creation and configuration of AWS Lambda instances. This abstraction is crucial because it allows our application to adapt dynamically to different environments — local development or cloud deployment—without altering the core business logic.

1. Understanding the LambdaFactory Implementation

Let’s examine the key components of the LambdaFactory service.

lambda-factory.service.ts:

...
import { Configuration } from 'src/config';

interface ILambdaClient {
  lambda: Lambda;
  functionName: string;
}

@Injectable()
export class LambdaFactory {
  private lambdaInstance: Lambda;

  constructor(private config: Configuration) {}

  public getLambda(service: string): ILambdaClient {
    // get function name and endpoint from configuration
    const { name: functionName, endpoint: endpoint } = this.config.getService(service);

    // for cloud
    if (!this.config.IsOffline) {
      this.lambdaInstance = this.lambdaInstance ?? new Lambda({});

      return {
        lambda: this.lambdaInstance,
        functionName: functionName,
      };
    }

    // for local development
    this.lambdaInstance = new Lambda({
      endpoint: endpoint,
    });

    return {
      lambda: this.lambdaInstance,
      functionName: functionName,
    };
  }
}
Enter fullscreen mode Exit fullscreen mode

The getLambda method in the LambdaFactory service is key to configuring the Lambda client instance for either local development or cloud deployment, as determined by the Configuration service. This is crucial for two main reasons:

  1. Local Development: For local testing, especially with the Serverless framework, each Lambda function often requires a unique endpoint. Therefore, the method creates a new Lambda object for each call, ensuring accurate local simulation.
  2. Cloud Deployment: In the cloud, Lambda functions are identified by their names, not endpoints. Here, the method optimizes performance by reusing the same Lambda instance for each invocation, following a singleton-like pattern.

This approach ensures flexibility and consistency across different environments, simplifying development and deployment processes.

2. Understanding the Configuration service

In LambdaFactory we used getService method from Configuration service to fetch specific settings for LambdaCommunicationService, based on defined enums.

configuration.service.ts:

@Injectable()
export class Configuration {
  constructor(private configService: ConfigService) {
    this.validateConfig();
  }

  get IsOffline(): boolean {
    return Boolean(this.configService.get<boolean>('IS_OFFLINE'));
  }

  public getService(service: string): IAcccessorConfig {
    try {
      switch (service) {
        case Accessor.Plan:
          return {
            name: this.configService.getOrThrow<string>('PLANACCESSOR_NAME'),
            endpoint: this.configService.getOrThrow<string>('PLANACCESSOR_ENDPOINT'),
          };
        // case Accessor.OtherAccessor:
        // ...
        default:
          throw new Error(
            `Unknown accessor type. Configuration.service misses accessor: ${service}`,
          );
      }
    } catch (e) {
      throw new Error(e);
    }
  }
  private validateConfig(): void {
    // when running application, this function checks that developer didn't forget to add necessary configs to configuration.service (mostly for enums)

    // Validate accessor configurations
    for (const accessor of Object.values(Accessor)) {
      this.getService(accessor as Accessor);
    }

    // Validate queue configuration not missed
    for (const queue of Object.values(Queue)) {
      this.getQueue(queue as Queue);
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

You will need to do some actions to get benefits from using Configuration service:

  • Define Enums: Developers need to create enums for different services to streamline configuration retrieval.
  • Maintain .env File: They must set up and update the .env file with all necessary variables.

Error Handling and Validation: The service includes error handling in the getService method and a validateConfig method to ensure all configurations are correct and complete.

3. Key Benefits of Using LambdaFactory

In summary, the LambdaFactory is a powerful pattern that simplifies the management of environment-specific configurations and enhances the robustness of our Lambda Communication Service, making it adaptable to both local and cloud environments. Let’s sum up advantages of this approach:

  1. Environment Agnostic: LambdaFactory seamlessly switches between local and cloud configurations, enhancing developer productivity and reducing environment-specific bugs.
  2. Configurational Flexibility: By centralizing Lambda configurations, it allows easy updates and maintenance of service configurations without altering the core logic.
  3. Enhanced Readability: The clear separation of concerns and abstraction of environment-specific details make the code more readable and maintainable.

In Conclusion

As we finish this guide on improving communication in AWS Lambda with NestJS, we’ve explored the details of creating a communication system that works well both for local development and when used in the cloud.

From understanding service design and building a robust Lambda Communication service to implementing a Lambda Factory for environment adaptability, this guide has provided a comprehensive pathway to mastering Lambda communications in a distributed system.

🔗 Explore the Demo Repository with the practical implementation of the concepts discussed.

🔍 If you’re keen to see where it all began, revisit the first part of this exploration in my article: “Local Development with AWS Lambda and NestJS: Docker, Debugging, and Hot Reload”.

🤝 Your feedback is invaluable! Feel free to drop comments, ask questions, or share your insights and optimizations. Every contribution helps to enhance our collective knowledge and build a resourceful developer community.

Happy Coding and looking forward to exploring asynchronous requests with SQS in our upcoming guide! 🚀

Top comments (0)