DEV Community: Rahul Lokurte

How to Validate Requests to the AWS API Gateway using CDK

Rahul Lokurte — Sat, 28 Aug 2021 21:08:53 +0000

Amazon API Gateway is an AWS service that helps to create REST and WebSocket APIs easily. It acts as the first entry point for applications to access data, business logic, or functionality from your backend services. It handles traffic management, authorization and access control, monitoring, and API version management.

If you have written backend services, you must have written lots of request validations and transformation logics in the backend services. When the request comes to the backend service, you validate the request and do the transformations and return the response. Can we avoid the request coming to the backend if it is invalid?. The answer to this question is provided by AWS API Gateway.

This blog post will show how to validate the request before it goes to the backend service. We will use the AWS Lambda function as a backend service and expose API with querystring parameters and request body validations.

Create API Gateway and Lambda using CDK

we will create an API Gateway and integrate it with Lambda Function using AWS CDK.

Create a new directory on your system.



mkdir cdk-apigateway && cd cdk-apigateway

We will use cdk init to create a new Javascript CDK project:



cdk init --language javascript

The cdk init command creates a number of files and folders inside the cdk-apigateway directory to help us organize the source code for your AWS CDK app.

We can list the stacks in our app by running the below command. It will show CdkApigatewayStack.



cdk ls

Let us install the AWS Lambda and API gateway construct.



npm install @aws-cdk/aws-lambda @aws-cdk/aws-apigateway

Edit the file lib/cdk-apigateway-stack.js to create an API Gateway and AWS lambda resource as shown below.



const cdk = require("@aws-cdk/core");
const apigateway = require("@aws-cdk/aws-apigateway");
const lambda = require("@aws-cdk/aws-lambda");

class CdkApigatewayStack extends cdk.Stack {
  /**
   *
   * @param {cdk.Construct} scope
   * @param {string} id
   * @param {cdk.StackProps=} props
   */
  constructor(scope, id, props) {
    super(scope, id, props);

    const greet = new lambda.Function(this, "GreetHandler", {
      runtime: lambda.Runtime.NODEJS_14_X,
      code: lambda.Code.fromAsset("lambda"),
      handler: "greet.handler",
      functionName: "greet",
    });

    const greetApi = new apigateway.LambdaRestApi(this, "GreetLambda", {
      handler: greet,
      proxy: false,
      restApiName: "greet-api",
    });

    const greetApiIntegration = new apigateway.LambdaIntegration(greet);
    const items = greetApi.root.addResource("greet");
    const users = greetApi.root.addResource("user");
    items.addMethod("GET", greetApiIntegration);
    users.addMethod("POST", greetApiIntegration);
  }
}

module.exports = { CdkApigatewayStack };

Now, we can deploy them using cdk deploy and then verify resources are created in AWS Console.

Request Validation

API Gateway can perform the basic validation. When the validation fails, API Gateway immediately fails the request, returns a 400 error response to the calling application. The AWS API Gateway contains three types of validators:

Validate body - This validator will only validate the body of the request.
Validate query string parameters and headers - This validator will only validate the parameters coming in the request.
Validate body, query string parameters, and headers - This validator will validate both body and parameters of the request.

Validate query string parameters

Let us see, how to validate the query string parameter.

If you see the below lambda code, it is expecting us to pass a query string named greetName



if (event.httpMethod === "GET") {
    let response = {
      statusCode: 200,
      body: `Hello ${event.queryStringParameters.greetName}. Welcome to CDK`,
    };
    return response;
  }

If we do not add the validations at the API Gateway level, the request goes to the lambda function and we get undefined for the variable greetName in the response.

Now, let us edit lib/cdk-apigateway-stack.js to add the request validation for the query string parameter to the API Gateway using CDK as shown below.



items.addMethod("GET", greetApiIntegration, {
      authorizationType: apigateway.AuthorizationType.IAM,
      requestParameters: {
        "method.request.querystring.greetName": true,
      },
      requestValidatorOptions: {
        requestValidatorName: "querystring-validator",
        validateRequestParameters: true,
        validateRequestBody: false,
      },
    });

We have added requestParameters where we need to specify required request parameters as a key-value pair. A key must match the format method.request.location.name, where the location may be querystring, path or header depending on what we want to validate. In our case, it is querystring which we want to validate. The name is the valid, unique parameter name. In our case, it is greetName. The Value is the boolean, which indicates, if the parameter is required or not. In our case, we want the request to contain a querystring, hence it is set to true.

Now deploy the code using the command cdk deploy and verify if the validation is added in the AWS console as shown below.

Now, let us send an invalid request, we immediately get an error indicating that the required querystring is not passed in the request.

The same steps can be followed for pathparameters and the headers validation.

Validate Request body

Let us see, how to validate the body of the request. This is a little bit complex than validating the querystring and headers.

If you see the below lambda code, it is expecting us to have username, department, departmentName.



if (event.httpMethod === "POST") {
    let request = JSON.parse(event.body);
    let username = request.username;
    let department = request.department;
    let departmentName = department.departmentName;

    let user = { username: username, departmentName: departmentName };
    const response = {
      statusCode: 200,
      body: JSON.stringify(user),
    };
    return response;
  }

We need to first create a schema model to validate the request. Let us add the below schema.



const greetModel = new apigateway.Model(this, "model-validator", {
      restApi: greetApi,
      contentType: "application/json",
      description: "To validate the request body",
      modelName: "greetmodelcdk",
      schema: {
        type: JsonSchemaType.OBJECT,
        required: ["username"],
        properties: {
          username: { type: "string" },
          department: {
            type: "object",
            properties: {
              departmentName: { type: "string" },
            },
          },
        },
      },
    });

Now, let us edit lib/cdk-apigateway-stack.js to add the request validation to the API Gateway using CDK as shown below.



users.addMethod("POST", greetApiIntegration, {
      requestValidator: new apigateway.RequestValidator(
        this,
        "body-validator",
        {
          restApi: greetApi,
          requestValidatorName: "body-validator",
          validateRequestBody: true,
        }
      ),
      requestModels: {
        "application/json": greetModel,
      },
    });

Let us deploy the code using cdk deploy and verify if resources are created in the AWS console.

The model schema is created as shown below.

Also, verify that the API resource has the model schema attached to it as shown below.

Let us test different scenarios for the request body validation.

Valid Request:



{
  "username": "Rahul",
  "department": {
    "departmentName": "Engineer"
  }
}

We get a valid response as seen below.

Invalid type for departmentName:



{
  "username": "Rahul",
  "department": {
    "departmentName": 1
  }
}

Here, we are passing the Integer value for departmentName. The schema we defined accepts a string for departmentName. So, we get 400 Bad requests with the message as shown below.

Missing Required Field:



{
  "department": {
    "departmentName": 1
  }
}

Here, we are missing the field username. The schema we defined has a mandatory field username. So, we get 400 Bad requests with the message as shown below.

Conclusion

In this blog post, we saw what are request validators in AWS API Gateway. We learned about how to create a validator for querystring and the request body using cloud development kit (CDK).

GitHub Repo: https://github.com/rahulmlokurte/aws-usage/tree/main/aws-cdk/cdk-apigateway

Ice-cream Flavor Picker using AWS Event Bridge

Rahul Lokurte — Wed, 18 Aug 2021 18:53:56 +0000

In a distributed microservice architecture, we have many services, each doing one thing and each service interacts with other services. Initially, it looks like, each service is independent and we can deploy the changes faster. But as the application starts to grow, it tends to have a lot of interactions between services making the whole application tightly coupled.

In this blog, we will see what event bridge is and how it overcomes some of the shortcomings of microservice architectures.

A Tale of Microservice

A client makes a request to the order service, which then, in turn, sends a request to the delivery service. The delivery service returns 201 as a successful response to the order service and then the order service sends a response to the client. We all start from here and it looks like each service is decoupled from one another.

What happens when we add more services that integrate them with order service. Now as you see, the order service will have more responsibilities.

Order service will have to coordinate with each of the services. If there are any retry scenarios, the order service has to handle them. When one of the services has updated, the order service needs to update the same, so now, other services are waiting for the update to complete. What if we add a new service and order service is not aware of this new service? How about reverting the transactions? What about if one of the services is not available. So, now, we are in a complex situation. Microservice does not look like our friend.

Event Bridge

Amazon EventBridge is a serverless event bus service that you can use to connect your applications with data from a variety of sources. Events are central to EventBridge and the events are observable. Previously in a microservice architecture, we had an API, which would instruct each service on what action to take. But with events being observable, now, we just notify that some changes have happened to the event bus. Other consumers can listen to the event bus and take the action based on their use case.

As we see, when an order has been placed, the event is triggered on the event bridge. Other consumers like inventory service check the inventory and the delivery service checks the delivery. We can also have other consumers subscribed to the same event bridge and are not interested in the event, so they won't take any action. So, tight coupling which we saw in a microservice architecture is minimized.

Event bridge also supports integration with many SAAS-based applications and third-party applications. It also supports sending events to other AWS services like Lambda functions, step functions.

key benefits of Event Bridge

Decoupling of applications
Simplified event routing
Improved availability
Third-party integrations

Terminology used in Event Bridge

Events - An event indicates a change. This can be a change in AWS services, a third-party service, or one of our own custom applications.
Event buses - An event bus receives events. When you create a rule, you associate it with a specific event bus, and the rule is matched only to events received by that event bus. By default, the AWS account has a default event bus, which receives events from AWS services. We can create custom event buses to receive events. You can also create partner event buses to receive events from third-party applications.
Rules - A rule matches incoming events and routes them to specific targets for further processing. A single rule can route to multiple targets. Processing of Rules is not in any order, they can even process parallelly. A rule can also filter or enhance the JSON sent to the target service.
Targets - A target is the one that does the processing of events. Targets can be Lambda functions, Step Function state machines, or any built-in target. A target receives events in JSON format.

Icecream Flavor Picker: An Example

Let us see how we can create an Event Bus, define the Rules and the targets using the Icecream Flavor Picker Example.

The step function is used to trigger the eventbridge. EventBridge defines the rules, which will execute a specific lambda function. If the input to the step function is banana, the eventbridge banana rule executes, which in turn, triggers the banana lambda function. If the input to the step function is almond, the eventbridge almond rule executes, which in turn, triggers the almond lambda function.

Create a Step Function

Create a state machine with the name IcecreamFlavourStateMachine.
Define the step function as shown below.

{
  "Comment": "A Ice Cream Flavor Chooser",
  "StartAt": "Create IceCream flavor",
  "States": {
    "Create IceCream flavor": {
      "Type": "Pass",
      "Next": "Flavor Choose"
    },
    "Flavor Choose": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.icecreamFlavor.nut",
          "StringEquals": "almond",
          "Next": "Almond Flavor"
        },
        {
          "Variable": "$.icecreamFlavor.nut",
          "StringEquals": "banana",
          "Next": "Banana Flavor"
        }
      ],
      "Default": "DefaultState"
    },
    "Almond Flavor": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [
          {
            "Detail.$": "$.icecreamFlavor",
            "DetailType": "almond_flavor",
            "EventBusName": "Ice_cream_flavour",
            "Source": "IcecreamFlavor"
          }
        ]
      },
      "End": true
    },
    "Banana Flavor": {
      "Type": "Task",
      "Resource": "arn:aws:states:::events:putEvents",
      "Parameters": {
        "Entries": [
          {
            "Detail.$": "$.icecreamFlavor",
            "DetailType": "banana_flavor",
            "EventBusName": "Ice_cream_flavour",
            "Source": "IcecreamFlavor"
          }
        ]
      },
      "End": true
    },
    "DefaultState": {
      "Type": "Fail",
      "Cause": "No Matches!"
    }
  }
}

Depending upon the type of flavor selected, it puts the events to a specific event Rule.

Create an EventBus and Event Rules

Event buses receive events from a variety of sources and match them to rules in your account.

Create an event bus with the name Ice_cream_flavour
Create a Rule with the name Almond_Rule
Select the custom pattern as shown below

{
  "source": ["IcecreamFlavor"],
  "detail-type": ["almond_flavor"]
}

Select the Target Lambda Function Almond_Icecream which we will create later.
Create a Rule with the name Banana_Rule
Select the custom pattern as shown below

{
  "source": ["IcecreamFlavor"],
  "detail-type": ["banana_flavor"]
}

Select the Target Lambda Function Banana_Icecream which we will create later.

Create lambda functions

Create a lambda function with the name Almond_Icecream and Banana_Icecream
Change the code in index.js as below.

exports.handler = async (event) => {
  console.log(event);
  var nutType = event.detail.nut;
  console.log("The type of Flavor is: ", nutType);
  return nutType;
};

Test the application

Let us send the following almond JSON to IcecreamFlavourStateMachine step function

{
  "icecreamFlavor": {
    "nut": "almond"
  }
}

When we start the execution, we see that it triggers the almond event bridge rule as can be seen below.

We also see the event Id ae9a19c8-72d7-e5e9-1fa0-87c3f4666faa, using which we can verify that Almond_Icecream lambda function is called.

Let us send the following banana JSON to IcecreamFlavourStateMachine step function

{
  "icecreamFlavor": {
    "nut": "banana"
  }
}

When we start the execution, we see that it triggers the banana event bridge rule as can be seen below.

We also see the event Id fcbfbe74-8a05-00fe-6aa5-01793b4c2c1f, using which we can verify that Banana_Icecream lambda function is called.

Conclusion

In this blog post, we looked at what event bridge is and how to create rules and apply a pattern to the rules. We have seen the example application of Icecream Flavour Picker using step functions, event bridge, and lambda functions. To know more about the event bridge, step functions and lambda functions head over to official documentation https://docs.aws.amazon.com/.

AWS Lambda using CDK

Rahul Lokurte — Thu, 12 Aug 2021 17:20:38 +0000

Infrastructure as code has become a go-to process to automatically provision and manage cloud resources. AWS provides two options for infrastructure as code.

AWS CloudFormation
AWS Cloud Development Kit

With CloudFormation, we have to write a lot of YAML templates or JSON files. As AWS adds more services, we have to add more files to CloudFormation. It becomes difficult to work with lots of files. YAML/JSON is based on data serialization and not an actual programming language. The AWS CDK will overcome the limitations of cloud formation by enabling the reuse of code and proper testing.

AWS CDK is a framework that allows developers to use familiar programming languages to define AWS cloud infrastructure and provision it. CDK provides the Constructs cloud component that cover many of the AWS services and features. It helps us to define our application infrastructure at high level.

we will create a Lambda Function and the infrastructure around lambda function using AWS CDK.

Create a new directory on your system.

mkdir cdk-greetapp && cd cdk-greetapp

We will use cdk init to create a new Javascript CDK project:

cdk init --language javascript

The cdk init command creates a number of files and folders inside the cdk-greetapp directory to help us organize the source code for your AWS CDK app.

We can list the stacks in our app by running the below command. It will show CdkGreetappStack.

$ cdk ls
CdkGreetappStack

Let us install AWS lambda construct library.

npm install @aws-cdk/aws-lambda

Edit the file lib/cdk-greetapp-stack.js to create an AWS lambda resource as shown below.

const cdk = require("@aws-cdk/core");
const lambda = require("@aws-cdk/aws-lambda");

class CdkGreetappStack extends cdk.Stack {
  /**
   *
   * @param {cdk.Construct} scope
   * @param {string} id
   * @param {cdk.StackProps=} props
   */
  constructor(scope, id, props) {
    super(scope, id, props);
    // defines an AWS Lambda resource
    const greet = new lambda.Function(this, "GreetHandler", {
      runtime: lambda.Runtime.NODEJS_14_X,
      code: lambda.Code.fromAsset("lambda"),
      handler: "greet.handler",
    });
  }
}

module.exports = { CdkGreetappStack };

Lambda Function uses NodeJS 14.x runtime
The handler code is loaded from the directory named lambda where we will add the lambda code.
The name of the handler function is greet.handler where greet is the name of file and handler is exported function name.

Lets create a directory name lambda in root folder and add a file greet.js.

mkdir lambda
cd lambda
touch greet.js

Add the lambda code to greet.js

exports.handler = async function (event) {
  console.log("request:", JSON.stringify(event, undefined, 2));
  let response = {
    statusCode: 200,
    body: `Hello ${event.path}. Welcome to CDK!`,
  };
  return response;
};

Before deploying the AWS resource, we can take a look on what resources will be getting created by using below command.

cdk diff

NOTE: If we have multiple profiles set in our system, we need to tell cdk to look into particular profile. This can be done, by adding below key-value in cdk.json which was generated when we created a CDK project.

"profile": "<YOUR_PROFILE_NAME>"

Now, once we are ok with the resources which will be created, we can deploy it using below command

cdk deploy

Let us open the AWS Lambda console

Select Amazon API Gateway AWS Proxy from the Event template list.

Click on Test, we can see that, we get the proper response as shown below.

Conclusion

we saw how to create a lambda function and also lambda resource by using AWS Cloud Development Kit. We also saw various commands related to CDK for initiating projects, deploying the resources to AWS. The code repository link is here

If you want to do hands-on, I have created a Youtube Video. You can check out here

Build Serverless Api Using lambda-api

Rahul Lokurte — Fri, 11 Jun 2021 00:00:00 +0000

Serverless application can be easily created with AWS lambda and AWS Gateway. We can write an lambda function and configure the gateway. We get a Rest endpoint which we can call to reach the lambda function. This approach is easy, but comes with a caveats. We have to configure API gateway, manage the deployments, publish to stages.

In this blog, we will see how to develop and deploy a serverless application using
Lambda API, a lightweight framework for developing serverless applications.

Lets create a new project directory.

mkdir serverless-samples
cd serverless-samples

We will initiate the project and Install lambda-api.

npm init
npm i lambda-api

The Lambda-api just needs the 3 steps.

Require the framework and Instantiate it.
Define a route.
Declare our Lambda Handler.

Create a new handler.js file and add the below contents.

//----------------------------------------------------------------------------//
// Require the framework
//----------------------------------------------------------------------------//

const app = require("lambda-api")({ version: "v1.0", base: "v1" });

//----------------------------------------------------------------------------//
// Define an API routes
//----------------------------------------------------------------------------//

// GET
app.get("/greet/:name", (req, res) => {
  // Send the response
  res.status(200).json({
    //name: req.params,
    status: "ok",
    body: "Hello " + req.params.name,
  });
});

//----------------------------------------------------------------------------//
// Lambda handler
//----------------------------------------------------------------------------//
module.exports.router = (event, context, callback) => {
  context.callbackWaitsForEmptyEventLoop = false;

  // Run the request
  app.run(event, context, callback);
}; // end router handler

In the above code, we have created an GET endpoint with /greet/:name. The Path parameter is name which will be passed to the GET endpoint. The API will return the response with statuscode as 200 and the body will greet a person with name that is passed in the pathparams. To run the request, we use app.run() method.

Deploy to AWS

Create a zip file of the project and upload the zip file to function serverless-sample and change the handler to handler.router.

Invoke the function with a test event. We need to choose the template that matches the service that triggers our function or we can also create a event document in Json.

As shown above, we have to select a New event option and select apigateway-aws-proxy template and give an test event name as gatewaysample. We will get a sample event which is compatible with the api-gateway.

Now test the endpoint by adding values to below keys.

"resource": "/v1/greet/rahul"
"path": "/v1/greet/rahul"
"httpMethod": "GET"

Once configured, click on Test button, We get the below greeting for the name passed into the function.

Getting Started with Apache Pulsar

Rahul Lokurte — Wed, 22 Jul 2020 16:56:49 +0000

Prerequisite

To run apache pulsar, you need to have the Docker installed on the system. You can download docker from https://docs.docker.com/docker-for-windows/install/

Getting Started

Note: Apache pulsar has a cluster and within cluster, we have tenants. Within Tenants, we have namespaces.

Start the pulsar instance in Docker

docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.6.0 bin/pulsar standalone

Check the container id of the pulsar

docker ps -a

Execute into the container bash

docker exec -it <container-id> bash

Get the cluster list

bin/pulsar-admin clusters list

Get the tenant list

bin/pulsar-admin tenants list

Get the namespaces list

bin/pulsar-admin namespaces list <tenant-name>

Get the topics

bin/pulsar-admin topics list <tenant-name>/<namespace-name>

Create a topic

bin/pulsar-admin topics create-partitioned-topic persistent://public/default/test-topic -p 1

To produce a message to the topic

bin/pulsar-client produce persistent://public/default/test-topic --num-produce 1 --messages "Hello pulsar 1"

To Consume a message from the topic

bin/pulsar-client consume persistent://public/default/test-topic --num-messages 0 --subscription-name test-topic-sub --subscription-type Exclusive

NOTE: All the above commands assume, you are working on default tenants and default namespaces.