API Gateway integration with AWS Services.

Sharad Khambe — Tue, 29 Oct 2024 06:42:37 +0000

In this blog post we will discuss on how API gateway can be used for integration with other AWS services such as dynamodb, lambda functions, step functions, SNS etc.

What is AWS API Gateway ?

AWS API Gateway us managed service that is highly available, scalable . API gateway provides easy integration with various AWS services like lambda, dynamodb, step functions and external/internal HTTP endpoints for integration purposes. It acts as entrypoint for your API traffic.
It provides features like authorization, throttling, CORS, caching, transformations, openAPI specs, custom domain names to name few.

API Gateway High Level Architecture

Integration Type

Lambda Function: lambda function can be integrated with API gateway in 2 different ways.

First called as Lambda Proxy and second one Lambda Direct Integration. In lambda proxy integration pattern, when client makes request to api, API Gw wraps body into events and some contextual data in context and pass it both event and context to lambda function. The response from lambda function is passed as-is back to api client which means lambda function will need to prepare response in way that includes status code, http headers, body etc.This is supported by HTTP and Rest api.

Second called Lambda Direct Integration in which case we can use VTL to perform some modifications on request body or headers and pass it lambda and then in response similar modifications can be done to response. VTL is bit complex and hard to understand at first place but its very powerful tool for message transformation and enrichment between request and response.

AWS Services Integration: This pattern allows you to integrate API Gw with various AWS Services directly without using lambda as middleware. This integration lets an API expose AWS service actions. In AWS integration, you will need to configure both the integration request and integration response and set up necessary data mappings from the method request to the integration request, and from the integration response to the method response.

HTTP Endpoints Integrations: In this pattern you place your API GW in front of http endpoints be it internal or external services. This can be either HTTP or HTTP_PROXY mode integration. In HTTP Proxy mode API Gateway passes the incoming request from the client to the HTTP endpoint and passes the outgoing response from the HTTP endpoint to the client.
In HTTP mode, you can use integration or VTL mapping to update request and responses.

Mock IntegrationThis type of integration lets API Gateway return a response without sending the request further to the backend. API response is defined at API gateway and used for mock responses for testing purposes.

Private Integrations: It allows you to integrate with services running in VPC by using VPC private links through network load balancer or to on premises services using AWS direct connect. You can also integrate with application load balancer using VPC private link in this case.

Now we will look at use case on how API gateway be integrated with AWS S3 bucket to list files, download or upload file to specific bucket. We will use CDK(if you want to know more about CDK,read it here https://dev.to/smkhub/aws-cdk--45l0

Complete CDK code in javascript for this use case can be found at GitHub repo https://github.com/SMKHub/cdk-s3-api

Create directory named "cdk-s3-api" and Initialise cdk app using "cdk init app --language=javascript" from cdk-s3-api directory.

In cdk-s3-app-stack.js file under constructor function first create S3 bucket, code below will create S3 bucket named s3-api-bucket with sane default values set by cdk as best practises.

   const s3ApiBucket = new s3.Bucket(this, 's3ApiBucket', {
      bucketName: 's3-api-bucket',
      removalPolicy: cdk.RemovalPolicy.DESTROY
   }
);

Create Rest API gateway with endpoint type as edge. Since we are dealing with s3 objects we will need to set values for "binaryMediaTypes". This is required to handle binary media types and for our API we are setting it to octet-stream and image/png types.

const restApi = new apigw.RestApi(this, 's3Api', {
      restApiName: "s3Api", 
      description: "Rest API for accessing S3 bucket and its object", 
      endpointConfiguration: {
        types:[ apigw.EndpointType.EDGE]
      }, 
      binaryMediaTypes: ["application/octet-stream", "image/png"]
    });

Next define AWS Integration with S3 service. Code below define following properties for integration.

service : specifies name of AWS service to be integrated in our case it's s3
path : s3 bucket name
integrationHttpMethod : specifies integration method type.
options. passthroughBehavior : Allows pass-through when the integration has NO content types mapped to templates.
role to be used by API Gateway while making calls to S3 API's
set integration request path bucket to method path folder from incoming api call.
options. integrationResponses : Response that API Gateway provides after a method's backend completes processing a request with additional information mentioned below.
set response headers to match integration for content type, content length and date headers. this is so that when client receives response it understand response content type as it might be xml, jpg image or normal text file stored in s3 bucket.

Note that setting up integration requests parameters and response parameters is necessary for API GW to successfully call S3 apis and send response back to client

const listBucketApiIntegration = new apigw.AwsIntegration({
      service:"s3", 
      region:"us-east-1", 
      path: s3ApiBucket.bucketName, 
      integrationHttpMethod: "GET", 
      options: {
        credentialsRole:this.apiGatewayRole, 
        passthroughBehavior: apigw.PassthroughBehavior.WHEN_NO_TEMPLATES,
        requestParameters: {'integration.request.path.bucket': 'method.request.path.folder'},
        integrationResponses: [
          {
            statusCode: '200', 
            responseParameters: { 
              'method.response.header.Date': 'integration.response.header.Date',
              'method.response.header.Content-Length': 'integration.response.header.Content-Length', 
              'method.response.header.Content-Type': 'integration.response.header.Content-Type'
            }
          }
        ]
      }
    });

Next add method to list bucket objects. Here we are using IAM authorization for making call to this http GET method.

bucketRootResource.addMethod('GET', listBucketApiIntegration, 
      { 
        authorizationType: apigw.AuthorizationType.IAM, 
        requestParameters: {
          'method.request.path.folder': true
        }, 
        methodResponses: [
          {
            statusCode: '200', 
            responseParameters: 
              {
                'method.response.header.Content-Type': true,
                'method.response.header.Date': true, 
                'method.response.header.Content-Length': true
              }
          }
        ]
      }
    );

Now we will add resource under root with path as "/{object}" and then define GET and PUT method separately for downloading object and uploading object to S3 bucket.

const bucketItemResource = bucketRootResource.addResource("{item}");
const getBucketItemApiIntegration = new apigw.AwsIntegration({
      service: "s3", 
      region:'us-east-1', 
      path: s3ApiBucket.bucketName+'/{object}',
      integrationHttpMethod: "GET", 
      options: {
        credentialsRole: this.apiGatewayRole, 
        passthroughBehavior: apigw.PassthroughBehavior.WHEN_NO_TEMPLATES, 
        requestParameters: {
          'integration.request.path.bucket': 'method.request.path.folder', 
          'integration.request.path.object': 'method.request.path.item',
          'integration.request.header.Accept': 'method.request.header.Accept'
        }, 
        integrationResponses: [          
            {
              statusCode: '200', 
              responseParameters: { 
                'method.response.header.Date': 'integration.response.header.Date',
                'method.response.header.Content-Length': 'integration.response.header.Content-Length', 
                'method.response.header.Content-Type': 'integration.response.header.Content-Type'
              }

          }
        ]
      }
    });

bucketItemResource.addMethod('GET', getBucketItemApiIntegration, 
      { 
        authorizationType: apigw.AuthorizationType.IAM, 
        requestParameters: {
          'method.request.path.folder': true,
          'method.request.path.item': true, 
          'method.request.header.Accept': true
        }, 
        methodResponses: [
          {
            statusCode: '200', 
            responseParameters: 
              {
                'method.response.header.Content-Type': true,
                'method.response.header.Date': true, 
                'method.response.header.Content-Length': true
              }
          }
        ]
      }
    );

Run cdk synth to validate if code is fine and then run cdk deploy to create stack resources in AWS account.

Integration response and request parameters that we have mapped in cdk code can be seen in API GW console as below.

Testing of API can be done using postman. Note that you will need to set access key and secret access key as API Gw authentication is set to IAM in CDK construct.

Finally don't forget to delete resources from AWS account using "cdk destroy" command. Note you might need to delete objects from S3 bucket manually before cdk destroy command if you have uploaded objects as part of testing API else stack deletion will fail stating S3 bucket is not empty.

AWS CDK

Sharad Khambe — Thu, 17 Oct 2024 14:17:01 +0000

In this blog, we will cover up basic of what is AWS CDK, it's features, use cases, cdk commands, how to use them and deploy into AWS.

AWS Cloud Development Kit

AWS Cloud Development Kit is open source development framework used to model and provision cloud resources using familiar programming languages such as JS, TS, Python, Go etc.
You can use any of supported languages to define your cloud resources and when deployed those resources are provisioned through AWS CloudFormation.

Key Benifits of using CDK

Familiar : Benefits of using programming language of your's/team's choice. This gives advantage of using same programming language for your infrastructure build as well as for your application development.

Tools Support : Lot of languages have in built support for linting, testing, debugging which is inherited by cdk framework.

Abstraction : CDK provides secure first defaults properties for most of resources according best practices and implementation. e.g. when you create S3 bucket it will have public access disabled by default. However you have option to set each properties as well if you chose to do so.

CDK Main Components

1. Core Framework : With core framework you can create App which can contain one more multiple structured components called as Stacks.
Stacks are logical unit of infrastructure which contains one or many resources. Resources are mapped one-to-one to Cloudformation resources. Stack in CDK is one to one relationship with CloudFormation stack. Similar to CloudFormation you can have nested stack in CDK(we will discuss this in another blog with example!).
It's good practice to divide resources into stack that have different purpose or lifecycles. E.g. stateless(SQS) vs stateful(RDS) OR front-end applications Vs backend API's for front end.

2. AWS Construct Library: AWS construct library is set of components designed and created by AWS to create resources for specific services. These are build considering best practices and security implementation for given services. AWS constructs are categorised into three levels

Level 1(L1) constructs : These constructs are mapped one to one with single CloudFormation resource specification. E.g. S3 bucket. They are named starting with "Cfn". e.g. CfnBucket

Level 2(L2) constructs : These constructs provide higher level abstraction compared to L1 constructs. L2 constructs is very simple and requires very little input to generate complex CloudFormation templates. However they can have many defaults properties and options to be set.

Level 3(L3) constructs : These constructs are known as patterns and it can contain collection of resources that are configured to work together to achieve specific functionality. E.g. Appsync API with dynamoDB and step functions integration.

3. AWS CDK CLI : CLI commands helps to interact with core framework such as initialise our project, inspect differences between various deployments versions. Let's look at workflow on how to use CLI to develop infrastructure and deployment into AWS.

Development and Deployment Workflow

cdk bootstrap
CDK requires specific dedicated resources to be created for you to be able to deploy resources using cdk. cdk bootstrap cli command helps with achieving necessary preconfiguration to be done. This is just one time activity.

Workflow

Create new project with name of directory e.g. "cdk-todo-app-api".
🔄 cdk init app --language=javascript

Build Project
🦾 npm run build

Create templates from given code
🧬 cdk synth

Validate change set that will result as part of deployment
🔎 cdk diff

Deploy changes to resources into cloud.
🚀 cdk deploy

DEV Community: Sharad Khambe