In Depth Guide to Serverless APIs with AWS Lambda and AWS API Gateway (Part 2)

TL;DR The repository with an example project can be found on GitHub

This is the last article in a two-part series about building a serverless API with AWS technology.

In the first part, we learned about authentication, request bodies, status codes, CORS and response headers. We set up an AWS SAM project that connected API-Gateway, Lambda, and Cognito so users could sign up and in.

In this article, we will talk about third-party integration, data storage, and retrieval and "calling Lambda functions from Lambda functions" (we will not actually do this, but learn about a method to achieve something similar.

Adding the Image upload

We will start with the image upload. It works as following:

1. Request a pre-signed S3 URL via our API
2. Upload an image directly to S3 via the pre-signed URL

So we need two new SAM/CloudFormation resources. A Lambda function that generates the pre-signed URL and an S3 bucket for our images.

Let's update the SAM template:

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: "A example REST API build with serverless technology"

Globals:

  Function:
    Runtime: nodejs8.10
    Handler: index.handler
    Timeout: 30 
    Tags:
      Application: Serverless API

Resources:

  ServerlessApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      Cors: "'*'"
      Auth:
        DefaultAuthorizer: CognitoAuthorizer
        Authorizers:
          CognitoAuthorizer:
            UserPoolArn: !GetAtt UserPool.Arn
      GatewayResponses:
        UNAUTHORIZED:
          StatusCode: 401
          ResponseParameters:
            Headers:
              Access-Control-Expose-Headers: "'WWW-Authenticate'"
              Access-Control-Allow-Origin: "'*'"
              WWW-Authenticate: >-
                'Bearer realm="admin"'

  # ============================== Auth ==============================
  AuthFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/auth/
      Environment:
        Variables:
          USER_POOL_ID: !Ref UserPool
          USER_POOL_CLIENT_ID: !Ref UserPoolClient
      Events:
        Signup:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /signup
            Method: POST
            Auth:
              Authorizer: NONE
        Signin:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /signin
            Method: POST
            Auth:
              Authorizer: NONE

  PreSignupFunction:
    Type: AWS::Serverless::Function
    Properties:
      InlineCode: |
        exports.handler = async event => {
          event.response = { autoConfirmUser: true };
          return event;
        };

  UserPool:
    Type: AWS::Cognito::UserPool
    Properties:
      UserPoolName: ApiUserPool
      LambdaConfig:
        PreSignUp: !GetAtt PreSignupFunction.Arn
      Policies:
        PasswordPolicy:
          MinimumLength: 6

  UserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      UserPoolId: !Ref UserPool
      ClientName: ApiUserPoolClient
      GenerateSecret: no

  LambdaCognitoUserPoolExecutionPermission:
    Type: AWS::Lambda::Permission
    Properties: 
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt PreSignupFunction.Arn
      Principal: cognito-idp.amazonaws.com
      SourceArn: !Sub 'arn:${AWS::Partition}:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${UserPool}'

  # ============================== Images ==============================
  ImageBucket:
    Type: AWS::S3::Bucket
    Properties: 
      AccessControl: PublicRead
      CorsConfiguration:
        CorsRules:
          - AllowedHeaders:
              - "*"
            AllowedMethods:
              - HEAD
              - GET
              - PUT
              - POST
            AllowedOrigins:
              - "*"

  ImageBucketPublicReadPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref ImageBucket
      PolicyDocument:
        Statement:
          - Action: s3:GetObject
            Effect: Allow
            Principal: "*"
            Resource: !Join ["", ["arn:aws:s3:::", !Ref "ImageBucket", "/*" ]]

  ImageFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/images/
      Policies:
        - AmazonS3FullAccess
      Environment:
        Variables:
          IMAGE_BUCKET_NAME: !Ref ImageBucket
      Events:
        CreateImage:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /images
            Method: POST

Outputs:

  ApiUrl:
    Description: The target URL of the created API
    Value: !Sub "https://${ServerlessApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
    Export:
      Name: ApiUrl

Okay, we ended up with three new resources. We also needed a BucketPolicy that allows public reads on our new image bucket.

The ImagesFunction has an API event so we can handle POST requests with it. The function gets an S3 access policy and an environment variable so it knows the ImageBucket.

We need to create a new file for the function code functions/images/index.js.

const AWS = require("aws-sdk");

exports.handler = async event => {
  const userName = event.requestContext.authorizer.claims["cognito:username"];
  const fileName = "" + Math.random() + Date.now() + "+" + userName;
  const { url, fields } = await createPresignedUploadCredentials(fileName);
  return {
    statusCode: 201,
    body: JSON.stringify({
      formConfig: {
        uploadUrl: url,
        formFields: fields
      }
    }),
    headers: { "Access-Control-Allow-Origin": "*" }
  };
};

const s3Client = new AWS.S3();
const createPresignedUploadCredentials = fileName => {
  const params = {
    Bucket: process.env.IMAGE_BUCKET_NAME,
    Fields: { Key: fileName }
  };
  return new Promise((resolve, reject) =>
    s3Client.createPresignedPost(params, (error, result) =>
      error ? reject(error) : resolve(result)
    )
  );
};

So, what happens in this function?

First, we extract the userName from the event object. API-Gateway and Cognito control access to our function, so we can be sure that a user exists in the event object when the function is called.

Next, we create a unique fileName based on a random number, the current time-stamp, and the username.

The createPresignedUploadCredentials helper function will create the pre-signed S3 URL. It will return an object with url and fields attributes.

Our API client has to send a POST request to the url and include all the fields and the file in its body.

Integrate the Image Recognition

Now, we need to integrate the third-party image recognition service.

When an image is uploaded S3 will fire an upload event that will be handled by a lambda function.

This brings us to one of the questions from the beginning of the first article.

How to call a Lambda from a Lambda?

Short answer: You don't.

Why? While you could call a Lambda directly from a Lambda via the AWS-SDK, it brings a problem. We would have to implement all the things that need to happen if something goes wrong, like retries and such. Also, in some cases, the calling Lambda would have to wait for the called Lambda to finish and we would have to pay for the waiting time too.

So what are the alternatives?

Lambda is an event-based system, our functions can be triggered by different event sources. The main course of action is to use another service to do so.

In our case, we want to call a Lambda when the file upload finished, so we have to use an S3 event as a source. But there are other event sources as well.

• Step Functions lets us coordinate Lambda functions with state machines
• SQS is a queue where we can push our results to so they can be picked up by other Lambdas
• SNS is a service that allows us to fan-out one Lambda result to many other Lambdas in parallel.

We add a new Lambda function to our SAM template that will be called by S3.

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: "An example REST API build with serverless technology"

Globals:

  Function:
    Runtime: nodejs8.10
    Handler: index.handler
    Timeout: 30 
    Tags:
      Application: Serverless API

Resources:

  ServerlessApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      Cors: "'*'"
      Auth:
        DefaultAuthorizer: CognitoAuthorizer
        Authorizers:
          CognitoAuthorizer:
            UserPoolArn: !GetAtt UserPool.Arn
      GatewayResponses:
        UNAUTHORIZED:
          StatusCode: 401
          ResponseParameters:
            Headers:
              Access-Control-Expose-Headers: "'WWW-Authenticate'"
              Access-Control-Allow-Origin: "'*'"
              WWW-Authenticate: >-
                'Bearer realm="admin"'

  # ============================== Auth ==============================
  AuthFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/auth/
      Environment:
        Variables:
          USER_POOL_ID: !Ref UserPool
          USER_POOL_CLIENT_ID: !Ref UserPoolClient
      Events:
        Signup:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /signup
            Method: POST
            Auth:
              Authorizer: NONE
        Signin:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /signin
            Method: POST
            Auth:
              Authorizer: NONE

  PreSignupFunction:
    Type: AWS::Serverless::Function
    Properties:
      InlineCode: |
        exports.handler = async event => {
          event.response = { autoConfirmUser: true };
          return event;
        };

  UserPool:
    Type: AWS::Cognito::UserPool
    Properties:
      UserPoolName: ApiUserPool
      LambdaConfig:
        PreSignUp: !GetAtt PreSignupFunction.Arn
      Policies:
        PasswordPolicy:
          MinimumLength: 6

  UserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      UserPoolId: !Ref UserPool
      ClientName: ApiUserPoolClient
      GenerateSecret: no

  LambdaCognitoUserPoolExecutionPermission:
    Type: AWS::Lambda::Permission
    Properties: 
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt PreSignupFunction.Arn
      Principal: cognito-idp.amazonaws.com
      SourceArn: !Sub 'arn:${AWS::Partition}:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${UserPool}'

  # ============================== Images ==============================
  ImageBucket:
    Type: AWS::S3::Bucket
    Properties: 
      AccessControl: PublicRead
      CorsConfiguration:
        CorsRules:
          - AllowedHeaders:
              - "*"
            AllowedMethods:
              - HEAD
              - GET
              - PUT
              - POST
            AllowedOrigins:
              - "*"

  ImageBucketPublicReadPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref ImageBucket
      PolicyDocument:
        Statement:
          - Action: s3:GetObject
            Effect: Allow
            Principal: "*"
            Resource: !Join ["", ["arn:aws:s3:::", !Ref "ImageBucket", "/*" ]]

  ImageFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/images/
      Policies:
        - AmazonS3FullAccess
      Environment:
        Variables:
          IMAGE_BUCKET_NAME: !Ref ImageBucket
      Events:
        CreateImage:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /images
            Method: POST

  TagsFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/tags/
      Environment:
        Variables:
          PARAMETER_STORE_CLARIFAI_API_KEY: /serverless-api/CLARIFAI_API_KEY_ENC
      Policies:
        - AmazonS3ReadOnlyAccess # Managed policy
        - Statement: # Inline policy document
          - Action: [ 'ssm:GetParameter' ]
            Effect: Allow
            Resource: '*'
      Events:
        ExtractTags:
          Type: S3
          Properties:
            Bucket: !Ref ImageBucket
            Events: s3:ObjectCreated:*

Outputs:

  ApiUrl:
    Description: The target URL of the created API
    Value: !Sub "https://${ServerlessApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
    Export:
      Name: ApiUrl

We add a new serverless function resource that has an S3 event that is called when a new S3 object was created in our ImageBucket.

The need to call a third-party API, namely the Clarifai API, in our Lambda function brings us to the next question.

How store Credentials for third-party APIs?

There are many ways to do so.

One way is to encrypt the credentials with KMS. We encrypt it via CLI, add the encrypted key to the template.yaml as an environment variable and then, inside the Lambda we use the AWS-SDK to decrypt the credentials before we use them.

Another way is to use the Parameter Store of the AWS Systems Manager. This service allows storing encrypted strings that can be retrieved via the AWS-SDK inside a Lambda. We just need to provide our Lambda with the name that defines where the credentials are stored.

In this example, we are going to use the Parameter Store.

If you haven't created a Clarifai account, now is the time. They will supply you with an API key we next need to store into the Parameter Store.

aws ssm put-parameter \
--name "/serverless-api/CLARIFAI_API_KEY" \
--type "SecureString" \ 
--value "<CLARIFAI_API_KEY>"

This command will put the key in the Parameter Store and encrypt it.

Next, we need to tell our Lambda function about the name via an environment variable and give it permission to call getParameter via the AWS-SDK.

Environment:
        Variables:
          PARAMETER_STORE_CLARIFAI_API_KEY: /serverless-api/CLARIFAI_API_KEY_ENC
      Policies:
        - Statement:
          - Action: [ 'ssm:GetParameter' ]
            Effect: Allow
            Resource: '*'

Let's look at the JavaScript side of things, for this, we create a new file functions/tags/index.js.

const AWS = require("aws-sdk");
const Clarifai = require("clarifai");

exports.handler =  async event => {
  const record = event.Records[0];
  const bucketName = record.s3.bucket.name;
  const fileName = record.s3.object.key;
  const tags = await predict(`https://${bucketName}.s3.amazonaws.com/${fileName}`);

  await storeTagsSomewhere({ fileName, tags });
};

const ssm = new AWS.SSM();
const predict = async imageUrl => {
  const result = await ssm.getParameter({
    Name: process.env.PARAMETER_STORE_CLARIFAI_API_KEY, 
    WithDecryption: true
  }).promise();

  const clarifaiApp = new Clarifai.App({
    apiKey: result.Parameter.Value
  });

  const model = await clarifaiApp.models.initModel({
    version: "aa7f35c01e0642fda5cf400f543e7c40",
    id: Clarifai.GENERAL_MODEL
  });

  const clarifaiResult = await model.predict(imageUrl);

  const tags = clarifaiResult.outputs[0].data.concepts
    .filter(concept => concept.value > 0.9)
    .map(concept => concept.name);
  return tags;
};

The handler is called with an S3 object creation event. There will be only one record, but we could also tell S3 to batch records together.

Then we create an URL for the newly created image and give it to the predict function.

The predict function uses our PARAMETER_STORE_CLARIFAI_API_KEY environment variable to get the name of the parameter inside the Parameter Store. This allows us to change the target parameter without a change to the Lambda code.

We get the API key decrypted and can do the calls to the third-party API. Then we store the tags somewhere.

Listing and deleting tagged Images

Now that we can upload images and they get automatically tagged, the next step would be to list all the images, filter them by tags and delete them if not needed anymore.

Let's update the SAM template!

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: "An example REST API build with serverless technology"

Globals:

  Function:
    Runtime: nodejs8.10
    Handler: index.handler
    Timeout: 30 
    Tags:
      Application: Serverless API

Resources:

  ServerlessApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      Cors: "'*'"
      Auth:
        DefaultAuthorizer: CognitoAuthorizer
        Authorizers:
          CognitoAuthorizer:
            UserPoolArn: !GetAtt UserPool.Arn
      GatewayResponses:
        UNAUTHORIZED:
          StatusCode: 401
          ResponseParameters:
            Headers:
              Access-Control-Expose-Headers: "'WWW-Authenticate'"
              Access-Control-Allow-Origin: "'*'"
              WWW-Authenticate: >-
                'Bearer realm="admin"'

  # ============================== Auth ==============================
  AuthFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/auth/
      Environment:
        Variables:
          USER_POOL_ID: !Ref UserPool
          USER_POOL_CLIENT_ID: !Ref UserPoolClient
      Events:
        Signup:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /signup
            Method: POST
            Auth:
              Authorizer: NONE
        Signin:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /signin
            Method: POST
            Auth:
              Authorizer: NONE

  PreSignupFunction:
    Type: AWS::Serverless::Function
    Properties:
      InlineCode: |
        exports.handler = async event => {
          event.response = { autoConfirmUser: true };
          return event;
        };

  UserPool:
    Type: AWS::Cognito::UserPool
    Properties:
      UserPoolName: ApiUserPool
      LambdaConfig:
        PreSignUp: !GetAtt PreSignupFunction.Arn
      Policies:
        PasswordPolicy:
          MinimumLength: 6

  UserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      UserPoolId: !Ref UserPool
      ClientName: ApiUserPoolClient
      GenerateSecret: no

  LambdaCognitoUserPoolExecutionPermission:
    Type: AWS::Lambda::Permission
    Properties: 
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt PreSignupFunction.Arn
      Principal: cognito-idp.amazonaws.com
      SourceArn: !Sub 'arn:${AWS::Partition}:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${UserPool}'

  # ============================== Images ==============================
  ImageBucket:
    Type: AWS::S3::Bucket
    Properties: 
      AccessControl: PublicRead
      CorsConfiguration:
        CorsRules:
          - AllowedHeaders:
              - "*"
            AllowedMethods:
              - HEAD
              - GET
              - PUT
              - POST
            AllowedOrigins:
              - "*"

  ImageBucketPublicReadPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref ImageBucket
      PolicyDocument:
        Statement:
          - Action: s3:GetObject
            Effect: Allow
            Principal: "*"
            Resource: !Join ["", ["arn:aws:s3:::", !Ref "ImageBucket", "/*" ]]

  ImageFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/images/
      Policies:
        - AmazonS3FullAccess
      Environment:
        Variables:
          IMAGE_BUCKET_NAME: !Ref ImageBucket
      Events:
        ListImages:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /images
            Method: GET
        DeleteImage:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /images/{imageId}
            Method: DELETE
        CreateImage:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /images
            Method: POST

  TagsFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/tags/
      Environment:
        Variables:
          PARAMETER_STORE_CLARIFAI_API_KEY: /serverless-api/CLARIFAI_API_KEY_ENC
      Policies:
        - AmazonS3ReadOnlyAccess # Managed policy
        - Statement: # Inline policy document
          - Action: [ 'ssm:GetParameter' ]
            Effect: Allow
            Resource: '*'
      Events:
        ExtractTags:
          Type: S3
          Properties:
            Bucket: !Ref ImageBucket
            Events: s3:ObjectCreated:*

Outputs:

  ApiUrl:
    Description: The target URL of the created API
    Value: !Sub "https://${ServerlessApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
    Export:
      Name: ApiUrl

We add some new API events to our ImagesFunction and now need to update the JavaScript for it too.

const AWS = require("aws-sdk");

exports.handler = async event => {
  switch (event.httpMethod.toLowerCase()) {
    case "post":
      return createImage(event);
    case "delete":
      return deleteImage(event);
    default: 
      return listImages(event);
  }
};

const createImage = async event => {
  const userName = extractUserName(event);
  const fileName = "" + Math.random() + Date.now() + "+" + userName;
  const { url, fields } = await createPresignedUploadCredentials(fileName);
  return response({
    formConfig: {
      uploadUrl: url,
      formFields: fields
    }
  }, 201);
};

const deleteImage = async event => {
  const { imageId } = event.pathParameters;
  await deleteImageSomewhere(imageId);
  return response({ message: "Deleted image: " + imageId });
};

// Called with API-GW event
const listImages = async event => {
  const { tags } = event.queryStringParameters;
  const userName = extractUserName(event);
  const images = await loadImagesFromSomewhere(tags.split(","), userName);
  return response({ images });
};

// ============================== HELPERS ==============================

const extractUserName = event => event.requestContext.authorizer.claims["cognito:username"];

const response = (data, statusCode = 200) => ({
  statusCode,
  body: JSON.stringify(data),
  headers: { "Access-Control-Allow-Origin": "*" }
});

const s3Client = new AWS.S3();
const createPresignedUploadCredentials = fileName => {
  const params = {
    Bucket: process.env.IMAGE_BUCKET_NAME,
    Fields: { Key: fileName }
  };

  return new Promise((resolve, reject) =>
    s3Client.createPresignedPost(params, (error, result) =>
      error ? reject(error) : resolve(result)
    )
  );
};

The delete function is straight forward, but also answers one of our questions.

How use a Query String or Route Parameters?

const deleteImage = async event => {
  const { imageId } = event.pathParameters;
  ...
};

The API-Gateway event object has a pathParameters attribute that holds all of the parameters we defined in our SAM template for the event. In our case imageId because we defined Path: /images/{imageId}.

A query string used in a similar way.

const listImages = async event => {
  const { tags } = event.queryStringParameters;
  ...
};

The rest of the code isn't too complicated. We will use the even data to load or delete the images.

The query string will be stored as an object inside the queryStringParameters attribute of our event object.

Conclusion

Sometimes it happens that new technology brings a paradigm shift. Serverless is one of these technologies.

The full power of it boils often down to using managed services whenever possible. Don't roll your own crypto, authentication, storage, or compute. Use what your cloud-provider has already implemented.

The big problem here is often that there are many ways to do things. There isn't only one right way. Should we use KMS directly? Should we let the Systems Manager handle things for us? Should we implement auth via Lambda? Should we use Cognito and be done with it?

I hope I could answer the questions that arise when using Lambda to some extent.

---

Originally published on the Moesif Blog

Comments

[Jack Dow]

Interesting, but little bit complicated :)