Implementing SQS Fifo Queue with lambda in AWS Amplify using Cloudformation.

In this tutorial, we are integrating SQS to send message in to lambda function in an Amplify project using Cloudformation.

git repo link: https://github.com/wahmd/sqs-with-lambda-using-aws-amplify

What We'll build:

Integrate Custom Resource (SQS) with amplify such that sending message to queue invokes lambda with the event message in body.
Receive same payload inside the lambda function.

https://user-images.githubusercontent.com/74547936/146831301-27c3f6eb-c6a3-4d4a-98be-10d6f96aac77.mp4

Table of Contents

• Why This tutorial
• Basic Project Setup (from amplify docs):
• Add Lambda function using Amplify CLI:
• Adding the SQS fifo Queue
• Linking SQS queue with Lambda
  • Adding Parameter (using value from another stack)
  • Implicit Way of Adding Parameter:
    • **<CATEGORY><RESOURCE_NAME><OUTPUTS_VARIABLE>**
  • Explicit Way of Adding Parameter:
  • Add Queue as Dependency to lambda (resources creation order)
• Working Demo

Important Tip:

If during the tutorial, you do any change in cloudformation, template or paramters file, make sure to amplify env checkout before doing amplify push. Otherwise, the cli doesn't detect change during amplify status.

Why This tutorial

SQS is not directly generated by the amplify cli like few other services. e.g we can add a lambda using command
amplify add function

But to add SQS, we do not have a command like amplify add queue etc.

There are multiple ways to add other resources not supported by CLI as Custom resources.

Amplify provides two major methods to integrate a custom resource in our amplify app.

1. Use CDK to add custom AWS resources
2. Use CloudFormation to add custom AWS resources

In the first one, you can write your custom resource as simple as in Javascript which on cdk synth will convert to cloudformation.
In the second, you simply provide a cloudformation which it deploys on amplify push.

Both of these methods are absolutely great. However, In my recent project, I found another way which I'd like to share with you guys. In this method, I created SQS using amplify folder structure and cloudformation without making a seperate folder for custom resources (like in the above methods).

Didn't find much of it online, so just sharing it here for learning purposes.

Basic Project Setup (from amplify docs):

First we need to have a basic amplify backend initialized.

In order to do so, complete all steps on Prerequisites and Set up fullstack project to have an empty amplify backend initialized.

Add Lambda function using Amplify CLI:

Now, we can start by adding a lambda function which will be used to poll from the fifo queue.
You can add lambda by
amplify add function

This will create an AWS lambda function that will be used to process messages from the queue.

Now we can see a function handleOrder is added into amplify/backend folder

This is present locally, so we need to amplify push it so that this lambda is created on the cloud.

After push, you can now go to aws console and check it. (make sure to select your region when viewing since lambda is region based service and it'll be only present in your region)

This backend folder containes all resources. So, if we were to add another (custom) resource, we need to create a folder inside it.

Adding the SQS fifo Queue

• Create a new folder inside the backend folder and name it to queue. ('queue' is not a reserved word, you can name anything but you need to update in other files as well -explain later in the tutorial ). This is Category
• Create a folder and name it to orderQueue (this is resource (queue))
• Any resource folder must have these two files:
  • template.yml
  • parameters.json

So create these files.

I'm using yml for cloudformation.
in the resources, add SQS:Queue type resource as

Resources:
  OrderQueue:
    Type: AWS::SQS::Queue
    Properties:
      FifoQueue: true
      ContentBasedDeduplication: true
      QueueName:
        Fn::Join:
          - ''
          - - orders-queue-
            - Ref: env
            - .fifo

Here, I'm using the First In, First Out (FIFO) queue with ContentBasedDeduplication currently on.

This will also dynamically generate queue name based on the runtime environment.
You can read more about intrinsic function Fn:join from the docs

We also need to attach SQS policy to this queue with permissions to send, receive, delete & more actions as:

SQSPolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - Ref: OrderQueue
      PolicyDocument:
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - SQS:SendMessage
              - SQS:ReceiveMessage
              - SQS:DeleteMessage
              - SQS:GetQueueAttributes
            Resource:
              Fn::GetAtt:
                - OrderQueue
                - Arn       

To keep it simple, we're using '*'. This policy allows all principals to perform the listed actions on OrderQueue.
Instead, in a real world application, you should only include those resources or accounts that need to have access on the queue.

So, now our complete template.yml looks like:

AWSTemplateFormatVersion: '2010-09-09'
Description: >-
  {"createdOn":"Windows","createdBy":"Amplify","createdWith":"7.3.6","stackType":"queue-SQS","metadata":{}}
Parameters:
  env:
    Type: String
Resources:
  OrderQueue:
    Type: AWS::SQS::Queue
    Properties:
      FifoQueue: true
      ContentBasedDeduplication: true
      QueueName:
        Fn::Join:
          - ''
          - - orders-queue-
            - Ref: env
            - .fifo

  SQSPolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - Ref: OrderQueue
      PolicyDocument:
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - SQS:SendMessage
              - SQS:ReceiveMessage
              - SQS:DeleteMessage
              - SQS:GetQueueAttributes
            Resource:
              Fn::GetAtt:
                - OrderQueue
                - Arn       
Outputs:
  QueueURL:
    Description: URL of new Amazon SQS Queue
    Value:
      Ref: OrderQueue
  QueueARN:
    Description: ARN of new Amazon SQS Queue
    Value:
      Fn::GetAtt:
        - OrderQueue
        - Arn
  QueueName:
    Description: Name new Amazon SQS Queue
    Value:
      Fn::GetAtt:
        - OrderQueue
        - QueueName

• Place an empty object in parameters.json as:
  {}

• Include queue into your backend-config folder. Because if your resource is not listed in backend-config, it won't appear in amplify status and hence will not be pushed on cloud.
  

{
  "function": {
    "handleOrder": {
      "build": true,
      "providerPlugin": "awscloudformation",
      "service": "Lambda",
    }
  },
  "queue": {
    "ordersQueue": {
      "providerPlugin": "awscloudformation",
      "service": "SQS"
    }
  }
}

• Now save the changes & do amplify push again to have queue on cloud.
• We can see our queue on console
• If everything is pushed on cloud with not issues, you can move to the next part.

Linking SQS queue with Lambda

Now, we have queue and 'handleOrder' on cloud but both are not configured. We it configured such that if SQS gets a message, it is sent to lambda as an event.
This is a perfect case for Type EventSourceMapping which is basically mapping an event from source(kineses, SQS anything that produces event etc) to a lambda function.

So we add this in the cloudformation of our handleOrder function under the Resources section.

"LambdaFunctionSQSMapping": {
      "Type": "AWS::Lambda::EventSourceMapping",
      "Properties": {
        "BatchSize": 1,
        "Enabled": true,
        "EventSourceArn": {
          "Ref": "queueordersQueueQueueARN"
        },
        "FunctionName": {
          "Ref": "LambdaFunction"
        }
      }
    }

file path: amplify/backend/function/handleOrder/handleOrder-cloudformation-template.json

Here, Important attributes to consider are:

• EventSourceArn - Its uniquely identifiable number of the source from which the event is going to come.
• FunctionName - The Name of the function that will be called when event the arrives.

Adding Parameter (using value from another stack)

Here, we currently don't have queueARN inside this file. We can access this using parameters and Outputs ability of the stacks.
We are exporting QueueARN from our queue in it's template.yml.
There are two ways to use parameters.

• implicit - amplify picks automatically if directory structure is followed)
• explicit - define exactly from which resource, get which value using intrinsic functions.

Implicit Way of Adding Parameter:

• Include a parameter queueordersQueueQueueARN in lambda cloudformation as:

    "queueordersQueueQueueARN": {
      "Type": "String"
    }

The parameter name structure is very important as amplify automatically picks it if used right.

**<CATEGORY><RESOURCE_NAME><OUTPUTS_VARIABLE>**

Directory Structure:

amplify 
  ├── backend
  │       ├── function
  │       │      └── handleOrder
  │       ├── queue
  │       │       └── ordersQueue
  │       │               ├── template.yml
  │       │               └── parameters.json

example: queueordersQueueQueueARN

Explicit Way of Adding Parameter:

Along with the implicit way, you also define in parameters.json exactly from where you will get this value.

• Include in file amplify/backend/function/handleOrder/parameters.json

{
    "queueordersQueueQueueARN": {
        "Fn::GetAtt": ["ordersQueue", "Outputs.QueueARN"]
    }
}

Here, GetAtt fetches QueueARN from resource ordersQueue which is being exported from stack using Outputs.

Add Queue as Dependency to lambda (resources creation order)

In backend-config, all resources are listed and generated in parallel if there is no dependency between them.

If we try to push our current app, we will get error:
An error occur during the push operation: Template error: instance of Fn:GetAtt references undefined resource ordersQueue

We are getting this in parameters.json, when we are trying to access QueueARN from its exports.

orderQueue is undefined and accessing one of its exports results in error.

Why is orderQueue undefined?

Because cloud is creating queue & lambda in parallel, but since lambda is dependent on queue (we're using queue's output in lambda), we have to tell cloud that create lambda only when queue is perfectly created & ready

We can define the order in which resources will be created on cloud in backend-config.json as:

• update the backend-config.json as:

{
  "function": {
    "handleOrder": {
      "build": true,
      "providerPlugin": "awscloudformation",
      "service": "Lambda",
      "dependsOn": [
        { 
          "category": "queue",
          "resourceName": "ordersQueue",
          "attributes": ["QueueARN"]
        }
      ]
    }
  },
  "queue": {
    "ordersQueue": {
      "providerPlugin": "awscloudformation",
      "service": "SQS"
    }
  }
}

Here, in dependsOn, we define that the current resource should not be created unless all resouces in dependsOn array are ready since it has dependency. First create dependent resources then create the original resource.

• Do amplfiy env checkout <INSERT_YOUR_ENV>
• Do amplify push -y

After a successful push, you'll have everything ready for demo.

Working Demo

We can see that sending message to queue invokes lambda with the event message in body.

• ✅ Receive same payload inside the lambda function.

https://user-images.githubusercontent.com/74547936/146826151-cc9d3e8e-9fd6-4f55-ae12-f1245b326e18.mp4

If you liked this, ⭐ this repo at:
https://github.com/wahmd/sqs-with-lambda-using-aws-amplify

Comments

[Teerasak Vichadee]

Cannot create SQS, amplify doesn't show queue when run amplify status

[Waleed Ahmad]

This usually happens when you edit 'backend-config.json' and dont checkout env.
Make sure to:

• include queue in backend-config.json (just as in tutorial with the right folder structure)
• run "amplify env checkout ENVNAME"

then 'amplify status' will show the queue.
Let me know if it helps.