DEV Community

Cover image for Day 47: Integrating AWS SQS and SNS for Reliable Messaging
Thu Kha Kyawe
Thu Kha Kyawe

Posted on

Day 47: Integrating AWS SQS and SNS for Reliable Messaging

#aws #100daysofcloudaws

Lab Information

The Nautilus DevOps team needs to implement priority queuing using Amazon SQS and SNS. The goal is to create a system where messages with different priorities are handled accordingly. You are required to use AWS CloudFormation to deploy the necessary resources in your AWS account. The CloudFormation template should be created on the AWS client host at /root/nautilus-priority-stack.yml, the stack name must be nautilus-priority-stack and it should create the following resources:

Two SQS queues named nautilus-High-Priority-Queue and nautilus-Low-Priority-Queue.
An SNS topic named nautilus-Priority-Queues-Topic.
A Lambda function named nautilus-priorities-queue-function that will consume messages from the SQS queues. The Lambda function code is provided in /root/index.py on the AWS client host.
An IAM role named lambda_execution_role that provides the necessary permissions for the Lambda function to interact with SQS and SNS.
Enter fullscreen mode Exit fullscreen mode

Once the stack is deployed, to test the same you can publish messages to the SNS topic, invoke the Lambda function and observe the order in which they are processed by the Lambda function. The high-priority message must be processed first.

topicarn=$(aws sns list-topics --query "Topics[?contains(TopicArn, 'nautilus-Priority-Queues-Topic')].TopicArn" --output text)

aws sns publish --topic-arn $topicarn --message 'High Priority message 1' --message-attributes '{"priority" : { "DataType":"String", "StringValue":"high"}}'

aws sns publish --topic-arn $topicarn --message 'High Priority message 2' --message-attributes '{"priority" : { "DataType":"String", "StringValue":"high"}}'

aws sns publish --topic-arn $topicarn --message 'Low Priority message 1' --message-attributes '{"priority" : { "DataType":"String", "StringValue":"low"}}'

aws sns publish --topic-arn $topicarn --message 'Low Priority message 2' --message-attributes '{"priority" : { "DataType":"String", "StringValue":"low"}}'
Enter fullscreen mode Exit fullscreen mode

Lab Solutions

Architecture

SNS Topic
│
├─(priority=high)──▶ High Priority SQS
└─(priority=low) ──▶ Low Priority SQS

Lambda Function
├─ Reads High Priority Queue FIRST
└─ Then processes Low Priority Queue

Step 1: Create CloudFormation Template

On the AWS client host, create the file exactly here:

vi nautilus-priority-stack.yml
Enter fullscreen mode Exit fullscreen mode

Step 2: Paste This CloudFormation Template

AWSTemplateFormatVersion: '2010-09-09'
Description: Priority Queue Processing Stack
Resources:

  HighPriorityQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: nautilus-High-Priority-Queue
      VisibilityTimeout: 60

  LowPriorityQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: nautilus-Low-Priority-Queue
      VisibilityTimeout: 60

  PriorityQueuesTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: nautilus-Priority-Queues-Topic

  HighPrioritySubscription:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref PriorityQueuesTopic
      Protocol: sqs
      Endpoint: !GetAtt HighPriorityQueue.Arn
      FilterPolicy:
        priority:
          - high

  LowPrioritySubscription:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref PriorityQueuesTopic
      Protocol: sqs
      Endpoint: !GetAtt LowPriorityQueue.Arn
      FilterPolicy:
        priority:
          - low 

  HighPriorityPolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref HighPriorityQueue
      PolicyDocument:
        Statement:
          - Effect: Allow
            Principal: "*"
            Action: SQS:SendMessage
            Resource: !GetAtt HighPriorityQueue.Arn
            Condition:
              ArnEquals:
                aws:SourceArn: !Ref PriorityQueuesTopic

  LowPriorityPolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:    
        - !Ref LowPriorityQueue
      PolicyDocument:
        Statement:
          - Effect: Allow
            Principal: "*"  
            Action: SQS:SendMessage
            Resource: !GetAtt LowPriorityQueue.Arn
            Condition:
              ArnEquals:
                aws:SourceArn: !Ref PriorityQueuesTopic

  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: lambda_execution_role
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
        - arn:aws:iam::aws:policy/AmazonSQSFullAccess
        - arn:aws:iam::aws:policy/AmazonSNSFullAccess

  PriorityLambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: nautilus-priorities-queue-function
      Runtime: python3.9
      Handler: index.lambda_handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Timeout: 5
      Environment:
        Variables:
          high_priority_queue: !Ref HighPriorityQueue
          low_priority_queue: !Ref LowPriorityQueue
      Code:
        S3Bucket: kklabsuser-911514
        S3Key: function-code.zip
Enter fullscreen mode Exit fullscreen mode

Save and exit.

Create the S3 bucket & Upload the file immediately

zip function.zip index.py
aws s3 mb s3://kklabsuser-911514
aws s3 cp function-code.zip s3://kklabsuser-911514/function-code.zip
Enter fullscreen mode Exit fullscreen mode

Step 3: Deploy the CloudFormation Stack

Run exactly:

aws cloudformation create-stack \
  --stack-name nautilus-priority-stack \
  --template-body file:///root/nautilus-priority-stack.yml \
  --capabilities CAPABILITY_NAMED_IAM
Enter fullscreen mode Exit fullscreen mode

Step 4: Wait for Stack Completion

aws cloudformation wait stack-create-complete \
  --stack-name nautilus-priority-stack
Enter fullscreen mode Exit fullscreen mode

✅ Stack status must be CREATE_COMPLETE

Step 5: Test Priority Processing
Get SNS Topic ARN

aws sns list-topics
Enter fullscreen mode Exit fullscreen mode

Publish Messages (As Given in Lab)

topicarn=$(aws sns list-topics --query "Topics[?contains(TopicArn, 'nautilus-Priority-Queues-Topic')].TopicArn" --output text)

aws sns publish --topic-arn $topicarn --message 'High Priority message 1' --message-attributes '{"priority" : { "DataType":"String", "StringValue":"high"}}'

aws sns publish --topic-arn $topicarn --message 'High Priority message 2' --message-attributes '{"priority" : { "DataType":"String", "StringValue":"high"}}'

aws sns publish --topic-arn $topicarn --message 'Low Priority message 1' --message-attributes '{"priority" : { "DataType":"String", "StringValue":"low"}}'

aws sns publish --topic-arn $topicarn --message 'Low Priority message 2' --message-attributes '{"priority" : { "DataType":"String", "StringValue":"low"}}'
Enter fullscreen mode Exit fullscreen mode

Step 7: Verify Priority Order
Check SQS Queues

aws sqs get-queue-attributes \
  --queue-url $(aws sqs get-queue-url \
    --queue-name nautilus-High-Priority-Queue \
    --query QueueUrl \
    --output text) \
  --attribute-names ApproximateNumberOfMessages



aws sqs get-queue-attributes \
  --queue-url $(aws sqs get-queue-url \
    --queue-name nautilus-Low-Priority-Queue \
    --query QueueUrl \
    --output text) \
  --attribute-names ApproximateNumberOfMessages
Enter fullscreen mode Exit fullscreen mode

Step 8: Test Lambda for Low Priority Order

Resources & Next Steps
📦 Full Code Repository: KodeKloud Learning Labs
📖 More Deep Dives: Whispering Cloud Insights - Read other technical articles
💬 Join Discussion: DEV Community - Share your thoughts and questions
💼 Let's Connect: LinkedIn - I'd love to connect with you
Credits
• All labs are from: KodeKloud
• I sincerely appreciate your provision of these valuable resources.

Top comments (0)