DEV Community

Cover image for 📜 AWS 147: Infrastructure as Code - Priority Queuing with SNS, SQS, and CloudFormation
Hritik Raj
Hritik Raj

Posted on

📜 AWS 147: Infrastructure as Code - Priority Queuing with SNS, SQS, and CloudFormation

#aws #cloudformation #sqs #100daysofcloud

🏗️ Automated Priorities: Deploying SQS & SNS with CloudFormation

Hey Cloud Architects 👋

Cloud Automation

Welcome to Day 47 of the #100DaysOfCloud Challenge!
Today, we are moving away from manual console configuration and embracing Infrastructure as Code (IaC). The Nautilus DevOps team needs a reliable priority queuing system where "High Priority" messages are processed before "Low Priority" ones. We'll be using AWS CloudFormation to deploy an entire stack consisting of SNS, multiple SQS queues, and a Lambda consumer.

This task is part of my hands-on practice on the KodeKloud Engineer platform, which is essential for mastering automated resource provisioning.

🎯 Objective

  • Author a CloudFormation template (nautilus-priority-stack.yml) to define the infrastructure.
  • Provision two SQS queues: nautilus-High-Priority-Queue and nautilus-Low-Priority-Queue.
  • Create an SNS Topic to act as the message dispatcher.
  • Deploy a Lambda function with an IAM role (lambda_execution_role) to consume and process messages.
  • Verify the priority logic by publishing attributed messages via the AWS CLI.

💡 Why Priority Queuing?

In production, not all data is created equal. Some messages (like payment processing) must be handled immediately, while others (like weekly reports) can wait. By using SNS message attributes and SQS subscriptions, we can route urgent data to a dedicated "Fast Track" queue.

🔹 Key Concepts

  • CloudFormation Stack: A single unit of management for a collection of AWS resources defined in a YAML or JSON template.
  • SNS Subscription Filter Policies: The logic that tells SNS: "Only send messages with the attribute priority: high to this specific queue."
  • Lambda Trigger: The mechanism that automatically invokes our function as soon as a message lands in either queue.

🛠️ Step-by-Step: IaC Workflow

🔹 Phase A: Authoring the Template

We define our resources in the nautilus-priority-stack.yml file. This includes the SQS queues, the SNS topic, and the subscription logic that separates the "High" from the "Low" priority traffic.

  • Template Path: /root/nautilus-priority-stack.yml
  • Resource Logic: Ensure the SNS subscriptions include FilterPolicy to look for the "priority" attribute.

🔹 Phase B: Deploying the Stack

Once the template is ready, we use the AWS CLI to launch the infrastructure.

  • Command: 
   aws cloudformation create-stack --stack-name nautilus-priority-stack --template-body file:///root/nautilus-priority-stack.yml --capabilities CAPABILITY_NAMED_IAM
Enter fullscreen mode Exit fullscreen mode
  • Note: The CAPABILITY_NAMED_IAM is required because our template creates a specific IAM role for the Lambda function.

🔹 Phase C: Message Publishing & Testing

With the stack CREATE_COMPLETE, we publish messages to the SNS topic. We include message attributes so the SNS Filter Policy knows which queue to target.

  • Publishing High Priority: 
aws sns publish --topic-arn $topicarn --message 'High Priority message' --message-attributes '{"priority" : { "DataType":"String", "StringValue":"high"}}'
Enter fullscreen mode Exit fullscreen mode

  • Publishing Low Priority: 
aws sns publish --topic-arn $topicarn --message 'Low Priority message' --message-attributes '{"priority" : { "DataType":"String", "StringValue":"low"}}'
Enter fullscreen mode Exit fullscreen mode

🔹 Phase D: Verification

The final step is observing the Lambda function's behavior.

  • Log Inspection: Check CloudWatch Logs for the nautilus-priorities-queue-function.
  • Verification: Confirm that the high-priority messages appear in the logs processed before the low-priority ones, validating our routing and processing order.

✅ Verify Success

  • Stack Status: nautilus-priority-stack is in the CREATE_COMPLETE state.
  • Routing: Messages are successfully filtered into the correct queues based on their attributes.
  • Processing: The Lambda function successfully consumes from both queues using its IAM permissions.

📝 Key Takeaways

  • 🚀 Repeatability: If we need to deploy this in another region, we just run the CloudFormation template again—no manual clicking required.
  • 🛡️ Granular Filtering: SNS Filter Policies are incredibly powerful for decoupled architectures, reducing the amount of "junk" data a Lambda has to parse.
  • 📦 IAM Lifecycle: By including the IAM role in the stack, the permissions are created and destroyed alongside the infrastructure, preventing "zombie" roles.

🚫 Common Mistakes

  • Missing Capabilities Flag: Forgetting --capabilities CAPABILITY_NAMED_IAM will cause the stack creation to fail immediately.
  • Incorrect ARN Reference: Hardcoding ARNs in the template instead of using !Ref or !GetAtt makes the template less portable.
  • Attribute Mismatch: If the StringValue in your publish command doesn't exactly match the FilterPolicy in the template (case sensitivity matters!), the message will be dropped or ignored.

🌟 Final Thoughts

Infrastructure as Code is the standard for modern DevOps. By mastering CloudFormation, you've moved from "Cloud Admin" to "Cloud Engineer." You now have the power to deploy complex, secure, and filtered messaging systems with a single command.

🌟 Practice Like a Pro

Want to master AWS CloudFormation? Get hands-on experience here:
👉 KodeKloud Engineer - Practice Labs

🔗 Let’s Connect

Top comments (0)