Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session covers how distributed systems communicate reliably — the backbone of every production-grade AWS architecture.
📋 Topics Covered
| # | Topic | Type |
|---|---|---|
| 1 | Monolithic vs Microservices Architecture | Concept + Interview |
| 2 | Why Messaging Services Exist | Concept |
| 3 | Amazon SQS — What It Is | Concept + Interview |
| 4 | SQS — Pull-Based vs Push-Based | Concept + Interview |
| 5 | SQS Configuration Settings | Concept + Lab |
| 6 | Visibility Timeout — Deep Dive | Concept + Interview |
| 7 | SQS Message Lifecycle — With and Without Failure | Concept + DevOps |
| 8 | Key SQS APIs | Concept + Lab |
| 9 | Dead Letter Queue (DLQ) | Concept + Interview |
| 10 | SQS Standard vs FIFO Queue | Concept + Cert |
| 11 | SQS FIFO Deduplication | Concept + Cert |
| 12 | Two Core SQS Integration Patterns | Concept + DevOps |
| 13 | Amazon SNS — What It Is | Concept + Interview |
| 14 | SNS vs SQS — Push vs Pull | Concept + Interview |
| 15 | SNS Security and Message Filtering | Concept + Cert |
| 16 | SNS + SQS Fan-Out Pattern | Concept + DevOps |
| 17 | Amazon SNS FIFO | Concept + Cert |
| 18 | SNS FIFO + SQS FIFO Fan-Out | Concept + Cert |
| 19 | Production Architecture View | DevOps |
| 20 | Lab — Deploy SNS and SQS Queues | Lab |
| 21 | Interview Questions | Interview |
| 22 | Assignment | Practice |
Monolithic vs Microservices Architecture
Before queues and topics make sense, you need to understand the problem they solve.
Monolithic Architecture
The entire application — UI, business logic, database layer, authentication, notifications, payment — is packaged and deployed as a single unit.
Early advantages: simple to develop, test, and deploy — everything is in one place.
Why it breaks down at scale:
- A bug in the notification module can crash the entire application — including payments
- Scaling one feature (payments under high load) requires scaling the entire application
- Deploying any small change means redeploying everything — risk and coordination overhead
- The codebase grows into something nobody fully understands anymore
Microservices Architecture
The application is divided into small, independently deployable services, each responsible for exactly one business capability.
What this looks like: Order Service → Payment Service → Notification Service → Inventory Service → User Service — each deployed separately, scaled independently, owned by its own team.
Advantages: fault isolation (Notifications failing doesn't affect Payments), independent scaling, each service uses the best technology for its job, teams deploy without coordinating with each other.
The challenge microservices introduce:
If Payment needs to tell Notification "payment completed, send email" — what happens if Notification is temporarily down? Does Payment fail too? This tight synchronous coupling defeats the purpose of separation.
This is exactly the problem SQS and SNS solve.
🎯 Interview tip: When asked "why microservices?" — cover independent deployment, fault isolation, team autonomy, and technology flexibility. When asked "what challenges come with microservices?" — inter-service communication, distributed tracing, and data consistency are the honest answers.
Why Messaging Services Exist
In a direct synchronous call: Order Service → HTTP → Notification Service → if Notification is down, Order fails too. Tight coupling.
With a queue: Order Service → puts message in queue → returns success immediately → Notification reads from queue when ready.
The queue acts as a buffer and decoupler — the producer doesn't care when the consumer processes, just that it eventually does. Traffic spikes are absorbed by the queue instead of crashing downstream services.
Amazon SQS — What It Is
Amazon SQS (Simple Queue Service) is a fully managed, distributed message queue that temporarily stores messages until a consumer is ready to process them.
The bank token system analogy:
You walk into a bank. You don't go directly to the cashier — you take a numbered token. The token machine doesn't solve your problem. It organizes the waiting line so cashiers can handle customers at a manageable pace. Customers (producers) keep arriving. Cashiers (consumers) process at their own pace. SQS is the token machine.
Key facts:
- Fully managed — no servers, clusters, or infrastructure to provision
- Automatically distributed across multiple backend partitions
- Replicated across multiple AZs — messages are durable even if one AZ fails
- Scales to billions of messages automatically
- Once a message is consumed and deleted, it's gone
SQS — Pull-Based vs Push-Based
SQS is pull-based. Consumers actively poll the queue using ReceiveMessage() when they're ready. SQS never pushes messages.
Think of it like checking your email inbox — you open it when you're ready. Your inbox doesn't force your browser open when mail arrives.
Why pull-based is powerful: consumers control their own processing rate, each consumer scales independently based on queue depth, if a consumer goes down messages safely queue up until it recovers.
Long Polling vs Short Polling:
Short Polling (Wait Time = 0, default): Asks "any messages?" → SQS responds immediately, even if empty → you pay for the API call even when nothing was there.
Long Polling (Wait Time up to 20s): Asks "any messages?" → SQS waits up to 20 seconds → if a message arrives, it returns immediately → if nothing arrives, it returns at timeout. Fewer API calls, lower cost, faster response when messages exist.
🎯 Production tip: Always use Long Polling (20 seconds). It reduces empty responses, lowers costs, and responds faster when messages actually arrive.
SQS Configuration Settings
| Configuration | One-Line Meaning |
|---|---|
| Visibility Timeout | Hides a message while one consumer is processing it — prevents double-processing |
| Message Retention Period | How long an unprocessed message stays before being auto-deleted (1 min – 14 days, default 4 days) |
| Delivery Delay | How long after sending before consumers can see the message (0 – 15 minutes) |
| Maximum Message Size | Max size per message — 256 KB |
| Receive Message Wait Time | How long ReceiveMessage waits when queue is empty — enables Long Polling (0 – 20 seconds) |
Visibility Timeout — Deep Dive
The mechanism that prevents two consumers from processing the same message simultaneously.
How it works:
Consumer A calls
ReceiveMessage()→ SQS returns Message X → SQS hides Message X from all other consumers for the Visibility Timeout duration → Consumer A processes → Consumer A callsDeleteMessage()→ Message permanently gone.If Consumer A crashes or takes too long → Visibility Timeout expires → Message reappears → Consumer B picks it up and retries.
The ChangeMessageVisibility pattern for long-running jobs:
Consumer picks the message → starts processing → periodically calls
ChangeMessageVisibility()to extend the timeout before it expires → processing completes →DeleteMessage().🎯 Interview tip: It's the consumer itself (Lambda/EC2/ECS) that calls
ChangeMessageVisibility— not a separate monitoring process. The consumer extends its own timeout while processing, then deletes the message on success.
Setting the right Visibility Timeout: If your job typically takes 45 seconds, set the timeout to at least 90 seconds — a 2× buffer accounts for occasional slowness without triggering false retries.
SQS Message Lifecycle — With and Without Failure
Without failure (happy path):
Customer → Producer →
SendMessage()→ SQS Queue → Consumer callsReceiveMessage()→ Visibility Timeout starts → Consumer processes (DB write, email, payment) → Consumer callsDeleteMessage()→ Done ✅
With failure (retry path):
Customer → Producer → SQS → Consumer picks message → Processing fails (crash, exception) → Visibility Timeout expires → Message reappears → Another consumer retries → Success →
DeleteMessage()✅If it keeps failing after maxReceiveCount attempts → Message moves to the Dead Letter Queue
Key SQS APIs
| API | What it does |
|---|---|
SendMessage() |
Put a message into the queue |
ReceiveMessage() |
Read messages from the queue (up to 10 at a time) |
ChangeMessageVisibility() |
Extend the Visibility Timeout for a message being processed |
DeleteMessage() |
Remove a successfully processed message permanently |
PurgeQueue() |
Delete all messages in the queue at once |
DeleteQueue() |
Remove the entire queue |
Dead Letter Queue (DLQ)
Some messages genuinely cannot be processed — corrupt data, an unrecoverable bug, an unexpected format. Without a DLQ, these messages retry forever, consuming consumer capacity and blocking healthy messages.
A Dead Letter Queue is a separate SQS queue that receives messages which have failed after exceeding the configured maxReceiveCount (maximum retry attempts).
Also called the poisoned pill defense — a message that would kill your consumer over and over gets quarantined instead of contaminating normal flow.
The flow:
Producer → SQS Queue → Consumer → Failure → Retry → Retry → Retry (maxReceiveCount reached) → Message moved to DLQ → Normal queue continues processing healthy messages
What to do with DLQ messages:
- Inspect the failed message — understand what was in it
- Find the root cause — bug in consumer? Bad data from producer?
- Fix the application or the message data
- Use Redrive to replay the message back to the source queue after the fix is deployed
🎯 Interview Q: Does DLQ automatically fix failed messages? → No. DLQ only stores them safely. A human (or automation) must investigate, fix the root cause, and replay the message.
Production DLQ best practices:
- Always configure a DLQ on every production queue — without one, failed messages silently disappear after retention expires
- Set DLQ retention to 14 days (maximum) — maximum time to investigate
- Set CloudWatch alarm on DLQ's
ApproximateNumberOfMessagesVisiblemetric — get paged the moment any message lands there - After fixing root cause, use Redrive to replay — don't manually reprocess
maxReceiveCount: Typical production value is 3–5 retries. Too low = transient errors moved to DLQ prematurely. Too high = a bad message wastes consumer resources for too long.
SQS Standard vs FIFO Queue
| Standard Queue | FIFO Queue | |
|---|---|---|
| Ordering | Best-effort (may arrive out of order) | Strict first-in, first-out — guaranteed |
| Delivery | At-least-once (duplicates possible) | Exactly-once (within 5-minute deduplication window) |
| Throughput | Nearly unlimited | 300 messages/second (3,000 with batching) |
| Deduplication | Not supported | Built-in |
| Queue name | Any name | Must end in .fifo
|
| Use case | Notifications, logs, parallel jobs where order doesn't matter | Financial transactions, order processing, inventory — where order and exactly-once matter |
How to choose: Is message order critical to correctness? → FIFO. Can you afford occasional duplicates? → Standard. Need maximum throughput? → Standard.
SQS FIFO Deduplication
FIFO queues guarantee exactly-once processing within a 5-minute deduplication window. If the same message is sent twice within 5 minutes, the duplicate is silently discarded.
Two deduplication methods:
Content-Based Deduplication: SQS generates a SHA-256 hash of the message body. Identical hash within 5 minutes = duplicate, dropped.
Message Deduplication ID: The producer explicitly sends a unique ID with each message. Same ID within 5 minutes = duplicate, dropped. Gives producers explicit control over deduplication.
Message Group ID:
Within a FIFO queue, messages with the same Group ID are processed in strict order. Messages with different Group IDs can be processed in parallel.
Example: order processing queue where Group ID = order_id. All events for order 1001 (created → paid → shipped → delivered) are in strict order. Events for order 1002 process in parallel with order 1001's events — FIFO within groups, parallelism across groups.
Two Core SQS Integration Patterns
Pattern 1 — Producer-Consumer Decoupling
Order Service sends a message to SQS when an order is placed → returns success to user immediately → Fulfillment Service reads from the queue at its own pace → processes independently.
If Fulfillment is slow or down, orders safely queue up — no data lost, no error shown to the user. When Fulfillment recovers, it processes the backlog.
The core value of SQS: services don't need to know about each other, don't need to be available at the same time, and traffic spikes are absorbed by the queue instead of crashing downstream services.
Pattern 2 — Workflow Orchestration
For multi-step processes (Order → Payment → Warehouse → Shipping → Notification), SQS handles message passing between steps while AWS Step Functions orchestrates the overall workflow — managing state, retries, and branching logic.
SQS: the messaging between steps. Step Functions: the sequencing, state tracking, and failure handling at the workflow level.
Amazon SNS — What It Is
Amazon SNS (Simple Notification Service) is a fully managed push-based Publish/Subscribe messaging service.
The pub/sub model:
Publishers send a message to an SNS Topic — not to individual consumers. SNS immediately fans out a copy to every subscriber of that topic simultaneously. Each subscription receives its own independent copy.
SNS is not a queue. It doesn't store messages long-term — once published, SNS immediately delivers to subscribers and moves on. SNS is a broadcasting megaphone, not a holding room.
Supported subscriber types:
| Subscriber | What happens |
|---|---|
| SQS queue | Message dropped into the queue for async processing |
| Lambda function | Function invoked immediately |
| Message sent to the address | |
| SMS | Text message sent |
| HTTP/HTTPS endpoint | POST request sent to the URL |
| Mobile push (APNs, GCM) | Push notification to iOS/Android |
SNS vs SQS — Push vs Pull
| SQS | SNS | |
|---|---|---|
| Model | Pull-based — consumers poll | Push-based — SNS delivers immediately |
| Storage | Yes — stores until consumed (up to 14 days) | No — delivers and moves on |
| Consumers | One message → typically one consumer | One message → all subscribers simultaneously |
| Use for | Decoupling, buffering, async work queues | Broadcasting, fan-out, real-time notifications |
They're complementary, not competing. SQS for reliable async processing; SNS for immediate fan-out broadcasting.
SNS Security and Message Filtering
Security
Encryption in transit: HTTPS by default on all SNS API calls and deliveries.
Encryption at rest: SSE via AWS KMS — encrypts message content before delivery.
Access control:
- SNS Topic Policy: Resource-based policy on the topic — defines who can publish and subscribe. Used for cross-account access (similar to S3 bucket policies).
-
IAM Policy: Controls which IAM users/roles can call SNS APIs (
sns:Publish,sns:Subscribe).
Message Filtering
By default, every subscriber receives every message. Filter policies let each subscription define which messages it actually receives, based on message attributes.
Example:
An Order topic receives all order events. Fulfillment only cares about
status: "PLACED". Analytics wants everything. Fraud only cares aboutamount > 10000.Without filtering: every service processes every event and discards irrelevant ones — wasteful.
With filtering: each subscription only receives matching messages — SNS filters before delivery.
The Fulfillment filter policy:
{ "status": ["PLACED"] }
SNS evaluates this filter before delivery. Messages where status is not "PLACED" are not delivered to the Fulfillment subscription at all.
SNS + SQS Fan-Out Pattern
One of the most important architectural patterns in AWS — appears constantly in interviews and certifications.
The problem: When an order is placed, Fulfillment, Inventory, Notifications, and Analytics all need to react. If you call each directly, you're back to tight coupling — one slow service makes the entire order placement slow.
The fan-out solution:
Producer publishes one message to an SNS Topic → SNS immediately delivers to all subscribed SQS queues simultaneously → each service has its own queue → each service processes at its own pace, independently.
Visualized:
New Order Event → SNS Topic (order-events)
→ SQS Queue → Fulfillment Service (has its own DLQ, scaling, retry logic)
→ SQS Queue → Inventory Service (has its own DLQ, scaling, retry logic)
→ SQS Queue → Notification Service (has its own DLQ)
→ SQS Queue → Analytics Service (has its own DLQ)
Why this pattern is powerful:
- Publish once, SNS handles delivery to all consumers
- Each consumer is fully isolated — if Analytics is slow, Fulfillment is unaffected
- Adding a new consumer = add a new SQS subscription to the topic, no producer changes needed
- Each queue has independent DLQ, retry logic, and scaling
🎯 Interview Q: "How would you design a system where one event triggers multiple independent services?" → SNS + SQS fan-out. One SNS topic, one SQS queue per service, each with its own consumer and DLQ.
Amazon SNS FIFO
For cases where message order matters at the broadcasting level.
| Standard SNS | SNS FIFO | |
|---|---|---|
| Ordering | Best-effort | Strict ordering within a Message Group |
| Deduplication | No | Yes — same mechanism as SQS FIFO |
| Throughput | Very high | Up to 300 published messages/second |
| Subscribers | Any type | Only SQS FIFO queues |
Key constraint: SNS FIFO can only fan out to SQS FIFO queues. If you need FIFO ordering, the entire delivery chain must be FIFO.
SNS FIFO + SQS FIFO Fan-Out
The most controlled and ordered fan-out pattern:
Publisher → SNS FIFO Topic → SQS FIFO Queue A (Fulfillment) → strict order guaranteed
→ SQS FIFO Queue B (Inventory) → strict order guaranteed
When to use this vs standard fan-out:
If order of events matters for correctness downstream — inventory updates where "add 10 units" then "remove 15 units" must process in exact order — use SNS FIFO + SQS FIFO.
If order doesn't matter (analytics, notifications), use standard SNS + SQS — higher throughput, simpler, cheaper.
Production Architecture View
This is what a real event-driven system looks like combining everything from this session:
User places order → Order Service → publishes to SNS Topic
SNS fans out simultaneously to:
- SQS Queue (Fulfillment) → consumer: ECS service, has DLQ monitored by CloudWatch
- SQS Queue (Inventory) → consumer: Lambda, has DLQ
- SQS FIFO Queue (Payment Audit) → strict order for audit trail
- Email subscription → sends order confirmation directly to customer
Each SQS queue uses Long Polling (20s), Visibility Timeout set to 2× expected processing time, DLQ with 14-day retention, CloudWatch alarm on DLQ depth, Redrive configured for replay after fixes.
🧪 Lab — Deploy SNS and SQS Queues
Step 1 — Create SQS Queue + DLQ
SQS Console → Create queue → Type: Standard → Name:
order-processing-queue→ Visibility Timeout: 30s → Message Retention: 4 days → Receive Message Wait Time: 20s → Create queue.Create second queue: Name:
order-processing-dlq→ Create queue.Go back to
order-processing-queue→ Edit → Dead-letter queue → Enable → Selectorder-processing-dlq→ maxReceiveCount: 3 → Save.
Step 2 — Create SNS Topic
SNS Console → Topics → Create topic → Type: Standard → Name:
order-events→ Create topic.
Step 3 — Subscribe SQS to SNS
SNS topic page → Create subscription → Protocol: SQS → Endpoint: ARN of
order-processing-queue→ Create subscription.Then: SQS queue → Access Policy → add the SNS topic ARN as a principal with
sqs:SendMessagepermission on this queue (allows SNS to write to SQS).
Step 4 — Test Fan-Out
SNS topic → Publish message → enter test payload → Publish.
Go toorder-processing-queue→ Send and receive messages → Poll for messages → see the message SNS delivered ✅
Step 5 — Test DLQ
Receive a message from the queue 3 times without deleting it (simulating failure) → Check
order-processing-dlq→ message should have moved there ✅
⚡ Quick Revision
Monolithic vs Microservices
- Monolithic: single unit, simple initially, hard to scale and maintain
- Microservices: per-capability services, independent deployment, fault isolation
SQS
- Pull-based distributed queue — consumers poll, SQS never pushes
- At-least-once delivery, best-effort ordering (Standard) vs exactly-once, strict FIFO (FIFO queue)
- Standard: unlimited throughput. FIFO: 300/sec (3000 with batching), name ends in
.fifo - Long Polling (20s): always use in production
SQS Key Config
- Visibility Timeout: hides message during processing, prevents double-processing
- DLQ: receives messages after maxReceiveCount failures, does NOT auto-fix, set 14-day retention, monitor with CloudWatch alarm
- ChangeMessageVisibility: consumer extends its own timeout for long jobs
SQS API Flow
SendMessage → ReceiveMessage → ChangeMessageVisibility (if needed) → DeleteMessage (on success)
SNS
- Push-based Pub/Sub — one publish → immediate delivery to all subscribers simultaneously
- Not a queue — no long-term storage
- Subscribers: SQS, Lambda, Email, SMS, HTTP, Mobile Push
- Message Filtering: per-subscription JSON filter policy on message attributes
Fan-Out (SNS + SQS)
- One SNS topic → multiple SQS queues → each consumer independent
- Best pattern for: one event → multiple independent services
- Add a new consumer: just add a new SQS subscription — no producer changes
FIFO
- SNS FIFO: strict ordering, deduplication, only delivers to SQS FIFO queues
- Use SNS FIFO + SQS FIFO when order correctness matters end-to-end
- Use Standard SNS + SQS when order doesn't matter — higher throughput, cheaper
💼 Interview Questions
Q1: What is the difference between SQS and SNS?
SQS is pull-based — consumers poll for messages, messages are stored until consumed (up to 14 days), and typically one consumer processes each message. SNS is push-based pub/sub — publishers send to a topic, SNS immediately pushes to all subscribers simultaneously, and messages aren't stored long-term. They're complementary: SNS for fan-out broadcasting, SQS for reliable async processing.
Q2: What is the Visibility Timeout in SQS and why is it important?
When a consumer reads a message, it becomes invisible to all other consumers for the Visibility Timeout duration. This prevents double-processing — only one consumer handles the message at a time. If the consumer succeeds and deletes the message, it's gone. If the consumer fails, the timeout expires, the message reappears, and another consumer can retry.
Q3: What is a Dead Letter Queue and when would a message end up there?
A DLQ is a separate SQS queue that receives messages which have failed processing after exceeding maxReceiveCount. A message lands there when a consumer repeatedly can't process it — due to a bug, corrupt data, or an unrecoverable error. The DLQ doesn't fix anything — it safely stores failed messages for investigation, root cause analysis, and optional replay via Redrive after the issue is fixed.
Q4: What is the difference between SQS Standard and SQS FIFO?
Standard offers at-least-once delivery (duplicates possible) and best-effort ordering with nearly unlimited throughput. FIFO guarantees exactly-once delivery (5-minute deduplication window) and strict ordering, but is limited to 300 messages/second (3,000 with batching). Use Standard when order and deduplication don't matter. Use FIFO for financial transactions, order processing, inventory — where sequence is critical.
Q5: Explain the SNS + SQS Fan-Out pattern and why it's useful.
A producer publishes one message to an SNS topic. SNS immediately delivers a copy to all subscribed SQS queues simultaneously. Each downstream service has its own queue, consumer, DLQ, and retry logic — completely independent. This means one event triggers multiple services without tight coupling. Adding a new consumer is just adding a new SQS subscription — no changes to the producer. Failures in one consumer don't affect others.
Q6: Your SQS consumer takes 10 minutes to process a message but the Visibility Timeout is 30 seconds. What happens and how do you fix it?
After 30 seconds, the timeout expires and the message reappears. Another consumer picks it up — now two consumers process the same message, risking data corruption or duplicate operations. Fix: the consumer should periodically call ChangeMessageVisibility() to extend the timeout before it expires. Alternatively, set Visibility Timeout to at least 2× the maximum expected processing time.
Q7: How does message filtering work in SNS?
Each subscription defines a filter policy — a JSON document specifying which message attribute values the subscription cares about. SNS evaluates the filter before delivery. If the message attributes don't match, SNS doesn't deliver that message to that subscription. This prevents every consumer from receiving and discarding irrelevant messages, reducing processing cost and simplifying consumer logic.
Q8: What is the difference between SNS FIFO and Standard SNS?
Standard SNS delivers with best-effort ordering at very high throughput and supports all subscriber types (SQS, Lambda, Email, SMS, HTTP). SNS FIFO guarantees strict ordering and deduplication within a Message Group, but is limited to 300 messages/second and can only deliver to SQS FIFO queues. Use SNS FIFO when the order of events matters for correctness downstream and the entire delivery chain must be ordered.
📝 Assignment
Create a CloudWatch alarm (on any metric — EC2 CPU, SQS queue depth, or a custom metric) and configure it to send notifications to a list of email addresses or aliases via an SNS topic. Verify that when the alarm triggers, all email addresses receive the notification.
AWS Session 14 — SQS & SNS Messaging | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer
Top comments (0)