SQS, SNS & EventBridge: Simple Queues and Fan-Out in AWS Data Pipelines

sqs sns eventbridge are the three AWS primitives you reach for the moment a pipeline stops being one script and becomes a set of services that must talk to each other without falling over. A producer should not care how slow, how many, or how broken its consumers are; a consumer should not have to be online at the exact instant a producer emits. The way you buy that independence on AWS is a queue, a pub/sub topic, or an event router — Amazon SQS, Amazon SNS, and Amazon EventBridge respectively — and knowing which of the three to use, and how each behaves under duplicates and retries, is the difference between a pipeline that self-heals and one that pages you at 3am.

These three services look similar from a distance — they all move messages between components — but they answer different questions. A queue answers "buffer this work so exactly one worker pool drains it at its own pace." A pub/sub topic answers "take this one message and hand a copy to every interested subscriber." An event router answers "look at the shape of this event and send it wherever its content says it should go." This guide walks through the four ideas an interviewer will actually probe — SQS standard vs FIFO queues, SNS pub/sub and the SNS-to-SQS fan-out pattern, EventBridge buses/rules/targets, and the at-least-once delivery semantics that force you to write idempotent consumers — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the fan-out practice library →, harden your retry logic on the idempotency practice set →, and rehearse event-driven consumers on the event-processing practice set →.

---

On this page

• Why SQS, SNS & EventBridge decouple AWS pipelines
• Amazon SQS — standard vs FIFO queues
• Amazon SNS — pub/sub topics & fan-out
• EventBridge — event bus, rules, patterns, targets
• Delivery semantics — at-least-once, idempotency & ordering
• Cheat sheet — SQS / SNS / EventBridge recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why SQS, SNS & EventBridge decouple AWS pipelines

Three shapes of decoupling — a queue, a topic, and a router each solve a different coupling problem

The one-sentence invariant: SQS is a point-to-point queue, SNS is a one-to-many pub/sub topic, and EventBridge is a content-based event router — you choose by counting consumers and asking whether routing depends on the message body. Everything else about the three services follows from that single distinction. Get it wrong and you will bolt a filter policy onto a queue, or fan out through a work queue and wonder why only one consumer ever sees each message.

What decoupling actually buys you.

• Time decoupling (buffering). A producer writes to a queue and moves on; the consumer drains it whenever it is ready. A traffic spike lands in the queue instead of knocking the consumer over — the queue is a shock absorber.
• Load-leveling. A queue lets a bursty producer (say, 10,000 clickstream events in one second) feed a steady consumer (say, 200 writes/second to a warehouse) without either side knowing the other's rate.
• Backpressure. When consumers fall behind, the queue depth grows instead of dropping data; depth is a first-class metric you alarm on, and it is your signal to scale consumers out.
• Failure isolation. If one downstream is down, its messages wait (or dead-letter) without taking the producer or the other downstreams with it.

The three primitives, in one line each.

• SQS — a queue. Messages sit in a durable buffer until exactly one consumer (or one consumer group) receives, processes, and deletes each one. One logical reader drains the queue. This is the work-queue / job-queue shape.
• SNS — a pub/sub topic. A publisher sends one message to a topic; SNS pushes a copy to every subscription. N independent subscribers each get their own copy. This is the fan-out / broadcast shape.
• EventBridge — an event router. Events land on a bus; rules match them against JSON patterns and route matching events to targets. Routing depends on event content, and one event can trigger many targets. This is the event-driven-choreography shape.

Where they compose.

• SNS → SQS is the canonical fan-out: SNS gives you the one-to-many copy, each SQS queue gives its own consumer independent buffering, retries, and a dead-letter queue.
• EventBridge → SQS / SNS / Lambda is the router in front of everything: match on content, then hand off to a queue for buffered work or a topic for further fan-out.
• Producer → SQS → worker is the simplest decoupling and often all you need.

What interviewers listen for.

• Do you say "SQS is one-to-one, SNS is one-to-many, EventBridge routes on content" in the first sentence? — senior signal.
• Do you reach for SNS→SQS fan-out when the requirement is "several teams each need every event," rather than reading the same queue from multiple services? — the classic correct answer.
• Do you bring up at-least-once delivery and idempotency unprompted when asked "what could go wrong?" — required framing.
• Do you pick EventBridge over SNS when routing depends on the event body or you need SaaS/AWS-service event sources and schedules? — senior signal.

Worked example — the same event, three delivery shapes

Detailed explanation. Imagine one business event — OrderPlaced — and three different downstream needs. A billing service must charge the card (one worker pool, must not double-charge). An analytics service, a fraud service, and a search-indexer each independently need every order. And a nightly reconciliation job must run on a schedule regardless of orders. That single scenario touches all three services, and mapping it out is the fastest way to internalise the distinction.

Question. For the OrderPlaced scenario above, which of SQS, SNS, or EventBridge fits each need, and why?

Input.

Need	Consumers	Routing depends on body?
Charge the card	one worker pool	no
Analytics + fraud + search each need every order	three independent teams	no
Route only high-value orders to a review queue	one, conditional	yes (amount > 1000)

Code.

Producer emits OrderPlaced
  ├─ SQS billing-queue            → one worker pool drains, charges once
  ├─ SNS orders-topic  ──fan-out──┬─ SQS analytics-queue
  │                               ├─ SQS fraud-queue
  │                               └─ SQS search-queue
  └─ EventBridge rule {detail.amount > 1000} → SQS review-queue

Step-by-step explanation. Billing is point-to-point work, so it gets its own SQS queue that exactly one worker pool drains. Analytics, fraud, and search each need every order independently, so the order is published to an SNS topic that fans a copy into three separate SQS queues — each team owns its buffer, retries, and DLQ. The "high-value only" routing depends on the message body (amount > 1000), which is content-based routing, so that is an EventBridge rule with an event pattern. Each service is used exactly where its shape matches.

Output.

Need	Service	Reason
Charge the card	SQS	one consumer group, buffered work
Every order to 3 teams	SNS → SQS fan-out	one-to-many, independent consumers
High-value orders only	EventBridge	routing depends on body

Rule of thumb. Count consumers first: one consumer group → SQS; many that each need every message → SNS; routing that depends on the message content → EventBridge.

---

2. Amazon SQS — standard vs FIFO queues

Standard scales infinitely with best-effort order; FIFO trades throughput for strict order and dedup

Amazon SQS is a fully managed message queue with two queue types, and picking between them is the first SQS interview question. Say it in one breath: standard queues give unlimited throughput, at-least-once delivery, and best-effort ordering; FIFO queues give strict per-group ordering and exactly-once processing, at a throughput ceiling. Most data pipelines run on standard queues and make consumers idempotent; you reach for FIFO only when order or de-duplication is a hard business requirement.

Standard queues.

• Unlimited throughput. Nearly unlimited transactions per second per API action — you never provision capacity.
• At-least-once delivery. A message is delivered at least once; occasionally a message is delivered more than once, so consumers must be idempotent (Section 5).
• Best-effort ordering. Messages are generally delivered in the order sent, but this is not guaranteed — reordering can happen.

FIFO queues.

• Strict ordering per message group. Order is preserved within a MessageGroupId; different groups are processed in parallel, so you get ordering and parallelism by choosing the group key (e.g. customer_id).
• Exactly-once processing. Duplicates sent within a 5-minute de-duplication interval are detected via MessageDeduplicationId (or content-based dedup, a SHA-256 of the body) and delivered once.
• Throughput ceiling. By default 300 messages/second per API action, 3,000/second with batching (10 per batch); high-throughput mode raises this substantially. The queue name must end in .fifo.

Knobs every SQS consumer must set.

• Visibility timeout. When a consumer receives a message, SQS hides it from other consumers for the visibility timeout (default 30s, max 12 hours). The consumer must DeleteMessage before the timeout expires, or the message becomes visible again and is redelivered. Set it just above your p99 processing time, and extend it with ChangeMessageVisibility for long jobs.
• Long polling. Set WaitTimeSeconds (up to 20s) on receive so the call waits for a message instead of returning empty immediately. Long polling cuts empty receives, lowers cost, and reduces latency versus short polling — always turn it on.
• Message retention. Messages live in the queue for a retention period (default 4 days, up to 14) if not deleted.
• Batching. ReceiveMessage returns up to 10 messages per call; SendMessageBatch / DeleteMessageBatch cut request count and cost.
• Message size. Up to 256 KB per message; larger payloads use the Extended Client Library (body in S3, pointer in the message).

Dead-letter queues.

• Redrive policy. Attach a DLQ with a RedrivePolicy naming a deadLetterTargetArn and maxReceiveCount. After a message is received maxReceiveCount times without being deleted (i.e. it keeps failing), SQS moves it to the DLQ instead of redelivering forever — this quarantines poison messages.
• Redrive back. Once you fix the bug, redrive moves messages from the DLQ back to the source queue to reprocess.

Worked example — poll a queue with long polling and delete on success

Detailed explanation. The canonical SQS consumer loop is receive → process → delete, with long polling to avoid busy-waiting. The delete is the acknowledgement: a message is only removed once you explicitly delete it, so a crash mid-processing leaves the message to reappear after the visibility timeout. That is what makes SQS reliable and also why consumers must be idempotent.

Question. Write a consumer that long-polls a standard queue, processes each message, and deletes it only on success. Show what happens to a message whose processing throws.

Input.

receipt	body	processing result
msg-a	{"order_id": 1}	success
msg-b	{"order_id": 2}	raises (bug)

Code.

import boto3

sqs = boto3.client("sqs")
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/orders"

while True:
    resp = sqs.receive_message(
        QueueUrl=QUEUE_URL,
        MaxNumberOfMessages=10,      # batch receive
        WaitTimeSeconds=20,          # long polling
        VisibilityTimeout=60,        # hide while we work
    )
    for msg in resp.get("Messages", []):
        try:
            handle(msg["Body"])                       # your idempotent work
            sqs.delete_message(                        # ack ONLY on success
                QueueUrl=QUEUE_URL,
                ReceiptHandle=msg["ReceiptHandle"],
            )
        except Exception:
            pass  # no delete → message reappears after VisibilityTimeout

Step-by-step explanation. receive_message with WaitTimeSeconds=20 waits up to 20 seconds for messages, returning a batch of up to 10. For msg-a, handle(...) succeeds and we call delete_message with its ReceiptHandle, so it leaves the queue. For msg-b, handle(...) raises, we skip the delete, and after the 60-second visibility timeout SQS makes it visible again and redelivers it. If msg-b keeps failing and the queue has a redrive policy with maxReceiveCount=5, it moves to the DLQ on the sixth failed receive.

Output.

message	delivered again?	final location
msg-a	no (deleted)	processed, gone
msg-b	yes, after 60s	source queue → DLQ after maxReceiveCount

Rule of thumb. Delete is your ack: never delete before the work is durably done, and set the visibility timeout just above your p99 processing time so retries are neither premature nor slow.

SQS interview question on ordered, deduplicated processing

Question. You process account-balance updates and two things are non-negotiable: updates for the same account must be applied in order, and a network retry that sends the same update twice must not be applied twice. Different accounts can be processed in parallel. Which queue type and settings, and how do you get both order and parallelism?

Solution Using a FIFO queue with MessageGroupId and MessageDeduplicationId

Code.

import boto3

sqs = boto3.client("sqs")
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/balances.fifo"

def send_update(account_id: str, delta: int, event_id: str):
    sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=f'{{"account": "{account_id}", "delta": {delta}}}',
        MessageGroupId=account_id,            # order preserved PER account
        MessageDeduplicationId=event_id,      # dedup within 5-min window
    )

# same event_id sent twice (a retry) is stored once
send_update("acct-7", -50, "evt-100")
send_update("acct-7", -50, "evt-100")   # duplicate, ignored
send_update("acct-7", +20, "evt-101")
send_update("acct-9", +5,  "evt-102")   # different group, parallel

Step-by-step trace.

send	MessageGroupId	MessageDeduplicationId	stored?	order effect
evt-100 (-50)	acct-7	evt-100	yes	1st in acct-7
evt-100 (-50) again	acct-7	evt-100	no (dup)	ignored
evt-101 (+20)	acct-7	evt-101	yes	2nd in acct-7 (after evt-100)
evt-102 (+5)	acct-9	evt-102	yes	1st in acct-9 (parallel)

1. The queue name ends in .fifo, so it is a FIFO queue with strict ordering and dedup enabled.
2. MessageGroupId="acct-7" guarantees all acct-7 updates are delivered and processed in send order; evt-101 cannot overtake evt-100.
3. MessageDeduplicationId="evt-100" sent twice within the 5-minute window is recognised as a duplicate and stored once, so the -50 is applied exactly once.
4. acct-9 is a different MessageGroupId, so it is an independent ordering lane processed in parallel with acct-7 — you get order and throughput by keying the group on the entity that needs ordering.

Output:

account	applied updates	final delta	duplicates applied
acct-7	-50 then +20 (in order)	-30	0
acct-9	+5	+5	0

Why this works — concept by concept:

• MessageGroupId — the ordering scope; SQS FIFO guarantees order only within a group, so choosing the group key (here account_id) is how you decide what must be sequential and what may run in parallel.
• MessageDeduplicationId — a caller-supplied identity checked over a 5-minute window; identical ids collapse to one message, giving exactly-once enqueue even when the network makes you retry a send.
• Parallel groups — distinct group ids are independent lanes, so FIFO does not force global serialization — throughput scales with the number of active groups.
• Content-based dedup alternative — if you cannot supply an id, enable content-based dedup and SQS hashes the body; a byte-identical retry is deduped automatically.
• Cost — FIFO is O(1) per message but capped (300/s, or 3,000/s batched) without high-throughput mode; if you do not need order, standard queues remove the ceiling and you dedup in the consumer instead.

Fan-out
Topic — fan-out
Queue and fan-out delivery problems

Practice →

Queue Topic — queue SQS-style buffering and visibility-timeout problems

Practice →

---

3. Amazon SNS — pub/sub topics & fan-out

One publish, many subscribers — SNS broadcasts a copy to every subscription, and SNS→SQS is the fan-out pattern

Amazon SNS is push-based publish/subscribe messaging. A publisher sends a message to a topic; SNS delivers a copy to every subscription on that topic. Where SQS is a buffer that one reader drains, SNS is a splitter that hands the same message to many independent consumers. The invariant to say out loud: SNS gives you one-to-many delivery, and combining SNS with SQS gives each subscriber its own durable, independently-retryable buffer — that combination is the AWS fan-out pattern.

Topics, subscriptions, protocols.

• Topic. The named channel you publish to; publishers know only the topic ARN, never the subscribers.
• Subscription. An endpoint bound to the topic. Supported protocols include SQS, Lambda, HTTP/HTTPS, email/email-JSON, SMS, mobile push, and Kinesis Data Firehose (for archival to S3/Redshift).
• Decoupling. Adding or removing a subscriber never touches the publisher — you wire a new consumer by subscribing a new endpoint.

The fan-out pattern (SNS → SQS).

• Why not just fan out to Lambda directly? You can, but subscribing SQS queues to the topic gives each consumer a durable buffer, its own retry policy, its own DLQ, and independent backpressure. If the analytics consumer is down for an hour, its queue simply fills while billing keeps flowing.
• Each subscriber gets its own copy. Publish once; SNS puts one copy in each subscribed queue. The queues are independent — a poison message in one does not affect the others.
• Ordered fan-out. Use a FIFO topic subscribed to FIFO queues when the whole fan-out must preserve order per group.

Message filtering.

• Filter policies. Attach a filter policy JSON to a subscription so it only receives messages whose attributes (or, with payload-based filtering, whose body fields) match. Filtering happens in SNS, so an uninterested queue never receives — and never pays for — irrelevant messages.
• Example. A subscription with {"event_type": ["order_placed"]} receives order-placed events but not order_cancelled, without the consumer writing any filter code.

Delivery details.

• At-least-once, with retries. SNS retries failed deliveries (with backoff for HTTP/S endpoints) and can route permanently-failed deliveries to a subscription-level DLQ.
• Raw message delivery. By default SNS wraps the payload in a JSON envelope; enable raw message delivery on SQS/Firehose subscriptions so the consumer receives the bare payload instead of the SNS wrapper.

Worked example — publish once, land in three queues

Detailed explanation. The clearest way to see fan-out is to publish a single message to a topic that has three SQS subscriptions and confirm all three queues receive an independent copy. Nothing about the publish call changes as you add or remove subscribers — that decoupling is the whole point.

Question. An orders-topic has three SQS subscriptions: analytics, fraud, search. Publish one order_placed message and show what each queue receives.

Input.

{"event_type": "order_placed", "order_id": 501, "amount": 42.5}

Code.

import boto3, json

sns = boto3.client("sns")
TOPIC_ARN = "arn:aws:sns:us-east-1:123456789012:orders-topic"

sns.publish(
    TopicArn=TOPIC_ARN,
    Message=json.dumps({"order_id": 501, "amount": 42.5}),
    MessageAttributes={
        "event_type": {"DataType": "String", "StringValue": "order_placed"},
    },
)
# analytics, fraud, search queues are each subscribed to orders-topic

Step-by-step explanation. The single publish call sends the message to orders-topic. SNS looks up every subscription and delivers a copy to each — one into the analytics SQS queue, one into fraud, one into search. The publisher never enumerates subscribers. Each queue now holds its own copy that its consumer drains independently, with its own visibility timeout and DLQ.

Output.

queue	copies received	independent consumer
analytics-queue	1	yes
fraud-queue	1	yes
search-queue	1	yes

Rule of thumb. If several teams each need every message, publish to an SNS topic and give each team its own subscribed SQS queue — never have multiple services compete on one shared queue, because a queue delivers each message to only one of them.

SNS interview question on selective fan-out

Question. Your orders-topic fans out to a fraud queue that should receive every order, and a high-value-review queue that should receive only orders over 1000. You do not want the review consumer filtering in code (it wastes reads and DLQ space). How do you route only the high-value orders to the review queue?

Solution Using an SNS subscription filter policy

Code.

import boto3, json

sns = boto3.client("sns")
TOPIC_ARN  = "arn:aws:sns:us-east-1:123456789012:orders-topic"
REVIEW_ARN = "arn:aws:sqs:us-east-1:123456789012:high-value-review"

# subscribe the review queue with a filter policy on a numeric attribute
sns.set_subscription_attributes(
    SubscriptionArn="<review-subscription-arn>",
    AttributeName="FilterPolicy",
    AttributeValue=json.dumps({"amount": [{"numeric": [">", 1000]}]}),
)

def place_order(order_id: int, amount: float):
    sns.publish(
        TopicArn=TOPIC_ARN,
        Message=json.dumps({"order_id": order_id, "amount": amount}),
        MessageAttributes={
            "amount": {"DataType": "Number", "StringValue": str(amount)},
        },
    )

place_order(501, 42.5)     # fraud only
place_order(502, 5000.0)   # fraud AND high-value-review

Step-by-step trace.

publish	amount attribute	fraud (no filter)	review (amount > 1000)
order 501	42.5	delivered	filtered out
order 502	5000.0	delivered	delivered

1. The fraud subscription has no filter policy, so it matches everything — it receives both orders.
2. The review subscription's filter policy {"amount": [{"numeric": [">", 1000]}]} is evaluated by SNS against each message's attributes.
3. Order 501 (amount=42.5) fails the numeric predicate, so SNS never delivers it to the review queue — no read, no DLQ churn on the consumer side.
4. Order 502 (amount=5000) passes, so it is delivered to both fraud and review. The routing decision happened in SNS, not in consumer code.

Output:

queue	messages received	filtered in SNS
fraud-queue	501, 502	no
high-value-review-queue	502	yes

Why this works — concept by concept:

• Filter policy — a declarative JSON predicate on the subscription; SNS evaluates it before delivery, so unmatched messages are dropped at the topic instead of clogging a consumer.
• Message attributes — filtering matches on structured attributes (or payload fields with payload-based filtering), so you expose the routing keys as attributes when you publish.
• Per-subscription scope — each subscription carries its own policy, so one topic can feed a catch-all fraud queue and a selective review queue simultaneously.
• Cost and cleanliness — filtering in SNS avoids paying for receives and DLQ handling of messages a consumer would have discarded anyway.
• Cost — filter evaluation is O(attributes) per subscription per message, negligible; it saves downstream SQS request and Lambda-invocation cost by not delivering non-matches.

Fan-out
Topic — fan-out
SNS-to-SQS fan-out and broadcast problems

Practice →

Events Topic — event-processing Pub/sub and event-processing problems

Practice →

---

4. EventBridge — event bus, rules, patterns, targets

EventBridge routes on content — events land on a bus, rules match a JSON pattern, and matching events fan out to targets

Amazon EventBridge is a serverless event bus that routes events by their content. Where SNS decides delivery by who subscribed, EventBridge decides delivery by what the event looks like: a rule holds a JSON event pattern, and every event on the bus that matches is routed to the rule's targets. Say the invariant plainly: EventBridge is content-based routing plus deep AWS/SaaS integration and schedules — you use it when the destination depends on the event body, or when the event source is an AWS service or a SaaS partner.

The event bus.

• Default bus. Every account has one; AWS services (S3, EC2, ECS, CodePipeline, and dozens more) automatically emit events to it.
• Custom bus. Create your own bus for your application's events to keep them isolated from AWS-service noise.
• Partner / SaaS bus. SaaS providers (e.g. Datadog, Zendesk, Shopify) can deliver events straight onto a partner event bus — no polling glue code.

Events and patterns.

• Event envelope. Every event is JSON with standard fields: source, detail-type, account, region, time, and a free-form detail object carrying your payload.
• Event pattern. A rule matches on any of those fields. {"source": ["orders"], "detail-type": ["OrderPlaced"], "detail": {"amount": [{"numeric": [">", 1000]}]}} matches high-value order events. Patterns support prefix, numeric, exists, anything-but, and $or matching — richer than SNS filter policies.

Rules and targets.

• Up to 5 targets per rule. One matching event can trigger multiple targets: Lambda, SQS, SNS, Step Functions, Kinesis Data Streams/Firehose, ECS tasks, API destinations (any HTTPS API), and more.
• Input transformer. Reshape the event before it hits a target — extract fields and build a custom payload so the target sees exactly what it expects.
• Target DLQ + retries. Configure a dead-letter queue and a retry policy per target so an event that a target repeatedly fails to accept is captured, not lost.

Schedules.

• Scheduled rules. A rule can fire on a cron(...) or rate(...) expression instead of an event pattern — the managed replacement for a cron server driving a Lambda.
• EventBridge Scheduler. The newer dedicated service for one-time and recurring schedules at scale, with time zones, flexible time windows, and 200+ target APIs — prefer it over scheduled rules for large or complex scheduling.

Reliability features.

• Archive & replay. Archive matching events and later replay them onto the bus — invaluable for reprocessing after a consumer bug or for backfills.
• Schema registry. Discover and version event schemas and generate typed bindings.

Worked example — a rule that routes matching events to a target

Detailed explanation. The core EventBridge object is a rule: an event pattern plus one or more targets. Here a rule matches OrderPlaced events from the orders source and routes them to an SQS queue, ignoring every other event on the bus. The pattern is pure JSON — no code runs to decide the match.

Question. Create a rule on a custom bus that routes OrderPlaced events (source orders) to an SQS order-processing queue, and show which events match.

Input.

{"source": "orders", "detail-type": "OrderPlaced",
 "detail": {"order_id": 501, "amount": 42.5}}

Code.

import boto3, json

eb = boto3.client("events")

eb.put_rule(
    Name="route-order-placed",
    EventBusName="app-bus",
    EventPattern=json.dumps({
        "source": ["orders"],
        "detail-type": ["OrderPlaced"],
    }),
)
eb.put_targets(
    Rule="route-order-placed",
    EventBusName="app-bus",
    Targets=[{
        "Id": "to-order-queue",
        "Arn": "arn:aws:sqs:us-east-1:123456789012:order-processing",
    }],
)

Step-by-step explanation. put_rule registers the rule route-order-placed on the app-bus with an event pattern that matches events whose source is orders and detail-type is OrderPlaced. put_targets binds the order-processing SQS queue as the destination. From then on, EventBridge evaluates every event published to app-bus: matches are delivered to the queue; non-matches (a PaymentFailed event, or an event from source: inventory) are ignored by this rule.

Output.

event on bus	source / detail-type	matches rule?	routed to
OrderPlaced	orders / OrderPlaced	yes	order-processing queue
PaymentFailed	billing / PaymentFailed	no	—
OrderPlaced	inventory / OrderPlaced	no (source ≠ orders)	—

Rule of thumb. An EventBridge rule is "pattern → targets"; keep patterns specific (match source and detail-type) so a rule fires only for the events it truly owns.

EventBridge interview question on scheduled event-driven work

Question. You must run a reconciliation Lambda every day at 02:00 UTC, and also trigger it immediately whenever an OrderRefunded event with amount > 500 appears. You want one Lambda, driven by both a schedule and a content match, with failures captured rather than dropped. How do you wire it with EventBridge?

Solution Using a scheduled rule plus a pattern rule with a target DLQ

Code.

import boto3, json

eb = boto3.client("events")
LAMBDA_ARN = "arn:aws:lambda:us-east-1:123456789012:function:reconcile"
DLQ_ARN    = "arn:aws:sqs:us-east-1:123456789012:reconcile-dlq"

# 1) schedule: every day at 02:00 UTC
eb.put_rule(Name="reconcile-daily",
            ScheduleExpression="cron(0 2 * * ? *)")

# 2) content match: large refunds, on the app bus
eb.put_rule(Name="reconcile-large-refund", EventBusName="app-bus",
            EventPattern=json.dumps({
                "source": ["orders"],
                "detail-type": ["OrderRefunded"],
                "detail": {"amount": [{"numeric": [">", 500]}]},
            }))

target = [{
    "Id": "to-reconcile",
    "Arn": LAMBDA_ARN,
    "RetryPolicy": {"MaximumRetryAttempts": 3},
    "DeadLetterConfig": {"Arn": DLQ_ARN},   # capture failed deliveries
}]
eb.put_targets(Rule="reconcile-daily", Targets=target)
eb.put_targets(Rule="reconcile-large-refund", EventBusName="app-bus",
               Targets=target)

Step-by-step trace.

trigger	rule	condition met	Lambda invoked	on repeated failure
clock 02:00 UTC	reconcile-daily	schedule due	yes	→ reconcile-dlq
OrderRefunded amount=750	reconcile-large-refund	750 > 500	yes	→ reconcile-dlq
OrderRefunded amount=100	reconcile-large-refund	100 > 500 false	no	—

1. The scheduled rule reconcile-daily uses a cron(0 2 * * ? *) expression, so EventBridge invokes the Lambda every day at 02:00 UTC with no server to run cron.
2. The pattern rule reconcile-large-refund matches OrderRefunded events whose detail.amount exceeds 500 and invokes the same Lambda ARN — one function, two independent triggers.
3. A refund of 100 fails the numeric predicate, so no invocation happens — content-based filtering keeps the Lambda from running on small refunds.
4. Both targets carry a RetryPolicy and a DeadLetterConfig, so if the Lambda keeps failing to accept an event, EventBridge parks it in reconcile-dlq after the retries instead of dropping it silently.

Output:

source of invocation	reconcile ran	failures preserved
daily 02:00 schedule	yes	in reconcile-dlq
large refund (>500) event	yes	in reconcile-dlq
small refund (≤500) event	no	—

Why this works — concept by concept:

• Event pattern — declarative JSON matching on source, detail-type, and nested detail fields routes only the events you care about, so the Lambda is invoked precisely, not on every event.
• Scheduled rule — a cron/rate expression turns EventBridge into a managed scheduler, removing the cron box you would otherwise babysit.
• Shared target — the same Lambda ARN is a target of two rules, so one implementation serves both the time-driven and event-driven paths.
• Target DLQ + retry policy — per-target retries and a dead-letter queue make delivery durable; a poison event is captured for inspection rather than lost.
• Cost — rule evaluation is O(rules × events) but serverless and pay-per-event; you pay for matched deliveries, and non-matches cost nothing to ignore.

Events
Topic — event-processing
Event-router and rule-matching problems

Practice →

Pipelines Topic — pipelines Event-driven pipeline-design problems

Practice →

---

5. Delivery semantics — at-least-once, idempotency & ordering

At-least-once means duplicates will happen — the fix is idempotent consumers, and strict order lives only inside a FIFO group

The single most-probed correctness topic across all three services is delivery semantics. Standard SQS, SNS, and EventBridge all deliver at-least-once: they guarantee a message arrives, and occasionally it arrives more than once. Say the consequence out loud: you cannot prevent duplicates at the transport layer, so you make processing safe under duplicates — an idempotent consumer. The second half is ordering: only FIFO queues (and FIFO topics) guarantee order, and only within a message group.

Why duplicates are unavoidable.

• Redelivery on non-ack. If a consumer processes a message but crashes before deleting it, SQS redelivers after the visibility timeout — the work happens twice unless it is idempotent.
• Producer retries. A network blip on a SendMessage/publish can make the producer retry, enqueuing the same logical event twice (unless you use a FIFO dedup id).
• Fan-out multiplies paths. In SNS→SQS and EventBridge→multiple-targets, each path can independently redeliver.

Building an idempotent consumer.

• Idempotency key. Give every message a stable business id (order_id, event_id). Process a key at most once by recording processed keys and skipping repeats.
• Dedup table (conditional write). Use a store with a uniqueness/conditional guarantee — e.g. DynamoDB PutItem with attribute_not_exists(pk) — so the first delivery writes the key and does the work, and a duplicate's conditional write fails and is skipped.
• Naturally idempotent operations. Prefer operations that are safe to repeat: PUT/upsert on a key rather than INSERT; SET balance = x rather than balance = balance - 50 (the latter double-applies on a duplicate).
• Scope the guarantee. Idempotency must span the effect, not just the write — if processing sends an email and writes a row, both must be guarded by the same key or you re-send the email on redelivery.

Ordering — what you actually get.

• Standard SQS / SNS / EventBridge — no ordering guarantee. Treat messages as an unordered set; if order matters, carry a sequence number and reorder in the consumer, or use FIFO.
• FIFO — order within a group. Ordered per MessageGroupId only. There is no global order across groups, and that is a feature: it is how FIFO gets parallelism.

Choosing a service — the decision table.

• One consumer group, buffered work, may be idempotent → SQS standard.
• Order and dedup are hard requirements → SQS FIFO (or FIFO topic → FIFO queue for ordered fan-out).
• Many independent consumers each need every message → SNS → SQS fan-out.
• Routing depends on event content, or the source is an AWS service / SaaS, or you need schedules → EventBridge.

Worked example — a duplicate arrives and the consumer processes once

Detailed explanation. The everyday idempotency pattern is a conditional write on the message's business id. The first delivery claims the id and does the work; a redelivered duplicate finds the id already claimed and short-circuits. This is what turns "at-least-once" into "effectively exactly-once" at the application layer.

Question. A payment message with event_id = pay-77 is delivered twice (redelivery after a crash). Show a consumer that charges the card exactly once.

Input.

delivery	event_id	amount
1	pay-77	30.00
2 (duplicate)	pay-77	30.00

Code.

import boto3
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb")

def handle(event_id: str, amount: float):
    try:
        ddb.put_item(
            TableName="processed_events",
            Item={"event_id": {"S": event_id}},
            ConditionExpression="attribute_not_exists(event_id)",  # claim key
        )
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return "skipped-duplicate"     # already processed
        raise
    charge_card(amount)                    # runs only on first delivery
    return "processed"

Step-by-step trace.

delivery	conditional put	outcome	charge_card called
1 (pay-77)	attribute_not_exists → succeeds	processed	yes (30.00)
2 (pay-77)	key exists → ConditionalCheckFailed	skipped-duplicate	no

1. On delivery 1, put_item with attribute_not_exists(event_id) succeeds because the key is new — the consumer atomically claims pay-77 and then charges the card.
2. On delivery 2, the same conditional put fails with ConditionalCheckFailedException because pay-77 already exists; the consumer returns early and never charges again.
3. The claim and the check are the same atomic operation, so even two concurrent deliveries cannot both pass — one wins the conditional write, the other fails.
4. Deleting the SQS message after either outcome is safe, because the effect (one charge) is already guaranteed by the dedup table.

Output:

event_id	times charged	duplicate handled
pay-77	1	yes (2nd skipped)

Rule of thumb. Make the dedup claim and the side effect share one atomic guard (a conditional write or a unique constraint); an idempotency check that is separate from the write has a race window that a duplicate will eventually hit.

Delivery-semantics interview question on safe retries under fan-out

Question. An order event fans out via SNS to an inventory queue and an email queue. Both consumers occasionally see the same event twice, and the email consumer must never send two confirmation emails for one order. How do you guarantee at-most-once effect per consumer while keeping the fan-out at-least-once delivery?

Solution Using a per-consumer idempotency key on a stable event id

Code.

import boto3
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb")

def process_once(consumer: str, event_id: str, effect):
    key = f"{consumer}#{event_id}"          # scope key PER consumer
    try:
        ddb.put_item(
            TableName="dedup",
            Item={"pk": {"S": key}},
            ConditionExpression="attribute_not_exists(pk)",
        )
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return "skip"
        raise
    effect()                                 # e.g. send_email / decrement_stock
    return "done"

# email consumer, order evt-900 delivered twice
process_once("email", "evt-900", lambda: send_confirmation("evt-900"))
process_once("email", "evt-900", lambda: send_confirmation("evt-900"))  # skip

Step-by-step trace.

consumer	event_id	dedup key	conditional put	effect
email	evt-900 (1st)	email#evt-900	succeeds	send email
email	evt-900 (2nd)	email#evt-900	fails (exists)	skipped
inventory	evt-900 (1st)	inventory#evt-900	succeeds	decrement stock

1. The dedup key is scoped per consumer (consumer#event_id), so the email consumer and the inventory consumer each process evt-900 once — the two fan-out paths do not block each other.
2. The email consumer's second delivery hits an existing email#evt-900 key and skips, so exactly one confirmation email is sent.
3. The inventory consumer's key inventory#evt-900 is independent, so it still processes the event once even though email already claimed its own key.
4. SNS/SQS keep delivering at-least-once; the effect is made at-most-once per consumer by the conditional write, giving effectively-exactly-once end to end.

Output:

consumer	effect applied	duplicates suppressed
email	1 confirmation email	yes
inventory	1 stock decrement	yes

Why this works — concept by concept:

• At-least-once delivery — the transport guarantees arrival, not uniqueness, so correctness must be enforced by the consumer, not assumed from the queue.
• Idempotency key — a stable business id (event_id) is the identity you dedup on; without one, you cannot tell a duplicate from a distinct event.
• Per-consumer scope — prefixing the key with the consumer name lets each fan-out branch process the same event independently, which is exactly what fan-out is for.
• Atomic conditional write — attribute_not_exists makes claim-and-check a single race-free operation, closing the window two concurrent deliveries would otherwise exploit.
• Cost — one conditional write per delivery, O(1); a tiny, bounded overhead that converts unavoidable duplicates into safe, repeatable processing.

Idempotency
Topic — idempotency
Idempotent-consumer and dedup problems

Practice →

Reliability Topic — fault-tolerance At-least-once and retry-safety problems

Practice →

---

Cheat sheet — SQS / SNS / EventBridge recipes

Poll a standard queue with long polling.

resp = sqs.receive_message(QueueUrl=url, MaxNumberOfMessages=10,
                           WaitTimeSeconds=20, VisibilityTimeout=60)
for m in resp.get("Messages", []):
    handle(m["Body"])
    sqs.delete_message(QueueUrl=url, ReceiptHandle=m["ReceiptHandle"])

FIFO send — order per group, dedup per id.

sqs.send_message(QueueUrl=fifo_url, MessageBody=body,
                 MessageGroupId=account_id,       # order scope
                 MessageDeduplicationId=event_id) # 5-min dedup window

Redrive policy (attach a DLQ).

{"deadLetterTargetArn": "arn:aws:sqs:...:orders-dlq",
 "maxReceiveCount": "5"}

SNS → SQS fan-out (publish once, land in many queues).

sns.publish(TopicArn=topic, Message=json.dumps(payload),
            MessageAttributes={"event_type":
                {"DataType": "String", "StringValue": "order_placed"}})

SNS subscription filter policy.

{"event_type": ["order_placed"], "amount": [{"numeric": [">", 1000]}]}

EventBridge rule + pattern + target.

eb.put_rule(Name="r", EventBusName="app-bus", EventPattern=json.dumps(
    {"source": ["orders"], "detail-type": ["OrderPlaced"]}))
eb.put_targets(Rule="r", EventBusName="app-bus",
    Targets=[{"Id": "t", "Arn": queue_arn}])

Service picker.

Situation	Service
One consumer group, buffered work	SQS standard
Strict order + dedup required	SQS FIFO
Many consumers each need every message	SNS → SQS fan-out
Route by event content / AWS / SaaS source / schedule	EventBridge

---

Frequently asked questions

What is the difference between SQS, SNS, and EventBridge?

SQS is a queue: messages sit in a durable buffer until one consumer group receives, processes, and deletes each one — point-to-point work distribution. SNS is a pub/sub topic: a publisher sends one message and SNS pushes a copy to every subscription — one-to-many broadcast. EventBridge is an event router: events land on a bus and rules route them to targets based on a JSON event pattern — content-based routing with deep AWS and SaaS integration. Count consumers and ask whether routing depends on the body: one group → SQS, many → SNS, content-driven → EventBridge.

When should I use an SQS FIFO queue instead of a standard queue?

Use FIFO only when strict ordering or exactly-once processing is a hard requirement — for example applying account-balance changes in order, or de-duplicating retried sends. FIFO preserves order within a MessageGroupId and de-duplicates on MessageDeduplicationId within a 5-minute window, but it caps throughput (300 messages/second, or 3,000 batched, higher with high-throughput mode). Standard queues give unlimited throughput and best-effort ordering; most pipelines use standard queues and make consumers idempotent instead of paying the FIFO throughput cost.

What is SNS fan-out?

Fan-out is the pattern where one message published to an SNS topic is delivered as an independent copy to many subscribers at once. The canonical form is SNS → SQS: you subscribe several SQS queues to a topic, and each publish drops one copy into every queue, so each consuming team gets its own durable buffer, retry policy, and dead-letter queue. It is the right answer when several services each need every event — far better than pointing multiple services at one shared queue, because a queue delivers each message to only one consumer.

How do I make an SQS consumer idempotent?

Give every message a stable business id (an event_id or order_id) and process each id at most once. The robust technique is a conditional write to a dedup store — for example DynamoDB PutItem with ConditionExpression="attribute_not_exists(event_id)" — so the first delivery claims the id and does the work while a duplicate's conditional write fails and is skipped. Keep the claim and the side effect under one atomic guard, and prefer naturally idempotent operations (upsert, SET balance = x) over ones that double-apply on repeat (balance = balance - 50).

What is a visibility timeout and a dead-letter queue?

The visibility timeout is the window (default 30 seconds, up to 12 hours) during which a received SQS message is hidden from other consumers; the consumer must delete it before the timeout, or it becomes visible again and is redelivered — which is how SQS recovers from a crashed consumer. A dead-letter queue is a separate queue that captures messages which fail repeatedly: with a redrive policy specifying maxReceiveCount, SQS moves a message to the DLQ after that many failed receives instead of redelivering a poison message forever. Together they give you automatic retry plus a quarantine for the messages that never succeed.

SNS vs EventBridge — which one for fan-out?

Use SNS when you need high-throughput, low-latency fan-out to a known set of subscribers and routing does not depend on the message body (a filter policy on message attributes is enough). Use EventBridge when routing depends on the event content, when the events come from AWS services or SaaS partners, when you need schedules, or when you want archive-and-replay and richer pattern matching (prefix, numeric, anything-but, $or). EventBridge has higher per-event latency and lower raw throughput than SNS, so a common design is EventBridge for content-based routing that then hands off to SNS or SQS for the actual buffered fan-out.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every messaging idea above, from the SNS-to-SQS fan-out to the FIFO message group and the conditional-write idempotency guard, maps to a hands-on practice room where you build the consumer against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you make this fan-out consumer idempotent under at-least-once delivery?" holds up under a senior interviewer's depth probes.

Practice fan-out problems now →
Event-processing drills →