A fintech startup is building a real-time payment processing platform. Three teams need to react when a payment completes: the notifications team (send push + email), the fraud team (score the transaction), and the analytics team (write to the data warehouse for reporting and reconciliation — including re-processing historical events whenever finance restates a report).
The backend architect picks EventBridge (serverless event bus — rules are evaluated at publish time; an event that matches no enabled rule is discarded, not stored) for all three. Each team wires up a rule that triggers their Lambda (serverless compute). Three teams, three rules, one bus. Clean.
Six months later, the data warehouse goes down for a planned 4-hour migration on Sunday night. To stop their Lambda from throwing 14,000 errors against a dead warehouse, the analytics team disables their EventBridge rule for the window. Standard procedure. They re-enable it Monday at 6 AM.
14,000 payment events fire during those 4 hours. The fraud and notifications rules process every one of them. For the analytics rule, EventBridge evaluates each event, finds no enabled match, and moves on. No delivery is attempted. Nothing fails. Nothing is retried. Nothing is stored.
$1.4M in transactions are missing from the Monday reconciliation report. The audit team flags it by 9 AM.
The architect chose EventBridge for all three consumers. That was the mistake. What should the analytics team have been wired to instead?
A) Enable a DLQ (Dead Letter Queue — captures events EventBridge attempted to deliver to a target but could not) on the analytics rule — Sunday's 14,000 events would have been captured and re-driven after the maintenance window
B) Use SQS (message queue — buffers messages until a consumer processes and deletes them; each message is consumed once, then gone) as the rule target — the queue absorbs events while the pipeline is down, and the Lambda drains it on resume without disabling anything
C) Consume from a Kinesis Data Stream (streaming log — records are retained 24 hours to 7 days regardless of whether anyone reads them; each consumer owns its read position and resumes from it) — during maintenance the pipeline simply stops, then resumes from its last checkpoint and reads everything it missed
D) Enable EventBridge Archive + Replay (stores events published to the bus; an operator can later replay them, scoped to selected rules) — the archive retains Sunday's events and a Monday-morning replay re-delivers them to the analytics rule
Answer in the comments.
Top comments (6)
The answer is C.
EventBridge (serverless event bus) is a router, not a store. Rules are evaluated at the moment an event is published. An event that matches an enabled rule is delivered to that rule's targets; an event that matches nothing is discarded on the spot. A disabled rule is nothing. So for those 4 hours there was no delivery attempt, no error, no retry, and no DLQ entry — because from EventBridge's perspective, nothing went wrong. The system did exactly what it was configured to do. That is the defining property of a router: if no one is listed to receive the event right now, the event ceases to exist.
A Kinesis Data Stream (streaming log — retains records independently of consumption) inverts the model. The producer writes a record; the record is stored for the retention window whether zero consumers or ten consumers are running. Each consumer tracks its own position in the stream — a checkpoint. Warehouse maintenance now requires disabling nothing: the analytics consumer stops, records accumulate in the stream, and on Monday it resumes from its checkpoint and processes the backlog. A finance restatement is the same motion — rewind the checkpoint and re-read.
The trade-offs are real: retention is finite (24 hours by default, 7 days extended, 365 days at additional cost — backfills beyond that need an S3 archive), you manage shard capacity, and the payment events must be produced into the stream — either the payment service writes to it directly, or a permanent EventBridge rule targets the stream. Note that this rule is one no team ever has a reason to disable: the stream absorbs regardless of consumer state.
The pattern: match each consumer's guarantee to the primitive. Notifications needs push delivery and tolerates loss — an EventBridge rule is right. Fraud needs delivery plus failure capture — EventBridge plus a Lambda failure destination. Analytics needs retention and re-consumption — that is a stream, and only a stream. This triad — bus (routes, no retention), queue (buffers, consume-once), log (retains, replayable) — is not an AWS quirk. It is the same distinction you make choosing between RabbitMQ, SQS, and Kafka.
A — Two mechanisms, both fatal. A rule DLQ captures events EventBridge attempted to deliver and could not — a throttled or unreachable target. A disabled rule attempts nothing, so there is no failure to capture. And even with the rule enabled, this DLQ would not have caught a warehouse outage: EventBridge invokes Lambda (serverless compute) asynchronously, and handing the event to Lambda's internal async queue counts as successful delivery. Errors thrown by the function afterward belong to Lambda's own retry cycle (2 additional attempts) and Lambda's failure destination — a different dead-letter mechanism on a different service than the one this option configures.
B — Honest credit first: had the team targeted SQS (message queue — consume-once) and simply left everything running, the queue would have absorbed Sunday's events. Buffering while a consumer is offline is exactly what queues are for, and this incident alone does not rule SQS out. The stated requirement does: reconciliation and restatements mean re-reading events that were already successfully processed. SQS deletes on consume — there is nothing left to re-read. Buffering is not replay.
D — The most defensible wrong answer. Archive + Replay, configured before the incident, would have recovered it — and a replay can be scoped to just the analytics rule, so the other teams see nothing. The difference is the operating model. Replay is a recovery procedure: an operator must notice the gap, start the replay, and scope it correctly, and the consumer still has no read position of its own. A stream makes the same recovery automatic — restart the consumer, it resumes — and turns backfills from an incident procedure into a routine, self-serve operation. When re-consumption is a weekly business requirement rather than a rare disaster, you want it to be a consumption model, not a break-glass tool.
Also, it would mean a lot to me if you could support my content and stay in touch 🙏
YouTube: youtube.com/@system-design-lab
LinkedIn: linkedin.com/in/joud-awad/
Medium Blog: joudwawad.medium.com/
Substack: joudawad.substack.com/
Some comments may only be visible to logged-in visitors. Sign in to view all comments.