A multi-tenant SaaS platform sends all background notifications — email, SMS, webhook calls — through a single SQS Standard queue (message queue — a message is removed only when a consumer explicitly deletes it or its retention period expires, 4 days by default). The queue feeds a Lambda function (serverless compute) through an event source mapping (polls the queue, invokes Lambda with batches, and deletes messages when the invocation succeeds). The team did things right: the queue has a Dead Letter Queue (DLQ — a second queue that receives messages after they fail delivery more than maxReceiveCount times) with maxReceiveCount of 5, and an alarm on DLQ depth.
Friday, 6 PM: a routine deployment ships two changes. One updates the email provider SDK — and breaks the provider credential lookup. The other is a "hardening" change from code review: the handler body is wrapped in a try/catch that logs any exception and returns normally, "so one bad notification can't poison the batch."
Saturday, 9 AM: support is flooded. No customer received anything overnight — 41,000 notifications gone. The team checks CloudWatch (AWS monitoring service): zero function errors all night. Queue depth: zero. The DLQ: empty. The DLQ alarm never fired. Every dashboard is green.
Where did 41,000 messages go?
A) The messages crossed an internal receive-count threshold and SQS deleted them — but no such mechanism exists; receive count only routes messages to a DLQ via a configured redrive policy, and SQS deletes nothing based on how many times a message was received
B) The handler caught every exception and returned success — Lambda reported a clean invocation, so the event source mapping did what success means: it called DeleteMessage on every message in the batch; the messages were deleted legitimately, one green invocation at a time
C) The queue's retention period expired overnight — but retention defaults to 4 days; messages a few hours old cannot age out, and expiry would not explain the zero-error, empty-DLQ picture either way
D) The deployment disabled the event source mapping, so messages were never consumed — but an unconsumed queue shows growing depth; a queue at zero depth with no errors means messages were received, "processed," and deleted
Answer in the comments.
Top comments (5)
The answer is B.
In the SQS (message queue) + Lambda (serverless compute) integration, your function's return value is not a status report. It is a delete instruction. When the function returns without throwing, the event source mapping (polls, invokes, deletes on success) treats every message in the batch as processed and calls
DeleteMessage. There is no second opinion — Lambda cannot see that the email was never sent. The only signal it understands is thrown-or-returned.The Friday deployment silenced that signal. The credential bug made every send fail; the new try/catch swallowed every failure and returned. From the platform's perspective, the night was flawless: invocation starts, exception raised, exception caught and logged, clean return, batch deleted. 41,000 times. The DLQ (catches messages that fail more than maxReceiveCount times) stayed empty for the most brutal reason possible — nothing ever failed. A DLQ is a net under the tightrope, and this deployment convinced the walker to report a safe landing while falling.
That is the pattern to internalize: in queue-consumer systems, the exception is not noise — it is the mechanism that keeps the message alive. Catch-log-and-return at the handler boundary converts every failure into a permanent, silent delete. Safety nets downstream of the success signal (DLQ, redrive, retries) are all blind to it.
Two fixes:
Fix 1 — Let failures fail. At the handler boundary, log and rethrow. If the goal of that try/catch was "one bad message shouldn't poison the batch," the correct tool is partial batch responses (
ReportBatchItemFailures): the function returns the IDs of the messages that failed, the mapping deletes only the successes, and the failures stay in the queue, retry, and reach the DLQ after 5 attempts — exactly the machinery the team had already built and then bypassed.Fix 2 — Alarm on the business signal, not just the error signal. "Zero errors" and "doing nothing" are indistinguishable to an error alarm. Add a CloudWatch (AWS monitoring service) alarm on notifications-sent-per-minute (or the provider's accepted count) with a floor threshold. A send rate of zero on a Saturday morning pages someone at 12:15 AM instead of letting support find out at 9.
A — This is a widely repeated myth, so it's worth killing precisely: SQS tracks
ApproximateReceiveCount, but the count triggers exactly one behavior — moving the message to a DLQ when a redrive policy is configured. It never triggers deletion. Without a redrive policy, a message can be received ten thousand times and will still sit in the queue until it is explicitly deleted or retention expires.C — Retention (default 4 days, max 14) is a slow clock. Messages enqueued Friday evening cannot expire Saturday morning. And expiry wouldn't produce this evidence pattern anyway — the messages didn't linger and die; they were consumed and deleted within seconds of arrival, which is why depth stayed at zero all night.
D — A disabled event source mapping starves the queue, and an unconsumed queue is a visibly growing queue. Depth pinned at zero means the opposite: consumption was running at full speed. The pipeline wasn't stopped — it was a wood chipper reporting success.
Also, it would mean a lot to me if you could support my content and stay in touch 🙏
YouTube: youtube.com/@system-design-lab
LinkedIn: linkedin.com/in/joud-awad/
Medium Blog: joudwawad.medium.com/
Substack: joudawad.substack.com/