A circuit breaker answers one question: should this job even attempt the call right now. It doesn't answer a second, equally important question: what happens to the work that would have been done. That second question is what a dead letter queue is for, and pairing the two is what turns "stop making the outage worse" into a pipeline that actually recovers cleanly once the dependency comes back.
Step 1: Understand What Each Piece Actually Owns
The breaker's job is narrow: track recent failures against one dependency, and once a threshold is crossed, stop attempting new calls for a cooldown period. It has no opinion about what happens to the record you were trying to process. That's the dead letter queue's job: catch anything the breaker deferred, or anything that failed outright, and hold it somewhere durable until it can be retried.
Keeping these responsibilities separate matters. A breaker that also tries to manage retries of deferred work ends up doing two jobs badly instead of one job well.
Step 2: Route Deferred Work to the Queue, Not the Void
When the breaker is open, don't silently skip the record. Write it to a table or a message queue with enough context to reprocess it later: the record's identifier, the operation that was attempted, a timestamp, and the reason it was deferred.
if breaker.is_open("shipping-api"):
dead_letter_queue.push({
"record_id": order.id,
"operation": "sync_shipping_status",
"deferred_at": time.time(),
"reason": "circuit_open",
})
continue
This is a small amount of extra code compared to the alternative, which is quietly losing whatever the job was supposed to do for that record.
Step 3: Build a Separate Sweep to Drain the Queue
Don't try to drain the dead letter queue from inside the same job run that populated it, the dependency is still down, that's why the breaker is open. Instead, build a scheduled sweep that runs independently, checks whether the breaker has closed, and if so, works through the queue in order.
def drain_dead_letters():
if breaker.is_open("shipping-api"):
return
for item in dead_letter_queue.pop_batch(100):
try:
sync_shipping_status(item["record_id"])
except Exception:
dead_letter_queue.push(item)
This sweep can run on its own schedule, every few minutes, and it naturally stops doing work once the breaker reopens if the dependency relapses.
Step 4: Watch for Queue Growth as Its Own Signal
A dead letter queue that never drains is a sign the breaker never closes, which usually means either the dependency is down for longer than you expected or your half-open probe logic has a bug. Alert on queue depth the same way you'd alert on any other backlog metric. A queue holding four items is normal during a short outage. A queue holding forty thousand items three days later is a different kind of problem, and one you want to know about long before a stakeholder asks why last week's numbers still look wrong.
Step 5: Order Matters More Than You'd Expect
If the records in your queue have any dependency on each other, an order update that needs to happen before a shipping status sync, for example, draining the queue out of order can create a different, quieter kind of data corruption than the original outage would have. Preserve insertion order in the queue implementation you choose, and process records in that order during the sweep, rather than reaching for whatever's fastest to pop.
Step 6: Instrument Both Halves
Log the breaker's state transitions and the queue's depth and drain rate together, ideally in the same dashboard. OpenTelemetry is a reasonable default for wiring this kind of event and metric data into whatever observability stack you already run. Seeing "breaker opened at 2:14, queue peaked at 3,200 items, breaker closed at 2:51, queue drained by 3:40" in one place tells you the whole story of an incident without cross-referencing three separate logs.
Step 7: Test the Full Loop, Not Just the Breaker
It's easy to test that a breaker opens correctly and stop there. Test the full loop instead: force the breaker open, confirm records land in the queue with the right shape, force the breaker closed again, and confirm the sweep drains the queue completely and in order. This is the part that tends to have bugs nobody notices until the first real outage, because the breaker-opening path gets tested far more often than the recovery path.
Picking Storage for the Queue
For most teams, a simple database table with a status column (pending, processing, done, failed) is enough, and it's easier to query and debug than a dedicated message queue if your volume is moderate. If you're already running something like a message broker for other parts of your pipeline, using it here is fine too, the pattern matters more than the specific storage mechanism. The AWS Builders' Library and Microsoft's Azure Architecture Center pattern catalog both have write-ups on queue-based load leveling worth reading if you want more depth on the tradeoffs.
If you're setting this up for the first time, 137Foundry has a longer guide covering circuit breaker thresholds and pairing strategy that works through a full nightly sync example, including how the numbers change once the breaker and queue are both in place.
The Combination Is the Point
Neither piece alone solves the whole problem. A breaker without a queue prevents wasted retries but silently drops work. A queue without a breaker still lets a job hammer a dead dependency before anything lands in the queue at all. Together, they turn an outage from a data-loss event into a delay, which is a much easier problem to have.
A Note on Idempotency
None of this works cleanly if reprocessing a record twice causes a duplicate side effect, a second charge, a duplicate email, a doubled inventory count. Before you wire up a dead letter queue, confirm the operation it's going to retry is safe to run more than once for the same record. If it isn't, add an idempotency key or a "already processed" check before the retry sweep goes live, not after the first duplicate charge shows up in a support ticket.
This is worth calling out explicitly because it's the detail most likely to bite a team that builds the breaker-plus-queue pattern correctly in every other respect. The queue does its job perfectly, replays a record that partially succeeded the first time, and the operation runs twice because nobody checked whether it was safe to.
Sizing the Sweep Frequency
How often the drain sweep runs is its own tuning decision. Too frequent, and you're polling a queue that's still empty because the breaker just opened thirty seconds ago, wasting scheduler slots. Too infrequent, and you're sitting on a drainable queue for longer than necessary after a short outage clears. Every five minutes is a reasonable starting point for most business-hours data pipelines; adjust based on how quickly your specific dependencies tend to recover once they come back.
Handling a Queue That Never Fully Drains
Occasionally a dependency doesn't cleanly recover, it comes back degraded, succeeding on some records and failing on others in a way that never lets the breaker fully close. In that case, treat a queue that's shrinking slowly but not reaching zero as its own alert condition distinct from a queue that's still growing. Both need attention, but they point to different root causes: one is an ongoing outage, the other is a partially recovered dependency that needs a support ticket of its own.
Top comments (0)