Every data pipeline that talks to a third-party API has retry logic somewhere. It's usually the first resilience feature anyone adds, and for good reason: a lot of failures really are transient, a dropped connection, a load balancer hiccup, a timeout that would have succeeded on the next attempt. But retries solve exactly one problem, and there's a second, more expensive problem they can't touch.
The Problem Retries Solve
A single failed call that would have succeeded a few seconds later is the textbook case for a retry. Exponential backoff, a handful of attempts, and you're done. This works well because the failure is genuinely random noise against a healthy system. Most libraries handle this pattern well out of the box, and if you're not already using one, something like Tenacity in Python handles backoff and jitter without much custom code.
The Problem Retries Make Worse
The failure mode retries can't fix, and actively worsen, is sustained unavailability. If a dependency is down for twenty minutes, not two seconds, every retry attempt during that window is wasted work. Worse, if enough callers are all retrying against the same struggling service at once, the retry traffic itself becomes part of the reason the service stays down longer. This is a well-documented pattern in distributed systems literature: retry storms amplify outages instead of riding them out.
For a data pipeline specifically, this shows up as job runtimes that balloon from minutes to hours, rate limits getting burned on calls that were never going to succeed, and outages that go unnoticed by monitoring because the job hasn't technically failed, it's just extremely slow.
Picture a nightly job syncing four thousand order records against a payments API. On a normal night it finishes in ten minutes. The night the payments API has an unannounced maintenance window, every one of those four thousand calls times out, and with three retries each at a fifteen-second timeout, that's potentially hours added to a job that should have finished before anyone was awake to notice. The job "succeeds" eventually, technically, just six hours later than the dashboard that depends on it expected fresh data.
How to Tell the Difference in Your Own Metrics
If you're trying to figure out whether your pipeline actually has this problem, look at job duration variance rather than average duration. A job whose runtime is consistently ten minutes plus or minus thirty seconds doesn't need a breaker yet. A job whose runtime is ten minutes on a good night and four hours on a bad one, with the bad nights always correlating to the same external dependency's status page showing an incident, is exactly the case a breaker is built for.
The other tell is your rate-limit consumption graph, if you track one. A spike in consumed quota that doesn't correspond to a spike in successful calls is retries burning through your allowance on calls that were failing anyway. That pattern usually means you're not just wasting time during an outage, you're also starting the recovery period already close to rate-limited, which delays how quickly you can catch up once the dependency comes back.
What Actually Solves the Second Problem
A circuit breaker is the pattern built for sustained unavailability specifically. It tracks recent failures against a dependency and, once a threshold is crossed, stops attempting the call entirely for a cooldown period. New calls fail immediately with no network request made, which means your job stops burning time and rate limit budget on a dependency that's confirmed to be down.
if breaker.is_open("payments-api"):
raise SkipCall("circuit open, deferring to pending queue")
result = call_payments_api(order_id)
The breaker periodically lets a small number of test calls through (the "half-open" state) to check whether the dependency has recovered, and closes again once they succeed. This gives you the retry behavior back once it's actually useful again, rather than disabling calls forever.
The Two Patterns Are Complementary, Not Competing
The right mental model isn't "retries versus circuit breakers." It's retries for the moment-to-moment noise, wrapped inside a breaker that catches the sustained outage case. A single call gets a few retries with backoff. If the failure rate across many calls crosses a threshold, the breaker opens and stops even the retried calls from going out. When the circuit breaker design pattern was formalized, it was explicitly designed to sit alongside retry logic, not replace it.
Where teams get this wrong is treating the two as interchangeable rather than layered. Increasing the retry count when a dependency starts failing more often feels like the intuitive fix, more attempts should mean more successes, but it's backwards for sustained outages. More retries against a truly down service just means more wasted attempts and, if the service is struggling rather than fully down, more load pushing it further into struggling. The breaker is what recognizes "this isn't noise anymore" and changes the job's behavior accordingly, something a retry count alone can never do because it has no memory of past attempts.
A reasonable rule of thumb: keep retries per call low, two or three attempts with backoff, and let the breaker's failure count across calls be the thing that decides whether the dependency is having a bad moment or a bad hour. Trying to solve the sustained-outage case by cranking retries up to ten or fifteen per call just delays the point where you notice the outage, it doesn't prevent the wasted work.
What Happens to the Work You Skipped
An open breaker means you're not attempting the call right now, but the underlying task, the record that needed to sync, the event that needed processing, still needs to happen. Pair the breaker with a durable queue or a pending table so deferred work gets picked up and reprocessed once the dependency recovers, rather than silently dropped. This is the piece that's easy to skip and expensive to have skipped: a breaker without a backlog strategy just becomes a faster way to lose data instead of a slower one.
Observability Ties It Together
Whichever combination you land on, log every state transition and pipe it into whatever tracing setup you already use. OpenTelemetry has become the default choice for this kind of event data across most modern backend stacks, and a breaker that trips silently provides a fraction of the value of one that pages someone the moment it opens.
If you want a fuller walkthrough of picking thresholds and pairing a breaker with a pending queue, 137Foundry put together a longer guide on adding circuit breakers to data automation jobs that works through a concrete nightly sync example end to end.
The Short Version
Retries handle the noise. Circuit breakers handle the outage. If your pipeline only has one of the two, you're covered for the failure mode that's usually annoying and exposed to the one that's usually expensive.
Auditing which of your scheduled jobs only has retries is a reasonable place to start. If you can name a dependency that's had a multi-hour outage in the last quarter and your job's response was "eventually finish, just very late," that's the dependency to wrap first.
Top comments (0)