Retry logic answers one question well: "should this run again." It's much worse at answering a different, more important question: "should a person look at this before it runs again." Most teams only build for the first question, which is why a job can retry the same doomed operation forty times before anyone notices it was never going to succeed.
This matters more as automation scales. A team running three jobs can get away with someone occasionally glancing at logs and catching the doomed retries by eye. A team running fifty jobs across a dozen integrations cannot, and the gap between "we have retries" and "we correctly route failures to the right response" becomes the difference between a system that mostly runs itself and one that quietly accumulates unresolved problems nobody has bandwidth to notice.
The default assumption is almost always "retry"
When an automation job fails, the reflexive fix is to add a retry. It's a reasonable default, most failures genuinely are transient, and automatically retrying a timeout or a brief network blip is the right call almost every time. The problem is that this default gets applied uniformly, to failures that are transient and failures that never will be, because the retry logic itself usually can't tell the difference.
A record with a malformed field, a reference to something that was deleted upstream, or a value that violates a downstream constraint isn't going to succeed on attempt six just because it failed on attempts one through five. Retrying it isn't buying time for the problem to resolve, because there's no version of "the problem resolves on its own" available. It just delays the moment someone actually looks at it, while burning compute and generating log noise on every attempt.
Photo by Tima Miroshnichenko on Pexels
Three questions that separate "retry" from "escalate"
Has this exact failure happened before, recently, on this same record? If the same record has failed the same way three times in the last hour, the fourth attempt is not meaningfully more likely to succeed than the third was. That's the clearest signal that the retry loop has stopped doing useful work.
Is the failure the kind that resolves with time, or the kind that requires a change? A connection timeout resolves with time, the network recovers, the downstream service comes back. A schema validation failure requires a change, either to the record or to the validation rule, and no amount of waiting fixes it. Classifying errors this way at the point they're caught, rather than treating every exception identically, is most of the work here.
Would a human looking at this right now actually be able to do something about it? If the answer is "no, they'd just see the same generic error and requeue it," escalating too early creates its own kind of alert fatigue, the same problem retrying forever creates, just moved earlier in the pipeline. The escalation needs to come with enough context (payload, error, attempt history) that a human reviewing it can actually make a decision, not just acknowledge that something is broken.
What "escalate" should actually mean
Escalating shouldn't mean paging someone at 2am for every quarantined record. It should mean moving the record out of the automatic retry loop and into a queue that's reviewed on a cadence, with enough detail attached that review is fast. PagerDuty and similar incident tools are the right layer for "this needs attention right now," reserved for backlog growth or systemic failure patterns, not for every individual record that stops retrying.
The distinction matters because pairing every single quarantined record with an urgent page trains the team to ignore the pages, which defeats the purpose of having them. Wikipedia's overview of incident management covers the broader discipline of separating "needs immediate response" from "needs eventual review," which is exactly the split that matters here.
A worked example
Consider a job that syncs customer records from a CRM into a billing system. Two kinds of failure show up in production: a timeout connecting to the CRM's API during its maintenance window, and a record whose email field is missing entirely, which the billing system's API rejects outright.
The first case is exactly what retries are for. The CRM's maintenance window ends, the same request that failed at 2:00am succeeds cleanly at 2:15am, and the retry loop closes out the failure without anyone needing to look at it. The second case is different in kind, not just severity. No amount of retrying fixes a missing email field. The record needs a human to either backfill the missing data at the source or decide the record should be excluded from billing entirely. Retrying it five times just delays that decision while generating five identical log entries.
The job's error handling needs to tell these two cases apart at the moment it catches the exception, not leave that determination to whoever eventually reads the logs. A timeout exception type gets the retry treatment. A validation exception type skips straight to the review queue.
Building the classification into the job, not the postmortem
The most effective version of this isn't a manual triage step applied after the fact, it's error classification built into the job itself at the point of failure. Wrap validation and business-rule failures in a distinct exception type from transient infrastructure failures, and let the retry logic branch on that type rather than treating every caught exception the same way. It's a small amount of extra structure that pays for itself the first time a genuinely permanent failure stops silently retrying for six hours before anyone notices.
The Google SRE book's chapter on handling overload makes a related point about designing systems that fail predictably rather than degrading in ways nobody planned for, and the same reasoning applies at the level of an individual job's error handling, not just system-wide load management.
What good escalation actually costs a team
Teams sometimes avoid building this classification because it feels like extra engineering effort for a problem that "mostly works" with generic retries. The honest accounting looks different once you count the cost on the other side: engineering time spent manually reconciling silently-dropped records, customer trust lost when a support ticket reveals data went missing for a week, and the alert fatigue that comes from every genuinely urgent failure being buried in a stream of retries that were never going to succeed.
Building the classification is a modest, one-time investment, usually a few distinct exception types and a branch in the retry decision. Living without it is a recurring cost that shows up every time a permanent failure gets treated like a transient one, which tends to happen more often than teams expect once a pipeline has been running for a year or two and accumulated a long tail of edge cases nobody anticipated at launch.
Where the retried-forever failures actually end up
Once a job correctly identifies that a failure needs a human rather than another retry, it needs somewhere durable to put that record while it waits for review. 137Foundry's automation team has written a detailed guide on building exactly that: a dead letter queue that captures what failed, why, and how many times, so the record isn't lost and isn't retried into oblivion either. It's the natural next step once a job stops treating every failure as identical.
Top comments (0)