The Retry Counter Said 7, The Config Said 3

A LangGraph pipeline I was reviewing last year. Token costs were running over projection, slow runs were taking about 3x the normal time. Nothing was throwing errors.

When I added a retry counter per step and ran a few slow cases through, the resolve step showed a count of 7. The config had max_retries=3.

The discrepancy came from two retry layers that had been written independently and never asked to coordinate. The step-level nodes had retry logic. The orchestrator graph had a recovery edge that re-invoked subgraphs when steps returned an error state — and that edge was triggering twice on ambiguous failure modes before the orchestrator gave up. So the actual retry count for those steps was whatever the step-level config said, plus two orchestrator re-invocations.

Once I had the counter, I could see it. Before that, I was looking at prompt quality and model selection because that's what the symptoms looked like.

The instrumentation was pretty simple: a counter object passed to every node, checked before retrying, incremented after.

import time
from collections import defaultdict

class RetryBudget:
    def __init__(self, run_limit=12, step_limit=3):
        self.run_limit = run_limit
        self.step_limit = step_limit
        self._total = 0
        self._by_step = defaultdict(int)

    def can_retry(self, step: str) -> bool:
        if self._total >= self.run_limit:
            return False
        return self._by_step[step] < self.step_limit

    def charge(self, step: str) -> None:
        self._total += 1
        self._by_step[step] += 1

    def state(self) -> dict:
        return {"total": self._total, "by_step": dict(self._by_step)}

run_limit=12 is meant to account for two layers interacting. If each step allows 3 retries and you have two layers that can both fire, you need the ceiling to be higher than 3 or the orchestrator layer will consume the whole budget on its first re-invoke. The team was using 12 which was basically 3 steps times 4 (step + orchestrator with some slack). It's not a principled number.

The orchestrator's recovery edge went through the same budget object. That was the main structural change — before, the orchestrator had no visibility into how many retries the step level had already used, and the step level had no idea the orchestrator was going to re-invoke. Sharing one budget object between both makes the interaction visible.

After a week of production data, _by_step showed that the resolve step was consuming most of the budget on slow runs. Looking at those specific calls, the upstream API was returning 200 with malformed JSON on a particular input pattern, and the retries were doing nothing because the response was consistently broken for that input. We ended up fixing input normalization before the call, not the retry config. But we wouldn't have known which step to look at without seeing where the retries were going.

---

The unresolved part is what to do when the run-level budget hits zero mid-run.

This pipeline accumulates meaningful intermediate state by the time it gets to the downstream steps. Early steps do entity resolution, later ones do candidate ranking, and by step 6 or 7 there's significant useful work already done. Failing the whole run discards that. Returning whatever completed without noting that the run didn't finish is wrong because the consumer runs aggregations on top of multiple runs, and incomplete results fed into aggregation without flagging produce incorrect totals.

What we ended up shipping was a result object with a complete: bool field and a list of which steps didn't finish. The consumer is supposed to check it before using the data. In practice, the check is inconsistent. The complete: False flag gets read in some call sites and ignored in others. The next thing we're considering is making the result type force the consumer to handle the incomplete case explicitly rather than just exposing a bool that can be ignored. Either a tagged union or a checked exception type. We haven't gotten there yet and I'm not certain it would actually fix the consistency problem even if we did.

There's probably a version where this becomes a dead letter queue and a human reviews the incomplete runs. We haven't needed that yet.