Your failover is a retry loop pretending to be resilience

A team I worked with had a clean SLA: agent responses in under 8 seconds, 99.5% of the time. Then the frontier model they routed everything through started rate-limiting during peak hours. Their "failover" was three retries with exponential backoff.

The SLA held on paper. In practice, users got a 24-second response that sometimes failed anyway. The retries weren't failover — they were the same request, replayed, hoping the wall had moved.

The counterintuitive part

Retry and failover look identical in a dashboard ("request failed → tried again"). They are not the same operation.

A retry assumes the system was momentarily unavailable and the task is intact. That's true for a database connection. It's false for an agent mid-task: the moment you re-send the whole conversation, you've thrown away the in-flight state — the half-formed tool call, the retrieved doc, the branch the agent had already chosen. You pay for the first attempt's tokens and the replay's tokens. Latency triples. And if the model is down (not just slow), every retry hits the same wall.

So the failure mode isn't "generation failed." It's state continuity failed. The task had accumulated context, and retry discards it.

This is a contract problem, not a wrapper problem

You don't fix this by adding retries=3 to your client. You fix it by defining, per sub-task, what "equivalent" means — and making the fallback consume the same handoff context rather than "start over."

That requires three things most single-model setups don't have:

1. An abstraction so the fallback speaks the same interface — if the backup model is OpenAI-compatible, swapping it in doesn't mean rewriting prompts or adapters. The task never learns a model went down.
2. A ranked fallback registry, not a single backup — one standby model is a single point of failure with a different name. A registry orders equivalents by capability and cost.
3. The routing layer carries state — the switch happens inside the task, not above it. The user sees one continuous attempt.

failover_policy:
  primary: frontier-model
  fallback_tier:
    - model: mid-tier-instruct
      equivalence: "handles same tool-calling schema"
      carries_state: true
    - model: small-instruct
      equivalence: "degrades to summary, preserves context"
      carries_state: true
  on_primary_unavailable: route_to_first_capable   # not: retry_primary

With a catalog of 25+ models behind one interface, "equivalent" stops being theoretical — there's a梯队 you can actually fail over to without dropping the task.

sequenceDiagram
    participant T as Task (in-flight)
    participant R as Router
    participant P as Primary (frontier)
    participant F as Fallback (mid-tier)
    T->>R: needs next step
    R->>P: call (carries handoff context)
    P-->>R: 429 / timeout
    R->>F: call (same handoff context)
    F-->>T: continuation (state preserved)
    Note over T,F: user sees ONE attempt, not a replay

The SLA only means something if the task survives

"99.9% availability" is meaningless if failover loses the task. A retry that returns after 3x the latency and a fresh start isn't a success — the user waited, paid, and got a do-over. The SLA number is green; the experience is red.

Real failover measures task completion, not request return. The fallback has to be able to pick up where the primary left off — which is exactly why the interface abstraction matters. Without it, you're not failing over, you're restarting.

def step(task, model_registry):
    for model in model_registry.ordered_for(task.capability):
        try:
            return model.call(task.handoff_context)   # state travels with the task
        except Unavailable:
            continue                                   # next equivalent, same context
    raise NoCapableModel

The router doesn't retry the model. It advances the task to the next model that can take its context.

The question I'll leave you with

Before you add a retry_on_failure wrapper: for the step you're protecting, what does "equivalent" mean — and does your fallback carry its state, or just rerun it?

No tool to sell, no link to click. Just this: most "resilient" LLM systems are retry loops with better logging. Resilience is a contract the fallback has to honor.

Comments

[Chris Sellers]

The distinction between discarding in-flight state and carrying the handoff context is the part that stuck. A retry that re-sends the whole conversation does look like failover in the metrics, and that is exactly why availability stayed green while the person on the other end got a do-over.

One edge that sits next to model failover: a tool call that timed out after send. Replaying the whole step under a new attempt can run the same side effect twice if the first call landed. Carrying state helps only if the pending tool outcome stays pending until the provider answers, instead of getting folded into "generation failed" and retried.

Curious whether your registry treats a mid-tool unknown as Unavailable (advance to the next model with the same intent still open), or parks the step until a reconciler closes it.

[TokenLat]

The metric-blindness is the scary part — availability stays green while the human on the other end gets a quiet do-over. And "pending stays pending until the provider answers" is the exact invariant that makes carrying state safe instead of dangerous. When you park the step waiting for that answer, does the reconciler have a deadline? A stuck provider could otherwise hold the step open forever, and "pending" turns into "forgotten" — does your reconciler have a deadline, or does a wedged provider just keep the step open?

[Chris Sellers]

State continuity is the right name for it. The piece I would be careful with is the half formed tool call you mention. If the primary model already emitted a tool call and the tool ran, the fallback model must see that result, or it will plan the same action again and you get a duplicate side effect instead of a slow response.

So I think the handoff context needs two parts: what the model was thinking, which is nice to keep, and what already happened in the world, which is mandatory to keep.

In your registry, does the fallback get the executed tool results as facts it cannot redo, or does it see the same transcript and decide for itself?

[TokenLat]

Splitting "what the model was thinking" from "what already happened in the world" is the right cut — the second is mandatory, the first is just nice to have. If the fallback re-decides from the transcript, you've rebuilt the exact risk you were trying to avoid: it'll plan the action that already ran. Do you stamp the executed result with something the fallback can't re-derive, or do you trust it purely because it's sitting in the handoff context?

[challan116-ux]

The test I like for this: if the second attempt goes to the same place with the same request, it is a retry no matter what the dashboard calls it. Real failover changes something material — provider, region, or model — and it should also change what you log, because blending retry and failover metrics is how a degrading primary hides for weeks. Your SLA-on-paper point is the tell: latency budgets spent re-asking a rate-limited service are not resilience spending.

[TokenLat]

The "changes what you log" point is the one dashboards miss — a failover that logs like a retry will never surface as a primary-degrade signal, and the primary quietly rots for weeks. Your SLA-on-paper tell is the cleanest red flag: latency spent re-asking a rate-limited service isn't resilience spending, it's just slow retry with a fancy name. Do you keep a separate "failover fired" counter per tier, or do you just tag the latency and hope someone slices it later?