Your failover is a retry loop pretending to be resilience

A team I worked with had a clean SLA: agent responses in under 8 seconds, 99.5% of the time. Then the frontier model they routed everything through started rate-limiting during peak hours. Their "failover" was three retries with exponential backoff.

The SLA held on paper. In practice, users got a 24-second response that sometimes failed anyway. The retries weren't failover — they were the same request, replayed, hoping the wall had moved.

The counterintuitive part

Retry and failover look identical in a dashboard ("request failed → tried again"). They are not the same operation.

A retry assumes the system was momentarily unavailable and the task is intact. That's true for a database connection. It's false for an agent mid-task: the moment you re-send the whole conversation, you've thrown away the in-flight state — the half-formed tool call, the retrieved doc, the branch the agent had already chosen. You pay for the first attempt's tokens and the replay's tokens. Latency triples. And if the model is down (not just slow), every retry hits the same wall.

So the failure mode isn't "generation failed." It's state continuity failed. The task had accumulated context, and retry discards it.

This is a contract problem, not a wrapper problem

You don't fix this by adding retries=3 to your client. You fix it by defining, per sub-task, what "equivalent" means — and making the fallback consume the same handoff context rather than "start over."

That requires three things most single-model setups don't have:

1. An abstraction so the fallback speaks the same interface — if the backup model is OpenAI-compatible, swapping it in doesn't mean rewriting prompts or adapters. The task never learns a model went down.
2. A ranked fallback registry, not a single backup — one standby model is a single point of failure with a different name. A registry orders equivalents by capability and cost.
3. The routing layer carries state — the switch happens inside the task, not above it. The user sees one continuous attempt.

failover_policy:
  primary: frontier-model
  fallback_tier:
    - model: mid-tier-instruct
      equivalence: "handles same tool-calling schema"
      carries_state: true
    - model: small-instruct
      equivalence: "degrades to summary, preserves context"
      carries_state: true
  on_primary_unavailable: route_to_first_capable   # not: retry_primary

With a catalog of 25+ models behind one interface, "equivalent" stops being theoretical — there's a梯队 you can actually fail over to without dropping the task.

sequenceDiagram
    participant T as Task (in-flight)
    participant R as Router
    participant P as Primary (frontier)
    participant F as Fallback (mid-tier)
    T->>R: needs next step
    R->>P: call (carries handoff context)
    P-->>R: 429 / timeout
    R->>F: call (same handoff context)
    F-->>T: continuation (state preserved)
    Note over T,F: user sees ONE attempt, not a replay

The SLA only means something if the task survives

"99.9% availability" is meaningless if failover loses the task. A retry that returns after 3x the latency and a fresh start isn't a success — the user waited, paid, and got a do-over. The SLA number is green; the experience is red.

Real failover measures task completion, not request return. The fallback has to be able to pick up where the primary left off — which is exactly why the interface abstraction matters. Without it, you're not failing over, you're restarting.

def step(task, model_registry):
    for model in model_registry.ordered_for(task.capability):
        try:
            return model.call(task.handoff_context)   # state travels with the task
        except Unavailable:
            continue                                   # next equivalent, same context
    raise NoCapableModel

The router doesn't retry the model. It advances the task to the next model that can take its context.

The question I'll leave you with

Before you add a retry_on_failure wrapper: for the step you're protecting, what does "equivalent" mean — and does your fallback carry its state, or just rerun it?

No tool to sell, no link to click. Just this: most "resilient" LLM systems are retry loops with better logging. Resilience is a contract the fallback has to honor.

Comments

[Arhan Canli]

The replay-versus-continue distinction is the right one, and one failure sits underneath it: tool calls with side effects. If the primary dies after emitting a tool call (a write, a payment, a ticket) but before the result reaches the router, "carry the handoff context to the fallback" can make the fallback issue the same call again. The context says the step is still open, because nothing recorded it as done. So the handoff needs an idempotency key per tool call, or a log of calls started versus completed, and the fallback should see "started, outcome unknown" rather than "not started".

The retry arithmetic in your opening example also checks out as a lower bound: three attempts at roughly 8s each is already 24s before any backoff sleeps, so a p99 target of 8s cannot survive even one retry at peak, whatever the success rate.

When you tested the fallback tier, did you compare tool-call schema compliance between the frontier model and the mid-tier one? "Handles the same schema" is where I would expect the equivalence claim to break first.